SWE-Chain: leaderboard
SWE-Chain (Lam et al., 2026) tests coding agents on 12 chains of consecutive release upgrades of real Python packages such as Flask, pytest and xarray (155 version transitions), where each step starts from the agent's own previous codebase and is checked against the target release's test suite. F1 in percent between resolving the upgrade-related tests and keeping previously passing tests green, averaged over the chains with one fix attempt allowed for execution errors; each row is a model run inside an agent CLI, and higher is better.
Metric: F1-score (%; 2TP / (2TP + FP + FN) over each chain's release steps, where TP are upgrade-related tests resolved, FN those left failing and FP previously passing tests broken, averaged over 12 chains; Build+Fix regime with one fix attempt for execution-level errors). Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.7 (xHigh) | 68.5 | #75 (Claude Opus 4.7) |
| 2 | GPT-5.5 (xHigh) | 64.8 | #48 (GPT-5.5) |
| 3 | GPT-5.3 Codex (High) | 59.2 | #114 (GPT-5.3 Codex) |
| 4 | GPT-5.4 (High) | 54.9 | #121 (GPT-5.4) |
| 5 | Claude Opus 4.6 (High) | 50.8 | #100 (Claude Opus 4.6) |
| 6 | Claude Sonnet 4.6 (High) | 45.9 | #158 (Claude Sonnet 4.6) |
| 7 | GLM-5.1 | 40.1 | #175 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=swe-chain · How It Works · Data refreshed daily, snapshot 2026-11-12.