SWE-Flux: leaderboard
SWE-Flux (2026) is a repository-level dynamic execution-reasoning benchmark of 480 instances over 12 real Python repositories, with gold answers harvested automatically from instrumented test executions rather than written by hand or judged by an LLM. It covers control flow, loops, program state, dataflow, exceptions and program invariants across single-test and multi-test questions; the metric is overall accuracy, in percent, higher is better.
Metric: Accuracy (%; execution-reasoning questions answered correctly over 480 instances in 12 Python repositories, gold answers harvested from instrumented executions). Source: arxiv.org. Saturation forecast: Around August 2027. 5 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 | 37.71 | #121 |
| 2 | GPT-5.3 Codex | 33.75 | #114 |
| 3 | GPT-OSS-20B | 20.62 | #1150 |
| 4 | Gemma 4 31B | 19.17 | #393 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=swe-flux · How It Works · Data refreshed daily, snapshot 2026-11-12.