SWE-Race - Hard: leaderboard
SWE-Race hard subset: repair real Python concurrency and async-lifecycle bugs using the BenchGen coding agent at medium effort and a 100-step offline budget. Score is mean per-task pass@1 over repeated attempts on shipped task versions; all hidden fail-to-pass tests must pass without regressions. Native task and attempt counts differ by model.
Metric: Mean per-task pass@1 (%). Source: labs.evaligo.com. Saturation forecast: Rough model projection: around 2026. 3 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GLM-5.3 Flash (Medium) (SWE-Race Run: v0.1 BenchGen) | 49.9 | |
| 2 | GPT-5.6 Luna (Medium) (SWE-Race Run: v0.1 BenchGen) | 44.7 | |
| 3 | DeepSeek V4 Flash (Medium) (SWE-Race Run: v0.1 BenchGen) | 23.2 |
Interactive version: theaggregate.ai/benchmark?slug=swe-race-hard · How It Works · Data refreshed daily, snapshot 2026-11-05.