RoadmapBench - Completion Score: leaderboard

RoadmapBench (2026) tests coding agents on 115 long-horizon tasks built from real open-source version upgrades in 17 repositories and five languages: from the earlier release and a roadmap of subtasks, the agent must implement the next release, with each subtask checked by its own tests. This board is the completion score: the complexity-weighted share of subtasks whose tests pass, averaged over tasks and given in percent; higher is better.

Metric: Completion score (%): mean over tasks of the complexity-weighted share of roadmap subtasks whose own test suites pass, times 100, 115 long-horizon tasks implementing a real release upgrade from a roadmap of subtasks, one trial per model, 2-hour wall-clock budget, extended thinking on; each row names its agent scaffold (OpenHands, the primary scaffold, or Terminus 2, the ablation); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 4.7xHigh69.2#85
2Claude Opus 4.6Thinking62.7#113
3GPT-5.4High49.7#135
4DeepSeek V4 ProHigh48.6#196
5GLM-5.145.3#198
6MiMo-V2.5-Pro44#284
7Gemini 3.1 Pro (Preview)High43.9#99
8Kimi K2.643.2#192
9Qwen 3.6 Plus42.4#238
10Qwen 3.5 397B A17B38.3#264
11Kimi K2.537.8#266
12MiniMax-M2.733.2#539
13Seed 2.0 ProHigh17.7#222

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=roadmapbench-completion-score · How It Works · Data refreshed daily, snapshot 2026-11-21.