RoadmapBench - Infra and Tooling: leaderboard

RoadmapBench (2026) tests coding agents on 115 long-horizon tasks built from real open-source version upgrades in 17 repositories and five languages: from the earlier release and a roadmap of subtasks, the agent must implement the next release, with each subtask checked by its own tests. This board is the percentage of the 23 infrastructure and tooling tasks the agent fully resolves, every subtask test suite passing; higher is better.

Metric: Resolved rate (%) on the 23 infrastructure and tooling tasks (every subtask test suite passes), 115 long-horizon tasks implementing a real release upgrade from a roadmap of subtasks, one trial per model, 2-hour wall-clock budget, extended thinking on; each row names its agent scaffold (OpenHands, the primary scaffold, or Terminus 2, the ablation); higher is better. Source: arxiv.org. Saturation forecast: Around December 2027. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 4.7xHigh43.5#85
2GPT-5.4High39.1#135
3Claude Opus 4.6Thinking30.4#113
4GLM-5.126.1#198
5MiniMax-M2.726.1#539
6Gemini 3.1 Pro (Preview)High26.1#99
7Kimi K2.621.7#192
8Qwen 3.6 Plus21.7#238
9MiMo-V2.5-Pro21.7#284
10Kimi K2.517.4#266
11DeepSeek V4 ProHigh17.4#196
12Qwen 3.5 397B A17B13#264
13Seed 2.0 ProHigh4.3#222

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=roadmapbench-infra-and-tooling · How It Works · Data refreshed daily, snapshot 2026-11-21.