RoadmapBench - ORM and Validation: leaderboard

RoadmapBench (2026) tests coding agents on 115 long-horizon tasks built from real open-source version upgrades in 17 repositories and five languages: from the earlier release and a roadmap of subtasks, the agent must implement the next release, with each subtask checked by its own tests. This board is the percentage of the 25 ORM and validation tasks the agent fully resolves, every subtask test suite passing; higher is better.

Metric: Resolved rate (%) on the 25 ORM and validation tasks (every subtask test suite passes), 115 long-horizon tasks implementing a real release upgrade from a roadmap of subtasks, one trial per model, 2-hour wall-clock budget, extended thinking on; each row names its agent scaffold (OpenHands, the primary scaffold, or Terminus 2, the ablation); higher is better. Source: arxiv.org. Saturation forecast: Around September 2027. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 4.6Thinking32#113
2Claude Opus 4.7xHigh32#85
3GLM-5.128#198
4Gemini 3.1 Pro (Preview)High24#99
5DeepSeek V4 ProHigh24#196
6Kimi K2.620#192
7GPT-5.4High20#135
8Kimi K2.512#266
9Qwen 3.5 397B A17B12#264
10Qwen 3.6 Plus12#238
11MiMo-V2.5-Pro12#284
12MiniMax-M2.78#539
13Seed 2.0 ProHigh8#222

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=roadmapbench-orm-and-validation · How It Works · Data refreshed daily, snapshot 2026-11-21.