RoadmapBench - ML and Data: leaderboard

RoadmapBench (2026) tests coding agents on 115 long-horizon tasks built from real open-source version upgrades in 17 repositories and five languages: from the earlier release and a roadmap of subtasks, the agent must implement the next release, with each subtask checked by its own tests. This board is the percentage of the 36 machine learning and data tasks the agent fully resolves, every subtask test suite passing; higher is better.

Metric: Resolved rate (%) on the 36 machine learning and data tasks (every subtask test suite passes), 115 long-horizon tasks implementing a real release upgrade from a roadmap of subtasks, one trial per model, 2-hour wall-clock budget, extended thinking on; each row names its agent scaffold (OpenHands, the primary scaffold, or Terminus 2, the ablation); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 4.7xHigh30.6#85
2GPT-5.4High27.8#135
3Claude Opus 4.6Thinking25#113
4GLM-5.18.3#198
5Gemini 3.1 Pro (Preview)High8.3#99
6MiMo-V2.5-Pro8.3#284
7DeepSeek V4 ProHigh8.3#196
8Kimi K2.65.6#192
9Qwen 3.6 Plus5.6#238
10MiniMax-M2.75.6#539
11Kimi K2.50#266
12Qwen 3.5 397B A17B0#264
13Seed 2.0 ProHigh0#222

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=roadmapbench-ml-and-data · How It Works · Data refreshed daily, snapshot 2026-11-21.