AutoRLBench - Issue: leaderboard
AutoRLBench gives a coding agent a real reinforcement-learning training run that went wrong, with its code, config, logs and metrics, and asks it to name the root cause. On tasks that have a repaired reference run the agent must also fix the workspace; the harness then reruns the training and a metric threshold on the new logs decides. Root-cause answers are graded by an LLM judge against a written oracle, and each model gets one run per task at maximum reasoning effort with no network access. Unlike MLE-bench, where agents build a model for a Kaggle task from scratch, every task starts from someone else's broken RL run and is scored on diagnosis and repair. This board is the share of the 33 failed training runs whose root cause the agent names.
Metric: Root cause found, 33 failed training runs (%). Source: thinking-space.github.io. Saturation forecast: Around November 2027. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-6 Astra (Max) | 45.5 | #6 (GPT-6 Astra) |
| 2 | GPT-6.1 Sol (Max) | 42.4 | #10 (GPT-6.1 Sol) |
| 3 | GLM-5.3 (Max) | 30.3 | #62 (GLM-5.3) |
| 4 | Qwen 3.8 Max (Max) | 27.3 | #55 (Qwen 3.8 Max) |
| 5 | Claude Fable 5.1 (Max) | 24.2 | #14 (Claude Fable 5.1) |
| 6 | Kimi K3.1 Pro (Max) | 24.2 | #28 (Kimi K3.1 Pro) |
| 7 | Kimi K3 (Max) | 0 | #53 (Kimi K3) |
| 8 | DeepSeek V4 Pro (Max) | 0 | #181 (DeepSeek V4 Pro) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=autorlbench-issue · How It Works · Data refreshed daily, snapshot 2026-11-12.