AutoRLBench: leaderboard
AutoRLBench gives a coding agent a real reinforcement-learning training run that went wrong, with its code, config, logs and metrics, and asks it to name the root cause. On tasks that have a repaired reference run the agent must also fix the workspace; the harness then reruns the training and a metric threshold on the new logs decides. Root-cause answers are graded by an LLM judge against a written oracle, and each model gets one run per task at maximum reasoning effort with no network access. Unlike MLE-bench, where agents build a model for a Kaggle task from scratch, every task starts from someone else's broken RL run and is scored on diagnosis and repair. This board is the share of all 53 runs passed: 33 root-cause judgments and 20 repair runs.
Metric: Pass rate over 33 issue judgments and 20 solution runs (%). Source: thinking-space.github.io. Saturation forecast: Around December 2027. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-6 Astra (Max) | 47.2 | #6 (GPT-6 Astra) |
| 2 | GPT-6.1 Sol (Max) | 47.2 | #10 (GPT-6.1 Sol) |
| 3 | GLM-5.3 (Max) | 37.7 | #62 (GLM-5.3) |
| 4 | Kimi K3.1 Pro (Max) | 34 | #28 (Kimi K3.1 Pro) |
| 5 | Qwen 3.8 Max (Max) | 32.1 | #55 (Qwen 3.8 Max) |
| 6 | Claude Fable 5.1 (Max) | 30.2 | #14 (Claude Fable 5.1) |
| 7 | DeepSeek V4 Pro (Max) | 5.7 | #181 (DeepSeek V4 Pro) |
| 8 | Kimi K3 (Max) | 3.8 | #53 (Kimi K3) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=autorlbench · How It Works · Data refreshed daily, snapshot 2026-11-12.