AutoRLBench 39 Issue / 20 Solution - Issue: leaderboard
AutoRLBench 39 Issue / 20 Solution - Issue — AutoRLBench native 39-issue/20-solution scored task-set edition, reviewed 16 November 2026. One run per model/task/level; issue compares root-cause answers against a hidden oracle, solution reruns the repaired workspace with metric thresholds and reward-hacking failure. Overall is total passes over59scored runs. README news retains32/24, but the exact shared task-ID sets and every one-decimal table score bind39/20. Native harness and effort are retained. Separate from33/20,32/18 and35/20 editions. Metric: Root cause found, 39 failed training runs (%). Source: https://thinking-space.github.io/AutoRLBench/
Metric: Root cause found, 39 failed training runs (%). Source: thinking-space.github.io. Saturation forecast: Around February 2028. 10 models tracked. Updated 16 Nov 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | GPT-6 Astra | Max | 38.5 | #6 |
| 2 | GPT-6.1 Sol | Max | 33.3 | #10 |
| 3 | GLM-5.3 | Max | 30.8 | #68 |
| 4 | Claude Fable 5.1 | Max | 23.1 | #15 |
| 5 | Qwen 3.8 Max | Max | 23.1 | #59 |
| 6 | Kimi K3.1 Pro | Max | 20.5 | #29 |
| 7 | Grok 4.7 | Max | 20.5 | #47 |
| 8 | Gemini 4 Argon | Max | 20.5 | #12 |
| 9 | Kimi K3 | Max | 0 | #56 |
| 10 | DeepSeek V4 Pro | Max | 0 | #191 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=autorlbench-39-issue-20-solution-issue · How It Works · Data refreshed daily, snapshot 2026-11-16.