AutoRLBench 39 Issue / 20 Solution: leaderboard
AutoRLBench 39 Issue / 20 Solution — AutoRLBench native 39-issue/20-solution scored task-set edition, reviewed 16 November 2026. One run per model/task/level; issue compares root-cause answers against a hidden oracle, solution reruns the repaired workspace with metric thresholds and reward-hacking failure. Overall is total passes over59scored runs. README news retains32/24, but the exact shared task-ID sets and every one-decimal table score bind39/20. Native harness and effort are retained. Separate from33/20,32/18 and35/20 editions. Metric: Pass rate over 39 issue judgments and 20 solution runs (%). Source: https://thinking-space.github.io/AutoRLBench/
Metric: Pass rate over 39 issue judgments and 20 solution runs (%). Source: thinking-space.github.io. Saturation forecast: Around April 2028. 10 models tracked. Updated 16 Nov 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | GPT-6 Astra | Max | 39 | #6 |
| 2 | GPT-6.1 Sol | Max | 37.3 | #10 |
| 3 | GLM-5.3 | Max | 33.9 | #68 |
| 4 | Gemini 4 Argon | Max | 33.9 | #12 |
| 5 | Claude Fable 5.1 | Max | 28.8 | #15 |
| 6 | Kimi K3.1 Pro | Max | 28.8 | #29 |
| 7 | Qwen 3.8 Max | Max | 25.4 | #59 |
| 8 | Grok 4.7 | Max | 22 | #47 |
| 9 | Kimi K3 | Max | 5.1 | #56 |
| 10 | DeepSeek V4 Pro | Max | 5.1 | #191 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=autorlbench-39-issue-20-solution · How It Works · Data refreshed daily, snapshot 2026-11-16.