AutoCodeBench: leaderboard
Each model writes one solution to each of 3,920 LLM-generated programming problems spread over 20 languages (184-200 per language), and the score is the average Pass@1 against hidden sandbox tests. Unlike HumanEval-style Python sets, the problems were generated and filtered by DeepSeek models (v1 set; AutoCodeBench-V2 is a later, different set).
Metric: Pass@1 (%). Source: arxiv.org. Saturation forecast: Around February 2027. 33 models tracked. Updated 17 Nov 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | O3 (2025-04-16) | High | 51.1 | #229 |
| 2 | Grok 4 | 50.9 | #315 | |
| 3 | Claude Opus 4 | Non-reasoning | 50.9 | #321 |
| 4 | O4 Mini (2025-04-16) | 50 | #376 | |
| 5 | Claude Sonnet 4 | Non-reasoning | 49.3 | #411 |
| 6 | GPT-4.1 | 48 | #505 | |
| 7 | Kimi K2 (0711) | 47.8 | #839 | |
| 8 | Qwen 3 235B A22B 2507 (Thinking) | 47.7 | #469 | |
| 9 | GLM-4.5 | Thinking | 46.6 | #537 |
| 10 | Qwen 3 235B A22B | Thinking | 45.9 | #631 |
| 11 | Doubao Seed 1.6 (250615) | Thinking | 45.3 | #394 |
| 12 | Doubao-Seed-1.6-thinking-250715 | 45 | #338 | |
| 13 | Qwen 3 Coder 480B A35B Instruct | 44.8 | #653 | |
| 14 | doubao-seed-1.6-thinking-250615 | 44.7 | #522 | |
| 15 | Hunyuan TurboS 0716 | 43.8 |
No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.
Interactive version: theaggregate.ai/benchmark?slug=autocodebench · How It Works · Data refreshed daily, snapshot 2026-11-17.