AutoCodeBench: leaderboard

Each model writes one solution to each of 3,920 LLM-generated programming problems spread over 20 languages (184-200 per language), and the score is the average Pass@1 against hidden sandbox tests. Unlike HumanEval-style Python sets, the problems were generated and filtered by DeepSeek models (v1 set; AutoCodeBench-V2 is a later, different set).

Metric: Pass@1 (%). Source: arxiv.org. Saturation forecast: Around February 2027. 33 models tracked. Updated 17 Nov 2026.

Top models

#ModelSettingScoreOverall rank
1O3 (2025-04-16)High51.1#229
2Grok 450.9#315
3Claude Opus 4Non-reasoning50.9#321
4O4 Mini (2025-04-16)50#376
5Claude Sonnet 4Non-reasoning49.3#411
6GPT-4.148#505
7Kimi K2 (0711)47.8#839
8Qwen 3 235B A22B 2507 (Thinking)47.7#469
9GLM-4.5Thinking46.6#537
10Qwen 3 235B A22BThinking45.9#631
11Doubao Seed 1.6 (250615)Thinking45.3#394
12Doubao-Seed-1.6-thinking-25071545#338
13Qwen 3 Coder 480B A35B Instruct44.8#653
14doubao-seed-1.6-thinking-25061544.7#522
15Hunyuan TurboS 071643.8

No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.

Interactive version: theaggregate.ai/benchmark?slug=autocodebench · How It Works · Data refreshed daily, snapshot 2026-11-17.