DevBench - Acceptance Testing: leaderboard

Each model writes acceptance tests (test inputs and expected outputs) from the requirements, design and reference source code for 22 repositories, with one review round; the score is the share of those tests that pass when run against the reference implementation, averaged uniformly over repositories. It grades the tests a model writes, not the code.

Metric: Oracle Test (%). Source: raw.githubusercontent.com. Saturation forecast: Estimated already saturated. 8 models tracked. Updated 17 Nov 2026.

Top models

#ModelScoreOverall rank
1GPT-4 Preview (0125)29.2#1197
2GPT-4 Preview (1106)25.9#980
3DeepSeek-Coder-33B-Instruct13.6#2726
4GPT-3.5 Turbo (1106)11.7#3361
5CodeLlama-34B-Instruct4.5#7944
6deepseek-coder-1.3B-instruct0#9476
7CodeLlama-7B-Instruct0#9490
8CodeLlama-13B-Instruct0#8718

No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.

Interactive version: theaggregate.ai/benchmark?slug=devbench-acceptance-testing · How It Works · Data refreshed daily, snapshot 2026-11-17.