DevBench - Implementation: leaderboard

Each model, driving the authors' ChatDev-based agent alone, writes a whole repository from a product requirement document, UML diagrams and an architecture design, for each of 22 repositories in Python, C/C++, Java and JavaScript. The score is the share of the repository's reference acceptance tests that pass, weighted by code lines; unlike HumanEval-style function completion, every task is a multi-file project.

Metric: Pass@ Accept. Test (%). Source: raw.githubusercontent.com. Saturation forecast: Around June 2028. 9 models tracked. Updated 17 Nov 2026.

Top models

#ModelScoreOverall rank
1GPT-4 Preview (0125)7.1#1197
2GPT-4 Preview (1106)6.9#980
3DeepSeek-Coder-33B-Instruct4.4#2726
4GPT-3.5 Turbo (1106)4.2#3361
5deepseek-coder-6.7B-instruct2.9#7057
6CodeLlama-34B-Instruct0.6#7944
7CodeLlama-13B-Instruct0.6#8718
8deepseek-coder-1.3B-instruct0#9476
9CodeLlama-7B-Instruct0#9490

No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.

Interactive version: theaggregate.ai/benchmark?slug=devbench-implementation · How It Works · Data refreshed daily, snapshot 2026-11-17.