DevBench - Environment Setup: leaderboard

Each model writes the dependency files for 12 Python, Java and JavaScript repositories; the score is the share of repositories whose example usage code then runs in a fixed base Docker image. With 12 repositories each one is 8.3 points, so the source marks these figures as noisy.

Metric: Pass@ Example Usage (%). Source: raw.githubusercontent.com. Saturation forecast: Estimated already saturated. 9 models tracked. Updated 17 Nov 2026.

Top models

#ModelScoreOverall rank
1GPT-4 Preview (1106)41.7#980
2GPT-4 Preview (0125)41.7#1197
3GPT-3.5 Turbo (1106)33.3#3361
4deepseek-coder-6.7B-instruct25#7057
5CodeLlama-13B-Instruct25#8718
6CodeLlama-34B-Instruct16.7#7944
7DeepSeek-Coder-33B-Instruct16.7#2726
8deepseek-coder-1.3B-instruct8.3#9476
9CodeLlama-7B-Instruct8.3#9490

No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.

Interactive version: theaggregate.ai/benchmark?slug=devbench-environment-setup · How It Works · Data refreshed daily, snapshot 2026-11-17.