DevBench - Environment Setup: leaderboard
Each model writes the dependency files for 12 Python, Java and JavaScript repositories; the score is the share of repositories whose example usage code then runs in a fixed base Docker image. With 12 repositories each one is 8.3 points, so the source marks these figures as noisy.
Metric: Pass@ Example Usage (%). Source: raw.githubusercontent.com. Saturation forecast: Estimated already saturated. 9 models tracked. Updated 17 Nov 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-4 Preview (1106) | 41.7 | #980 |
| 2 | GPT-4 Preview (0125) | 41.7 | #1197 |
| 3 | GPT-3.5 Turbo (1106) | 33.3 | #3361 |
| 4 | deepseek-coder-6.7B-instruct | 25 | #7057 |
| 5 | CodeLlama-13B-Instruct | 25 | #8718 |
| 6 | CodeLlama-34B-Instruct | 16.7 | #7944 |
| 7 | DeepSeek-Coder-33B-Instruct | 16.7 | #2726 |
| 8 | deepseek-coder-1.3B-instruct | 8.3 | #9476 |
| 9 | CodeLlama-7B-Instruct | 8.3 | #9490 |
No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.
Interactive version: theaggregate.ai/benchmark?slug=devbench-environment-setup · How It Works · Data refreshed daily, snapshot 2026-11-17.