Defects4J — leaderboard
Defects4J: Evaluates software-engineering agents on realistic issue resolution, repository navigation, testing, or maintenance workflows.
Metric: Defects4J Plausible @1 (self-reported). Source: benchmarklist.com. Status: saturation imminent. 33 models tracked.
Top models
| # | Model | Score |
|---|---|---|
| 1 | O4 Mini (2025-04-16) (High) | 53.8 |
| 2 | O3 Mini (High) | 48.8 |
| 3 | Claude 3.7 Sonnet | 47.8 |
| 4 | GPT-4.1 | 45.2 |
| 5 | Claude 3.5 Sonnet | 44.1 |
| 6 | Gemini 2.5 Flash (Preview 05-20) | 43.4 |
| 7 | DeepSeek V3 (0324) | 43 |
| 8 | Gemini 2.5 Pro (Preview 05-06) | 41.4 |
| 9 | DeepSeek V3 | 39.9 |
| 10 | Gemini 1.5 Pro | 36.4 |
| 11 | GPT-4o | 35 |
| 12 | Llama 4 Maverick | 33.7 |
| 13 | Gemini 2.0 Flash | 33 |
| 14 | Grok 2 (1212) | 31 |
| 15 | Gemini 1.5 Pro (001) | 30.3 |
Interactive version: theaggregate.ai/benchmark?slug=defects4j · How the rankings work · Data refreshed daily, snapshot 2026-07-22.