NL2Repo: leaderboard
Each task hands an agent a requirements document in natural language and an empty workspace, and it must build a complete installable Python library on its own. The 104 tasks are graded by running the upstream project's pytest suite, so the number is the average share of tests that pass; this board collects figures from benchmarklist.com, some self-reported, and higher is better. Source: arxiv.org/abs/2512.12730.
Metric: Score (mixed sources). Source: z.ai. Source: benchmarklist.com. Saturation forecast: Around September 2027. 53 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 5 (Max) | 75.3 | #12 (Claude Opus 5) |
| 2 | Claude Opus 5 | 72.3 | #12 |
| 3 | Claude Fable 5 | 70.2 | #16 |
| 4 | Claude Opus 4.8 | 69.7 | #37 |
| 5 | DeepSeek V4.1 Flash (1017) | 68.2 | #20 |
| 6 | DeepSeek V4.1 Flash | 65.4 | #36 |
| 7 | Qwen 3.8 Max (0902) | 64.9 | #29 |
| 8 | DeepSeek V4 Pro (0813) (Max) | 61.5 | #53 (DeepSeek V4 Pro (0813)) |
| 9 | DeepSeek V4 Pro (0813) | 61.1 | #53 |
| 10 | Hy4 preview | 58.9 | #38 |
| 11 | Kimi K3 | 58 | #30 |
| 12 | GLM-5.3 | 58 | #39 |
| 13 | Kimi K3 (Max) | 58 | #30 (Kimi K3) |
| 14 | GLM-5.3 (Max) | 58 | #39 (GLM-5.3) |
| 15 | DeepSeek-V4-Flash-Vision-Exp | 57.7 | #76 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=nl2repo · How It Works · Data refreshed daily, snapshot 2026-10-20.