CAPA - First-Turn Success: leaderboard
CAPA tests whether a coding assistant learns the recurring ambiguity of one user: each held-out session opens with an ambiguous coding request whose missing detail the same user has resolved the same way before, and the assistant may ask for clarification or submit code. This board is the share of sessions solved at the first turn, with no clarification asked, when the assistant is also given five resolved sessions from the same user. A simulated user answers clarifying questions and an execution-grounded judge decides whether a submission is correct.
Metric: First-turn executable success (%; share of the 300 held-out coding sessions whose first assistant turn is an accepted submission, with no clarification asked; same-user history: the model also sees the dialogue traces of five resolved sessions from the same user, whose recurring ambiguity pattern the held-out request repeats). Source: arxiv.org. Saturation forecast: Around 2028. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.8 | 60.3 | #56 |
| 2 | GLM-5.2 | 46.7 | #138 |
| 3 | Gemini 3.5 Flash | 38.7 | #101 |
| 4 | Kimi K2.6 | 32.3 | #178 |
| 5 | GPT-5.5 | 31 | #49 |
| 6 | Qwen 3.7 Max | 29.7 | #145 |
| 7 | Qwen 3 8B | 23 | #1830 |
| 8 | Qwen 3.5 27B | 21.3 | #359 |
| 9 | GPT-5.6 Sol | 18.3 | #30 |
| 10 | DeepSeek V4 Pro | 17.7 | #188 |
| 11 | Llama 3.3 70B Instruct | 16 | #1357 |
| 12 | Claude Sonnet 4.6 | 14 | #161 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=capa-first-turn-success · How It Works · Data refreshed daily, snapshot 2026-11-13.