CAPA (No History) - First-Turn Success: leaderboard
CAPA tests whether a coding assistant learns the recurring ambiguity of one user: each held-out session opens with an ambiguous coding request whose missing detail the same user has resolved the same way before, and the assistant may ask for clarification or submit code. This board is the share of sessions solved at the first turn, with no clarification asked, when the assistant sees only the new request. A simulated user answers clarifying questions and an execution-grounded judge decides whether a submission is correct.
Metric: First-turn executable success (%; share of the 300 held-out coding sessions whose first assistant turn is an accepted submission, with no clarification asked; no history: the model sees only the new ambiguous request). Source: arxiv.org. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.8 | 24.3 | #56 |
| 2 | Kimi K2.6 | 22.3 | #178 |
| 3 | GLM-5.2 | 18.7 | #138 |
| 4 | Qwen 3 8B | 18.3 | #1830 |
| 5 | Qwen 3.7 Max | 17.7 | #145 |
| 6 | Gemini 3.5 Flash | 17 | #101 |
| 7 | DeepSeek V4 Pro | 15.3 | #188 |
| 8 | Qwen 3.5 27B | 10.7 | #359 |
| 9 | Claude Sonnet 4.6 | 6.3 | #161 |
| 10 | Llama 3.3 70B Instruct | 5.7 | #1357 |
| 11 | GPT-5.6 Sol | 2.7 | #30 |
| 12 | GPT-5.5 | 2.3 | #49 |
Interactive version: theaggregate.ai/benchmark?slug=capa-no-history-first-turn-success · How It Works · Data refreshed daily, snapshot 2026-11-13.