CAPA: leaderboard
CAPA tests whether a coding assistant learns the recurring ambiguity of one user: each held-out session opens with an ambiguous coding request whose missing detail the same user has resolved the same way before, and the assistant may ask for clarification or submit code. This board is the share of sessions solved within eight turns when the assistant is also given five resolved sessions from the same user. A simulated user answers clarifying questions and an execution-grounded judge decides whether a submission is correct.
Metric: Executable success (%; share of the 300 held-out coding sessions solved within an eight-turn budget, a submission counting when the execution-grounded judge accepts it from the hidden tests and the reference solution, and a simulated user agent answering clarification questions; same-user history: the model also sees the dialogue traces of five resolved sessions from the same user, whose recurring ambiguity pattern the held-out request repeats). Source: arxiv.org. Saturation forecast: Around February 2027. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.8 | 90 | #56 |
| 2 | GLM-5.2 | 89.7 | #138 |
| 3 | GPT-5.5 | 84.3 | #49 |
| 4 | Claude Sonnet 4.6 | 83.7 | #161 |
| 5 | Qwen 3.7 Max | 82.7 | #145 |
| 6 | Kimi K2.6 | 82 | #178 |
| 7 | Gemini 3.5 Flash | 81.7 | #101 |
| 8 | DeepSeek V4 Pro | 79 | #188 |
| 9 | GPT-5.6 Sol | 78.7 | #30 |
| 10 | Qwen 3.5 27B | 74.3 | #359 |
| 11 | Llama 3.3 70B Instruct | 30 | #1357 |
| 12 | Qwen 3 8B | 27.7 | #1830 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=capa · How It Works · Data refreshed daily, snapshot 2026-11-13.