CAPA (No History): leaderboard
CAPA tests whether a coding assistant learns the recurring ambiguity of one user: each held-out session opens with an ambiguous coding request whose missing detail the same user has resolved the same way before, and the assistant may ask for clarification or submit code. This board is the share of sessions solved within eight turns when the assistant sees only the new request. A simulated user answers clarifying questions and an execution-grounded judge decides whether a submission is correct.
Metric: Executable success (%; share of the 300 held-out coding sessions solved within an eight-turn budget, a submission counting when the execution-grounded judge accepts it from the hidden tests and the reference solution, and a simulated user agent answering clarification questions; no history: the model sees only the new ambiguous request). Source: arxiv.org. Saturation forecast: Around February 2027. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.8 | 88 | #56 |
| 2 | GLM-5.2 | 85.3 | #138 |
| 3 | Gemini 3.5 Flash | 79.3 | #101 |
| 4 | GPT-5.6 Sol | 79 | #30 |
| 5 | Qwen 3.7 Max | 77.7 | #145 |
| 6 | Kimi K2.6 | 77 | #178 |
| 7 | DeepSeek V4 Pro | 76.3 | #188 |
| 8 | Claude Sonnet 4.6 | 76 | #161 |
| 9 | GPT-5.5 | 74.3 | #49 |
| 10 | Qwen 3.5 27B | 56 | #359 |
| 11 | Qwen 3 8B | 23.3 | #1830 |
| 12 | Llama 3.3 70B Instruct | 10 | #1357 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=capa-no-history · How It Works · Data refreshed daily, snapshot 2026-11-13.