Evolving Requirements - Full StrictPass: leaderboard
61 frozen repository tasks, 207 requirement states. ClaudeCode2.1.74, 200ktoken context, high reasoning, provider-default sampling, network-disabled. Full one complete session; Evo one response/state; Evo+Comp initial plus<=2completion revisions/state; Hy4-Preview high feedbackgenerator20ktokens and<=2publicobservations/message. StrictPass allchecks; NodeCredit partialstate credit; FDAC feedback-discounted adaptivecorrectness. Pooled row withheld. Metric: Native score (%). Source: https://arxiv.org/html/2610.35488v1
Metric: Native score (%). Source: arxiv.org. Saturation forecast: Around January 2028. 12 models tracked. Updated 17 Nov 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | GPT-6 Astra | High | 59 | #8 |
| 2 | GPT-5.6 Sol | High | 52.5 | #33 |
| 3 | Hy4 preview | High | 47.5 | #75 |
| 4 | GLM-5.3 | High | 44.3 | #73 |
| 5 | DeepSeek V4.1 Flash | High | 42.6 | #69 |
| 6 | Kimi K3 | High | 39.3 | #59 |
| 7 | Qwen 3.8 Flash | High | 39.3 | #128 |
| 8 | DeepSeek V4 Flash | High | 37.7 | #265 |
| 9 | DeepSeek V4 Pro | High | 36.1 | #198 |
| 10 | GLM-5.3 Flash | High | 36.1 | #136 |
| 11 | Grok 4.6 | High | 27.9 | #47 |
| 12 | Hy3 | High | 21.3 | #211 |
No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.
Interactive version: theaggregate.ai/benchmark?slug=evolving-requirements-full-strictpass · How It Works · Data refreshed daily, snapshot 2026-11-17.