MathConstraint: leaderboard
We introduce MathConstraint, a hard, adaptive benchmark for evaluating the combinatorial reasoning capabilities of LLMs. We combine constraint satisfaction problems with rigorous solver-based verification and design an adaptive generator to create instances that remain challenging as the LLMs improve in their reasoning capabilities. Unlike existing benchmarks that quickly saturate on fixed dataset
Metric: Accuracy (self-reported). Source: benchmarklist.com. Saturation forecast: Around August 2027. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.5 | 66.9 | #31 |
| 2 | Grok 4.20 | 58.1 | #175 |
| 3 | Gemini 3.1 Pro (Preview) | 51.7 | #58 |
| 4 | Qwen 3.6 Plus | 47.4 | #120 |
| 5 | Claude Opus 4.7 | 42.6 | #49 |
| 6 | GPT-OSS-120B | 39.5 | #341 |
| 7 | DeepSeek V4 Pro | 39.5 | #102 |
| 8 | Kimi K2.6 | 35.9 | #104 |
| 9 | DeepSeek V4 Flash | 34 | #137 |
| 10 | Gemini 3.1 Flash Lite | 24.9 | #191 |
| 11 | Claude Sonnet 4.6 | 18.5 | #90 |
| 12 | Llama 3.3 70B Instruct | 8.5 | #534 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=mathconstraint · How It Works · Data refreshed daily, snapshot 2026-10-20.