MathIF: leaderboard

Models solve 420 math problems (GSM8K to AIME) that each carry one to three Python-checkable instructions on length, wording, format or how the answer starts and ends; the score is the share of questions where every instruction is met, one sample per question. Unlike IFEval the instructions ride on math problems the model must also solve.

Metric: Hard accuracy (%): share of 420 constrained math questions whose every constraint is satisfied, 1 sample per question, T=1.0, top-p 0.95, 16k tokens. Source: raw.githubusercontent.com. Saturation forecast: Estimated already saturated. 23 models tracked. Updated 17 Nov 2026.

Top models

#ModelSettingScoreOverall rank
1Qwen 3 14BReasoning50.71#1264
2Qwen 3 4BReasoning44.05#2337
3Qwen 3 32BThinking43.81#965
4DeepSeek R1 Distill Qwen 32B42.62#1806
5DeepSeek R1 Distill Llama 70B41.43#1669
6QwQ-32B40.24#908
7DeepSeek R1 Distill Qwen 14B39.28#2683
8Qwen 3 8BThinking37.86#1868
9OlympicCoder-32B35.95#2519
10Qwen 3 1.7BThinking30.24#5807
11Qwen 3 0.6BThinking27.86#8337
12DeepSeek-R1-Distill-Qwen-7B26.43#7957
13DeepSeek R1 Distill Llama 8B22.14#6937
14s1-32B20.95#1728
15L1-Qwen-1.5B-Exact19.76

No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.

Interactive version: theaggregate.ai/benchmark?slug=mathif · How It Works · Data refreshed daily, snapshot 2026-11-17.