MHPP: leaderboard

Mostly Hard Python Problems (arXiv:2405.11430), a code generation benchmark of 210 manually curated challenging Python programming problems spanning seven challenge types (distraction, redefinition, shortcut, commonsense, cornercase, complex, and codesense). Scored by greedy pass@1 (%) against private held-out unit tests.

Metric: Pass@1 with greedy decoding (%, 210 hard Python problems). Source: sparksofagi.github.io. Saturation forecast: Estimated already saturated. 26 models tracked. Updated 20 Nov 2026.

Top models

#ModelScoreOverall rank
1O1 Preview68.1#494
2O1 Mini66.2#766
3GPT-4o (2024-05-13)51.4#735
4Claude 3.5 Sonnet (20241022)48.1#654
5Claude 3.5 Sonnet (20240620)45.2#696
6GPT-4 Turbo43.8#1012
7GPT-4o Mini (2024-07-18)42.4#1493
8Mistral Large 2 (Jul)41.4#1178
9DeepSeek V2.540#1231
10DeepSeek-Coder-33B-Instruct32.9#2583
11Gemma 2 27B (IT)28.6#1826
12GPT-3.5 Turbo (0125)28.1#3541
13Codestral 22B25.2#2619
14Claude 3 Haiku (20240307)24.8#2405
15Mixtral 8x22B Instruct21.9#2114

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=mhpp · How It Works · Data refreshed daily, snapshot 2026-11-20.