MHPP: leaderboard
Mostly Hard Python Problems (arXiv:2405.11430), a code generation benchmark of 210 manually curated challenging Python programming problems spanning seven challenge types (distraction, redefinition, shortcut, commonsense, cornercase, complex, and codesense). Scored by greedy pass@1 (%) against private held-out unit tests.
Metric: Pass@1 with greedy decoding (%, 210 hard Python problems). Source: sparksofagi.github.io. Saturation forecast: Estimated already saturated. 26 models tracked. Updated 20 Nov 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | O1 Preview | 68.1 | #494 |
| 2 | O1 Mini | 66.2 | #766 |
| 3 | GPT-4o (2024-05-13) | 51.4 | #735 |
| 4 | Claude 3.5 Sonnet (20241022) | 48.1 | #654 |
| 5 | Claude 3.5 Sonnet (20240620) | 45.2 | #696 |
| 6 | GPT-4 Turbo | 43.8 | #1012 |
| 7 | GPT-4o Mini (2024-07-18) | 42.4 | #1493 |
| 8 | Mistral Large 2 (Jul) | 41.4 | #1178 |
| 9 | DeepSeek V2.5 | 40 | #1231 |
| 10 | DeepSeek-Coder-33B-Instruct | 32.9 | #2583 |
| 11 | Gemma 2 27B (IT) | 28.6 | #1826 |
| 12 | GPT-3.5 Turbo (0125) | 28.1 | #3541 |
| 13 | Codestral 22B | 25.2 | #2619 |
| 14 | Claude 3 Haiku (20240307) | 24.8 | #2405 |
| 15 | Mixtral 8x22B Instruct | 21.9 | #2114 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=mhpp · How It Works · Data refreshed daily, snapshot 2026-11-20.