CodeGolf Bench - Python Pass@1: leaderboard
CodeGolf Bench Python Pass@1 asks LLMs to solve code.golf problems in Python with as few characters as possible and reports the share of problems whose first solution passes all hidden platform tests, estimated from 10 samples (CodeGolf Bench). Higher is better.
Metric: Pass@1 (%): share of problems whose Python solution passes every hidden test, estimated from 10 samples per problem; 115 code.golf problems (holes), solutions checked against the platform's hidden tests through its API; golfing prompt, temperature 0.2, top-p 0.95, up to 32,768 output tokens, 10 samples per problem; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 9 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek R1 | 43.47 | #482 |
| 2 | Llama 4 Maverick | 38.69 | #1110 |
| 3 | Gemini 2.5 Pro (Preview 03-25) | 38.2 | #241 |
| 4 | DeepSeek V3 (0324) | 34.78 | #715 |
| 5 | Gemini 2.5 Flash (Preview 04-17) | 34.18 | #453 |
| 6 | Qwen 3 235B A22B | 32.94 | #628 |
| 7 | Llama 4 Scout | 30.34 | #1818 |
| 8 | DeepSeek R1 Distill Llama 70B | 29.69 | #1669 |
| 9 | Gemma 3 27B (IT) | 23.47 | #1324 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=codegolf-bench-python-pass-1 · How It Works · Data refreshed daily, snapshot 2026-11-21.