CodeGolf Bench - Python Pass@1: leaderboard

CodeGolf Bench Python Pass@1 asks LLMs to solve code.golf problems in Python with as few characters as possible and reports the share of problems whose first solution passes all hidden platform tests, estimated from 10 samples (CodeGolf Bench). Higher is better.

Metric: Pass@1 (%): share of problems whose Python solution passes every hidden test, estimated from 10 samples per problem; 115 code.golf problems (holes), solutions checked against the platform's hidden tests through its API; golfing prompt, temperature 0.2, top-p 0.95, up to 32,768 output tokens, 10 samples per problem; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 9 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1DeepSeek R143.47#482
2Llama 4 Maverick38.69#1110
3Gemini 2.5 Pro (Preview 03-25)38.2#241
4DeepSeek V3 (0324)34.78#715
5Gemini 2.5 Flash (Preview 04-17)34.18#453
6Qwen 3 235B A22B32.94#628
7Llama 4 Scout30.34#1818
8DeepSeek R1 Distill Llama 70B29.69#1669
9Gemma 3 27B (IT)23.47#1324

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=codegolf-bench-python-pass-1 · How It Works · Data refreshed daily, snapshot 2026-11-21.