BIRD-Python: leaderboard
BIRD-Python (Hu, Hou, Cao and Li, 2026) rebuilds the BIRD text-to-SQL dev set, with 259 of its 1,534 questions corrected, as a text-to-Python benchmark: the model gets a database schema, a question and hints and must return code whose executed output an LLM validator judges equivalent to the reference result. This board is execution accuracy in percent for pandas code on the corrected set; higher is better.
Metric: LLM-validated execution accuracy (%; generated pandas code executed against the corrected ('verified') 1,534 BIRD dev questions (925 simple, 464 moderate, 145 challenging), an answer counting when an LLM validator judges its output equivalent to the reference result; one generation per question). Source: arxiv.org. Saturation forecast: Around February 2027. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 Max | 63.43 | #403 |
| 2 | DeepSeek R1 | 62.52 | #473 |
| 3 | Qwen 3 Coder Plus | 61.6 | #717 |
| 4 | Qwen Max | 60.82 | #1051 |
| 5 | Qwen 3 32B (Thinking) | 60.1 | #960 (Qwen 3 32B) |
| 6 | Qwen 3 14B (Reasoning) | 47.98 | #1233 (Qwen 3 14B) |
| 7 | Qwen 3 Coder 30B A3B Instruct | 45.83 | #972 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=bird-python · How It Works · Data refreshed daily, snapshot 2026-11-13.