BIRD-Python - SQL: leaderboard
BIRD-Python (Hu, Hou, Cao and Li, 2026) rebuilds the BIRD text-to-SQL dev set, with 259 of its 1,534 questions corrected, as a text-to-Python benchmark: the model gets a database schema, a question and hints and must return code whose executed output an LLM validator judges equivalent to the reference result. This board is execution accuracy in percent when the model writes SQL instead, on the same corrected set; higher is better.
Metric: LLM-validated execution accuracy (%; generated SQL executed against the corrected ('verified') 1,534 BIRD dev questions (925 simple, 464 moderate, 145 challenging), an answer counting when an LLM validator judges its result equivalent to the reference result; one generation per question). Source: arxiv.org. Saturation forecast: Around February 2027. 14 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 Max | 63.1 | #403 |
| 2 | DeepSeek R1 | 62.32 | #473 |
| 3 | Qwen 3 32B (Thinking) | 61.8 | #960 (Qwen 3 32B) |
| 4 | Qwen 3 Coder 30B A3B Instruct | 61.6 | #972 |
| 5 | Qwen 3 Coder Plus | 61.15 | #717 |
| 6 | Qwen 3 14B (Reasoning) | 58.34 | #1233 (Qwen 3 14B) |
| 7 | starcoder2-7B | 0.91 | #9496 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=bird-python-sql · How It Works · Data refreshed daily, snapshot 2026-11-13.