BIRD-Python - SQL: leaderboard
Metric: LLM-validated execution accuracy (%; generated SQL executed against the corrected ('verified') 1,534 BIRD dev questions (925 simple, 464 moderate, 145 challenging), an answer counting when an LLM validator judges its result equivalent to the reference result; one generation per question). Source: arxiv.org. Saturation forecast: Around December 2026. 14 models tracked.
Top models
| # | Model | Score |
|---|---|---|
| 1 | Qwen 3 Max | 63.1 |
| 2 | DeepSeek R1 | 62.32 |
| 3 | Qwen 3 32B (Thinking) | 61.8 |
| 4 | Qwen 3 Coder 30B A3B Instruct | 61.6 |
| 5 | Qwen 3 Coder Plus | 61.15 |
| 6 | Qwen 3 14B (Reasoning) | 58.34 |
| 7 | starcoder2-7B | 0.91 |
Interactive version: theaggregate.ai/benchmark?slug=bird-python-sql · How It Works · Data refreshed daily, snapshot 2026-09-29.