BacktestBench - SQL Execution Accuracy: leaderboard
The SQL generation stage of BacktestBench (2026), a quantitative-backtesting benchmark: the share of SQL queries a model writes to fetch market data for a natural-language strategy whose result set matches the reference result set, ignoring row order. Percent; higher is better.
Metric: Execution accuracy (%) of the intermediate SQL: the share of generated queries whose result set matches the ground-truth result set ignoring order, synthetic BacktestBench test split (the expert-crafted subset excluded), each model driving the AutoBacktest pipeline of the authors (Summarizer, SQL Retriever and Python Coder agents) over a PostgreSQL market database, temperature 0.6; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 23 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | Gemini 3 Pro | 98.7 | #145 | |
| 2 | Seed 1.8 | 97.45 | #252 | |
| 3 | Qwen 3 235B A22B 2507 (Thinking) | 97.11 | #466 | |
| 4 | GPT-OSS-120B | 96.58 | #704 | |
| 5 | Kimi K2 (Thinking) | 96.06 | ||
| 6 | MiniMax-M2.1 | 95.02 | #575 | |
| 7 | GLM-4.7 | 94.86 | #380 | |
| 8 | Qwen 3 Next 80B A3B (Thinking) | 94.79 | #594 | |
| 9 | Qwen 3 Max | 94.73 | #415 | |
| 10 | Qwen 3 Coder Plus | 94.03 | #720 | |
| 11 | DeepSeek V3.2 | 92.86 | #419 | |
| 12 | Qwen 3 30B A3B 2507 (Thinking) | 92.86 | #764 | |
| 13 | Qwen 3 32B | 90.72 | #972 | |
| 14 | GPT-OSS-20B | 90.56 | #1183 | |
| 15 | Seed-OSS-36B-Instruct | Non-reasoning | 87.96 | #851 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=backtestbench-sql-execution-accuracy · How It Works · Data refreshed daily, snapshot 2026-11-21.