PharmaBatchDB NLQ-to-SQL - SQL Compliance: leaderboard
Metric: SQL compliance rate (%): share of the 60 questions whose generated query is a SELECT with no blocked DML or DDL keyword that references only the active module tables (sqlglot T-SQL AST check), 60 manufacturing questions over a synthetic MS SQL Server database (Batch, MES and CIP modules, 20 each; 18 easy, 24 medium, 18 hard), zero-shot T-SQL generation with the full module schema in the prompt, temperature 0.1, 4-bit quantized models served by Ollama on consumer CPU hardware; higher is better. Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 4 models tracked.
Top models
| # | Model | Score |
|---|---|---|
| 1 | llama3.1:8b-instruct-q4_K_M (Ollama) | 93.33 |
| 2 | qwen2.5-coder:7b-instruct-q4_K_M (Ollama) | 88.33 |
| 3 | mistral:7b-instruct-q4_K_M (Ollama) | 25 |
| 4 | meditron:7b-q4_0 (Ollama) | 0 |
Interactive version: theaggregate.ai/benchmark?slug=pharmabatchdb-nlq-to-sql-sql-compliance · How It Works · Data refreshed daily, snapshot 2026-09-29.