llmbenchmarks.io Local GPU - SimpleQA Verified: leaderboard
SimpleQA Verified (Google DeepMind, 2025) on local hardware: 1,000 short fact-seeking questions curated from OpenAI's SimpleQA with corrected answers. Dataset, revision, anti-hedging prompt, empty system prompt and grading template follow Epoch AI's setup; the grader differs: Gemini 3.1 Flash-Lite at temperature 0 instead of Epoch's GPT-5.6 Terra, never the model being measured, and on Epoch's 1,000 reference answers it grades 1.1 points more generously. Score: correct answers (%) over all 1,000 questions, so a declined answer counts as not correct. llmbenchmarks.io runs this on its own hardware (2026): open-weight models, mostly 4-bit and 8-bit quantised checkpoints (NVFP4, AWQ, QAT) pinned to a Hugging Face repository and revision, served by vLLM 0.29.0 on one consumer machine (2x RTX 5060 Ti, 32 GB VRAM, tensor parallel 2), each with thinking on or off. Unlike vendor or leaderboard scores for the same test, a row measures the quantised deployment a home user would run, so a quantised checkpoint is its own model, not the full-precision original. Labels carry the mode as (Thinking) or (Non-thinking); Llama 3.1 rows (no thinking mode) and Ministral 3 14B Reasoning rows (always reasoning) stay bare. A run on a subset of the test is not read. Higher is better. Source: https://llmbenchmarks.io/local-gpu-benchmarks/
Metric: Correct answers (%) of 1,000 questions. Source: llmbenchmarks.io. Saturation forecast: Around 2030. 24 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Llama 3.1 8B Instruct | 8.3 | #1017 |
| 2 | Qwen 3.5 4B (Thinking) | 7.9 | #514 (Qwen 3.5 4B) |
| 3 | Qwen 3.5 4B (Non-reasoning) | 6.1 | #514 (Qwen 3.5 4B) |
| 4 | Gemma 4 E4B (IT) (Thinking) | 5.4 | |
| 5 | Gemma 4 E4B (IT) (Non-reasoning) | 3.3 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=llmbenchmarks-io-local-gpu-simpleqa-verified · How It Works · Data refreshed daily, snapshot 2026-11-05.