llmbenchmarks.io Local GPU - Chess Puzzles: leaderboard
Epoch AI's Chess Puzzles task (version 2.0.1) on local hardware: 100 positions, each with one best continuation, answered through Epoch's qa_agent scaffold and graded by exact comparison with the solution, with no grader model. The positions, task texts, scaffold and grading are Epoch's; the run differs in serving the model from a local endpoint with a 131,072-token window (compaction at 90% of the window and a 1,000,000-token output budget, as in Epoch's settings), in using vendor sampling settings, and in ending a puzzle without an answer when the server rejects a call at the window limit. Score: puzzles solved (%). Runs before 2026-10-25 (an older task version on a substitute puzzle set) are not read. llmbenchmarks.io runs this on its own hardware (2026): open-weight models, mostly 4-bit and 8-bit quantised checkpoints (NVFP4, AWQ, QAT) pinned to a Hugging Face repository and revision, served by vLLM 0.29.0 on one consumer machine (2x RTX 5060 Ti, 32 GB VRAM, tensor parallel 2), each with thinking on or off. Unlike vendor or leaderboard scores for the same test, a row measures the quantised deployment a home user would run, so a quantised checkpoint is its own model, not the full-precision original. Labels carry the mode as (Thinking) or (Non-thinking); Llama 3.1 rows (no thinking mode) and Ministral 3 14B Reasoning rows (always reasoning) stay bare. A run on a subset of the test is not read. Higher is better. Source: https://llmbenchmarks.io/local-gpu-benchmarks/
Metric: Puzzles solved (%), exact match. Source: llmbenchmarks.io. Saturation forecast: Around 2029. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 4 E4B (IT) (Non-reasoning) | 3 | |
| 2 | Llama 3.1 8B Instruct | 0 | #1017 |
| 3 | Qwen 3.5 4B (Non-reasoning) | 0 | #514 (Qwen 3.5 4B) |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=llmbenchmarks-io-local-gpu-chess-puzzles · How It Works · Data refreshed daily, snapshot 2026-11-05.