HalluWorld - Chess (No FEN): leaderboard

Hallucination rate on HalluWorld-Chess (2026), probes about chess positions and move histories whose ground truth comes from a chess engine, with no FEN string in the prompt. Percent; lower is better.

Metric: Hallucination rate (%) over the seven HalluWorld-Chess probes (about 348 questions, 50 board positions per probe, ground truth from a chess engine) with no FEN string given, temperature 0 where supported, thinking models at medium effort with up to 16k thinking tokens, every model at most 256 answer tokens; lower is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 14 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1GPT-5.5Medium0.3#54
2O32#237
3GPT-5.42.6#135
4O4 Mini5.7#368
5O3 Mini18.7#546
6Claude Sonnet 4.6Medium19.8#173
7GPT-5.4 MiniMedium22.4#406
8Claude Opus 4.6Medium24.7#113
9Kimi K2.6Non-reasoning44.5#192
10DeepSeek V3.1Non-reasoning53.4#511
11GLM-5Non-reasoning56.9#269
12Qwen 3 30B A3BNon-reasoning59.8#1156
13GPT-4o62.9#751
14GPT-4o Mini62.9#1667

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=halluworld-chess-no-fen · How It Works · Data refreshed daily, snapshot 2026-11-21.