HalluWorld - Chess (Incorrect FEN): leaderboard

Hallucination rate on HalluWorld-Chess (2026) probes when the prompt also contains an incorrect FEN string, which tests whether the model trusts conflicting testimony over the board. Percent; lower is better.

Metric: Hallucination rate (%) over the seven HalluWorld-Chess probes (about 348 questions) when the prompt also includes an incorrect FEN string, temperature 0 where supported, thinking models at medium effort with up to 16k thinking tokens, every model at most 256 answer tokens; lower is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 14 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1GPT-5.44#135
2GPT-5.5Medium4#54
3O36#237
4O4 Mini8.3#368
5GPT-5.4 MiniMedium22.1#406
6O3 Mini24.7#546
7Claude Opus 4.6Medium25.9#113
8Claude Sonnet 4.6Medium26.1#173
9Kimi K2.6Non-reasoning42.2#192
10GLM-5Non-reasoning47.4#269
11DeepSeek V3.1Non-reasoning50#511
12GPT-4o57.8#751
13Qwen 3 30B A3BNon-reasoning59.5#1156
14GPT-4o Mini62.9#1667

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=halluworld-chess-incorrect-fen · How It Works · Data refreshed daily, snapshot 2026-11-21.