Chess APO Bench: leaderboard
Chess APO Bench (Lesort et al., imec, 2026) is a chess-puzzle benchmark built from 1,118 Lichess puzzles (559 in the test split, stratified by rating and length) to study automatic prompt optimization for frozen LLMs. Given a position in FEN, the model must play the solver's move at every step of the puzzle's reference line; a puzzle counts as solved only if every one of its positions is answered with the single reference move, as on Lichess, so another winning move counts as wrong. The board is the paper's baseline-prompt evaluation (Table 2): puzzle accuracy on the 559-puzzle test set with the unoptimized base prompt, mean of three runs, at the reasoning level listed for each model.
Metric: Puzzle accuracy (%) on the 559-puzzle Lichess test split with the unoptimized base prompt (every position must match the single reference move), mean of three runs. Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | Gemini 3.5 Flash | Medium | 55.4 | #106 |
| 2 | Muse Spark 1.2 Contributor | Low | 31.9 | |
| 3 | GPT-5.6 Luna | Low | 31.72 | #131 |
| 4 | Gemini 3.5 Flash Lite | Low | 26.3 | #299 |
| 5 | DeepSeek V4 Pro (0813) | 14.49 | #97 | |
| 6 | Claude Haiku 4.5 | 11.63 | #583 | |
| 7 | Qwen 3.8 27B | 8.47 | #221 | |
| 8 | Jev 1.13.0 | 7.27 | ||
| 9 | GPT-4o Mini | 5.84 | #1667 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=chess-apo-bench · How It Works · Data refreshed daily, snapshot 2026-11-21.