Which chess player is stronger?
Compare two anonymized games and choose the target player with the higher established Lichess rapid rating.
Run Oct 13, 2026, 25 items, $6.14 to run.
Method
25 random player pairs drawn from the same 25 games as Guess the chess rating. Each of the 50 players appears in exactly one primary comparison. Each prompt identifies one target color in each of two different games and supplies anonymized PGNs, result and time control. Ratings, names, links and earlier model predictions are hidden.
The answer key is which target had the higher pre-game Lichess rapid Glicko-2 rating. Models choose A or B. Accuracy counts all 25 choices equally; higher is better. All sixteen contestants completed the same frozen sample, with medium reasoning and an 8,192-token output cap including reasoning. No tools or model-to-model judging are involved.
This is a separate task from numerical rating prediction. The two boards reuse games and should not be treated as independent evidence. The models' earlier numerical guesses are retained as a comparison baseline, not additional model scores on this board.
Results
| # | Model | Accuracy |
|---|---|---|
| 1 | Claude Opus 5.5 (medium reasoning · 25 pairs) | 88.0% |
| 2 | Claude Sonnet 5.5 (medium reasoning · 25 pairs) | 88.0% |
| 3 | Gemini 3.8 Flash (medium reasoning · 25 pairs) | 88.0% |
| 4 | Gemini 3.1 Pro (Preview) (medium reasoning · 25 pairs) | 84.0% |
| 5 | Gemini 3.5 Flash (medium reasoning · 25 pairs) | 84.0% |
| 6 | Gemini 3.7 Flash (medium reasoning · 25 pairs) | 84.0% |
| 7 | Qwen3.8 Max Prime (medium reasoning · 25 pairs) | 84.0% |
| 8 | Claude Fable 5.1 (medium reasoning · 25 pairs) | 80.0% |
| 9 | Gemini 3 Flash (Preview) (medium reasoning · 25 pairs) | 80.0% |
| 10 | Gemini 3.6 Flash (medium reasoning · 25 pairs) | 80.0% |
| 11 | GPT-5.6 Sol (medium reasoning · 25 pairs) | 80.0% |
| 12 | GPT-6 Sol (medium reasoning · 25 pairs) | 76.0% |
| 13 | GPT-6.1 Sol (medium reasoning · 25 pairs) | 76.0% |
| 14 | GPT-5.6 Luna (medium reasoning · 25 pairs) | 72.0% |
| 15 | GPT-6 Astra (medium reasoning · 25 pairs) | 72.0% |
| 16 | GLM-5.3 Flash (medium reasoning · 25 pairs) | 72.0% |
| Always choose Game B | 60.0% | |
| Unbiased random choice (expected) | 50.0% |
- Eighteen pairs have rating gaps of at least 500; only two have gaps below 200. Ten pairs contain exactly one player below 1000; none contain two. This small, deliberately broad sample does not represent the Lichess population.
- Five reversed-order repeats per model are consistency diagnostics and do not enter the score. The displayed cost includes these repeats. A changed answer can reflect order effects or variation between runs.
- A recorded validation amendment retained categorical choices when the confidence number contradicted them. All choices were kept unchanged, with no retries or inversions; contradictory confidence values are excluded from confidence diagnostics. The public evidence records the amendment and every answer.
- Chance and constant-choice rows are references, not model observations. Rank eligibility requires all 25 primary comparisons.
Hardest pairs, games, prompt and every choice
In the main leaderboard
These results are collected for the main leaderboard alongside external benchmarks. New scores enter the rating through the regular audited data updates. Comparisons with that rating therefore share some of its inputs.
Interactive version: theaggregate.ai/our-benchmarks/lichess-pairwise · How It Works · Data refreshed daily, snapshot 2026-10-12.