Which chess player is stronger?

Sixteen models compared 25 random pairs of players, one anonymized game per player. The score is whether the chosen player had the higher established pre-game Lichess rapid rating.

322/400 correct choices (80.5%), compared with 80.0% when ordering the same players by the models’ earlier numerical guesses. Each model answered the same 25 pairs; these are not 400 independent cases.

Full evidence JSON · All primary choices CSV · Exact prompt template

Most frequently wrong pairs

Sorted by the number of models that chose the lower-rated player. Links identify the target color and open the original game. Each source game contributes two players, compared against different games.

ComparisonModels wrongTarget ATarget BRating gap
Pair 0714/16Black, rated 1188White, rated 1310122
Pair 0513/16White, rated 2193Black, rated 1593600
Pair 0212/16Black, rated 1159White, rated 762397
Pair 167/16Black, rated 1273White, rated 1575302
Pair 206/16White, rated 1228White, rated 557671
Pair 096/16Black, rated 1811Black, rated 190190
Pair 245/16White, rated 1782Black, rated 2154372
Pair 193/16White, rated 1514White, rated 26891175
Pair 123/16Black, rated 1867Black, rated 1382485
Pair 222/16Black, rated 2718Black, rated 15511167
Pair 252/16Black, rated 2474White, rated 1862612
Pair 131/16White, rated 717White, rated 25081791
Pair 111/16Black, rated 743Black, rated 25011758
Pair 231/16White, rated 738White, rated 24261688
Pair 011/16Black, rated 1569White, rated 26681099
Pair 081/16Black, rated 2410Black, rated 1903507
Pair 100/16Black, rated 564Black, rated 21771613
Pair 210/16White, rated 739White, rated 21571418
Pair 140/16Black, rated 2641White, rated 12491392
Pair 040/16Black, rated 557Black, rated 17911234
Pair 180/16Black, rated 765White, rated 18771112
Pair 150/16White, rated 1120White, rated 1932812
Pair 060/16Black, rated 1639Black, rated 838801
Pair 170/16White, rated 1126White, rated 1752626
Pair 030/16White, rated 1363White, rated 1112251

Model results

ModelCorrectAccuracyEarlier estimatesSame choice after reversal
Claude Opus 5.522/2588%74%5/5
Claude Sonnet 5.522/2588%76%3/5
Gemini 3.8 Flash22/2588%86%4/5
Gemini 3.1 Pro (Preview)21/2584%92%4/5
Gemini 3.5 Flash21/2584%80%3/5
Gemini 3.7 Flash21/2584%80%4/5
Qwen3.8 Max Prime21/2584%58%3/5
Claude Fable 5.120/2580%82%3/5
Gemini 3 Flash (Preview)20/2580%82%5/5
Gemini 3.6 Flash20/2580%86%4/5
GPT-5.6 Sol20/2580%80%5/5
GPT-6 Sol19/2576%80%4/5
GPT-6.1 Sol19/2576%80%4/5
GPT-5.6 Luna18/2572%78%4/5
GPT-6 Astra18/2572%80%5/5
GLM-5.3 Flash18/2572%86%4/5

Method and limits

Every target player appears once. Pairing used seed 20261013 without consulting ratings or model guesses; same-game comparisons and duplicate game pairs were rejected. All models received the same prompts, medium reasoning, an 8,192-token output cap and no tools. Names, ratings, links and prior model guesses were hidden.

The labels describe established strength, not a verdict on who played the better individual game. Eighteen pairs differ by at least 500 rating points; only two differ by less than 200. Ten pairs include one player below 1000, and none include two. The sample is small and deliberately broad.

Five reversed-order repeats per model are diagnostics and never enter accuracy. Models chose the same underlying player in 64/80 repeats. Order effects and ordinary variation between generations are not separated by this check.

Validation amendment: 35 primary responses supplied a confidence number that contradicted the A/B choice. We retained the categorical choice unchanged and excluded those confidence values from confidence diagnostics. The change was recorded after early instances were seen; no call was retried and no answer inverted. The JSON preserves the amendment and all categorical answers, including reversed repeats.

Earlier-estimate accuracy gives tied numerical guesses half credit. Always choosing B scores 60%; unbiased random choice has expected accuracy 50%. New API spend, including repeats: $6.1434.

Exact prompt template

The placeholders are replaced with the target colors and anonymized PGNs shown for each pair.

Compare the established playing strength of two chess players from one complete game each. Both games are Lichess rapid games.

The target in Game A is WHITE/BLACK as specified below. The target in Game B is separately specified. Judge only these two target players, using the moves and their opponents' play as evidence. Infer which target player likely had the higher established Lichess rapid rating before these games. A single game's winner need not be the stronger player across games.

Choose A or B even if uncertain. Give the probability that target A was stronger, from 0 to 1, consistent with your choice (0.5 is allowed for a forced guess). Give a concise explanation, at most 120 words, citing concrete moves with game and color. Do not provide numerical rating estimates. Use only the supplied games; no external tools.

GAME A — target player: {color_a}
{pgn_a}

GAME B — target player: {color_b}
{pgn_b}

Return JSON with exactly: stronger ("A" or "B"), probability_a_stronger (number), explanation (string).

Pair 07: 14 of 16 models wrong

A: Black, rated 1188 · B: White, rated 1310. Higher pre-game rating: B.

Game A: Black, rated 1188 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 d6 2. Bc4 e6 3. Nf3 h6 4. e5 c6 5. exd6 Bxd6 6. O-O Be7 7. d4 Nf6 8. Nh4
O-O 9. Qf3 Nd5 10. Bxd5 Bxh4 11. Bc4 Bg5 12. Bxg5 Qxg5 13. Nc3 Re8 14. Ne4 Qf5
15. Qg3 Rd8 16. Bd3 Na6 17. Nd6 Qf6 18. Ne4 Qg6 19. Qe3 f5 20. Nc5 Nxc5 21.
dxc5 Rd5 22. c4 Rd8 23. Rae1 Bd7 24. f4 Re8 25. Rf3 Qf6 26. b3 Rad8 27. Rg3 Bc8
28. Qe5 Qxe5 29. fxe5 Rd4 30. Rg6 Rxd3 0-1
Game B: White, rated 1310 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. Nc3 e5 2. d3 d5 3. e3 Bb4 4. Bd2 Nf6 5. g3 O-O 6. Bg2 c6 7. Nf3 Bg4 8. O-O
Bxc3 9. Bxc3 e4 10. dxe4 Nxe4 11. Bb4 Re8 12. Qd4 Be6 13. Rad1 Nd7 14. Rfe1 c5
15. Bxc5 Ndxc5 16. Ne5 Qb6 17. Bxe4 Nxe4 18. Qxb6 axb6 19. a3 f6 20. Nf3 Bg4
21. Kg2 Ng5 22. Rd4 Bxf3+ 23. Kg1 Be4 24. h4 Nf3+ 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — wrong

Game A Black played an odd opening (1...d6 2...e6 3...h6) but punished White's 10.Bxd5?? with 10...Bxh4, winning a piece. Afterward Black consolidated reasonably, and 24...Re8 and 26...Rad8 led to 29...Rd4 and 30...Rxd3, picking up the bishop. Game B White played a passive setup and then blundered: 15.Bxc5?? simply dropped a piece. That was followed by 21.Kg2?, allowing 22...Bxf3+, and then 23.Kg1, allowing the fork 24...Nf3+. That is repeated tactical blindness. Both opponents were weak, so the evidence is limited. Still, A's target showed better tactical awareness, while B's target made several unforced blunders.

Reported P(A stronger): 0.6. Consistent with choice.

Claude Sonnet 5.5: chose A — wrong

Game A, Black: answered the early e5 with ...c6 and ...Be7 and developed sensibly. Played ...Nd5 and ...Bxh4 to win the knight pair, then ...Qf5 and ...Qg6 to improve the queen. ...f5 and ...Rd5 were active, and ...Rd4 followed by ...Rxd3 won a piece after White's Rg6 blunder. This was a solid, purposeful game against a mediocre opponent. Game B, White: played the opening passively and allowed ...e4 with tempo. 15.Bxc5 and 16.Ne5 were questionable, 18.Qxb6 axb6 left a worse structure, and 21.Kg2 walked into ...Ng5. 22.Rd4 and 24.h4 lost to ...Nf3+ smothered-style tactics. White's play had more inaccuracies and a tactical oversight, while Black in Game A was steadier. The evidence is modest, and both opponents seem to be around the same level.

Reported P(A stronger): 0.64. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game A features severe tactical oversights by both players characteristic of lower ratings. In Game A, Black (Target A) missed winning a free knight with 15...Qxe4 (playing 15...Rd8 instead), and on move 17 played 17...Qf6, completely hanging the undefended rook on e8 (which White inexplicably failed to capture). In Game B, White (Target B) played a somewhat passive opening and lost a piece to an alert fork (14...c5), but Black in Game B demonstrated strong tactical pressure (20...Bg4 and 21...Ng5). Target B's play was flawed, but avoiding basic one-move blunders suggests a higher rating than Target A.

Reported P(A stronger): 0.28. Consistent with choice.

Gemini 3.1 Pro (Preview): chose A — wrong

Target A (Black) played a somewhat passive but solid game, avoiding major tactical errors before eventually capitalizing on White's blatant blunder (30...Rxd3). Target B (White) demonstrated a very passive and disjointed opening setup (1.Nc3, 2.d3, 3.e3, 5.g3), fell for a simple pawn fork (14...c5), and later allowed a devastating knight fork (24.h4 Nf3+). Target A's ability to maintain a solid position and avoid basic tactical oversights suggests a higher rating than Target B's error-prone and excessively passive play.

Reported P(A stronger): 0.65. Consistent with choice.

Gemini 3.5 Flash: chose A — wrong

In Game A, Black (Target A) displayed reasonable tactical awareness, capitalizing on White's mistake to win a knight with 10... Bxh4 and safely converting the material advantage to a win. In contrast, in Game B, White (Target B) played a very passive opening and exhibited poor tactical vision. Target B overlooked a simple pawn fork, losing a piece to 14... c5, and later allowed a devastating pin with 20. Nf3 Bg4 and 21... Ng5, leading to a quick collapse. Target A's play was significantly more solid and tactically alert.

Reported P(A stronger): 0.75. Consistent with choice.

Gemini 3.7 Flash: chose A — wrong

Game A Black demonstrated solid positional fundamentals, defending their center, contesting the d-file, and cleanly punishing White's final blunder with 30...Rxd3 (though missing the free knight earlier on move 15). In contrast, Game B White played an extremely passive and disjointed opening setup (Nc3, d3, e3, g3), walked into a simple pawn fork with 14...c5 losing a piece on c5 without compensation, and subsequently collapsed to the pin on f3 with 20...Bg4 and 21...Ng5. Target A displayed much better board vision and coordination.

Reported P(A stronger): 0.72. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

Game A Black wins mostly from White blundering away a rook: 29...Rd4 30.Rg6?? Rxd3, with Black's play (16...Na6, 18...Qg6, 22...Rd8) competent but unexceptional. Game B White, despite losing, played 16.Ne5 and 17.Bxe4, creating real complications, and only lost to a slick finish 22...Bxf3+ 23.Kg1 Be4 24...Nf3+. B's active middlegame ideas and tighter calculation against stronger opposition suggest a slightly higher rating.

Reported P(A stronger): 0.42. Consistent with choice.

Claude Fable 5.1: chose A — wrong

Game A Black played a passive setup (1...d6 2...e6 3...h6, 5...Bxd6 6...Be7 retreat) but found reasonable simplifications (10...Bxh4, 11...Bg5) and won material after White's 29.fxe5?? 30.Rg6?? blunder (30...Rxd3). Black also made odd shuffles (15...Rd8, 21...Rd5, 22...Rd8). Game B White built a solid Van Geet/fianchetto setup, but 14.Rfe1? allowed 14...c5 winning the Bb4 and a piece via 15.Bxc5 Ndxc5, then 20.Nf3 Bg4 21.Kg2 Ng5 22.Rd4?? Bxf3+ dropped more material. White B's errors were unforced piece-losing blunders against modest opposition; Black A's play, while passive, contained fewer outright material losses. Slight edge to A.

Reported P(A stronger): 0.55. Consistent with choice.

Gemini 3 Flash (Preview): chose A — wrong

Target A (Black) demonstrated solid fundamentals, winning material early (10... Bxh4) and maintaining a clear advantage through the endgame. While Black missed a winning tactical opportunity on move 15, their play remained cohesive and focused. Conversely, Target B (White) displayed poor tactical awareness, blundering a full piece to a basic pawn fork (14... c5). Target B subsequently failed to manage the pin on the f3-knight, allowing the decisive fork 24. Nf3+. The disparity in tactical oversight suggests Target A is the stronger player; Target B's loss of material to a simple one-move threat indicates a lower level of established play compared to Target A's overall stability.

Reported P(A stronger): 0.65. Consistent with choice.

Gemini 3.6 Flash: chose A — wrong

Target A (Black) played a solid positional game despite a passive opening (6...Be7) and missing a free knight with 15...Qxe4. However, A maintained structural integrity, defended well, and accurately punished White's 30. Rg6 blunder with 30...Rxd3. In contrast, Target B (White) demonstrated severe tactical oversights early in Game B. B allowed a troubling pin with 9...e4, missed the basic 14...c5 fork that lost a bishop, and later allowed 20...Bg4 and 21...Ng5, walking into a fatal knight fork on f3. Target A showed far better fundamental tactical awareness and stability.

Reported P(A stronger): 0.8. Consistent with choice.

GPT-5.6 Sol: chose A — wrong

Game A’s Black played more consistently and purposefully: 9...Nd5 answered White’s pressure, 19...f5 challenged the centralized knight, and 21...Rd5 followed by ...Rd8/...Rad8 coordinated both rooks before 30...Rxd3 punished the loose bishop. Game B’s White showed reasonable development and simplification with 17.Bxe4 and 18.Qxb6, but the passive opening conceded initiative, and 22.Rd4? overlooked 22...Bxf3+, where the bishop could escape because ...Ng5 protected f3. Then 24.h4? allowed 24...Nf3+, also forking the rook on d4. These tactical oversights make A more likely to have had the higher established rapid strength.

Reported P(A stronger): 0.78. Consistent with choice.

GPT-6 Sol: chose A — wrong

Game A’s Black handled the exchanges coherently with 20...Nxc5 and 28...Qxe5, then exploited White’s exposed bishop with 29...Rd4 and 30...Rxd3. Game B’s White made more consequential errors: 15.Bxc5 gave up a bishop for a pawn after 15...Ndxc5, and 17.Bxe4 Nxe4 lost the other bishop. Black then won White’s remaining knight with 22...Bxf3+. A looks substantially stronger despite an unpolished opening.

Reported P(A stronger): 0.95. Consistent with choice.

GPT-6.1 Sol: chose A — wrong

Game A Black showed better tactical awareness: 10...Bxh4 answered White’s capture on d5 by recovering a knight, and 20...Nxc5 removed White’s advanced knight. Black then coordinated the rooks with 26...Rad8 and 29...Rd4, exploiting 30.Rg6 with 30...Rxd3. The opening was slow, notably ...h6 and the bishop retreat to e7, but subsequent play was coherent. Game B White developed reasonably, yet 15.Bxc5 surrendered a bishop for a pawn without adequate compensation. Later, 22.Rd4 left the knight on f3 available to ...Bxf3+, and 24.h4 allowed ...Nf3+, forking both rooks. These repeated material oversights favor A.

Reported P(A stronger): 0.87. Consistent with choice.

GPT-5.6 Luna: chose A — wrong

Game A’s Black showed more purposeful coordination and tactical awareness: after 9...Nd5, 10...Bxh4 exploited the cleared diagonal, and the rooks became very active with 21...Rd5, 29...Rd4, and 30...Rxd3. Game B’s White developed passively with 1.Nc3 and 2.d3, then allowed Black’s initiative to grow; 22.Rd4 permitted 22...Bxf3+, followed by the decisive-looking 24...Nf3+. Although Game B’s White made some useful tactical exchanges, Game A’s Black appeared more consistently effective, so A is more likely to have had the higher established rapid strength.

Reported P(A stronger): 0.68. Consistent with choice.

GPT-6 Astra: chose A — wrong

Game A Black played a slow, somewhat aimless opening, but showed tactical awareness with 10...Bxh4, developed both rooks, and used 28...Qxe5 followed by 29...Rd4 to exploit White’s position, taking the loose bishop with 30...Rxd3. Game B White made more serious material errors: 15.Bxc5? Ndxc5 surrendered a bishop for a pawn, while 22.Rd4? allowed ...Bxf3+, losing another minor piece. Finally, 24.h4 permitted ...Nf3+, forking the king and both rooks. Despite A benefiting from weak opposition, B’s repeated failures to track threats make A the likelier stronger established player.

Reported P(A stronger): 0.86. Consistent with choice.

GLM-5.3 Flash: chose A — wrong

Game A's Black played coherently: good trades (10...Bxh4, 12...Qxg5), sensible regrouping with ...Na6, ...Bd7, and careful maneuvering (17.Ne4... wait, Black's ...f5 and ...Qf6 redeployment), then converted a small edge cleanly, with accurate rook play 29...Rxd4 30...Rxd3 winning a pawn. Game B's White blundered chronically: 14.Bxc5 allowed ...Ndxc5 regaining, and after 22.Rd4 Bxf3+ 23.Kg1 Be4 24.h4?? White lost to ...Nf3+ with the rook on d4 hanging and king unsafe—decisive carelessness. Black also erred little. A's target showed better technique and fewer mistakes, so A was likely higher-rated.

Reported P(A stronger): 0.72. Consistent with choice.

Pair 05: 13 of 16 models wrong

A: White, rated 2193 · B: Black, rated 1593. Higher pre-game rating: A.

Game A: White, rated 2193 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. d4 Nf6 2. c4 c5 3. d5 d6 4. Nc3 g6 5. e4 Bg7 6. g3 O-O 7. Bg2 Na6 8. a3 Bd7
9. Nge2 Nc7 10. O-O b5 11. cxb5 Nxb5 12. Bd2 Rb8 13. Qc1 Ng4 14. h3 Ne5 15. Kh2
Nc4 16. Rb1 Nxd2 17. Qxd2 Qa5 18. Qc2 Rfc8 19. f4 Nd4 20. Qd3 Nxe2 21. Qxe2
Bxc3 22. bxc3 Qxa3 23. Ra1 Qxc3 24. Rfc1 Qd4 25. Rxa7 Rb2 26. Qf3 Qd2 27. Rg1
Rd8 28. e5 Bf5 29. g4 Bb1 30. f5 dxe5 31. fxg6 hxg6 32. Rxe7 Qf4+ 33. Qxf4 exf4
34. Kh1 Ba2 35. Rc1 Bxd5 36. Bxd5 Rxd5 37. Re8+ Kg7 38. Rc8 Rdd2 39. R8xc5 Rh2+
40. Kg1 Rbg2+ 41. Kf1 f3 0-1
Game B: Black, rated 1593 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. d4 d5 2. c4 c6 3. e3 Bf5 4. Nf3 Nf6 5. Nbd2 h6 6. Be2 Nbd7 7. O-O e6 8. cxd5
exd5 9. h3 Bd6 10. b3 O-O 11. a4 Ne4 12. Nxe4 Bxe4 13. Ba3 Bxa3 14. Rxa3 Nf6
15. Ne5 Nd7 16. Nxd7 Qxd7 17. b4 a6 18. Rb3 Qe6 19. b5 axb5 20. axb5 Qg6 21.
Bf3 Bc2 22. Bh5 Bxd1 23. Bxg6 Bxb3 24. Bd3 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Game A White handled a Benoni-type opening soundly (6.g3, 9.Nge2, 10.O-O) against a clearly competent opponent, but was outplayed tactically: 21...Bxc3 22.bxc3 Qxa3 cost two pawns, and the later 32.Rxe7 Qf4+ allowed a winning endgame for Black. Still, White kept active play (28.e5, 30.f5) against strong opposition. Game B Black played a solid Slav and found 21...Bc2, forking queen and rook, but this was against a very weak opponent: 21.Bf3?? and 22.Bh5?? simply dropped the queen. That gives little evidence about Black's real strength. The tougher opposition in Game A slightly favors White there.

Reported P(A stronger): 0.6. Consistent with choice.

Claude Sonnet 5.5: chose A — correct

Game A (White) played a reasonable King's Indian/Benoni setup with 9.Nge2, 10.O-O and 12.Bd2. It was then outplayed: 13.Qc1 and 14.h3 allowed ...Ng4-e5-c4 and ...Qxa3 (22...Qxa3 won a pawn), and 19.f4 allowed 19...Nd4. White still fought on with activity (25.Rxa7, 32.Rxe7), so this looks like a solid mid-level player with some tactical lapses. Game B (White) made a serious blunder, 21.Bf3 followed by 22.Bh5, which lost the queen for a bishop and the exchange. The earlier play was passive, and 13.Ba3 and 17.b4 were not harmful but gave Black easy equality. Black in Game B played accurately, but that player is not the target. Target B (Black) played a standard Slav/QGD structure well, winning 12...Bxe4 and 21...Bc2 tactically, though against a weak opponent. Evidence for B is limited because the opponent was weak, while Game A's Black was stronger. Slight lean to A.

Reported P(A stronger): 0.6. Consistent with choice.

Gemini 3.8 Flash: chose B — wrong

Game B Black showed excellent positional handling of the Slav/London structure, trading pieces harmoniously and setting up a decisive tactical strike with 20...Qg6! (threatening mate on g2) followed by 21...Bc2! to cleanly net a rook. In Game A, White handled the Benoni setup sluggishly (13. Qc1, 14. h3, 15. Kh2), allowing Black free rein on the queenside. White then blundered a pawn with 21. Qxe2? and drifted into a completely passive, dominated position along the b-file and second rank. Target B exhibited cleaner tactical precision and overall coordination than Target A.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — wrong

Target A (White in Game A) plays a passive and slightly awkward opening (8. a3, 12. Bd2, 13. Qc1) and is outplayed positionally before blundering material with 21. Qxe2 Bxc3. They are eventually ground down in a lost endgame. Target B (Black in Game B) plays a solid and very well-developed Slav Defense, easily equalizing. They demonstrate excellent tactical awareness by exploiting White's inaccuracies, culminating in the sharp sequence 20... Qg6 and 21... Bc2!, which cleanly wins a rook through an intermediate exchange of queens. Target B's clean, mistake-free play strongly suggests a higher rating.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.5 Flash: chose B — wrong

Game B Black showed excellent positional play and tactical alertness in a Slav Defense. Black equalized smoothly, then launched a decisive tactical sequence with 20...Qg6 (threatening mate) and 21...Bc2, utilizing a fork on White's queen and rook to win a full rook. Conversely, Game A White played passively in the opening and middlegame (such as 15. Kh2 and 16. Rb1), allowed Black to dominate the queenside, blundered a pawn on a3, and defended poorly, ultimately getting mated on the back rank.

Reported P(A stronger): 0.2. Consistent with choice.

Gemini 3.7 Flash: chose B — wrong

Target B (Black) demonstrated clean, mature handling of the Slav Defense, effortlessly completing development, coordinating pieces toward White's kingside, and instantly exploiting tactical weaknesses with 20...Qg6! threatening g2 and setting up the decisive 21...Bc2 fork winning a rook. Target A (White) played rather passively against the Benoni, lost queenside control, blundered a pawn via 21.Qxe2? Bxc3 22.bxc3 Qxa3, and allowed Black's heavy pieces to completely dominate the 2nd rank and finish the game.

Reported P(A stronger): 0.28. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

Game A White handled a Modern Benoni with reasonable theory (6.g3, 9.Nge2) but collapsed tactically: 15.Kh2? allowed ...Nc4, and 19.f4? lost the d4-knight sequence, leaving Black a clear pawn up. Despite losing, White found the resourceful 28.e5 and kept resistance into a complex rook ending. Game B Black was outplayed positionally; 21...Bc2?? hangs material to 22.Bh5! and Black resigned after 22...Bxd1 23.Bxg6 Bxb3 24.Bd3, losing the exchange and position outright. A's errors were subtle; B blundered a full exchange in a quiet position, indicating lower strength.

Reported P(A stronger): 0.72. Consistent with choice.

Claude Fable 5.1: chose B — wrong

Game A White played a reasonable Benoni setup but drifted: 12.Bd2/13.Qc1 was passive, 15.Kh2 and 16.Rb1 allowed Nxd2 and the collapse of the queenside pawns (22...Qxa3, 23...Qxc3), then 28.e5/29.g4 opened the king and 32.Rxe7 lost to 32...Qf4+ trading into a hopeless rook ending. Game B Black played solid Slav development, correctly recaptured 8...exd5, equalized cleanly, and spotted the tactic 21...Bc2 winning the exchange; White's 22.Bh5?? blundered the queen, which Black handled accurately. Black in B showed sounder positional play against a fairly competent opponent; White in A was outplayed throughout.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3 Flash (Preview): chose B — wrong

Target B showed superior tactical vision and efficiency. The maneuver 18... Qe6 and 20... Qg6 prepared a clever tactical shot, 21... Bc2, which won the exchange and forced resignation. B's play was solid and purposeful throughout the Slav structure. Target A, playing White, struggled with coordination in a Benoni. Allowing the loss of the bishop pair with 16. Rb1 and failing to handle the pressure on the kingside (culminating in the 39-41 sequence) suggests a lower level of tactical and positional awareness compared to Target B. A's 13. Qc1 and 19. f4 were also somewhat imprecise, leading to a disadvantageous position.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3.6 Flash: chose B — wrong

Game A White played passively out of the opening against a Benoni setup with an early 6. g3. White struggled for a plan, conceded the queenside initiative, lost a pawn with 22. bxc3, and allowed Black's pieces to dominate the position down to a hopeless rook endgame. In contrast, Game B Black played a very solid and classical Slav Defense structure, maneuvered smoothly in the middlegame, and showed sharp tactical alertness by punishing White's blunder (21. Bf3) with 21...Bc2!, winning the queen after 22. Bh5 Bxd1. Game B Black displayed much cleaner overall technique and tactical awareness.

Reported P(A stronger): 0.2. Consistent with choice.

GPT-5.6 Sol: chose B — wrong

Game B’s Black played a coherent Slav setup and showed clear tactical awareness with 20...Qg6 and 21...Bc2, exploiting White’s misplaced rook and queen; after 22.Bh5, 22...Bxd1 23.Bxg6 Bxb3 converted the sequence cleanly. Game A’s White showed good opening familiarity and active ideas such as 25.Rxa7 and 28.e5, but 19.f4 allowed ...Nd4 followed by exchanges that exposed the queenside, while 22...Qxa3 and 23...Qxc3 won pawns. Later, White’s rooks became vulnerable to the coordinated invasion 38...Rdd2, 39...Rh2+, and 40...Rbg2+. B’s game was shorter and opponent-assisted, but B displayed fewer weaknesses.

Reported P(A stronger): 0.38. Consistent with choice.

GPT-6 Sol: chose B — wrong

Game B’s Black played a sound opening and found the tactical 21...Bc2, attacking White’s queen and rook; after 22.Bh5, Black collected both with 22...Bxd1 and 23...Bxb3. Game A’s White fought back actively with 25.Rxa7 and 32.Rxe7, but allowed Black’s queen to take pawns on a3 and c3 and later struggled against active rooks. B’s play looks more controlled, though White’s abrupt collapse in Game B limits confidence.

Reported P(A stronger): 0.25. Consistent with choice.

GPT-6.1 Sol: chose B — wrong

Game B Black showed coordinated tactical play: 18...Qe6 and 20...Qg6 created kingside threats, while 21...Bc2 forked White’s queen and rook. After 22.Bh5, Black correctly chose 22...Bxd1 and 23...Bxb3, emerging a rook ahead despite losing the queen. Game A White handled a more demanding position and generated activity with 25.Rxa7 and 28.e5, but allowed the sequence 19...Nd4 through 23...Qxc3 to cost two pawns, then failed to contain Black’s second-rank rooks and mating net. B’s opponent supplied a major tactical opportunity, so the comparison remains uncertain, but B’s cleaner execution slightly outweighs A’s greater complexity.

Reported P(A stronger): 0.64. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — wrong

Game B’s target Black showed the clearer tactical understanding: after 21 Bf3, ...Bc2 attacked both the queen on d1 and rook on b3, and Black correctly collected them with 22...Bxd1 and 23...Bxb3. The opening play was also coherent. Game A’s target White played actively, including 28 e5 and 32 Rxe7, but allowed Black’s queenside pawn gains with 22...Qxa3 and 23...Qxc3, then was overwhelmed by the coordinated rooks after 37...Kg7. Thus B has the stronger evidence, though the short game and White’s major blunder make the comparison uncertain.

Reported P(A stronger): 0.65. Contradicts choice; excluded from confidence diagnostics.

GPT-6 Astra: chose B — wrong

Game B Black developed coherently, activated the queen with 18...Qe6 and 20...Qg6, and exploited the queen–rook fork with 21...Bc2. After 22.Bh5, Black correctly played 22...Bxd1 and 23...Bxb3, winning a rook through the queen exchange. Game A White showed more ambitious play, including 28.e5 and 30.f5, but allowed damaging simplification after 20.Qd3 and ultimately missed the decisive mating attack with 39.R8xc5? Rh2+. B gets a modest edge for tactical control, although B’s opponent provided a much easier opportunity than A’s.

Reported P(A stronger): 0.59. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — wrong

Game B's Black plays a clean, forcing tactic: 20...Qg6 21...Bc2! 22...Bxd1 23...Bxb3 nets a full rook with back-rank themes, executed instantly in rapid time. Game A's White shows decent Benoni handling (12.Bd2, 19.f4) but repeatedly errs: 15.Kh2 lets the knight fork, 27.Rg1 paralyzes the rook, and 35.Rc1/36.Bxd5 drops the bishop, with passive defense throughout. Black B's tactical accuracy under time pressure suggests the higher established strength, while White A's errors look more typical of a lower-rated rapid player.

Reported P(A stronger): 0.62. Contradicts choice; excluded from confidence diagnostics.

Pair 02: 12 of 16 models wrong

A: Black, rated 1159 · B: White, rated 762. Higher pre-game rating: A.

Game A: Black, rated 1159 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 e5 2. d3 d5 3. Nc3 dxe4 4. Nxe4 f5 5. Nc3 Nc6 6. Nf3 Bd6 7. Bg5 Qd7 8.
Qe2 h6 9. Bd2 g5 10. O-O-O Nf6 11. h4 g4 12. Nh2 h5 13. g3 b6 14. Bg5 Nh7 15.
Nd5 Nxg5 16. Nf6+ Ke7 17. Nxd7 Kxd7 18. hxg5 Bb7 19. f3 Nd4 20. Qf2 gxf3 21.
Nxf3 Nxf3 22. Bg2 Rab8 23. Bxf3 Bxf3 24. Qxf3 Ke6 25. Rdf1 Rbf8 26. Qc6 a5 27.
Re1 e4 28. Qc4+ Kd7 29. Qa4+ c6 30. dxe4 Bxg3 31. e5 Bxe1 32. Rxe1 h4 33. Qd4+
Kc7 34. Qd6+ Kb7 35. Qd7+ Ka6 36. Qd3+ b5 37. Qd6 Kb6 38. e6 h3 39. e7 Re8 40.
Qb8+ Rxb8 41. e8=Q Rbxe8 42. Rxe8 Rxe8 0-1
Game B: White, rated 762 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 e5 2. Nf3 d6 3. Nc3 c6 4. Bc4 b5 5. Bb3 a5 6. a4 b4 7. Ne2 f6 8. Ng3 h5
9. d3 g5 10. Nd2 h4 11. Nh5 c5 12. Nc4 Nc6 13. O-O Nd4 14. Ba2 Be6 15. g3 Qd7
16. gxh4 Bg4 17. Nxf6+ Nxf6 18. Qe1 gxh4 19. Bg5 Nh7 20. Nb6 Qc6 21. Nxa8 Qxa8
22. Bd5 Qd8 23. Bxd8 1-0

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Both games are low-level. In Game A, Black allowed a fork chain with 15.Nd5 and 16.Nf6+, where 7...Qd7 left the queen exposed, giving up the queen for minor pieces. Afterwards Black played reasonably: 19...Nd4, 20...gxf3 and 30...Bxg3 picked up material, and Black converted after White's 40.Qb8+?? blunder. In Game B, White faced a very weak opponent (5...a5, 7...f6, 8...h5) but played dubiously, with 15.g3 and 16.gxh4. White also gave up a knight with 17.Nxf6+ for little, then won because Black blundered the queen with 22...Qd8?? into 23.Bxd8. Game A's Black showed somewhat more tactical resourcefulness against a stronger opponent, so I lean slightly to A.

Reported P(A stronger): 0.55. Consistent with choice.

Claude Sonnet 5.5: chose A — correct

Both targets are fairly weak, but Black in Game A played more purposefully. In Game A, Black's 15...Nxg5 and 18...Bb7 were reasonable, and 19...Nd4 and 30...Bxg3 showed tactical awareness. Black then converted the endgame with 38...h3 and 40...Rxb8. Black did make loose moves such as 2...d5 and 12...h5, and White was also weak (17.Nxd7 was a blunder-ish trade). In Game B, White won only because Black blundered with 20...Qc6 and 22...Qd8. White's own play was sloppy: 16.gxh4 and 17.Nxf6+ were fine, but 11.Nh5 and 20.Nb6 were speculative. Black in Game B played extremely poorly, which suggests a low-rated pool, and White's win says little about their strength. Game A's opponent was somewhat more competent and Black still outplayed them, which points to A being stronger.

Reported P(A stronger): 0.62. Consistent with choice.

Gemini 3.8 Flash: chose B — wrong

Player B showed much sounder opening fundamentals (developing pieces naturally with Nf3, Nc3, Bc4, and castling) compared to Black in Game A, who pushed kingside pawns recklessly (4...f5, 8...h6, 9...g5). Furthermore, Black in Game A committed an elementary tactical oversight with 15...Nxg5??, directly walking into the 16. Nf6+ fork that dropped their queen for just a minor piece, only winning because White completely hung their queen on move 40. In contrast, White in Game B maintained solid piece coordination, navigated Black's flank aggression, and capitalized on tactics with 20. Nb6 and 22. Bd5 to decisively win material.

Reported P(A stronger): 0.22. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — wrong

Black in Game A demonstrates very low tactical awareness, blundering their queen to a basic one-move knight fork with 15... Nxg5?? 16. Nf6+. Black only wins because White commits an equally egregious queen blunder (40. Qb8+??). White in Game B, despite some awkward maneuvering like 11. Nh5, shows much better tactical vision by finding resources like 17. Nxf6+ to escape a trap and 20. Nb6 to win material. White B's ability to spot and execute these tactics suggests a higher rating than Black A's beginner-level blunders.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.5 Flash: chose B — wrong

In Game A, Black (Target A) committed a severe tactical oversight on move 15, playing 15... Nxg5, which completely ignored the threat on f6 and allowed the simple knight fork 16. Nf6+ winning the queen. In Game B, White (Target B) displayed far better tactical awareness and solid positional understanding. White successfully navigated Black's kingside threats, found a resource with 17. Nxf6+ to save the queen, executed a strong fork with 20. Nb6, and promptly punished Black's queen blunder with 23. Bxd8. Target B's play suggests a much higher rating.

Reported P(A stronger): 0.25. Consistent with choice.

Gemini 3.7 Flash: chose B — wrong

Player A (Black) exhibited serious tactical oversights, notably blundering their queen on move 15 with 15...Nxg5?? allowing the elementary fork 16. Nf6+, and only winning after White threw away a winning position with 40. Qb8+??. In contrast, Player B (White) played a much more coherent opening, handled Black's kingside thrust reasonably, and seized practical chances with tactical awareness (20. Nb6 and 22. Bd5 winning material before capturing Black's blundered queen on 23. Bxd8).

Reported P(A stronger): 0.18. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

Target A (Black, Game A) handled a chaotic middlegame competently, finding accurate trades (15...Nxg5, 17...Kxd7) and converting a winning endgame. Target B (White, Game B) won only through Black's severe blunder 22...Qxd8??, losing the queen; before that, Black had outplayed White with 13...Nd4 and 16...Bg4. Black's 8...Qd7 and 26...a5 in Game A show coherent plans, while White in Game B played passively (7.Ne2, 10.Nd2) and relied on an opponent's one-move oversight. A's opponent offered a tougher fight; A's play against it was stronger.

Reported P(A stronger): 0.68. Consistent with choice.

Claude Fable 5.1: chose B — wrong

Game A Black played a dubious setup (2...d5, 4...f5, 7...Qd7, 12...h5) and allowed 15.Nd5 Nxg5 16.Nf6+ losing the queen for two minor pieces; White then blundered badly (26.Qc6?, 27.Re1?? allowing e4 and Bxg3-Bxe1, and finally 40.Qb8+??), so Black's win reflects a weak opponent more than skill. Game B White handled a bizarre Black setup (3...c6, 4...b5, 7...f6, 8...h5, 9...g5) soundly, developed, castled, and punished tactically with 17.Nxf6+, 20.Nb6 winning the exchange and 22.Bd5 trapping/winning the queen. White in B showed better fundamentals and tactical alertness; Black in A showed poor opening understanding and missed basic tactics.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3 Flash (Preview): chose B — wrong

Target B shows better fundamental understanding and tactical alertness. In Game B, White develops logically (4. Bc4, 13. O-O) and exploits Black's blunders, such as the fork with 20. Nb6 and the hanging queen 23. Bxd8. In contrast, Target A (Black) in Game A commits a severe tactical oversight by allowing the simple fork 16. Nf6+, losing the queen. Although Black eventually wins, it is primarily due to White's even more egregious blunder (40. Qb8+), which hangs the queen for nothing. Target B's play is significantly more stable and demonstrates a higher floor of tactical competence than Target A.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.6 Flash: chose B — wrong

Game A Black played overly aggressive pawn pushes early on (4...f5, 9...g5) and committed a major tactical oversight with 15...Nxg5, allowing White's 16. Nf6+ fork to win Black's queen for a knight. Black only won because White blundered back heavily later (including hanging the queen with 40. Qb8+). In contrast, Game B White played a solid opening (1. e4, 2. Nf3, 4. Bc4) and demonstrated far superior tactical alertness, successfully capturing Black's hanging rook with 21. Nxa8 and taking Black's queen after 22...Qd8 23. Bxd8 to seal a clean victory without committing gross blunders.

Reported P(A stronger): 0.2. Consistent with choice.

GPT-5.6 Sol: chose B — wrong

Game A Black showed fighting resourcefulness with 19...Nd4, 27...e4 and the promotion defense 40...Rxb8 41...Rbxe8, but the elementary queen loss after 15...Nxg5? 16.Nf6+ Ke7 17.Nxd7 is strong evidence against established strength; victory depended on White’s later collapse. Game B White handled an eccentric attack calmly, castled, and exploited tactics accurately: 17.Nxf6+ cleared lines, 20.Nb6 drove the queen while threatening the rook, 21.Nxa8 won material, and 23.Bxd8 punished Black’s final blunder. Although Game B’s opponent was very weak, White made fewer serious errors and displayed clearer tactical awareness.

Reported P(A stronger): 0.27. Consistent with choice.

GPT-6 Sol: chose B — wrong

Game A’s Black played actively, but 15...Nxg5 allowed 16.Nf6+ and 17.Nxd7, losing the queen. Black’s eventual win depended heavily on White’s later errors, including 40.Qb8+??. Game B’s White handled the tactics more reliably: 20.Nb6 attacked Black’s queen while threatening the a8 rook, and 22.Bd5 kept pressure on the queen before 23.Bxd8 won it. B therefore looks likelier to have been the stronger established player.

Reported P(A stronger): 0.16. Consistent with choice.

GPT-6.1 Sol: chose B — wrong

Game B White developed more coherently and spotted 20.Nb6, forking Black’s queen and rook, followed by 21.Nxa8. White also exploited 22...Qd8 with 23.Bxd8, though 15.g3 and the subsequent knight sacrifice show imperfect play. Game A Black weakened the position with early pawn pushes and, crucially, allowed 15...Nxg5 16.Nf6+ followed by 17.Nxd7, losing the queen to a straightforward fork. Black showed resourcefulness with 27...e4 and 30...Bxg3–31...Bxe1, but the eventual win depended heavily on White’s errors, especially 40.Qb8+. B’s play provides stronger evidence of established playing strength.

Reported P(A stronger): 0.79. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — wrong

Game B’s White showed more reliable tactical control: after 20 Nb6, White captured the rook with 21 Nxa8 and then exploited 22...Qd8 with 23 Bxd8, winning decisively without an evident major error. Game A’s Black eventually found useful tactics, especially 40...Rxb8 followed by 41...Rbxe8, but 15...Nxg5 allowed 16 Nf6+ and 17 Nxd7, losing the queen for a minor piece. Black’s eventual win therefore depended heavily on White’s later tactical mistakes, whereas Game B’s White more cleanly converted the opponent’s errors.

Reported P(A stronger): 0.35. Consistent with choice.

GPT-6 Astra: chose B — wrong

Game B White showed better tactical awareness: 17.Nxf6+ captured with tempo, 20.Nb6 forked queen and rook, and 22.Bd5 attacked the queen before 23.Bxd8 punished its retreat. Black’s weak play limits how much this proves, but White exploited the opportunities coherently. Game A Black’s 15...Nxg5 overlooked 16.Nf6+ and 17.Nxd7, losing the queen for inadequate compensation. Black later found useful counterplay with 27...e4 and 30...Bxg3, but the eventual victory depended heavily on White’s errors, especially 40.Qb8+. B’s play is stronger evidence of established strength.

Reported P(A stronger): 0.86. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose A — correct

Game A's Black plays a coherent strategic game: 9...g5-11...g4 secures a strong pawn shield, 15...Nxg5/16...Nf6+ wins material via a calculated knight fork line (15.Nd5?? blunder punished), then Black converts cleanly in the endgame (34-41 precise checks, 38...h3, 39...e7) to promote and force resignation. Game B's White wins mainly because Black self-destructs: 17...Nxf6 allowing Nxf6+, 20...Qc6? losing the rook to 21.Nxa8. White's play (12...Nc4-to-13...Nd4 setup is awkward) shows reasonable tactics but less sustained quality. A's demonstrated technique and punishment of a smaller error suggests higher strength.

Reported P(A stronger): 0.65. Consistent with choice.

Pair 16: 7 of 16 models wrong

A: Black, rated 1273 · B: White, rated 1575. Higher pre-game rating: B.

Game A: Black, rated 1273 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 e5 2. d5 Bc5 3. e4 Qf6 4. Nf3 c6 5. Nc3 cxd5 6. exd5 Ne7 7. Ne4 Qb6 8.
Nxc5 Qxc5 9. Be3 Qa5+ 10. c3 Nxd5 11. Bd2 Qc7 12. Bd3 O-O 13. O-O d6 14. Bc4
Nf4 15. Be2 Nxe2+ 16. Qxe2 Bg4 17. h3 Bh5 18. Qe4 Nd7 19. Qh4 Bxf3 20. gxf3
Rfe8 21. Qg3 g6 22. Bg5 Nc5 23. b3 e4 24. f4 d5 25. f5 Qxg3+ 26. fxg3 Re5 27.
g4 a5 28. Be3 Nd3 29. Bd4 Re7 30. f6 Re6 31. c4 dxc4 32. bxc4 a4 33. a3 Rd8 34.
Bb6 Rdd6 35. Bc7 Rc6 36. Bh2 Rxc4 37. Rab1 b6 38. Rb5 Rc1 39. Rxc1 Nxc1 40. Rd5
e3 41. Rd8+ Re8 42. Rxe8# 1-0
Game B: White, rated 1575 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. Nc3 b5 6. Bb3 Be7 7. O-O O-O 8. Nd5
d6 9. c3 Nxe4 10. d4 exd4 11. cxd4 Nf6 12. Nxf6+ Bxf6 13. Be3 Na5 14. Bc2 Nc4
15. Qd3 g6 16. Bh6 Re8 17. Qb3 Bb7 18. h4 Bxh4 19. Nxh4 Qxh4 20. Qh3 Qxh3 21.
gxh3 Re2 22. Bd3 Rxb2 23. a3 Rb3 24. Bc2 Rxh3 25. Bg5 Rh1# 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — wrong

Game A Black chose an odd opening (2...Bc5, 3...Qf6) but then played coherently: 10...Nxd5 won a pawn, 15...Nxe2+ took the bishop, and the endgame was handled actively with 31...dxc4 and 36...Rxc4. Black only collapsed at the end with 40...e3??, allowing the back-rank mate Rd8+. Game B White made more basic errors. 8.Nd5 followed by 9.c3 let Black play 9...Nxe4 for a free pawn. 18.h4?? dropped another pawn, and 20.Qh3 traded queens into a wrecked structure. White then let a rook into e2 and b2, and 25.Bg5?? allowed Rh1#. Black B's poorer play overall points to A being the stronger target.

Reported P(A stronger): 0.68. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A, Black: played an early Qf6/Qb6 opening and lost the queen trade with 8...Qxc5 fine, but then 23...e4 and 24...d5 let White gain f-pawn space. Black allowed 25.f5 and 30.f6, then 36...Rxc4 and 38...Rc1 lost to a back-rank mate after 41...Re8. Black showed decent tactics like 15...Nxe2+ but was passive and missed the mating net. Game B, White: knew the Ruy Lopez to move 7, but 8.Nd5 and 9.c3 lost a pawn to 9...Nxe4. Then 16.Bh6, 18.h4 and 20.Qh3 were poor, allowing 18...Bxh4 and 21...Re2, and White lost badly with a hung b2 pawn. White in Game B played worse, with more blunders, while Black in Game A was only moderately weak. Slight edge to B's target being weaker, so A is slightly favored, but A's Black also lost. The choice is close.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Player B displays far superior foundational opening and strategic understanding. Playing a standard Ruy Lopez, Game B White followed sensible development schemes (8.Nd5, 10.d4, 15.Qd3) before collapsing in tactical defense and suffering 25...Rh1#. In contrast, Game A Black plays an unprincipled early queen sortie with Scholar's Mate aspirations (3...Qf6), misses White's hanging piece after 34.Bb6? by retreating with 34...Rdd6, and finally commits a catastrophic back-rank blunder with 40...e3?? walking directly into mate in two (41.Rd8+ Re8 42.Rxe8#). These errors are hallmarks of a much lower rating pool.

Reported P(A stronger): 0.22. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Black in Game A plays a very dubious opening (1...e5, 3...Qf6) and ultimately blunders a simple back-rank mate (40...e3??). White in Game B plays a standard Ruy Lopez but hangs pawns (9. c3, 18. h4) and completely misses a mating net (24. Bc2??). Both players show significant tactical weaknesses, but White in Game B exhibits slightly better opening principles than Black's early queen sorties in Game A, suggesting B might be slightly stronger.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

Game A Black plays the Englund Gambit (1...e5) and follows with naive, beginner-like early queen development (3...Qf6), which strongly suggests a lower established rating. Although Game A Black later coordinates their pieces reasonably well, they blunder a basic mate in two with 40...e3??. Game B White plays a highly standard and theoretical Ruy Lopez opening up to move 8, suggesting a more established understanding of chess principles, even though they suffer a tactical collapse, blundering the e4-pawn on move 9, the h4-pawn on move 18, and mate in one with 25. Bg5??. Game B White's opening competence makes them likely the higher-rated player overall.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Player B demonstrates standard Ruy Lopez opening principles and reasonable piece coordination before collapsing to tactical oversights in the endgame (allowing 24...Rxh3 and mate). In contrast, Player A plays the suspect 1...e5 against 1.d4 followed by an early queen sortie (3...Qf6), fails to capture a hanging bishop (34...Rdd6 instead of 34...Rxb6), and blunders into an elementary back-rank checkmate on move 40 with 40...e3?? allowing 41.Rd8+.

Reported P(A stronger): 0.28. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

Target A (Black, Game A) mishandled a dubious opening (1...e5 2...Bc5 3...Qf6), allowed White a comfortable edge after 8.Nxc5, and later blundered into 42.Rxe8#. Target B (White, Game B) chose the passive 8.Nd5, lost a pawn after 9...Nxe4, then allowed 18...Bxh4 winning material, and finally permitted 25...Rh1# with the rook on h1. Both lost, but A's opening was objectively poor and the mate resulted from a clear oversight in a simplified position, whereas B's errors were more tactical in a complex middlegame. B's play suggests a somewhat higher baseline.

Reported P(A stronger): 0.22. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A Black opened with 1...e5 2...Bc5 3...Qf6 (dubious, queen out early), then shuffled the queen (Qb6, Qc5, Qa5+, Qc7) and later allowed an elementary back-rank mate with 41...Re8 42.Rxe8#, having also lost material to tactics like 36...Rxc4 37.Rab1. Game B White played a coherent Ruy Lopez with Nd5/c3/d4 ideas, but blundered a pawn to 9...Nxe4 and then hung material with 18.h4?? Bxh4, collapsing afterward. Both are weak, but Game B White's opening knowledge and structure look more developed than Game A Black's queen-wandering and back-rank oversight; slight edge to B.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Target B (White) demonstrates a more mature understanding of opening principles, navigating a main-line Ruy Lopez (5. Nc3, 7. O-O) and coordinating pieces logically. Despite the catastrophic one-move blunder (25. Bg5?? allowing 25... Rh1#), the prior play was coherent. In contrast, Target A (Black) chose a dubious opening (1... e5 vs 1. d4) and an early queen development (3... Qf6) that typically characterizes lower-rated play. Target A also blundered a back-rank mate (41... Re8#) in a relatively simple endgame. Target B’s sounder development and opening knowledge suggest a higher established rating.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Target A (Game A Black) plays the dubious Englund Gambit (1...e5), misses a clean piece win on move 14 (14...Nf4 instead of 14...Qxc4 winning the bishop on c4), and ultimately blunders a simple back-rank checkmate with 41...Re8??. Target B (Game B White) demonstrates solid theoretical opening understanding in the Ruy Lopez through move 17. Although Target B collapses quickly after 18. h4? and gets mated on move 25, the overall quality of opening play and lack of gross early tactical blunders suggest Target B has a higher established rating than Target A.

Reported P(A stronger): 0.25. Consistent with choice.

GPT-5.6 Sol: chose A — wrong

Game A Black showed more sustained competence: despite the dubious 1...e5 and early queen moves, they completed development, castled, found useful play with 23...e4, 24...d5, 28...Nd3, and won the c4-pawn with 36...Rxc4. The final 40...e3? overlooked the back-rank mating sequence 41.Rd8+ Re8 42.Rxe8#, but it was one late tactical failure. Game B White played a coherent opening but deteriorated sharply: 18.h4?! exposed the king, 20.Qh3 invited a damaging queen trade, and 25.Bg5?? allowed the immediate 25...Rh1#. Missing such a direct mate, after already permitting ...Re2 and ...Rxh3, is stronger evidence of lower established strength.

Reported P(A stronger): 0.72. Consistent with choice.

GPT-6 Sol: chose A — wrong

Game A’s Black played an eccentric opening but then coordinated well: 10...Nxd5 recovered a pawn, 23...e4 challenged the center, and 25...Qxg3+ simplified. The late 40...e3?? allowed a back-rank mate. Game B’s White made more basic errors: 9.c3 allowed ...Nxe4, 18.h4 led to the loss of a knight after 18...Bxh4 19.Nxh4 Qxh4, and 24.Bc2 failed to address the rook’s attack on h3. Despite A’s final blunder, A appears more likely to have been the stronger established player.

Reported P(A stronger): 0.88. Consistent with choice.

GPT-6.1 Sol: chose A — wrong

Game A Black showed more sustained middlegame competence: 23...e4 and 24...d5 gained central space, 28...Nd3 established an outpost, and 33...Rd8 followed by ...Rdd6 and ...Rc6 activated both rooks. The decisive 40...e3 overlooked the back-rank mating threat, but followed substantial purposeful play. Game B White developed reasonably, then 18.h4 and 19.Nxh4 led to a pawn-down queen trade without a convincing attack. White allowed ...Re2–...Rxb2–...Rb3–...Rxh3 and finally missed the immediate mate with 25.Bg5. A’s broader positional coordination is stronger evidence despite both targets losing.

Reported P(A stronger): 0.79. Consistent with choice.

GPT-5.6 Luna: chose A — wrong

Target A (Black) showed more sustained, purposeful play: after developing, Black generated active counterplay with 23...e4 and 24...d5, then used the rooks actively on the c-file. The late 40...e3 and failure to handle 41 Rd8+ led to 42 Rxe8#, but much of the game was competently conducted. Target B (White) produced a plausible opening, yet the attack beginning with 18 h4 was poorly judged; after 24 Bc2? Rxh3, 25 Bg5?? allowed the immediate 25...Rh1#. B’s failure to address the rook invasion and mating threat suggests weaker established rapid strength.

Reported P(A stronger): 0.78. Consistent with choice.

GPT-6 Astra: chose A — wrong

Game A Black showed more coherent middlegame and endgame play: 23...e4 and 24...d5 established central control, 28...Nd3 created an active outpost, and 33...Rd8 followed by rook activity won material. However, 40...e3 overlooked the decisive 41.Rd8+. Game B White developed reasonably but lost direction: 18.h4 allowed ...Bxh4, while 20.Qh3 abandoned attacking prospects for a vulnerable, pawn-down ending. White then allowed ...Re2 and ...Rxb2, and 25.Bg5 missed mate in one. A's sustained coordination outweighs the final tactical failure.

Reported P(A stronger): 0.8. Consistent with choice.

GLM-5.3 Flash: chose A — wrong

Game A's Black plays coherently in a tricky line: 8.Nxc5 Qxc5 keeps material even, and Black trades actively (16...Bg4, 19...Bxf3) and creates counterplay with 23...e4 and 24...d5. The loss comes late (40...e3 41.Rd8+) in a bad endgame, suggesting weaker endgame technique but decent opening/middlegame understanding. Game B's White blunders immediately: 18.h4?? allows 18...Bxh4! 19.Nxh4 Qxh4 winning a piece, then drifts to 21...Re2, 22...Rxb2 and gets mated (25...Rh1#). Conceding a piece to a primitive tactic on move 18 is a clear sign of lower strength.

Reported P(A stronger): 0.65. Consistent with choice.

Pair 20: 6 of 16 models wrong

A: White, rated 1228 · B: White, rated 557. Higher pre-game rating: A.

Game A: White, rated 1228 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 e5 2. d5 Bc5 3. e4 Qf6 4. Nf3 c6 5. Nc3 cxd5 6. exd5 Ne7 7. Ne4 Qb6 8.
Nxc5 Qxc5 9. Be3 Qa5+ 10. c3 Nxd5 11. Bd2 Qc7 12. Bd3 O-O 13. O-O d6 14. Bc4
Nf4 15. Be2 Nxe2+ 16. Qxe2 Bg4 17. h3 Bh5 18. Qe4 Nd7 19. Qh4 Bxf3 20. gxf3
Rfe8 21. Qg3 g6 22. Bg5 Nc5 23. b3 e4 24. f4 d5 25. f5 Qxg3+ 26. fxg3 Re5 27.
g4 a5 28. Be3 Nd3 29. Bd4 Re7 30. f6 Re6 31. c4 dxc4 32. bxc4 a4 33. a3 Rd8 34.
Bb6 Rdd6 35. Bc7 Rc6 36. Bh2 Rxc4 37. Rab1 b6 38. Rb5 Rc1 39. Rxc1 Nxc1 40. Rd5
e3 41. Rd8+ Re8 42. Rxe8# 1-0
Game B: White, rated 557 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 d5 2. Bf4 Nf6 3. h3 Nc6 4. e3 Bf5 5. Bb5 e6 6. Bg5 Be7 7. Bxf6 Bxf6 8.
Nf3 O-O 9. O-O a6 10. Bxc6 bxc6 11. Nbd2 e5 12. dxe5 Be7 13. Nd4 Bxh3 14. gxh3
f5 15. Nxf5 Bh4 16. Nxh4 Qxh4 17. f3 Qg5+ 18. Kh1 Qxe5 19. Re1 Qg3 20. Qe2
Qxh3+ 21. Qh2 Qe6 22. Qe5 Qg6 23. Rg1 Qh6+ 24. Qh2 1-0

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Game A White handled a messy game soundly. After Black's odd 2...Bc5 and 3...Qf6 he won the bishop pair with 8.Nxc5, though 15.Be2 (allowing Nxe2+) and 19.Qh4 (allowing Bxf3 to wreck his pawns) were loose. In the endgame he played actively with 30.f6, 34.Bb6, 38.Rb5 and 40.Rd5, then finished with the back-rank mate 41.Rd8+. Game B White played aimlessly early: 2.Bf4, 5.Bb5, 6.Bg5 moves the bishop around. Weakening moves like 17.f3 let Black's queen roam. He mainly won material because Black gave pieces away (13...Bxh3, 15...Bh4). Neither player is convincing, but A showed somewhat more technique.

Reported P(A stronger): 0.57. Consistent with choice.

Claude Sonnet 5.5: chose A — correct

Game A White (A) played a long, coherent game. They traded the dark-squared bishop with 8.Nxc5 and handled the queen trade on move 25 well. They pushed f6 (30.f6) and the passed pawns, then found the finishing 40.Rd5 and 42.Rxe8#. Black blundered the endgame, though White did make a few inaccuracies, such as 17.h3 and the weakened structure after 19...Bxf3 20.gxf3. Game B White (B) played the slow, passive-looking 3.h3 and 5.Bb5, and 10.Bxc6 gave up the bishop pair. They then allowed 13...Bxh3 14.gxh3 f5, which exposed their king. White won only because Black blundered with 14...f5 15.Nxf5 and 15...Bh4, and the opponent's play was weak throughout (16...Qxh4 and the later queen moves). B's win was largely a gift, and A's performance was more consistent and showed better technique, so A is slightly more likely the stronger player.

Reported P(A stronger): 0.62. Consistent with choice.

Gemini 3.8 Flash: chose A — correct

Player A demonstrates vastly superior tactical awareness and board vision compared to Player B. In Game A, White develops logically, navigates the opening soundly, and finishes cleanly with an alert mating net (40. Rd5! e3 41. Rd8+). In contrast, Game B features severe tactical blunders by both players. White in Game B hung a piece with 14. Nxf5? (walked into bishop coverage from e7) and later inexplicably hung their queen in broad daylight with 22. Qe5?? (which Black failed to capture with 22...Qg6). The density of basic, one-move blunders in Game B is characteristic of a much lower rating tier.

Reported P(A stronger): 0.88. Consistent with choice.

Gemini 3.1 Pro (Preview): chose A — correct

Target A demonstrates a better understanding of development and tactics, finding nice moves like 7. Ne4 to exploit the opponent's early Queen sortie, though they do hang a pawn later with 10. c3. Target B plays the opening decently but then makes several massive blunders that both players fail to notice. For instance, in Game B, White plays 15. Nxf5 hanging the knight to Black's rook on f8, and later 22. Qe5?? hanging the Queen outright to Black's Queen on e6, which Black completely ignores with 22... Qg6. This mutual blindness to hanging pieces suggests Target B is a beginner, making Target A the stronger player.

Reported P(A stronger): 0.85. Consistent with choice.

Gemini 3.5 Flash: chose A — correct

Game A White showed better overall coordination and successfully executed a back-rank mate with 41. Rd8+ and 42. Rxe8#. Although White made errors like hanging a bishop with 14. Bc4 and 34. Bb6, the general structure of the game was reasonable. In contrast, Game B White committed an egregious one-move blunder by playing 22. Qe5, which hung the queen directly to Black's queen on e6 (which Black incredibly missed with 22... Qg6). This level of basic tactical oversight suggests Game B is of a lower rating band than Game A.

Reported P(A stronger): 0.65. Consistent with choice.

Gemini 3.7 Flash: chose B — wrong

Game B White demonstrates much better tactical awareness and defensive control. In Game B, White played a coherent opening, cleanly won material after Black's positional errors (11. dxe5 and 14. gxh3), and calmly diffused Black's kingside counterplay to force resignation (20. Qe2 and 24. Qh2). In contrast, Game A White made egregious one-move tactical blunders, outright hanging a bishop on move 14 with 14. Bc4 (which Black failed to capture with 14...Qxc4) and again on move 34 with 34. Bb6 (allowing 34...Rxb6). Game A White only won because Black overlooked a basic back-rank mate (41. Rd8+).

Reported P(A stronger): 0.22. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

In Game A, White shows solid middlegame understanding: developing naturally, exploiting the bishop pair, and converting with precise tactics like 30.f6 and 41.Rd8+ Re8 42.Rxe8#. In Game B, White plays passively (Bb5 then Bxc6 voluntarily, 6.Bg5 awkwardly), allows 13...Bxh3! winning a pawn with attacking chances, and only wins because Black blunders repeatedly (15...Bh4?, 20...Qxh3+? losing the queen after 21.Qh2). Game A's White demonstrates clearer strategic control and cleaner conversion than Game B's White, who relied on opponent errors.

Reported P(A stronger): 0.72. Consistent with choice.

Claude Fable 5.1: chose B — wrong

Game A White faced a weak opponent (1...e5 2.d5 Bc5, Qf6/Qb6 wandering) and played loosely: 7.Ne4 lost the d5 pawn, 19.Qh4 allowed Bxf3 wrecking the kingside, 34.Bb6?/35.Bc7? dropped the c4 pawn, and the win came only via Black's blunder 41...Re8?? allowing Rxe8#. Game B White played a coherent London setup, calmly refuted Black's 13...Bxh3 sacrifice with 15.Nxf5 and 16.Nxh4 winning two pieces, then consolidated (19.Re1, 20.Qe2, 23.Rg1) forcing resignation. Game B's opponent also played more reasonably (Nf6, Bf5, castling), suggesting a higher rating pool. B shows sounder technique.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3 Flash (Preview): chose B — wrong

Target B demonstrates a more solid and principled understanding of the opening (London System) and tactical consolidation. 13. Nd4 in Game B is a strong centralizing move that exploits Black's weaknesses (c6 and f5). Target B also showed good technique by offering a queen trade with 22. Qe5 while a piece up. In contrast, Target A's play was more erratic. Move 14. Bc4 in Game A was a wasted tempo that allowed Black to trade off an influential knight, and White’s kingside structure was unnecessarily shattered (20. gxf3). Target A was technically losing the endgame and only won due to Black’s major back-rank oversight (41. Rd8+), whereas Target B maintained control throughout.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.6 Flash: chose A — correct

Player A demonstrates superior positional structure and tactical awareness compared to Player B. In Game A, White quickly seized an advantage against the Englund Gambit with 7. Ne4! and maintained good piece activity, eventually capitalizing cleanly on Black's back-rank oversight with 41. Rd8+. While White blundered a bishop with 14. Bc4 (which Black missed), Player A's general play was much more cohesive. In Game B, White played passive early moves (3. h3, 7. Bxf6) and blundered a full queen with 22. Qe5, which Black miraculously failed to take with 22... Qxe5. Game B exhibits a significantly lower level of tactical oversight.

Reported P(A stronger): 0.65. Consistent with choice.

GPT-5.6 Sol: chose A — correct

Game A White showed more coherent strategic and tactical play: 19.Qh4 followed by f4-f5, g4, f6 created a dangerous kingside bind, while 31.c4 and the rook activation 37.Rab1–38.Rb5–40.Rd5 culminated accurately in 41.Rd8+ and 42.Rxe8#. Game B White correctly accepted Black’s unsound 13...Bxh3 sacrifice and found 15.Nxf5 and 16.Nxh4, but the opening plan with early h3, Bb5, Bg5 and repeated exchanges was less convincing, and 17.f3 exposed the king unnecessarily. Game B’s result depended heavily on Black’s crude sacrifice and premature resignation, whereas A demonstrated stronger sustained planning.

Reported P(A stronger): 0.76. Consistent with choice.

GPT-6 Sol: chose A — correct

Game A White showed more sustained planning against a relatively resilient opponent: 25.f5 secured a queen trade, 30.f6 restricted Black, and 38.Rb5 followed by 40.Rd5 created the decisive rook threat. Game B White correctly accepted Black’s unsound bishop play with 14.gxh3 and 16.Nxh4, then handled the queen checks, but much of that advantage came from Black giving up material. A’s play across the longer, contested game is stronger evidence of established strength.

Reported P(A stronger): 0.77. Consistent with choice.

GPT-6.1 Sol: chose B — wrong

Game B White showed cleaner tactical accounting: 14.gxh3, 15.Nxf5 and 16.Nxh4 converted Black’s unsound attack into a material advantage. White then handled queen pressure coherently with 20.Qe2, 21.Qh2 and 23.Rg1, offering exchanges while ahead. Game A White found useful active play with 30.f6 and 38.Rb5, then spotted 41.Rd8+ and mate. However, 19.Qh4 allowed damaged kingside pawns, and the subsequent play conceded central activity and the c4 pawn. B’s opponent was much more reckless, so the inference is modest rather than decisive.

Reported P(A stronger): 0.6. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose A — correct

Game A White appears more likely to have the higher established strength. White handled the opening actively with 7.Ne4 and 8.Nxc5, then coordinated an effective kingside/central attack with 22.Bg5, 23.b3, 25.f5, and 30.f6. The rook activity and finish, 38.Rb5 through 42.Rxe8#, show sustained tactical awareness, despite some loose play such as 20.gxf3. Game B White defended well after Black’s unsound 13...Bxh3, notably with 14.gxh3, 17.f3, 18.Kh1, and 21.Qh2, but the game was short and Black’s sacrifice and subsequent attack were highly questionable. Thus B’s evidence is less informative about established strength.

Reported P(A stronger): 0.65. Consistent with choice.

GPT-6 Astra: chose B — wrong

B’s White showed more consistent coordination: 13.Nd4 centralized effectively, and after Black’s dubious 13...Bxh3, White defended with 19.Re1, 20.Qe2 and 21.Qh2, then sought queen exchanges with 22.Qe5 and 24.Qh2. A’s White found useful counterplay with 25.f5 and 30.f6, but 14.Bc4 followed by 15.Be2 wasted time, and the later bishop maneuver allowed Black to collect c4. A’s mating finish was alert, though it depended on Black ignoring the threat with 40...e3. B faced less convincing opposition, so the preference is modest.

Reported P(A stronger): 0.62. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — wrong

Both whites won against weak opposition, but Black in B was catastrophically weak (13...Bxh3?? and 15...Bh4?? giving away two pieces). White B nonetheless played crisply, punishing accurately (14.gxh3, 15.Nxf5, 16.Nxh4) and safely neutralizing checks. White A also played reasonably but had slack moments (24.f4 instead of fxe4, 27.g4 weakening), and his opponent erred mainly in mutual complicity (23...e4?, 38...Rc1?). B's clean, accurate exploitation against even weaker resistance slightly suggests the stronger established player, though the evidence is thin.

Reported P(A stronger): 0.45. Consistent with choice.

Pair 09: 6 of 16 models wrong

A: Black, rated 1811 · B: Black, rated 1901. Higher pre-game rating: B.

Game A: Black, rated 1811 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 g6 2. Nf3 Bg7 3. e3 d6 4. Bd3 Nc6 5. a3 Bg4 6. Nbd2 e5 7. h3 Bxf3 8. Nxf3
exd4 9. exd4 Qe7+ 10. Be3 Nxd4 11. Nxd4 Bxd4 12. Bb5+ c6 13. Qxd4 Nf6 14. Bd3
O-O-O 15. O-O c5 16. Qa4 Kb8 17. b4 Nd5 18. Bd2 Nb6 19. Qb3 f5 20. bxc5 dxc5
21. a4 c4 22. Bf4+ Ka8 23. Bxc4 Rd4 24. Rfe1 Qc5 25. Be5 Rd2 26. Re3 Nxc4 27.
Bxh8 Nxe3 28. Qg8+ 1-0
Game B: Black, rated 1901 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 f5 2. Bf4 Nf6 3. e3 e6 4. Nd2 b6 5. Ngf3 Bb7 6. c3 Be7 7. Be2 O-O 8. h4
d6 9. Ng5 Qc8 10. O-O h6 11. Nh3 c5 12. dxc5 bxc5 13. Nc4 d5 14. Ne5 Bd6 15.
Ng6 Rd8 16. Bxd6 Kf7 17. Ne5+ Kg8 18. Be7 Re8 19. Bxf6 gxf6 20. Ng6 Qd7 21. Bh5
Qg7 22. Nhf4 e5 23. Nxd5 Nd7 24. Qb3 Bxd5 25. Qxd5+ Kh7 26. Rfd1 Nb6 27. Qxc5
Rac8 28. Qb4 Rc4 29. Qb5 Rg8 30. b3 Rc7 31. c4 f4 32. exf4 exf4 33. Qf5 Nc8 34.
Ne7+ Kh8 35. Nxg8 Qxg8 36. Qxf6+ Rg7 37. Rd8 Ne7 38. Rxg8+ Nxg8 39. Qd4 Kh7 40.
Qxf4 Ne7 41. Re1 Ng6 42. Qe4 a5 43. Bxg6+ Rxg6 44. h5 Kh8 45. Qxg6 a4 46. Re8#
1-0

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Both targets are weak, so this is close to a guess. Game A Black's 10...Nxd4 combination fails to 12.Bb5+, and 13.Qxd4 wins a piece. Black then missed 13...cxb5, which would have limited the damage, and later ignored his back rank: 26...Nxc4 allowed 27.Bxh8 and 28.Qg8+. Game B Black blundered with 14...Bd6 into the 15.Ng6 fork, losing material, and later dropped the exchange to 34.Ne7+. Still, he defended longer and more coherently against a cleaner opponent, with ideas like 22...e5 and 24...Bxd5. That gives a slight edge to Game B's target.

Reported P(A stronger): 0.45. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A Black (Black) played a Modern setup and traded well early, but 12...c6 and 14...O-O-O left the king exposed. 16...Kb8 was passive, and 19...f5 and 20...dxc5 let White's pieces dominate. 23...Rd4 and 24...Qc5 allowed 25.Be5 with a decisive attack. Black blundered into 28.Qg8+ by playing 26...Nxc4, which lost quickly. Game B Black (Black) played a Stonewall-like Dutch with 1...f5 and 8...d6. It was passive early, but Black defended tenaciously for 46 moves, with 22...e5 and 24...Bxd5, and survived into a long endgame. Black blundered on 33...Nc8 and 34.Ne7+, yet resisted longer than Game A's Black and kept a more coherent structure. Game A's collapse in 28 moves suggests weaker play, though both players are probably similar in strength.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game A Black exhibits severe tactical blind spots typical of lower ratings. After falling for the elementary deflection tactic 10...Nxd4? 11.Nxd4 Bxd4 12.Bb5+!, Black orchestrated a dynamic counterattack but suffered complete tactical tunnel vision at the climax: Black missed a mate-in-two (27...Qxf2+ 28.Kh1 Qxg2#) and instead played 27...Nxe3??, walking straight into 28.Qg8# on the back rank. Game B Black played a coherent Dutch setup; although Black overlooked the tactical hook 15...Rd8? permitting 16.Bxd6, Black alertly spotted the fork threat and avoided immediate disaster with 16...Kf7. Overall, Game B Black demonstrates better strategic coherence and board awareness than Game A Black.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Black in Game A blunders a full piece early to a basic tactic (12... c6 13. Qxd4) and later walks into a simple back-rank mate (27... Nxe3 28. Qg8+). Black in Game B displays a better positional understanding in the Dutch Defense, completing development and castling before breaking down to a complex knight maneuver by White. Although Black B also drops a piece with 16... Kf7 (missing 16... Rxd6), the overall opening play and resistance suggest a more experienced player than Black A's basic tactical blindness.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

Game A Black exhibits weak tactical vision. They blunder a piece early to a simple two-ply combination (10... Nxd4? 11. Nxd4 Bxd4 12. Bb5+ and 13. Qxd4) and later succumb to a basic back-rank mate (26... Nxc4? 27. Bxh8 Nxe3 28. Qg8+). In contrast, Game B Black demonstrates superior opening knowledge and positional understanding in a complex Dutch Defense setup. Although Game B Black also makes a tactical oversight on move 15 (15... Rd8? allowing 16. Bxd6 with a Ne7+ fork threat), their overall strategic play and standard developmental plans suggest a significantly higher established rating than Game A Black.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Game A Black displayed low tactical awareness early on, blundering a full piece to a simple deflection motif on move 10 (10...Nxd4? 11.Nxd4 Bxd4 12.Bb5+ c6 13.Qxd4) and later allowing a direct back-rank mate with 27...Nxe3? 28.Qg8#. In contrast, Game B Black demonstrated a solid understanding of Dutch Defense structures (1...f5 through 7...O-O) and coordinated play. While Game B Black was outplayed by White's active knights starting around 15...Rd8, they showed better positional understanding, calculation, and board vision than Game A Black.

Reported P(A stronger): 0.22. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

Game A Black played dubious opening moves like 8...exd4 and 9...Qe7+, and blundered material at 25...Nxc4, but stayed competitively active until then. Game B Black committed severe strategic and tactical errors: 16...Kf7?? allowed 17.Ne5+ forking king and queen, 23...Nd7?? dropped the queen, 31...f4?? and 36...Rg7?? lost more, and ultimately got checkmated. B Black's repeated blunders and king exposure indicate a substantially weaker player than A Black.

Reported P(A stronger): 0.35. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A Black played a coherent Modern setup and won a pawn with 10...Nxd4, but then collapsed: 14...O-O-O into a half-open b/a file position, 19...f5 and 21...c4 lost material to 22.Bf4+ and 23.Bxc4, and 24...Qc5?? allowed 25.Be5 forking the rooks. Game B Black (Dutch) made positional errors (13...d5 allowing 14.Ne5, 15...Rd8? dropping the bishop to 16.Bxd6, 18...Re8 losing the exchange after Bxf6/Ng6) and lost badly, but the losses were gradual and the opponent played competently; Game A's White was also weak (5.a3, 17.b4), yet Black still fell for simple tactics. Both look low-rated; slight edge to B for longer resistance and more normal piece development.

Reported P(A stronger): 0.42. Consistent with choice.

Gemini 3 Flash (Preview): chose A — wrong

Player A (Black) shows better tactical resourcefulness and coordination despite losing a piece to a common discovery (12. Bb5+). Their counterplay, particularly with 23... Rd4 and 25... Rd2, forced White to find precise defensive moves. In contrast, Player B (Black) mismanaged the pressure in the Dutch Defense, committing a severe positional and safety error with 16... Kf7, which led to a passive and structural collapse. While both players suffered from tactical oversights (the back-rank mate in Game A and the exchange loss in Game B), Player A's ability to generate complications while down material indicates a slightly higher level of established strength compared to Player B's more passive deterioration.

Reported P(A stronger): 0.58. Consistent with choice.

Gemini 3.6 Flash: chose A — wrong

Player A played a coherent opening (Modern Defense) and reached a reasonable position before falling victim to a two-move tactical trick with 10...Nxd4, missing White's 12. Bb5+ intermediate check which won the d4-bishop. In contrast, Player B displayed much worse tactical vision in Game B: after White played 16. Bxd6, Black played 16...Kf7??, failing to even recapture the bishop on d6 and giving away a full piece for free. Player B continued to make severe one-move blunders such as 22...e5, hanging the d5-pawn. Player A's mistake was a classic intermediate check oversight, while Player B demonstrated a lack of basic board awareness.

Reported P(A stronger): 0.65. Consistent with choice.

GPT-5.6 Sol: chose A — wrong

Game A Black showed more coherent development and active counterplay with 14...O-O-O, 21...c4, and the rook invasion 23...Rd4–25...Rd2. They did blunder a piece through 10...Nxd4?! 11.Nxd4 Bxd4 12.Bb5+ c6 13.Qxd4 and later missed 28.Qg8+. Game B Black’s tactical awareness looked substantially weaker: 15...Rd8 ignored the attack on f8, 16...Kf7 failed to address the lost bishop and rook, and 18...Re8 allowed 19.Bxf6. Further material losses followed with little compensation. Thus A is more likely to have had the higher established rapid strength.

Reported P(A stronger): 0.78. Consistent with choice.

GPT-6 Sol: chose A — wrong

Both Black players made serious tactical mistakes, but A showed more purposeful counterplay after losing material to White’s 12.Bb5+ and 13.Qxd4, using ...c5, ...c4 and ...Rd4 to create threats. B’s 16...Kf7 left White’s bishop on d6 unchallenged; it then took the knight with 19.Bxf6. Later, B allowed 34.Ne7+ to win a rook and 38.Rxg8+ to win the queen. A’s play looks flawed, but B’s repeated missed threats suggest lower established strength.

Reported P(A stronger): 0.82. Consistent with choice.

GPT-6.1 Sol: chose B — correct

B looks somewhat stronger despite both losses. Game A Black’s 10...Nxd4 and 11...Bxd4 overlooked 12.Bb5+ followed by 13.Qxd4, losing a piece. Later, 27...Nxe3 allowed the decisive back-rank check 28.Qg8+. Game B Black showed more coherent central play with 11...c5 and 13...d5, then purposeful rook activity through 27...Rac8, 28...Rc4 and 29...Rg8. B’s 16...Kf7 and subsequent king exposure were serious weaknesses, but the overall development and counterplay suggest a somewhat higher established strength than A’s repeated tactical oversights.

Reported P(A stronger): 0.66. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B’s Black showed somewhat more coherent development and defensive play: castling on move 7, central counterplay with ...c5 and ...d5, and active rook moves (...Rc4 and ...Rc7). However, Black still missed serious tactical issues, notably 16...Kf7 followed by 18...Re8 19.Bxf6, and was eventually mated after 38.Rxg8+. Game A’s Black appears weaker: 10...Nxd4 11.Nxd4 Bxd4 12.Bb5+ c6 13.Qxd4 loses two minor pieces for one, and 26...Nxc4 allows 27.Bxh8 to win a rook. Thus B is more likely to have had the higher established rapid strength, though the evidence is limited.

Reported P(A stronger): 0.35. Consistent with choice.

GPT-6 Astra: chose A — wrong

Game A Black showed more purposeful coordination: 15...c5, 17...Nd5, 18...Nb6 and 21...c4 generated queenside counterplay, followed by active rook penetration with 23...Rd4 and 25...Rd2. A still made serious tactical errors, notably 10...Nxd4 and allowing 28.Qg8+. Game B Black displayed more persistent weaknesses: 14...Bd6 left the bishop vulnerable to Bxd6, 16...Kf7–17...Kg8 wasted time while exposed, and 33...Nc8 allowed 34.Ne7+ followed by Nxg8. Both games contain substantial mistakes, but A’s coherent counterplay suggests the higher established strength.

Reported P(A stronger): 0.86. Consistent with choice.

GLM-5.3 Flash: chose A — wrong

Black in A plays a coherent game: good tactics (8...exd4, 10...Nxd4 trading into a clean position), active queenside play with 15...O-O-O and 17...Nd5, and only succumbs to White's late mating attack (28.Qg8+) after material was even. Black in B handles the Stonewall structure passively: 9...Qc8 and 10...h6 misplace pieces, 15...Rd8/16...Kf7 mishandle the Bxd6 trade, and repeated tactical lapses lose material (22...e5 allowing 23.Nxd5, then 23...Nd7?? hanging pieces to 27.Qxc5). Black B's errors look much more amateur than Black A's solid play.

Reported P(A stronger): 0.7. Consistent with choice.

Pair 24: 5 of 16 models wrong

A: White, rated 1782 · B: Black, rated 2154. Higher pre-game rating: B.

Game A: White, rated 1782 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 f5 2. Bf4 Nf6 3. e3 e6 4. Nd2 b6 5. Ngf3 Bb7 6. c3 Be7 7. Be2 O-O 8. h4
d6 9. Ng5 Qc8 10. O-O h6 11. Nh3 c5 12. dxc5 bxc5 13. Nc4 d5 14. Ne5 Bd6 15.
Ng6 Rd8 16. Bxd6 Kf7 17. Ne5+ Kg8 18. Be7 Re8 19. Bxf6 gxf6 20. Ng6 Qd7 21. Bh5
Qg7 22. Nhf4 e5 23. Nxd5 Nd7 24. Qb3 Bxd5 25. Qxd5+ Kh7 26. Rfd1 Nb6 27. Qxc5
Rac8 28. Qb4 Rc4 29. Qb5 Rg8 30. b3 Rc7 31. c4 f4 32. exf4 exf4 33. Qf5 Nc8 34.
Ne7+ Kh8 35. Nxg8 Qxg8 36. Qxf6+ Rg7 37. Rd8 Ne7 38. Rxg8+ Nxg8 39. Qd4 Kh7 40.
Qxf4 Ne7 41. Re1 Ng6 42. Qe4 a5 43. Bxg6+ Rxg6 44. h5 Kh8 45. Qxg6 a4 46. Re8#
1-0
Game B: Black, rated 2154 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1/2-1/2"]
[TimeControl "600+0"]

1. d4 d5 2. c4 c6 3. Nc3 dxc4 4. e3 b5 5. a4 b4 6. Nb1 Nf6 7. Bxc4 e6 8. Nf3 a5
9. Nbd2 Be7 10. O-O O-O 11. Nb3 Ba6 12. Nfd2 Nbd7 13. Bxa6 Rxa6 14. Nc4 Nb6 15.
Ncxa5 Qd5 16. Nb7 Rxa4 17. Rxa4 Nxa4 18. N3a5 Nb6 19. Qc2 Rc8 20. Bd2 h6 21.
Rc1 Ne4 22. Be1 c5 23. dxc5 Nxc5 24. Rd1 Nxb7 25. Qxc8+ Nxc8 26. Rxd5 exd5 27.
Nxb7 Nb6 28. b3 Kf8 29. Na5 Nd7 30. Nc6 Nc5 31. Bxb4 Bd6 32. Kf1 Ke8 33. Ke2
Kd7 34. Bxc5 Bxc5 35. Ne5+ Ke6 36. Nd3 Bd6 37. g3 g5 38. b4 Kd7 39. Nc5+ Bxc5
40. bxc5 Kc6 41. Kd3 Kxc5 42. e4 d4 43. f4 f6 44. h4 gxh4 45. gxh4 h5 46. Kd2
Kc4 47. Ke2 d3+ 48. Kd2 Kd4 49. e5 fxe5 50. fxe5 Kxe5 51. Kxd3 Kf4 52. Ke2 Kg3
53. Kf1 Kxh4 54. Kg1 Kh3 55. Kh1 Kg3 56. Kg1 h4 57. Kh1 Kf2 58. Kh2 1/2-1/2

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — wrong

In Game A, White played a coherent London setup and punished Black's loosening play with clean tactics: 15.Ng6, 16.Bxd6, 23.Nxd5 winning material, and 34.Ne7+ with 35.Nxg8. White then converted smoothly, finishing with 43.Bxg6+ and 46.Re8#. In Game B, Black's play was error-prone. 15...Qd5 allowed White's knights to invade, and 24...Nxb7?? opened the c-file for 25.Qxc8+, dropping a rook to a simple tactic. Black only drew because White also misplayed. The king-and-pawn ending was also handled imprecisely, since the rook pawn let White's king reach the drawing corner. Overall, A's target looks more tactically reliable.

Reported P(A stronger): 0.66. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A White (Stronger? unclear): played a Stonewall-like Bf4 setup with an early h4 and Ng5. The play was aggressive but mostly sound, and Black's defence was loose (e.g. 9...Qc8, 20...Qd7, 23...Nd7 allowing Nxd5). White won with Nxd5 and Qxc5, then cleanly finished with Rd8 and Re8#. Overall this is a club-level performance with tactical alertness. Game B Black: handled a Slav and Queen's Gambit Accepted line with ...b5/...b4 accurately, and found ...Qd5 and ...Rxa4. Black navigated the tactical sequence (15.Ncxa5 through 26.Rxd5 exd5) and held an ending a pawn down against a stronger-looking White; the 41...Kxc5 pawn race was precise, and the draw was secure. Black's play was more accurate and consistent against a competent opponent, so B is slightly more likely to be stronger.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Player B exhibits substantially higher chess understanding and tactical refinement. In an advanced Slav/a4-b4 line, Black handled Queen's-side piece coordination cleanly and executed the tactical liquidation 24...Nxb7! 25.Qxc8+ Nxc8 26.Rxd5 exd5, transitioning into a slightly worse but holdable endgame that was drawn with textbook king activity and pawn-play technique. In contrast, White in Game A plays an aggressive early h4/Ng5 London attack against a severely underperforming opponent who collapsed with 15...Rd8? and 16...Kf7?. While White efficiently converted the material advantage, the level of opposition and strategic complexity in Game B reflects a significantly higher established rating.

Reported P(A stronger): 0.22. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Target B (Black) demonstrates deep positional and tactical understanding in the Slav Defense, navigating a highly complex sequence from moves 22-26 that results in an equal endgame. The thematic 22...c5 break and the ensuing mass exchanges (24...Nxb7 25. Qxc8+ Nxc8 26. Rxd5 exd5) show precise calculation. Target A (White) plays a decent attacking game in the Dutch, punishing Black's blunders (such as 15...Rd8? allowing 16. Bxd6), but early moves like 8.h4 and 11.Nh3 are somewhat committal and less refined than B's play. B's ability to maintain equality in a sharp theoretical variation suggests a higher rating.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

Game B (Black) demonstrates a much higher level of chess understanding and technical skill. Black navigates a sharp, theoretical Slav Defense (3...dxc4, 5...b4), coordinates pieces accurately in a queenless middlegame (15...Qd5, 16...Rxa4), and calculates a pawn-down minor piece endgame into a precise drawn king-and-pawn endgame (39...Bxc5, 48...Kd4). In contrast, Game A (White) plays a superficial opening with early flank attacks (8. h4, 11. Nh3) and wins primarily due to Black's major tactical blunder (16...Kf7?? allowing 17. Ne5+). Game B's positional accuracy and endgame precision suggest a significantly higher established rating.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Game B demonstrates much higher-level chess understanding and precision from both players. Target B navigates a complex Slav structure smoothly, executing accurate piece coordination (14...Nb6, 15...Qd5) and sharp liquidating tactics (22...c5! through 26...exd5). Target B then displays flawless endgame calculation to secure a pawn-ending draw (42...d4!, 47...d3+). In contrast, Game A features cruder opening play with an aggressive flank thrust (8. h4, 9. Ng5), where White capitalized easily on Black's severe tactical blunders (such as 16...Kf7 and passive piece placement) without facing substantial resistance.

Reported P(A stronger): 0.22. Consistent with choice.

Qwen3.8 Max Prime: chose A — wrong

Game A White shows strong tactical awareness: 16.Bxd6 wins material, 19.Bxf6 shatters Black's kingside, and 23.Nxd5 exploits the pin, converting to a clean win. Game B Black plays the Slav solidly but drifts passively; after 15...Qd5 and 16...Rxa4, Black trades into a simplified endgame (24...Nxb7 25.Qxc8+) without generating winning chances, ultimately drawing a king-and-pawn race. White's active piece coordination and decisive finishing in Game A suggest a higher established rating than Black's competent but reactive play in Game B.

Reported P(A stronger): 0.72. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A White plays a loose London setup with speculative 8.h4 and 9.Ng5/11.Nh3 shuffling, then 14.Ne5/15.Ng6 tactics work only because Black (Game A) blundered with 14...Bd6?? and 16...Kf7; White's conversion (24.Qb3, 37.Rd8) is competent but the opponent's level was low. Game B Black handles a theoretical Slav line (3...dxc4, 4...b5, 8...a5, 11...Ba6), keeps material balance through complex trades (15...Qd5, 16...Rxa4, 24...Nxb7 25.Qxc8+ Nxc8), and plays a long, accurate king-and-pawn endgame, though 56...h4?? spoiled a win with ...Kf2/...h3 ideas earlier. Overall Black in B shows more structured, sound chess against a stronger opponent.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Target B shows superior technical skill, navigating a complex Slav Defense mainline (4... b5, 5... b4). Black's maneuvering (14... Nb6) and the thematic 22... c5 break indicate deep theoretical knowledge. The tactical sequence 24... Nxb7 and the flawless transition to a drawn pawn endgame (39... Bxc5, 48... Kd4) suggest a high rating. Conversely, Target A wins against a weak Dutch Defense (1... f5) where Black blunders early (9... Qc8, 15... Rd8). While Target A efficiently capitalizes with 15. Ng6 and 24. Qb3, the overall quality of opposition and the lack of complex positional requirements in Game A make Target B the likely stronger player.

Reported P(A stronger): 0.2. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Game B's target (Black) demonstrates high positional understanding and theoretical fluency in a sharp Slav Defense line. Black executes the tactical break 22...c5! smoothly, leading to a forcing sequence (24...Nxb7! 25.Qxc8+ Nxc8 26.Rxd5 exd5) that transforms into a balanced endgame, which Black defends flawlessly to a draw. In contrast, Game A's target (White) plays a somewhat premature early flank advance with 8. h4 and 11. Nh3, though White effectively capitalizes on Black's major blunder (16...Kf7??). Overall, Black in Game B shows superior overall chess culture, calculation depth, and endgame technique.

Reported P(A stronger): 0.35. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B’s Black showed stronger positional and tactical control. The sequence 24...Nxb7! 25.Qxc8+ Nxc8 26.Rxd5 exd5 deliberately simplified into a sound endgame, followed by active king play with 32...Ke8, 33...Kd7 and the well-timed 42...d4. Game A’s White spotted several clear opportunities—16.Bxd6, 23.Nxd5 and 34.Ne7+—and converted decisively, but these arose largely from Black’s loose play, especially 14...Bd6, 15...Rd8 and the exposed king. White A’s early 8.h4 and repeated knight maneuvers were effective but less convincing as evidence of established strength than Black B’s sustained, coherent play.

Reported P(A stronger): 0.2. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s Black shows more sustained positional control: 16...Rxa4 finds counterplay, 21...Ne4 and 22...c5 challenge White’s center, and 39...Bxc5 leads to an active king-and-pawn ending. Game A’s White exploits Black’s errors well, notably with 17.Ne5+ and 34.Ne7+, but the early 8.h4, 9.Ng5, 11.Nh3 sequence spends several tempi on the same knight. A’s tactical win is impressive, but B’s play against a more resistant opponent is slightly stronger evidence of established strength.

Reported P(A stronger): 0.34. Consistent with choice.

GPT-6.1 Sol: chose A — wrong

Game A White showed sustained tactical coordination: 16.Bxd6 exploited the potential Ne7+ queen fork, while 34.Ne7+ followed by 35.Nxg8 won material. White then used 37.Rd8 and finished accurately with 46.Re8#. Game B Black demonstrated competence with the liquidation 24...Nxb7 25...Nxc8 26...exd5 and active king play. However, 44...gxh4 and 45...h5 simplified the remaining chances into an ultimately drawn rook-pawn ending. A’s stronger tactical execution gives White a modest edge, tempered by Game A Black’s weaker resistance.

Reported P(A stronger): 0.61. Consistent with choice.

GPT-5.6 Luna: chose A — wrong

Game A’s white showed more convincing tactical initiative and conversion: 15.Ng6, 18.Be7, and 19.Bxf6 created lasting pressure, followed by 23.Nxd5 and 27.Qxc5. The attack was converted accurately with 35.Nxg8, 38.Rxg8+, and the mating finish 46.Re8#. Game B’s black played solidly and found a useful simplification with 24...Nxb7, 25...Nxc8, and 26...exd5, then handled the pawn ending competently. However, that game provides mostly evidence of resilience and endgame technique rather than the stronger player creating winning chances. Thus A is favored, though only slightly because Game A’s opponent made substantial defensive errors.

Reported P(A stronger): 0.55. Consistent with choice.

GPT-6 Astra: chose A — wrong

A’s White showed sustained tactical awareness: 16.Bxd6 exploited the potential Ne7+ queen fork, 23.Nxd5 increased central pressure, and 34.Ne7+ followed by 37.Rd8 decisively converted the attack. The early h4–Ng5–Nh3 maneuver was less convincing. B’s Black demonstrated good coordination with 22...c5 and the liquidation beginning 24...Nxb7, then activated the king effectively. However, 47...d3+ and the subsequent exchanges allowed White’s e5 resource and a drawn rook-pawn ending. A’s cleaner tactical execution gives a slight edge, tempered by the weaker defensive resistance in Game A.

Reported P(A stronger): 0.59. Consistent with choice.

GLM-5.3 Flash: chose B — correct

Game B's Black navigated a tricky Semi-Slav line (15...Qd5, 16...Rxa4 counting out the knight fork ideas), simplified correctly (24...Nxb7, 25...Nxc8), and reached a dead-drawn pawn endgame — solid, theory-aware play with clean technique (42...d4, 47...d3+ showing accuracy). Game A's White won but with less convincing chess: 8.h4 and 9.Ng5 are crude attacking setups, though 21.Bh5 and the mating net (34.Ne7+, 45.Qxg6) show tactical alertness. Black's opposition in A was notably passive (10...h6, 12...bxc5 opening lines against himself), inflating A's result. B's clean handling of concrete lines suggests the higher established strength.

Reported P(A stronger): 0.6. Contradicts choice; excluded from confidence diagnostics.

Pair 19: 3 of 16 models wrong

A: White, rated 1514 · B: White, rated 2689. Higher pre-game rating: B.

Game A: White, rated 1514 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 e5 2. Bc4 Nf6 3. Nf3 Bc5 4. d3 O-O 5. O-O Nc6 6. a3 d5 7. exd5 Nxd5 8.
Nbd2 Bg4 9. Ne4 Bb6 10. h3 Bh5 11. g4 Bg6 12. Neg5 h6 13. Ne4 Nf4 14. Bxf4 exf4
15. Kg2 Qe7 16. Re1 Bxe4 17. Rxe4 Qf6 18. c3 g5 19. Bb3 Rfe8 20. Bc2 Rxe4 21.
dxe4 Rd8 22. Qe2 Ne5 23. Rd1 Rd6 24. Rd5 c6 1-0
Game B: White, rated 2689 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. c4 e6 2. Nf3 d5 3. g3 Nf6 4. Bg2 Nbd7 5. O-O Be7 6. b3 O-O 7. Bb2 c5 8. cxd5
exd5 9. d4 b6 10. Nc3 Bb7 11. Rc1 Rc8 12. dxc5 Nxc5 13. Nd4 Qd7 14. Qd2 Rfe8
15. Qf4 Ne6 16. Nxe6 fxe6 17. Rfd1 Bd6 18. Qh4 Rf8 19. e3 Qf7 20. Nb5 Be7 21.
Qf4 Qg6 22. Rxc8 Bxc8 23. Qc7 Bc5 24. Nxa7 Qc2 25. Rf1 Ba6 26. Be5 Bxf1 27.
Bxf1 Ne8 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A (White) played a sound but simple Italian. However, 11.g4 loosened the kingside, and the win came abruptly after 24.Rd5 c6 (likely a flag or resignation), not from clear technique. Game B (White) handled a double-fianchetto English/Catalan structure competently: 12.dxc5, 13.Nd4 and 16.Nxe6 created targets, and 20.Nb5 with 24.Nxa7 won material. White then erred with 25.Rf1? allowing ...Ba6 to win the exchange, and the queen wanderings (Qf4-h4-f4) were imprecise. Game B's opening and middlegame understanding look somewhat more sophisticated, though both players show errors. The evidence is close, so this is a slight lean toward B.

Reported P(A stronger): 0.45. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A White played an Italian-style opening but allowed 11...Bg6 and 13...Nf4 trades. The 24 moves are not very accurate: 11.g4 loosened the king, and 15.Kg2 left it exposed. White then won against a weak Black who dropped material (24...c6 ended the game, probably after a blunder). Black's play (18...g5, 22...Ne5) was loose, suggesting a modest opponent. Game B White played a coherent English with 3.g3, Bb2, Rc1, and 13.Nd4, so the structure and development look sound. But 20.Nb5 and 24.Nxa7 grabbed a pawn and left White's pieces offside and exposed to 24...Qc2 and 26...Bxf1, and White lost. Black in Game B played well and punished White's mistakes, which suggests a stronger opponent. Overall B's play was more theoretically sound but A's opponent was weaker. I lean slightly to B.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game B exhibits high-level strategic comprehension from both sides in a sophisticated English Opening. White coordinates pieces harmoniously with 11. Rc1, 13. Nd4, and 17. Rfd1, maintaining strong pressure until a tactical oversight with 24. Nxa7? allowed 24...Qc2!. In contrast, Game A features more rudimentary play: White unnecessarily weakens the kingside with 11. g4 and 12. Neg5, misplacing pieces and leaving holes that Black could have exploited. Even though White won Game A (likely by an early resignation or time-out on move 24), Player B's opening sophistication and positional maneuvering reflect a considerably higher established rating.

Reported P(A stronger): 0.23. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Game B White demonstrates a sophisticated understanding of positional play in a Reti/Catalan system, developing harmoniously and maneuvering well (13. Nd4, 15. Qf4) before a tactical oversight at the end (24. Nxa7 allowing Qc2). In contrast, Game A White plays a less refined opening, pointlessly wasting tempi (12. Neg5 h6 13. Ne4) and unnecessarily weakening the kingside with 11. g4 to break a pin. Therefore, Game B White likely has a higher rating despite losing.

Reported P(A stronger): 0.2. Consistent with choice.

Gemini 3.5 Flash: chose A — wrong

Game A White played a solid, blunder-free game. While the opening was relatively simple, White maintained active pieces, coordinated well, and capitalized on Black's late-game errors to secure a win. In contrast, Game B White displayed good positional understanding in an English Opening but suffered a severe tactical collapse. 24. Nxa7?? was a decisive blunder that overlooked 24...Qc2, which forks the b2-bishop and f1-rook and immediately loses the game. Player A's tactical consistency and lack of critical errors suggest a more reliable established playing strength.

Reported P(A stronger): 0.58. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Game B features much deeper opening understanding and strategic sophistication in a standard English/Reti structure (1. c4 through 18. Qh4) before White tactical slip with 24. Nxa7? allowing 24...Qc2. In contrast, Game A displays significantly lower-tier play, with White playing crude pawn-pushing and awkwardly placed knights (11. g4, 12. Neg5 h6 13. Ne4), facing an opponent who resigned prematurely after 24. Rd5. The overall structural and positional competence in Game B strongly points to a higher established rating.

Reported P(A stronger): 0.32. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

Game A's White played solid, accurate moves—central control, timely exchanges (16...Rxe4 20...Rxe4), and a clean rook lift ending with 24.Rd5, earning a win with minimal errors. Game B's White collapsed badly: 24.Nxa7?? hangs material and structure, allowing 24...Qc2, then 26.Bxf1?? loses to 26...Ne8, resigning immediately. Despite losing, B's opening (Catalan setup, 12...Nxc5, 15...Ne6) showed higher-level strategic understanding; the blunder was time-pressure induced. A won but with less demanding play and a simpler structure.

Reported P(A stronger): 0.32. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A White played 2.Bc4 then 6.a3 and the aimless 10.h3/11.g4/12.Neg5/13.Ne4 knight shuffle, and 24.Rd5 (Black can take with ...cxd5) looks like a blunder; the win came from Black resigning abruptly, not strong play. Game B White handled a Catalan/Reti structure with coherent plans (8.cxd5, 12.dxc5, 13.Nd4, 15.Qf4 pressuring d5), only collapsing after 23.Qc7?/24.Nxa7 allowed 24...Qc2 and 25...Ba6 winning the exchange. Game B's positional understanding and opponent quality suggest a higher rating despite the loss.

Reported P(A stronger): 0.38. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

White B demonstrates sophisticated positional play in an English/Catalan structure (1. c4, 3. g3, 9. d4), showing high-level opening knowledge. The coordination and maneuvering (13. Nd4, 15. Qf4, 20. Nb5) suggest a strong established rating. White A plays a standard Italian with a somewhat aggressive but loosening kingside expansion (11. g4). While White A capitalized on Black's late blunder (24... c6) to win, the strategic depth in Game B is more characteristic of a higher-rated player. Even with White B's tactical oversight (24. Nxa7), the overall quality of move selection in the first 20 moves points towards B having a higher established Lichess rapid rating.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.6 Flash: chose A — wrong

Player A played a solid, clean positional game without major mistakes, smoothly consolidating after 11. g4 with 15. Kg2 and maintaining strong central/d-file control (21. dxe4, 23. Rd1, 24. Rd5), ultimately forcing Black to resign in a lost position. In contrast, Player B built a nice English opening setup but suffered a catastrophic tactical oversight with 24. Nxa7??, missing Black's double attack 24...Qc2! which attacked both the bishop on b2 and the rook on f1. Player B failed to defend the tactics, losing material rapidly (26...Bxf1) and collapsing immediately.

Reported P(A stronger): 0.65. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B White showed stronger established opening and positional habits: 6.b3/7.Bb2, 11.Rc1, 14.Qd2, and 17.Rfd1 formed a coherent setup, while 20.Nb5 created active pressure. The decisive error was concrete but isolated: after saving the rook from 24...Qc2 with 25.Rf1, White overlooked 25...Ba6 and allowed 26...Bxf1. Game A White remained tactically sound, but the plan with 6.a3, 11.g4, and the 12.Neg5–13.Ne4 retreat was comparatively loose and inefficient. Game A’s unexplained finish after 24...c6 also provides little evidence of conversion skill. Thus B looks likelier to have had the higher established rapid strength despite losing.

Reported P(A stronger): 0.35. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s White shows more coherent development and planning: 7.Bb2, 11.Rc1, 17.Rfd1 and 22.Rxc8. The later 25.Rf1? allowed 25...Ba6 and 26...Bxf1, but that oversight does not erase the stronger earlier play. Game A’s White spent several moves retreating or repositioning the same pieces—12.Neg5 13.Ne4 and 19.Bb3 20.Bc2—after weakening the kingside with 11.g4. Game A’s result alone is not persuasive evidence of greater established strength.

Reported P(A stronger): 0.18. Consistent with choice.

GPT-6.1 Sol: chose B — correct

B White showed more coherent opening and positional play: 10.Nc3, 11.Rc1, 13.Nd4 and 17.Rfd1 coordinated development and central pressure. However, 24.Nxa7 followed by 26.Be5 overlooked Black’s ...Ba6–...Bxf1 tactic, losing the exchange. A White’s 11.g4 weakened the king, while 12.Neg5 h6 13.Ne4 wasted tempi; 15.Kg2 compounded the awkward king placement. A subsequently played reasonably with 18.c3 and 24.Rd5, but the recorded win after ...c6 does not demonstrate a decisive advantage. B’s stronger positional foundation narrowly outweighs the tactical lapse.

Reported P(A stronger): 0.61. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose A — wrong

Game A White played more forcefully and tactically, with 11.g4, 14.Bxf4, and the active 16.Re1 and 20.Bc2. The sequence 20...Rxe4 21.dxe4 kept the initiative, and Black resigned after 24...c6. Game B White showed reasonable positional development, but the tactical sequence 25...Ba6 26.Be5? Bxf1 allowed Black to take the rook, with only 27.Bxf1 recovering the bishop. That substantial one-move oversight makes Game B White look less established, despite the otherwise coherent opening and middlegame.

Reported P(A stronger): 0.64. Consistent with choice.

GPT-6 Astra: chose B — correct

B’s White showed more coherent positional development: 9.d4, 11.Rc1, 13.Nd4 and 17.Rfd1 coordinated the pieces and contested the center. The later 24.Nxa7 and especially 26.Be5, allowing ...Bxf1, were serious tactical lapses. A’s White avoided comparable material losses, but 6.a3 and the sequence 11.g4, 12.Neg5, 13.Ne4 spent time and weakened the kingside without a clear payoff. A’s 23.Rd1 and 24.Rd5 were purposeful, though the unexplained ending after 24...c6 supplies no evidence of a winning position. B’s stronger opening organization narrowly outweighs the later mistakes.

Reported P(A stronger): 0.36. Consistent with choice.

GLM-5.3 Flash: chose B — correct

Game B's White plays a coherent English/Reversed-Sicilian structure, trades well, and only loses after 24.Nxa7 grabs a poisoned pawn into Black's organized counterplay (24...Qc2, 25...Ba6, 26...Bxf1), suggesting Black was genuinely strong rather than White weak. Game A's White wins mainly because Black self-destructs: 17...Qf6 allows Rxe4 pressure, 18...g5 weakens the king, and 24...c6 simply drops the pawn/position. A's own play (12.Neg5 then retreating 13.Ne4, passive 21.Qe2) is unremarkable. B's positional handling looks stronger overall.

Reported P(A stronger): 0.4. Consistent with choice.

Pair 12: 3 of 16 models wrong

A: Black, rated 1867 · B: Black, rated 1382. Higher pre-game rating: A.

Game A: Black, rated 1867 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 c5 2. Bc4 e6 3. Nf3 a6 4. c3 b5 5. Bd3 c4 6. Be2 Bb7 7. d3 cxd3 8. Bxd3
Nf6 9. Nbd2 Be7 10. e5 Nd5 11. Be4 f5 12. Bd3 O-O 13. b3 Nc6 14. Bb2 Qc7 15. c4
Nf4 16. Be4 Nxg2+ 17. Kf1 fxe4 18. Nxe4 Nf4 19. Nfg5 h6 20. Nf3 Nb4 21. a3 Nbd3
22. Bc3 Bxe4 23. Nd4 Bxh1 24. Kg1 Qb7 25. f3 Nh3+ 26. Kf1 Bxf3 27. Qxd3 Be4+
28. Nf3 Bxd3+ 29. Kg2 Qxf3# 0-1
Game B: Black, rated 1382 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. c4 d6 2. Nc3 Nf6 3. e3 Bd7 4. Bd3 e5 5. a3 Be7 6. Bb1 O-O 7. Qc2 g6 8. h4 h5
9. b3 Bf5 10. e4 Bd7 11. Bb2 Nc6 12. Nf3 Bg4 13. Ng5 Nd4 14. Qd3 Nd7 15. f3 Be6
16. f4 Bxg5 17. hxg5 Nc5 18. Qe3 Ncxb3 19. f5 Nxa1 20. Bxa1 Bxc4 21. fxg6 fxg6
22. Nd5 Ne6 23. Qc3 Ba6 24. Nf6+ Kg7 25. Rf1 Bxf1 26. Kxf1 Nxg5 27. Ke1 Qxf6
28. Qxc7+ Kh6 29. Bd3 Rac8 30. Qxb7 Rc1+ 31. Ke2 Qf2# 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Game A Black played purposeful moves: the knight raid ...Nf4, ...Nxg2+ winning material, ...fxe4, then a coordinated attack with ...Nbd3, ...Bxh1, ...Nh3+ and a clean finish with ...Qxf3#. The opponent was weak (Bc4-d3-e2 wandering, 16.Be4 allowing Nxg2+). Game B Black played more passively and loosely, with odd bishop shuffles (...Bd7-f5-d7-g4-e6) and 16...Bxg5 giving up the bishop. Black still won material via ...Ncxb3 and ...Nxa1 and later mated with ...Qf2#, but White blundered heavily (Rf1?). Both targets exploited weak opponents, but Game A's attack looked more coherent, so A is slightly favoured.

Reported P(A stronger): 0.58. Consistent with choice.

Claude Sonnet 5.5: chose B — wrong

Game A (Black): White played poorly, with 10.e5 and 11.Be4 allowing 11...f5, then 15.c4 and 16.Be4. Black found 16...Nxg2+ and later 23...Bxh1, 25...Nh3+ and 29...Qxf3#, a strong attack. But White's opening and defense were weak, so the opponent looks low-rated, and Black's 2...e6 and 4...b5 are standard Sicilian-gambit-like play at club level. Game B (Black): Black played a solid King's Indian/Philidor setup and took the chance after 13.Ng5 with 13...Nd4, 16...Bxg5 and 18...Ncxb3, winning material. White blundered with 14.Qd3 and 19.f5, and Black then converted accurately with 25...Bxf1, 27...Qxf6 and 31...Qf2#. Black's 30...Rc1+ was a precise finish. Both opponents were weak and both targets played tactically well, so it is close. Black in B showed slightly more positional understanding and calculation in a more complex game, so I lean B.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3.8 Flash: chose B — wrong

Both target players won cleanly with Black, but Game B faced a more coherent kingside setup and demonstrated superior coordination. Game B Black neutralized White's h-pawn thrust, found energetic knight outposts (13...Nd4, 17...Nc5), and executed a dominant queenside infiltration (18...Ncxb3, 19...Nxa1). The conversion was completely clinical through 29...Rac8 and mate. In Game A, White wasted several tempi shuffling the bishop (Bd3-Be2, Be4-Bd3-Be4) and blundered outright with 16. Be4?? allowing 16...Nxg2+, leaving Black with an easier tactical task to finish the game.

Reported P(A stronger): 0.42. Consistent with choice.

Gemini 3.1 Pro (Preview): chose A — correct

Both targets played excellent attacking games against opponents who misplayed the opening. In Game A, Black was extremely sharp, finding tactical shots like 15... Nf4, 16... Nxg2+, and the crushing 24... Qb7! to create mating nets. In Game B, Black also punished White's bizarre setup (6. Bb1) efficiently with knights maneuvering to d4 and c5. However, Game A Black's tactical execution was slightly crisper and more complex, suggesting a higher level of rating.

Reported P(A stronger): 0.6. Consistent with choice.

Gemini 3.5 Flash: chose A — correct

In Game A, Black demonstrates excellent opening understanding with an active Sicilian setup, followed by precise, energetic tactical calculation. Moves like 15...Nf4, the tactical sequence 16...Nxg2+ and 17...fxe4, followed by 20...Nb4! show high-level tactical vision and a direct, punishing attack. In Game B, Black plays a somewhat passive and disjointed opening, including the repetitive maneuver 3...Bd7, 9...Bf5, and 10...Bd7. While Game B's Black successfully exploits White's passive play later on, the overall cohesion, opening preparation, and tactical sharpness of Game A's Black suggest a significantly higher established rating.

Reported P(A stronger): 0.7. Consistent with choice.

Gemini 3.7 Flash: chose A — correct

Game A Black displayed excellent opening understanding and tactical sharpness against White's unorthodox Sicilian play. Moves like 4...b5, 5...c4, 11...f5, and the precise tactical combination starting with 16...Nxg2+ followed by 17...fxe4 and 20...Nb4/21...Nbd3 demonstrated high-level positional control and tactical accuracy. In Game B, both players looked less seasoned; Black's passive 3...Bd7 and retreating tempo loss 9...Bf5 and 10...Bd7 signaled less sophisticated opening knowledge, even though Black converted well later with 13...Nd4 and 17...Nc5. Game A Black demonstrated higher overall proficiency.

Reported P(A stronger): 0.64. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

Game A Black shows sharper tactical vision: 15...Nf4! sets up 16...Nxg2+!, and after 17.Kf1 the sequence 21...Nbd3, 22...Bxe4, 23...Bxh1 wins decisively, finishing with 29...Qxf3#. Game B Black plays solidly but relies heavily on opponent blunders: White's 13.Ng5?, 17.Qe3?, and especially 24.Nf6+? sacrifice material gratuitously. B's 18...Ncxb3 is good but less forcing than A's coordinated knight-bishop attack. A's opponent resisted longer and made fewer outright hangs, so A's win required more accurate, creative play.

Reported P(A stronger): 0.68. Consistent with choice.

Claude Fable 5.1: chose B — wrong

Game A Black played a coherent Sicilian with ...b5/...c4 harassing the bishop, then exploited White's loose play with Nf4/Nxg2+ tactics and a clean mating attack; but White in A was very weak (Bc4/Bd3/Be2 shuffles, Be4 hanging to Nxg2+). Game B Black faced a slightly more organized opponent (Nc3, Qc2, Bb2 setup) and converted via ...Bxg5, ...Ncxb3 winning the exchange, ...Bxc4, and the final ...Rc1+ ...Qf2# mating net, though ...Bf5/Bd7/Bg4/Be6 bishop wandering and ...Bxg5 were loose. Both are similar club level; B's calculation of multiple material-winning tactics against a less cooperative opponent gives a slight edge.

Reported P(A stronger): 0.42. Consistent with choice.

Gemini 3 Flash (Preview): chose A — correct

Black in Game A demonstrates superior opening knowledge and tactical precision. Black plays a thematic Sicilian setup and immediately punishes White's inaccuracies (13. b3 and 16. Be4) with the powerful 15... Nf4 and 16... Nxg2+. The subsequent coordination of the knights (20... Nb4, 21... Nbd3) and the clinical finish (25... Nh3+ and 27... Be4+) show a high level of proficiency. In Game B, Black plays more passively with 3... Bd7 and 10... Bd7. While Black B exploits White's eccentric play (6. Bb1, 8. h4) well with 13... Nd4, the overall level of theoretical accuracy and calculated aggression in Game A suggests a stronger established player.

Reported P(A stronger): 0.65. Consistent with choice.

Gemini 3.6 Flash: chose A — correct

Both players capitalized well on unorthodox opening play by their opponents, but Black in Game A displayed superior dynamic precision and tactical awareness. Game A Black launched a energetic Sicilian expansion with 4...b5 and 5...c4, then efficiently punished White's clumsy piece coordination (13. b3?) with a series of sharp tactical blows starting with 15...Nf4!, 16...Nxg2+!, 17...fxe4!, and 20...Nb4!. In Game B, Black played slightly passively early on (3...Bd7) before taking advantage of White's poor setup (4. Bd3, 6. Bb1) to win material. Game A Black showed higher tactical sharpness and dynamic initiative overall.

Reported P(A stronger): 0.58. Consistent with choice.

GPT-5.6 Sol: chose A — correct

Game A Black showed more purposeful calculation: 16...Nxg2+! 17.Kf1 fxe4 won the bishop while preserving the knight, and 22...Bxe4 23...Bxh1 followed by ...Nh3+ and the forced mating sequence was precise. Game B Black also converted well, especially with 18...Ncxb3 19...Nxa1 and 25...Bxf1, but the opening involved several inefficient bishop moves (...Bd7-f5-d7), and much of the win followed White’s conspicuous tactical errors. A’s play looks more consistently coordinated and tactically sophisticated.

Reported P(A stronger): 0.64. Consistent with choice.

GPT-6 Sol: chose A — correct

Game A’s Black played a more coherent game: 11...f5 drove back White’s bishop, 16...Nxg2+ set up 17...fxe4, and 20...Nb4 followed by 21...Nbd3 supported the decisive 22...Bxe4. Game B’s Black also spotted strong tactics, notably 18...Ncxb3 and 25...Bxf1, but the early 9...Bf5 10...Bd7 lost time. Both opponents made serious mistakes, so the comparison is uncertain.

Reported P(A stronger): 0.68. Consistent with choice.

GPT-6.1 Sol: chose A — correct

Game A Black showed more coherent development and sustained tactical coordination: 15...Nf4, 16...Nxg2+ and 17...fxe4 exploited White’s bishop placement, while 21...Nbd3 and 22...Bxe4 led to winning the rook. The finish with 24...Qb7, 25...Nh3+ and 27...Be4+ was particularly coordinated. Game B Black also found effective tactics—17...Nc5, 18...Ncxb3 and 25...Bxf1—and finished accurately with 30...Rc1+. However, the repeated early bishop moves (...Bd7–f5–d7–g4–e6) suggest less efficient opening handling. Both benefited from substantial White errors, so the preference for A is moderate.

Reported P(A stronger): 0.67. Consistent with choice.

GPT-5.6 Luna: chose A — correct

Game A’s Black showed a more coherent attacking plan and tactical coordination: 16...Nxg2+ followed by 17...fxe4, 21...Nbd3, and 23...Bxh1 created sustained pressure and a forced mating net. Game B’s Black also exploited major oversights, especially 18...Ncxb3 and 19...Nxa1, then 25...Bxf1 and 27...Qxf6. However, the repeated bishop retreats in the opening (3...Bd7, 9...Bf5, 10...Bd7) and dependence on White’s loose play make B’s performance look slightly less polished. A therefore appears marginally more likely to have had the higher established rapid strength.

Reported P(A stronger): 0.58. Consistent with choice.

GPT-6 Astra: chose A — correct

Game A Black showed more coherent development and sustained tactical coordination: 15...Nf4, 16...Nxg2+, and 17...fxe4 won material, while ...Nb4–d3 intensified the attack. The finish 27...Be4+ cleverly combined a discovered rook check with an attack on White’s queen. Game B Black also found effective tactics, notably 17...Nc5 and ...Ncxb3–a1, but the opening featured repeated bishop moves (...Bd7–f5–d7–g4–e6) and less economical development. Both opponents supplied substantial errors; B’s opponent additionally donated a rook with 25.Rf1. A’s more consistent coordination makes A the likelier stronger established player.

Reported P(A stronger): 0.73. Consistent with choice.

GLM-5.3 Flash: chose A — correct

Black in A played an active, coherent game: queenside space (4...a6/4...b5), the tactical shots 15...Nf4 and 16...Nxg2+!, then 21...Nbd3 forking, winning material cleanly and concluding with a forced mating net (25...Nh3+, 26...Bxf3, 27...Be4+, 29...Qxf3#). Black in B also won but largely by punishing obvious White errors (13.Ng5?, 18...Ncxb3/19...Nxa1 exploiting loose moves), and needed 30 moves against passive play; A's attack was more precise and self-created, suggesting the higher rating.

Reported P(A stronger): 0.68. Consistent with choice.

Pair 22: 2 of 16 models wrong

A: Black, rated 2718 · B: Black, rated 1551. Higher pre-game rating: A.

Game A: Black, rated 2718 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 c6 2. d4 d5 3. exd5 cxd5 4. Bd3 Nc6 5. c3 Qc7 6. Ne2 Bg4 7. O-O e6 8. Qe1
Nf6 9. Bg5 Bh5 10. f4 Bxe2 11. Qxe2 Be7 12. Nd2 h6 13. Bh4 O-O-O 14. b4 Kb8 15.
Nb3 Rc8 16. a4 Ne4 17. Be1 f5 18. Nc5 Bxc5 19. dxc5 g5 20. a5 a6 21. c4 gxf4
22. b5 Nd4 23. Qb2 Qxc5 24. Bb4 Ne2+ 25. Kh1 Qg1+ 26. Rxg1 Nf2# 0-1
Game B: Black, rated 1551 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. d4 e6 2. Nf3 c5 3. dxc5 Qa5+ 4. Bd2 Qxc5 5. Be3 Qb5 6. Nc3 Qxb2 7. Rb1 Qa3
8. Rb3 Qa5 9. Bd2 Bb4 10. Qa1 Ne7 11. Nb1 Bxd2+ 12. Nfxd2 f6 13. Ra3 Qc7 14.
Rc3 Nbc6 15. e3 Nd5 16. Rc5 Ndb4 17. c3 b6 18. Qb2 bxc5 19. cxb4 cxb4 20. Ne4
O-O 21. Bb5 Qe5 22. Qxe5 Nxe5 23. Nd6 a6 24. Ba4 Nc6 25. O-O f5 26. Re1 Rf6 27.
Rc1 e5 28. Ne8 Re6 29. Nc7 Ra7 30. Nxe6 dxe6 31. Rxc6 Bd7 32. Rc4 Bxa4 33. Rxb4
Bb5 34. Nc3 Rc7 35. Nxb5 Rc1# 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Game A Black played a coherent Caro-Kann exchange plan (…Bg4xe2, …O-O-O, …Ne4, …f5, …g5) and finished with the sharp queen sacrifice 25…Qg1+ 26.Rxg1 Nf2#, a smothered mate. Game B Black won mainly because White blundered repeatedly (8.Rb3, 16.Rc5 losing the exchange, 35.Nxb5?? allowing …Rc1#). Black's own play there was loose: early queen wandering with 6…Qxb2, and the odd 26…Rf6/28…Re6 that dropped the exchange to Nc7. That suggests a weaker player benefiting from a weak opponent. A's opponent was more competent, and A's tactical finish was cleaner.

Reported P(A stronger): 0.68. Consistent with choice.

Claude Sonnet 5.5: chose A — correct

Game A Black played the Caro-Kann sensibly (4...Nc6, 6...Bg4, 10...Bxe2) and then punished White's weak kingside play. 18...Bxc5 and 19...g5 were active, 21...gxf4 opened lines, and 22...Nd4 followed by 23...Qxc5 and 24...Ne2+ led to a clean smothered-style mate with 26...Nf2#. The opponent was weak, so the win is only modest evidence. Game B Black grabbed the b2 pawn with 6...Qxb2 and won, but the opponent was also weak: White's 16.Rc5 lost an exchange, and 28.Ne8 and 30.Nxe6 were poor. Black's play was solid, yet 21...Qe5 and 26...Rf6 were fairly ordinary, and the endgame mate was handed over. Both opponents look similar in strength. A's attack was slightly more convincing, so I lean to A.

Reported P(A stronger): 0.6. Consistent with choice.

Gemini 3.8 Flash: chose A — correct

Player A demonstrates sound opening knowledge in the Caro-Kann Exchange Variation and excellent strategic understanding (16...Ne4, 17...f5, 19...g5), culminating in a flawlessly executed smothered mate combination starting with 24...Ne2+. In contrast, Player B exhibits low-tier opening habits with early, risky queen sorties (3...Qa5+, 5...Qb5, 6...Qxb2) and poor tactical awareness, allowing the devastating 28. Ne8 and 29. Nc7 fork to drop an entire rook. Player B only won because White committed an elementary back-rank blunder on move 35. Player A is far more polished and established.

Reported P(A stronger): 0.88. Consistent with choice.

Gemini 3.1 Pro (Preview): chose A — correct

Target A played a principled Caro-Kann, castled queenside, and executed a beautiful smothered mate sequence starting with 22... Nd4! Target A demonstrated strong positional understanding and advanced tactical vision. In contrast, Target B played a dubious opening (3... Qa5+), went pawn-grabbing early (6... Qxb2), and blundered material (28... Re6? leading to 31. Rxc6). Target B only won because White blundered a basic back-rank mate at the end (35. Nxb5?? Rc1#). Target A's cohesive, high-level play heavily outclasses Target B's erratic and opportunistic style.

Reported P(A stronger): 0.95. Consistent with choice.

Gemini 3.5 Flash: chose A — correct

Game A Black demonstrates excellent positional understanding and high tactical sharpness, playing a solid Caro-Kann and executing a flawless smothered mate sequence (24...Ne2+ 25.Kh1 Qg1+ 26.Rxg1 Nf2#). In contrast, Game B Black plays an erratic opening with early queen moves (3...Qa5+, 5...Qb5) and exhibits weak tactical vision, missing a simple winning queen fork with 17...Nc2+ after 17.c3, instead playing 17...b6. Game B only ends because White blunders into a simple back-rank mate with 35.Nxb5?? Rc1#.

Reported P(A stronger): 0.95. Consistent with choice.

Gemini 3.7 Flash: chose A — correct

Game A Black displayed high-level Caro-Kann opening understanding, purposeful strategic play on both flanks, and flawless tactical calculation, concluding with a textbook smothered mate starting with 24...Ne2+ and 25...Qg1+. In contrast, Game B featured sloppy opening play with risky queen excursions (5...Qb5, 6...Qxb2). Game B Black later missed a tactical sequence, falling into a knight fork and dropping an exchange on move 29 (28.Ne8 Re6 29.Nc7), only winning due to White blundering back-rank mate with 35.Nxb5??

Reported P(A stronger): 0.85. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

Game A Black plays a coherent Caro-Kann, coordinates queenside castling with ...O-O-O and ...Kb8, then executes a clean tactical finish: 21...gxf4, 22...Nd4, 23...Qxc5, and the forcing 24...Ne2+ 25...Qg1+ 26...Nf2#. Game B Black's opening is awkward: the queen raid 3...Qa5+, 5...Qb5, 6...Qxb2 wastes tempos, and later ...b6/...bxc5 leaves a clumsy structure; Black wins mainly because White self-destructs with 28.Ne8? and 30.Nxe6?. A's active piece play and precise smothered-mate finish show stronger established technique.

Reported P(A stronger): 0.72. Consistent with choice.

Claude Fable 5.1: chose A — correct

Game A Black played a coherent Caro-Kann exchange setup (6...Bg4, 10...Bxe2 trading the bishop, 16...Ne4, 17...f5) and finished with a clean 25...Qg1+ 26.Rxg1 Nf2# smothered mate; White's play (Qe1, Bg5-h4-e1) was weak but not absurd. Game B Black grabbed material with 6...Qxb2 and won, but allowed a repeatedly harassed queen and lost an exchange to 28.Ne8/29.Nc7 (28...Re6?), winning only because White blundered terribly (Rc5?, Nxb5?? allowing Rc1#). White in Game B was very weak (Qa1, Rb3-a3-c3-c5), suggesting a lower-rated pairing. Game A's opposition and Black's technique look somewhat higher-level.

Reported P(A stronger): 0.6. Consistent with choice.

Gemini 3 Flash (Preview): chose A — correct

Target A (Black) exhibits a sophisticated understanding of the Caro-Kann, coordinating a complex kingside attack while managing queenside pressure (16...Ne4, 19...g5). The execution of the classic smothered mate (24...Ne2+ through 26...Nf2#) demonstrates high tactical awareness. Target B (Black) plays a suspicious opening, making multiple early queen moves (3...Qa5+, 5...Qb5, 6...Qxb2) to grab a pawn, a strategy often punished at higher levels. While Target B won via a back-rank mate, it was facilitated by a significant oversight by White (34. Nc3??). Target A’s strategic cohesion and tactical finish suggest a higher established rating.

Reported P(A stronger): 0.85. Consistent with choice.

Gemini 3.6 Flash: chose A — correct

In Game A, Black played a very high-quality Caro-Kann defense, castling queenside and accurately handling White's pawn storm before executing a brilliant smothered mate combination (24...Ne2+ 25. Kh1 Qg1+ 26. Rxg1 Nf2#). In contrast, Game B's Black displayed poor tactical awareness, blundering material with 28...Re6 29. Nc7! and losing a piece on c6 (31. Rxc6). Game B Black was heavily losing and only won because White committed a fatal back-rank blunder with 35. Nxb5??, allowing 35...Rc1#. Game A Black demonstrated vastly superior tactical calculation and positional judgment.

Reported P(A stronger): 0.88. Consistent with choice.

GPT-5.6 Sol: chose A — correct

Game A Black showed a coherent plan: 13...O-O-O, 15...Rc8, 16...Ne4 and 17...f5 coordinated the pieces before the kingside pawn attack. The finish 24...Ne2+ 25.Kh1 Qg1+ 26.Rxg1 Nf2# was a precise forced mating combination. Game B Black found useful tactics with 17...b6 and ultimately 35...Rc1#, but the early queen expedition 5...Qb5 6...Qxb2 cost many tempi, and later 28...Re6 29.Nc7 Ra7 30.Nxe6 allowed White to recover material and gain the initiative. B’s win depended more heavily on White’s final mating blunder.

Reported P(A stronger): 0.82. Consistent with choice.

GPT-6 Sol: chose A — correct

Game A’s Black built a coherent attack with 16...Ne4, 17...f5 and 22...Nd4, then finished with the calculated 24...Ne2+ 25...Qg1+ 26...Nf2#. Game B’s Black won after White ignored the threat to the rook on c5, but the early queen excursion with 5...Qb5 and 6...Qxb2 was risky, and 28...Re6 allowed 29.Nc7 to fork the rooks. A’s play is more convincing evidence of established strength.

Reported P(A stronger): 0.88. Consistent with choice.

GPT-6.1 Sol: chose A — correct

Game A Black showed coherent development and coordinated attacking play: 16...Ne4, 19...g5 and 22...Nd4 established active central knights, culminating in the accurately calculated 24...Ne2+ 25...Qg1+ 26...Nf2# combination. Game B Black also recognized a mating opportunity with 34...Rc7 and 35...Rc1#, but benefited substantially from White's rook wandering and 18.Qb2, which allowed ...bxc5. More tellingly, B's 28...Re6 allowed 29.Nc7 to fork both rooks, followed by 30.Nxe6 and 31.Rxc6, surrendering much of the earlier material advantage. A's more consistent coordination and tactical accuracy suggest the higher established strength.

Reported P(A stronger): 0.86. Consistent with choice.

GPT-5.6 Luna: chose B — wrong

Game B’s black handled a more complicated tactical battle and still converted after White’s active sequence 28.Ne8, 30.Nxe6, and 31.Rxc6. The queen excursion with 6...Qxb2–8...Qa5 was risky, but Black consistently found practical resources and finished with 35...Rc1#. Game A’s black played a coherent attack, especially 23...Qxc5, 24...Ne2+, and 25...Qg1+, but White’s 22.b5 and the resulting mating net were comparatively straightforward errors. Thus B appears slightly more likely to have had the higher established strength.

Reported P(A stronger): 0.43. Consistent with choice.

GPT-6 Astra: chose A — correct

Game A Black showed coherent development and attacking coordination: 16...Ne4, 17...f5 and 19...g5 established central control and kingside pressure. The finish 22...Nd4 followed by 24...Ne2+ and 25...Qg1+ demonstrated precise tactical vision. Game B Black exploited White’s loose rook with 17...b6 and found 35...Rc1#, but played less consistently. Repeated early queen moves delayed development, and 28...Re6 allowed 29.Nc7, forking the rooks and forcing material concessions. Both opponents supplied opportunities, but A’s sustained coordination is stronger evidence of established playing strength than B’s recovery against erratic opposition.

Reported P(A stronger): 0.9. Consistent with choice.

GLM-5.3 Flash: chose B — wrong

Both Black targets beat weaker-seeming White opposition, but B's play looks more sustained. In Game B, Black punished White's queen retreats (6...Qxb2, 8...Qa5) and found the in-between tactic 18...bxc5! netting a piece, then converted cleanly: 20...O-O, trades with 30...dxe6, and finished with 34...Bb5 and 35...Rc1#, showing endgame technique. In Game A, Black played well (22...Nd4 opening lines, 24...Ne2+ leading to the smothered mate 26...Nf2#), but the mate relied on White's cooperative 25.Kh1 Qg1+ allowing Rxg1, and earlier Black moves (17...f5, 19...gxf4) were more speculative. B's longer, controlled domination suggests a slightly higher rating.

Reported P(A stronger): 0.35. Consistent with choice.

Pair 25: 2 of 16 models wrong

A: Black, rated 2474 · B: White, rated 1862. Higher pre-game rating: A.

Game A: Black, rated 2474 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 Nf6 2. c4 e6 3. Nc3 d5 4. cxd5 exd5 5. Bg5 c6 6. e3 Bf5 7. Qf3 Bg6 8.
Bxf6 Qxf6 9. Qxf6 gxf6 10. h4 h5 11. Nh3 Bd6 12. Kd2 Nd7 13. Bd3 b5 14. Rac1
Ke7 15. Ne2 Rac8 16. Nhf4 Rhg8 17. Ng3 Bxf4 18. exf4 Bxd3 19. Kxd3 Kd6 20. Rhe1
Rce8 21. Rxe8 Rxe8 22. Nxh5 c5 23. Ng3 c4+ 24. Kd2 Nf8 25. h5 Ne6 26. Nf5+ Kd7
27. g3 Rh8 28. h6 a5 29. b3 Nd8 30. bxc4 dxc4 31. d5 Nb7 32. Re1 Nd6 33. Re7+
Kd8 34. Ra7 Nxf5 35. Ra8+ Kd7 36. Rxh8 b4 37. h7 1-0
Game B: White, rated 1862 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. d4 d5 2. c4 c6 3. e3 Bf5 4. Nf3 Nf6 5. Nbd2 h6 6. Be2 Nbd7 7. O-O e6 8. cxd5
exd5 9. h3 Bd6 10. b3 O-O 11. a4 Ne4 12. Nxe4 Bxe4 13. Ba3 Bxa3 14. Rxa3 Nf6
15. Ne5 Nd7 16. Nxd7 Qxd7 17. b4 a6 18. Rb3 Qe6 19. b5 axb5 20. axb5 Qg6 21.
Bf3 Bc2 22. Bh5 Bxd1 23. Bxg6 Bxb3 24. Bd3 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Game A Black played a coherent QGD Exchange with ...Bf5-g6 and ...h5 into an endgame. The errors came late: 22...c5 allowed Nxh5 to win a pawn, and 33...Kd8 and 34...Nxf5 let the h-pawn promote. Overall this was reasonable play against a competent opponent. Game B White made gross blunders. 21.Bf3?? allowed 21...Bc2, trapping the queen, and 22.Bh5 made it worse, losing the queen for little. Earlier play such as 18.Rb3 and 19.b5 was also aimless. Hanging the queen to a simple bishop tactic suggests a weaker player than A's endgame inaccuracies do.

Reported P(A stronger): 0.75. Consistent with choice.

Claude Sonnet 5.5: chose B — wrong

Game A, Black: allowed a queen trade and doubled f-pawns, then played passively. 13...b5 and 14...Ke7 were okay, but 22...c5 and 23...c4+ let White build a kingside attack. Black blundered with 36...b4, overlooking the h-pawn advance; 34...Nxf5 gave up the knight and the rook trade was poor, so the h-pawn was decisive. Black's play looks around club level. Game B, White: played a standard Slav/Queen's Gambit structure sensibly through move 16, but 21.Bf3 allowed 21...Bc2, and 22.Bh5 Bxd1 23.Bxg6 Bxb3 lost the exchange and more, a major tactical oversight (the queen was en prise to a deflection). Both are weak; White in B blundered a queen-trade sequence, whereas Black in A held a roughly equal game for a long time. Slight lean to B being stronger because B's opening play was more structured, though the evidence is thin.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3.8 Flash: chose A — correct

Player A (Black) demonstrated a solid grasp of structural and positional nuances in an offbeat Queen's Gambit Declined endgame, handling doubled f-pawns and coordinating pieces reasonably well until collapsing under heavy pressure late in the game (33...Kd8? and 34...Nxf5?). In contrast, Player B (White) adopted a timid, passive setup with little initiative and committed a severe one-move tactical blunder with 21. Bf3?, completely missing the direct fork 21...Bc2, which immediately dropped a full exchange and prompted resignation on move 24.

Reported P(A stronger): 0.72. Consistent with choice.

Gemini 3.1 Pro (Preview): chose A — correct

Target A (Black) played a solid opening and navigated a complex endgame reasonably well until falling for a multi-step tactical trap (33... Kd8 34. Ra7 Nxf5 35. Ra8+) which lost the rook. Target B (White) played a rather passive opening (3. e3, 5. Nbd2) and then completely missed a basic one-move double attack with 21. Bf3? Bc2, forking the Queen and Rook. Target A demonstrated deeper strategic understanding and positional play compared to Target B's passive play and elementary tactical oversight.

Reported P(A stronger): 0.75. Consistent with choice.

Gemini 3.5 Flash: chose A — correct

Player A (Black) demonstrates a much higher baseline of chess understanding than Player B (White). In Game A, Black navigates a highly theoretical Queen's Gambit Declined Exchange line, maintaining a reasonable position through the middlegame (e.g., 10... h5, 12... Nd7) before succumbing to White's strong endgame play. In contrast, Player B plays very passively in the opening (3. e3, 5. Nbd2) and exhibits poor tactical vision. After Black's 20... Qg6, which threatens mate on g2, White plays 21. Bf3? completely overlooking the basic fork 21... Bc2, which wins the exchange and the game on the spot. Player A is clearly the stronger, more established player.

Reported P(A stronger): 0.75. Consistent with choice.

Gemini 3.7 Flash: chose A — correct

Player A (Black) played a complex endgame against a strong opponent, demonstrating reasonable strategic ideas (queenside pawn expansion, king activity) before succumbing to tactical tricks on move 34 (34...Nxf5? allowing 35. Ra8+). Player B (White) played passively with unambitious developing moves and committed a basic tactical oversight with 21. Bf3??, walking straight into the 21...Bc2 fork winning the rook on b3 and leading to an immediate resignation.

Reported P(A stronger): 0.72. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

Game A Black defended the Exchange Caro-Kann solidly; the loss resulted from a concrete tactical slip (30...dxc4 allowing 31.d5 and the rook raid 33.Re7+/37.h7), not systemic errors. Game B White played passively (b3, a4), lost time after 12.Nxe4, then blundered a whole queen with 22.Bh5??, ignoring that 22...Bxd1 captures the queen. Game B White's queen oversight indicates weaker tactical alertness than Game A Black's eventual endgame lapse, so A was likely the higher-rated player.

Reported P(A stronger): 0.68. Consistent with choice.

Claude Fable 5.1: chose A — correct

Game B White blundered a full exchange tactically with 21. Bf3?? allowing 21...Bc2 forking queen and rook, then resigned early after 23...Bxb3 — a clear tactical oversight by a lower-level player, though earlier play (8.cxd5, 10.b3, 17.b4 minority plan) was coherent. Game A Black lost a long endgame after a sound opening (6...Bf5 7...Bg6, trading into a doubled-pawn structure), with errors like 22...c5 and 29...Nd8 letting the h-pawn and d5 break decide; no single piece-losing blunder. Both appear modest, but Game A Black's play was more consistent and lacked a one-move tactical collapse.

Reported P(A stronger): 0.6. Consistent with choice.

Gemini 3 Flash (Preview): chose A — correct

Game A Black demonstrates a reasonable understanding of the QGD Exchange structure, employing active king centralization (19... Kd6) and seeking counterplay in a strategically difficult endgame. Although Black ultimately loses to a skewer (35. Ra8+), the game reached a high degree of technical complexity. In contrast, Game B White plays a very passive opening and commits a basic tactical oversight by missing the 21... Bc2 fork, which immediately loses the exchange in a stable, quiet position. The nature of White's blunder in Game B, occurring without significant pressure, suggests a lower established rating compared to the more sophisticated defensive effort shown by Black in Game A.

Reported P(A stronger): 0.6. Consistent with choice.

Gemini 3.6 Flash: chose A — correct

Player A (Black) demonstrates good opening familiarity in a Carlsbad Queen's Gambit Declined (6...Bf5) and plays logically against White's flank pawn storm, only collapsing in a difficult, already-losing endgame (34...Nxf5?). In contrast, Player B (White) plays passively in a symmetrical Slav structure and commits a basic tactical oversight in a balanced middlegame with 21. Bf3??, completely missing Black's 21...Bc2 which forks the queen and rook and immediately loses significant material. Player A shows significantly better baseline positional understanding and tactical awareness throughout the main phase of the game.

Reported P(A stronger): 0.65. Consistent with choice.

GPT-5.6 Sol: chose A — correct

Game A Black showed more sustained positional competence: 13...b5, 16...Rhg8, king centralization through ...Ke7-d6, and the timely 22...c5 created active counterplay in the queenless middlegame. Black ultimately mishandled White’s h-pawn and blundered with 34...Nxf5?, allowing 35.Ra8+ and 36.Rxh8. Game B White also played a coherent Slav structure, but 21.Bf3? permitted 21...Bc2, a straightforward double attack on Qd1 and Rb3; 22.Bh5 Bxd1 23.Bxg6 Bxb3 then left White decisively down material. A’s longer stretch of purposeful play slightly outweighs the comparable final tactical oversight.

Reported P(A stronger): 0.62. Consistent with choice.

GPT-6 Sol: chose B — wrong

Game B’s White developed coherently and pursued a clear queenside plan with 17.b4 and 19.b5, though 21.Bf3 and 22.Bh5 missed that ...Bc2 would let Black take the queen and then the rook on b3. Game A’s Black also played a reasonable opening, but allowed 22.Nxh5, made little progress against White’s advancing h-pawn, and overlooked 35.Ra8+ followed by 36.Rxh8. B’s play before the tactical mistake looks more convincing.

Reported P(A stronger): 0.27. Consistent with choice.

GPT-6.1 Sol: chose A — correct

A Black showed more sustained positional and endgame competence: 22...c5 and 23...c4+ created counterplay, while ...Nf8–e6 and ...Nd8–b7–d6 improved the knight. However, 34...Nxf5 allowed 35.Ra8+, losing the rook and permitting promotion. B White developed sensibly and pursued a coherent minority attack with b4–b5, but mishandled the tactical threat after 21...Bc2: 22.Bh5 Bxd1 23.Bxg6 Bxb3 traded queens while losing a rook. Both made serious errors, but A’s longer stretch of purposeful play provides somewhat stronger evidence of established strength.

Reported P(A stronger): 0.65. Consistent with choice.

GPT-5.6 Luna: chose A — correct

Game A’s black player showed a coherent strategic plan: developing actively with ...Bf5, ...b5, ...Rac8, then using ...c5 and ...c4+ to create counterplay. Although Black later missed the rook-saving resource after 34.Ra7, the game remained competitive and featured meaningful endgame play. Game B’s white player made a much more decisive tactical oversight: after 21.Bf3 Bc2, 22.Bh5?? allowed ...Bxd1, losing the queen, and 23.Bxg6 was followed by ...Bxb3, losing another major piece. This indicates substantially weaker tactical awareness than Game A’s black player.

Reported P(A stronger): 0.82. Consistent with choice.

GPT-6 Astra: chose A — correct

Game A Black showed more sustained endgame coordination: 14...Ke7 and 19...Kd6 activated the king, while 22...c5 and 23...c4+ generated counterplay. Black nevertheless mishandled White’s passed h-pawn, and 34...Nxf5 allowed 35.Ra8+ and the loss of the rook. Game B White developed reasonably and pursued a coherent queenside expansion, but 21.Bf3 allowed 21...Bc2, attacking the queen and skewering the rook on b3. The resource 22.Bh5 avoided losing a queen outright but still conceded an exchange after 23...Bxb3. A’s longer stretch of purposeful play gives Black a modest edge, despite the eventual collapse.

Reported P(A stronger): 0.64. Consistent with choice.

GLM-5.3 Flash: chose A — correct

Game A's Black handled the semi-Slav structure reasonably, exploiting White's 9.Qxf6?? blunder, showed decent endgame technique (22...c5, 23...c4+ creating counterplay), and only erred late (34...Nxf5 allowing 35.Ra8+ and 37.h7). Game B's White played weakly: 21.Bf3 allowed 21...Bc2, then 22.Bh5 Bxd1 23.Bxg6 Bxb3 simply dropped a full bishop, prompting resignation despite no catastrophic earlier mistake. Black's clean tactical conversion (Bc2/Bxd1/Bxb3) and White's piece-loss blunder suggest B's target is weaker than A's, who competed for ~37 moves against competent opposition.

Reported P(A stronger): 0.72. Consistent with choice.

Pair 13: 1 of 16 models wrong

A: White, rated 717 · B: White, rated 2508. Higher pre-game rating: B.

Game A: White, rated 717 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 Nf6 2. Qf3 e5 3. g4 Nc6 4. g5 Nd4 5. Qd3 Ng4 6. h3 Nxf2 7. Kxf2 h6 8.
gxh6 Qf6+ 9. Ke3 Bc5 10. hxg7 Qxg7 11. c3 Nb5+ 12. Ke2 Qf6 13. Qxb5 Qf2+ 14.
Kd1 c6 15. Qc4 d5 16. exd5 cxd5 17. Qxc5 Bf5 18. Qxd5 Ke7 19. Bb5 Kf6 20. Ne2
Qf4 21. Qxb7 Qg5 22. Rg1 Qxg1+ 23. Nxg1 1-0
Game B: White, rated 2508 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 c6 2. d4 d5 3. exd5 cxd5 4. Bd3 Nc6 5. c3 Qc7 6. Ne2 Bg4 7. O-O e6 8. Qe1
Nf6 9. Bg5 Bh5 10. f4 Bxe2 11. Qxe2 Be7 12. Nd2 h6 13. Bh4 O-O-O 14. b4 Kb8 15.
Nb3 Rc8 16. a4 Ne4 17. Be1 f5 18. Nc5 Bxc5 19. dxc5 g5 20. a5 a6 21. c4 gxf4
22. b5 Nd4 23. Qb2 Qxc5 24. Bb4 Ne2+ 25. Kh1 Qg1+ 26. Rxg1 Nf2# 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A White opens with the unorthodox 2.Qf3 and 3.g4. It wins mainly because Black collapses: 6...Nxf2?? is unsound, and 22...Qxg1+?? gives away the queen. White's own play (13.Qxb5, 17.Qxc5, 21.Qxb7) just collects loose material, which says little about skill against a weak opponent. Game B White plays a coherent Exchange Caro-Kann setup (4.Bd3, 5.c3, Ne2, O-O) and a sensible queenside pawn storm (14.b4, 16.a4, 20.a5) against a competent opponent. It does err: 23.Qb2 lets Black win c5, and 24.Bb4? allows 24...Ne2+ 25.Kh1 Qg1+ 26.Rxg1 Nf2#, a smothered mate. Even so, B's overall understanding and the stronger opposition point to B being the stronger target.

Reported P(A stronger): 0.35. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A: White opened 2.Qf3 and 3.g4, which are dubious. Black's 6...Nxf2 sacrifice and 8...Qf6+ gave Black a strong attack, yet White survived. Black then blundered with 13...Qf2+ and 16...cxd5, and White won with 17.Qxc5 and 21.Qxb7. This points to a weak game from both sides, and White's opening moves were poor. Game B: White played a reasonable Caro-Kann Exchange with solid development (Bd3, c3, Ne2, Qe1), but 10.f4 and 18.Nc5 gave Black chances. The decisive error was 21.c4 and 22.b5, which ignored Black's threats, and White lost to a knight-and-queen-sacrifice mate. White in B played more principled chess for most of the game, while White in A relied on opponent errors. B is slightly more likely stronger, but with low confidence.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game A features very low-level, chaotic play: White plays dubious early queen moves like 2. Qf3 and 3. g4, allowing their king to wander to e3 on move 9, while both sides repeatedly hang material. In contrast, Game B shows solid positional opening knowledge in the Caro-Kann Exchange Variation (3. exd5, 4. Bd3, 5. c3) and coherent opposite-side castling plans. Although White in Game B blundered into a classic smothered mate tactic starting with 24. Bb4? allowing 24...Ne2+, their strategic understanding and move quality across the middlegame reflect a vastly higher established rating than White in Game A.

Reported P(A stronger): 0.08. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

White A plays a nonsensical opening (2. Qf3, 3. g4) and exposes their king (9. Ke3), surviving only because Black inexplicably hangs pieces (13. Qxb5, 17. Qxc5). This indicates a beginner level. In contrast, White B plays a theoretical Caro-Kann Exchange, develops logically, and launches a credible queenside attack (14. b4, 20. a5). Although White B falls for a classic smothered mate trap (24...Ne2+ 25.Kh1 Qg1+ 26.Rxg1 Nf2#), their overall positional understanding and structural play are vastly superior to White A.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

Game A White plays highly irregular, low-level chess, with an early queen excursion (2. Qf3), a premature pawn storm (3. g4), and walking their king to e3 by move 9. Both players commit severe errors. Conversely, Game B White demonstrates solid opening and positional understanding in the Caro-Kann Exchange, developing logically (4. Bd3, 6. Ne2) and organizing a queenside attack (14. b4, 20. a5). Although Game B White ultimately misses a tactical smothered mate sequence starting with 24...Ne2+, their level of play prior to this is vastly superior, indicating a much higher established rating.

Reported P(A stronger): 0.03. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Game A features chaotic, low-level opening play with 2. Qf3 and 3. g4 by White, followed by reckless piece sacrifices and mutual tactical blunders throughout a wild king hunt. In contrast, Game B displays coherent positional understanding in a Caro-Kann Exchange Variation, with White executing standard strategic concepts like queenside expansion (14. b4, 15. Nb3, 16. a4). Although White in Game B blundered into a classic smothered mate sequence starting with 24. Bb4?? Ne2+, their overall opening preparation, pawn structure handling, and strategic coordination indicate a much higher established rating than Game A's White.

Reported P(A stronger): 0.08. Consistent with choice.

Qwen3.8 Max Prime: chose A — wrong

Game A White survived a chaotic opening (6...Nxf2 7.Kxf2) and converted material with accurate tactics: 13.Qxb5, 17.Qxc5, 22.Rg1 repulsing 21...Qg5. The win against speculative Black shows practical strength. Game B White chose a passive Exchange Caro-Kann (4.Bd3, 5.c3, 8.Qe1), drifted after 13...O-O-O, and lost to a standard smothered pattern (24...Ne2+ 25.Kh1 Qg1+ 26.Rxg1 Nf2#) without counterplay. A's active resourcefulness exceeds B's generic play, though A's 2.Qf3/3.g4 opening was also dubious.

Reported P(A stronger): 0.66. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A White played 2.Qf3 and 3.g4 against the Alekhine, then walked the king to e3 after 6...Nxf2; the win came only because Black (Game A) blundered pieces with 11...Nb5+?, 13...Qf2+ and 21...Qg5?? dropping the queen. Both A players look weak. Game B White played a coherent Caro-Kann Exchange setup (4.Bd3, 5.c3, 6.Ne2, 7.O-O), a sensible queenside pawn storm (14.b4, 16.a4, 20.a5, 22.b5), and lost to a sharp smothered-mate combination (24...Ne2+, 25...Qg1+) by a competent opponent. Game B White's structured play and stronger opposition suggest a higher rating than A's chaotic opening.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Game A White shows a lack of fundamental principles, developing the queen prematurely (2. Qf3), pushing flank pawns (3. g4), and walking the king to the center early (9. Ke3) in a chaotic sequence. In contrast, Game B White plays a standard Exchange Caro-Kann with logical development (4. Bd3, 6. Ne2, 12. Nd2) and positional awareness, establishing a strong outpost with 18. Nc5. Although Game B White loses to a classic smothered mate tactic (24. Bb4?? Ne2+), their overall opening knowledge and structural play suggest a significantly higher established rating than Game A White.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Game A White displays a clear lack of basic opening and tactical principles, playing dubious moves like 2. Qf3, 3. g4, and moving the king to e3 on move 9. Game A White wins only because Black blunders the queen multiple times (17...Bf5 missing 17...Qxc5, and 22...Qxg1+). In contrast, Game B White demonstrates solid positional understanding in a standard Caro-Kann Exchange Variation, executing a coherent queenside pawn storm with 14. b4, 15. Nb3, and 16. a4. Although Game B White falls for a classic smothered mate tactic starting with 24...Ne2+, Game B White shows vastly superior established playing strength compared to Game A White.

Reported P(A stronger): 0.05. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B White showed much more coherent opening and strategic play: normal development, castling, and a purposeful queenside expansion with 14.b4, 16.a4, 18.Nc5, and 22.b5. The decisive error was tactical—24.Bb4? allowed 24...Ne2+ and the forced mating pattern. Game A White played extremely unsoundly with 2.Qf3, 3.g4, 6.h3, and an exposed king walk to e3/d1, succeeding mainly because Black repeatedly hung material, notably 13...Qf2+ allowing 17.Qxc5 and 22...Qxg1+ losing the queen to 23.Nxg1. B is therefore much likelier to have had the stronger established rapid level.

Reported P(A stronger): 0.08. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s White showed more coherent play: 7.O-O completed development, and 14.b4 and 20.a5 pursued a queenside plan. White then missed the mating threat with 24.Bb4, but that single oversight is less telling than Game A’s White playing 2.Qf3 and 3.g4, then exposing the king with 9.Ke3. Game A’s win also relied heavily on Black leaving pieces available to 13.Qxb5 and 17.Qxc5.

Reported P(A stronger): 0.2. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game B White shows substantially more coherent development and planning: 4.Bd3, 6.Ne2 and 7.O-O, followed by the coordinated queenside expansion 14.b4, 15.Nb3 and 18.Nc5. White ultimately overlooks the mating combination beginning 24...Ne2+, but the preceding play suggests stronger established understanding. Game A White’s 2.Qf3, 3.g4 and 4.g5 neglect development and king safety; 6.h3 allows ...Nxf2, forcing an exposed king walk. A’s win relies heavily on Black’s poor attack and especially 22...Qxg1+, surrendering the queen for a rook, rather than demonstrating comparable overall strength.

Reported P(A stronger): 0.94. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B’s White shows a much more coherent opening and positional plan: 1.e4 c6, 4.Bd3, 7.O-O, and the queenside expansion with 14.b4 and 20.a5. However, the game ends with a severe tactical oversight: 24.Bb4? Ne2+ 25.Kh1 Qg1+ 26.Rxg1 Nf2#. Game A’s White plays very dubious early moves, especially 2.Qf3 and 3.g4, and exposes the king with 7.Kxf2 and 9.Ke3, but consistently exploits Black’s hanging pieces, notably 13.Qxb5 and 17.Qxc5. Overall B appears stronger despite the decisive blunder.

Reported P(A stronger): 0.35. Consistent with choice.

GPT-6 Astra: chose B — correct

Game B White showed substantially stronger opening principles and planning: 4.Bd3, 5.c3, 6.Ne2 and 7.O-O developed coherently, followed by queenside expansion against Black’s long-castled king and the thematic 18.Nc5. White ultimately missed the mating combination 24...Ne2+ 25.Kh1 Qg1+ 26.Rxg1 Nf2#, but Black’s coordinated attack was considerably more demanding. Game A White’s 2.Qf3, 3.g4 and 4.g5 neglected development and exposed the king, which reached e3 by move nine. A’s eventual win involved collecting loose pieces and benefited from Black’s much weaker play, including 22...Qxg1+ allowing 23.Nxg1.

Reported P(A stronger): 0.03. Consistent with choice.

GLM-5.3 Flash: chose B — correct

Game B's White showed coherent positional chess: normal Caro-Kann structure, safe O-O, planned a4/b5 expansion, and sensible trades (11.Qxe2, 18.Nc5). The loss came mainly from one late blunder 24.Bb4?? overlooking the smothered mate after 24...Ne2+ 25.Kh1 Qg1+ 26.Rxg1 Nf2#. Game A's White played chaotic opening moves (2.Qf3, 4.g4) and merely profited from Black's absurd blunders (6...Nxf2?, 7...h6? allowing gxh6/Qf6+), with the win largely gifted. B's target demonstrated stronger fundamental play, so likely higher rating despite losing.

Reported P(A stronger): 0.6. Contradicts choice; excluded from confidence diagnostics.

Pair 11: 1 of 16 models wrong

A: Black, rated 743 · B: Black, rated 2501. Higher pre-game rating: B.

Game A: Black, rated 743 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. d4 d5 2. Qd3 e5 3. dxe5 Qe7 4. f4 f6 5. Qxd5 fxe5 6. fxe5 Bf5 7. Qxb7 Qb4+
8. Qxb4 Bxb4+ 9. Bd2 Bxd2+ 10. Nxd2 Bxc2 11. Rc1 Ba4 12. Rxc7 Na6 13. Rc4 Bb5
14. Rd4 Rd8 15. Rxd8+ Kxd8 16. Nb3 Ba4 17. Nc1 Nb4 18. b3 Bb5 19. a3 Nc2+ 20.
Kd1 Ne3+ 21. Kd2 Nxf1+ 22. Ke1 Ne3 23. Kd2 Nxg2 24. a4 Nf4 25. e3 Bxa4 26. bxa4
Nd5 27. Nb3 Nh6 28. e4 Ng4 29. exd5 Nf2 30. Ke2 Nxh1 31. Nf3 Re8 32. Ng5 h6 33.
Nf7+ Ke7 34. Nd6 Rd8 35. Nb5 Rxd5 36. Nxa7 Rxe5+ 37. Kd3 Rh5 38. Nc6+ Kd7 39.
Ne5+ Rxe5 40. Kd4 Re2 41. h4 Rh2 42. Nc5+ Kc6 43. Ne4 Rxh4 44. a5 Kb7 45. Kd5
Ka6 46. Nc5+ Kxa5 47. Nd7 Rh2 48. Ne5 Rd2+ 49. Kc4 Ka4 50. Nf7 Rc2+ 51. Kd3 Kb3
52. Ke4 Rc3 53. Kf5 Re3 54. Kg6 h5 55. Kxg7 Rg3+ 56. Kh6 Rh3 57. Ne5 Rh2 58.
Nf3 Rh3 59. Ng5 Rh2 60. Nf3 Rf2 61. Ng5 Ng3 62. Nh3 Rh2 63. Nf4 h4 64. Kg5 h3
65. Kg4 Ne4 66. Nxh3 Nf2+ 67. Kg3 Rxh3+ 68. Kxf2 Rd3 69. Ke2 Kc2 0-1
Game B: Black, rated 2501 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. c4 e6 2. Nf3 d5 3. g3 Nf6 4. Bg2 Nbd7 5. O-O Be7 6. b3 O-O 7. Bb2 c5 8. cxd5
exd5 9. d4 b6 10. Nc3 Bb7 11. Rc1 Rc8 12. dxc5 Nxc5 13. Nd4 Qd7 14. Qd2 Rfe8
15. Qf4 Ne6 16. Nxe6 fxe6 17. Rfd1 Bd6 18. Qh4 Rf8 19. e3 Qf7 20. Nb5 Be7 21.
Qf4 Qg6 22. Rxc8 Bxc8 23. Qc7 Bc5 24. Nxa7 Qc2 25. Rf1 Ba6 26. Be5 Bxf1 27.
Bxf1 Ne8 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A Black played a dubious opening that dropped pawns: 3...Qe7 and 4...f6 allowed 5.Qxd5 and 7.Qxb7. Black then let a knight go with 26...Nd5 and 28...Ng4, allowing 29.exd5. The rook-versus-knight ending was also handled slowly, and White blundered constantly, so the result says little. Game B Black played a coherent Queen's Gambit-style setup with ...Nbd7, ...c5 and ...b6/Bb7. Black kept the structure sound and found active ideas: 24...Qc2 hit the rook and 25...Ba6 won the exchange (26...Bxf1). This shows tactical awareness against a competent opponent. B's play looks considerably more controlled.

Reported P(A stronger): 0.25. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A Black met 2.Qd3 with 2...e5 and 4...f6, which left Black down a pawn after 7.Qxb7. Black then found 7...Qb4+ and 8...Bxb4+, followed by 10...Bxc2, which won back material and gave Black activity. Black won a long endgame. White's play was weak, and Black's conversion was uneven, with a long rook-and-knight ending that took until move 69. Game B Black played a solid, principled Queen's Gambit-style setup (Nbd7, Be7, ...c5, ...b6, ...Bb7). Black played 12...Nxc5, 15...Ne6, and 17...Bd6 sensibly. Black then tactically punished White's 24.Nxa7 with 24...Qc2 and 26...Bxf1, winning quickly. Black in B looks more coherent and cleaner, though White in B also blundered. Game A's opponent also played poorly, and Black's play there was shakier. I lean B.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game B demonstrates high-level, mature positional chess and sharp tactics. Black in Game B navigated a classical Catalan/English structure cleanly, coordinated pieces purposefully, and executed a decisive tactical finish starting with 24...Qc2!, culminating in 27...Ne8!, leaving White completely lost. In stark contrast, Game A features chaotic, low-rated beginner/intermediate play: Black played the unsound 2...e5, 3...Qe7, and 4...f6, blundering early on (allowing 7. Qxb7), followed by both sides missing simple tactical themes (e.g., White dropping the h1-rook to ...Nxh1). Black in Game B plays with vastly superior chess understanding.

Reported P(A stronger): 0.04. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Game B Black demonstrates a solid understanding of positional chess, smoothly navigating a complex Reti/Tarrasch structure and executing a sharp tactical sequence (24... Qc2!). In contrast, Game A Black plays dubiously in the opening (2... e5?!), blunders a pawn with 6... Bf5 allowing 7. Qxb7, and relies on basic knight forks against an equally weak opponent in a chaotic, low-level game. Target B is significantly stronger.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

Game A features a chaotic, lower-rated game. Black plays a highly questionable opening with 2...e5 and 3...Qe7, which blocks development, and the game proceeds with numerous tactical and positional inconsistencies from both sides. In contrast, Game B shows a high level of theoretical understanding and positional control from Black. Black navigates a standard English Defence setup flawlessly, developing logically, maintaining a solid pawn structure, and capitalizing on White's inaccuracies with sharp tactical precision starting with 24...Qc2, 25...Ba6, and 27...Ne8, forcing an immediate resignation. Player B is clearly the more established and stronger player.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Game B Black demonstrates significantly higher chess understanding, playing a classical, principled setup against White's English Opening (1...e6, 4...Nbd7, 7...c5). Black coordinated their pieces harmoniously in the middlegame and executed a decisive tactical sequence with 24...Qc2!, exploiting the fork on d1 and b2, followed cleanly by 25...Ba6 and 27...Ne8. In contrast, Game A Black plays chaotic and suboptimal chess from the opening (2...e5, 3...Qe7, 4...f6), dropping pawns and allowing Queen tactics (7.Qxb7), surviving only through White's blunders in an erratic low-level endgame.

Reported P(A stronger): 0.08. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

Game A's Black wins but against a very weak opponent (White's 2.Qd3, 4.f4, and 10.Rxc1 blundering Bxc2). Black plays solidly but nothing exceptional; the win stems from White's opening disaster. Game B's Black demonstrates stronger strategic play: 12...Nxc5 recapture, 15...Ne6 redeployment, 18...Rf8 prophylaxis, and the decisive 23...Qc2 and 26...Bxf1 tactics exploiting White's overextension (24.Nxa7). Game B's opponent played a coherent opening and middlegame, making Black's clean conversion more impressive evidence of higher established strength.

Reported P(A stronger): 0.18. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A Black won a very weak opponent's gambit (2.Qd3, 5.Qxd5 losing material) but then played erratically: 23...Nxg2 and 24...Nf4 followed by 25...Bxa4?? and 27...Nh6 lost pieces, and the knight on h1 was stranded for 30+ moves in a sloppy endgame. Game B Black handled a mainstream Catalan/Réti setup soundly (7...c5, 11...Rc8, 15...Ne6), then found 23...Bc5 and 24...Qc2 25...Ba6 winning the exchange with a crushing attack, forcing resignation after 27...Ne8. The opponent in B also played a coherent, higher-level game. B's play is markedly more consistent and tactically sharp.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Game B Black demonstrates significantly higher skill, playing a sophisticated English defense with sound development (10... Bb7, 14... Rfe8). Black's tactical execution is precise, notably 24... Qc2! which wins material by targeting the loose rook and bishop. In contrast, Game A Black plays an erratic opening (3... Qe7, 4... f6) and succeeds largely because White blunders a piece (21. Kd2? allowing Nxf1+). The quality of calculation and opening knowledge in Game B suggests a much higher established rating than the scrappy, blunder-prone play seen in Game A.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Player B demonstrates high positional understanding and opening knowledge in a standard English/Catalan setup (1...e6, 2...d5, 7...c5, 10...Bb7). Player B executed a decisive tactical sequence starting with 24...Qc2!, exploiting White's back rank and bishop on b2, leading to a quick victory. In contrast, Player A played very low-level opening moves (2...e5, 3...Qe7, 4...f6), giving away pawns early for no compensation against White's eccentric 2. Qd3. Although Player A eventually won after White blundered a rook and knight, Player B's clean, high-level tactical and strategic play indicates a much higher established rating.

Reported P(A stronger): 0.05. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B’s Black showed much stronger positional coherence: sound development through 14...Rfe8, purposeful pressure with 18...Rf8 and 19...Qf7, then accurate tactics after 22.Rxc8 Bxc8. The sequence 24...Qc2, 25...Ba6 and 26...Bxf1 decisively exploited White’s loose back rank. Game A’s Black found good ideas such as 7...Qb4+ and the knight incursion 19...Nc2+–23...Nxg2, but the opening was highly irregular and the large advantage required a long, meandering conversion, with the knight stranded on h1 and many inefficient rook maneuvers. B looks substantially more established.

Reported P(A stronger): 0.08. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s Black played a coherent opening against a normally developing opponent, then found active moves: 24...Qc2, 25...Ba6 and 26...Bxf1 won a rook, while 27...Ne8 attacked White’s queen. Game A’s Black exploited major errors by an opponent who brought the queen out on move 2, but chose a risky early ...f6 and later allowed 29.exd5 to take a knight. B’s play is stronger evidence of established rapid strength.

Reported P(A stronger): 0.04. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game B Black shows coherent development and purposeful coordination: 14...Rfe8, 17...Bd6 and 19...Qf7 improve the position, while 23...Bc5, 24...Qc2 and 25...Ba6 exploit White’s pawn grab to win the exchange with 26...Bxf1. Game A Black’s opening, especially 2...e5 and 3...Qe7, allows substantial pawn losses without convincing compensation. Although 19...Nc2+ and 20...Ne3+ exploit White’s mistakes effectively, 25...Bxa4 simply permits 26.bxa4, surrendering a bishop. A’s eventual victory relies much more heavily on the opponent’s repeated tactical errors; B’s play is substantially more consistent and sophisticated.

Reported P(A stronger): 0.96. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B’s black showed cleaner, more coherent opening development and strong tactical awareness: after 22 Rxc8, Bxc8 preserved activity, and 24 Nxa7 Qc2 followed by 25...Ba6 and 26...Bxf1 won White’s rook through a clear tactical sequence. Game A’s black was resourceful and ultimately converted actively, including 7...Qb4+ and 21...Nxf1+, but the opening with 2...e5, 4...f6, and allowing 5 Qxd5 and 7 Qxb7 was strategically risky. The long game also featured considerable maneuvering and depended on substantial white inaccuracies. B therefore more likely had the higher established rapid strength.

Reported P(A stronger): 0.68. Contradicts choice; excluded from confidence diagnostics.

GPT-6 Astra: chose B — correct

Game B Black shows coherent development and purposeful coordination: 14...Rfe8, 17...Bd6 and 18...Rf8 build activity, while 24...Qc2 and 25...Ba6 exploit White’s exposed rook to win the exchange with 26...Bxf1. This comes against substantially more organized opposition. Game A Black finds the useful 19...Nc2+–21...Nxf1 maneuver, but later simply loses a bishop with 25...Bxa4 26.bxa4. The eventual victory depends heavily on White’s further errors, notably 39.Ne5+ allowing ...Rxe5. B’s play is much stronger evidence of established playing strength.

Reported P(A stronger): 0.97. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose A — wrong

Game A's Black won material early with 5...fxe5/6...Bf5, exploited tactics (17...Nb4, 19...Nc2+ winning the knight via fork ideas), then converted a long R+N vs N endgame with solid technique: king activity (44...Kb7-a6), the...

Reported P(A stronger): 0.75. Consistent with choice.

Pair 23: 1 of 16 models wrong

A: White, rated 738 · B: White, rated 2426. Higher pre-game rating: B.

Game A: White, rated 738 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 d5 2. e3 Bd7 3. Qf3 c6 4. g4 b5 5. h3 Na6 6. h4 Qc7 7. g5 Rd8 8. h5 c5 9.
c3 b4 10. b3 c4 11. bxc4 dxc4 12. cxb4 Nxb4 13. Nh3 Nc2+ 14. Kd2 Nxa1 15. Bb2
Qa5+ 16. Nc3 Bxh3 17. Rxh3 Qxg5 18. Bxa1 Qg1 19. Rh1 Qg5 20. Bxc4 Rxd4+ 21. Kc2
Rxc4 22. Rd1 Qc5 23. Qa8+ Qc8 24. Qd5 Rxc3+ 25. Bxc3 f6 26. f4 e6 27. Qb5+ Kf7
28. e4 e5 29. f5 Nh6 30. Rd7+ Be7 31. Qd5+ Ke8 32. Rxa7 Nxf5 33. Ra8 Bd8 34.
Rxc8 Ke7 35. Qb7+ Kd6 36. Qc6+ Ke7 37. Bb4+ Kf7 38. Qd5+ Ke8 39. Rxd8# 1-0
Game B: White, rated 2426 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 c5 2. Nf3 e6 3. b3 a6 4. g3 Nc6 5. Bg2 d5 6. exd5 exd5 7. O-O Nf6 8. d4
cxd4 9. Bb2 Bc5 10. Re1+ Kf8 11. Nxd4 h5 12. Nxc6 bxc6 13. Bd4 Bd6 14. c4 h4
15. Nc3 hxg3 16. hxg3 Bh3 17. Bxh3 Rxh3 18. Kg2 Rh6 19. Rh1 Rxh1 20. Qxh1 Ng8
21. cxd5 c5 22. Be3 Be5 23. Rc1 Bd4 24. Qe1 Nf6 25. f3 Qd7 26. Qd2 Bxc3 27.
Bxc5+ Kg8 28. Qxc3 Nxd5 29. Qd4 Re8 30. Rd1 Re2+ 31. Kf1 Rh2 32. Qxd5 Qh3+ 33.
Ke1 Qxg3+ 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A is very low-level on both sides. White's 3.Qf3, 4.g4 and 6.h4 neglect development, and 13.Nh3 allowed 13...Nc2+ forking king and rook. White won mainly because Black blundered repeatedly, e.g. 16...Bxh3 and the later collapse of Black's king. Game B's White played a coherent double-fianchetto setup, including 8.d4, 9.Bb2 and 13.Bd4, and handled a sharp middlegame reasonably. White then collapsed with 32.Qxd5?? into ...Qh3+ and ...Qxg3+, a serious error. Even so, Game B's overall structure and the opponent's quality suggest a stronger level of play than Game A's chaotic game.

Reported P(A stronger): 0.3. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A: White played the opening poorly (2.e3 and 4.g4 gambits, 5.h3 and 6.h4 pawn-pushing, 3.Qf3 early queen). Black's 13...Nc2+ and 16...Bxh3 were strong, and White was probably losing around move 20. White won only because Black blundered, e.g. 23...Qc8 and 24...Rxc3+. The result suggests a low-rated, chaotic game. Game B: White played a solid Sicilian setup with 3.b3, 4.g3 and 7.O-O, and the moves were reasonable, though 10.Re1+ and 14.c4 led to a worse position. The loss came from tactics: 31.Kf1? and 32.Qxd5? allowed 32...Qh3+ and 33...Qxg3+. White's opening play was more principled in B, so B is likely stronger, but the evidence is weak and the margin small.

Reported P(A stronger): 0.35. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Player B exhibits solid positional understanding and coherent opening play (2. Nf3, 3. b3, 4. g3, 7. O-O), followed by purposeful piece activity and central play, only collapsing tactically late with 30. Rd1? allowing the decisive ...Re2+. In contrast, Player A plays like a beginner: violating opening principles with 3. Qf3 and an absurd flurry of flank pawn pushes (4. g4, 5. h3, 6. h4, 7. g5, 8. h5), while completely neglecting development and overlooking the elementary fork 13...Nc2+ dropping a full rook. Despite Black later blundering in Game A, Player A's overall grasp of chess fundamentals is vastly inferior to Player B's.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Game A White shows a lack of opening fundamentals with bizarre early moves like 3. Qf3, 4. g4, and pushing flank pawns to h5 and g5, later falling for a simple knight fork with 13... Nc2+. In contrast, Game B White demonstrates a solid understanding of opening principles, piece development, and structural play against the Sicilian Defense (e.g., fianchetto, challenging the center with d4). Although Game B White blunders at the end with 32. Qxd5 allowing a mating net, their overall positional understanding and tactical awareness throughout the game indicate a significantly higher rating than Game A White.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

In Game A, White's opening play is very weak, including premature queen moves (3. Qf3) and random pawn pushes (4. g4, 6. h4), quickly blundering a rook to a basic fork with 13. Nh3 Nc2+. In contrast, Game B's White plays a coherent and logical game, employing a recognized sideline (3. b3) against the Sicilian, developing pieces harmoniously (7. O-O, 10. Re1+), and executing sound plans. While Game B's White eventually blunders the king's safety with 30. Rd1?, their overall strategic and tactical competence is vastly superior to the beginner-level play shown by White in Game A.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Game A White shows weak opening fundamentals with random flank pawn pushes (3. Qf3, 4. g4, 6. h4, 8. h5) and outright blundered a full rook on move 13 (13. Nh3 Nc2+), winning only because Black repeatedly blundered back. In contrast, Game B White plays a coherent opening with standard piece development and central control (3. b3, 5. Bg2, 7. O-O, 8. d4). Even though Game B White eventually lost due to tactical oversights in time trouble (30. Rd1 Re2+ 31. Kf1 Rh2), the overall strategic and tactical comprehension is significantly higher than Game A White's.

Reported P(A stronger): 0.15. Consistent with choice.

Qwen3.8 Max Prime: chose A — wrong

Game A's White faced absurd openings (2…Bd7, 3…c6, 4…b5) yet played coherently, winning material cleanly (18.Bxa1, 20.Bxc4) and converting with 30.Rd7+, 38.Qd5+, 39.Rxd8#. Game B's White made real errors: 11.g4? weakening the kingside, 16.hxg3 allowing 16…Bh3, and losing after 30…Re2+ 31.Kf1 Rh2 32.Qxd5 Qh3+ 33.Ke1 Qxg3#. B's White collapsed against a much more competent opponent, suggesting a substantially lower established rating than A's White.

Reported P(A stronger): 0.82. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A White played a bizarre, unsound opening (2.e3, 3.Qf3, 4.g4, 5.h3, 6.h4, 7.g5, 8.h5), lost the exchange and rook on a1 to 13...Nc2+, and only won because Black blundered badly (23...Qc8?? losing the queen and later Rxc3+ for nothing). The opponent's play (2...Bd7, 3...c6, 4...b5, 5...Na6) indicates a very low-rated pairing. Game B White showed coherent development (b3/g3 fianchetto, 10.Re1+ Kf8, 11.Nxd4, 14.c4, 21.cxd5 c5) and was even better before 29.Qd4?? allowed 29...Re8 and 30...Re2+ with a losing attack. Despite losing, Game B White's overall move quality and opponent's Sicilian handling suggest a higher established rating than Game A's target.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Game A White plays a primitive opening, making five pawn moves and an early queen sortie (3. Qf3) before developing any minor pieces. White misses a basic fork (13. Nh3 Nc2+), losing a full rook, and only wins because Black fails to convert. Conversely, Game B White exhibits sophisticated positional play in a 3. b3 Sicilian, prioritizing development (9. Bb2, 10. Re1+) and central control. Game B involves high-level themes like the h-pawn thrust and piece maneuvering. Although Game B White blunders at the end (32. Qxd5??), the technical proficiency and opening knowledge displayed are significantly higher than the disorganized play seen from Game A White.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Game A White shows extremely weak opening fundamentals, pushing flank pawns recklessly with 4. g4, 6. h4, 7. g5, and 8. h5 while neglecting piece development, and blundering a full rook with 13. Nh3 Nc2+. Although White eventually won after Black blundered back, the initial quality of play indicates a much lower rating. Game B White plays a sound, disciplined opening (Snyder Sicilian), castles early, and manages a complex positional battle logically up to move 29. Even though Game B White loses to a late-game tactical miscalculation (30. Rd1 Re2+), their overall chess understanding and move quality are vastly superior to Game A White.

Reported P(A stronger): 0.15. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B White showed substantially more coherent development and positional play: 7.O-O, 8.d4, 9.Bb2, 14.c4, and 23.Rc1 followed recognizable plans, while 27.Bxc5+ and 28.Qxc3 handled tactics sensibly. The late 32.Qxd5? allowed ...Qh3+ and ...Qxg3+, but it was one tactical failure. Game A White’s 3.Qf3, 4.g4, repeated h-pawn moves, and 13.Nh3? allowed ...Nc2+ and ...Nxa1, indicating much weaker fundamentals. A’s eventual mating attack, including 30.Rd7+ and 34.Rxc8, was resourceful but depended heavily on Black’s repeated defensive errors.

Reported P(A stronger): 0.1. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s White developed and contested the center with 7.O-O, 8.d4 and 11.Nxd4, then navigated exchanges with 27.Bxc5+ and 28.Qxc3. The late 32.Qxd5? was a serious mistake, but Game A’s White showed more persistent problems: 3.Qf3 and 4.g4 neglected development, and 13.Nh3 allowed 13...Nc2+ 14.Kd2 Nxa1. A’s win also depended on Black’s 33...Bd8, which allowed 34.Rxc8. B looks more likely to have had the higher established strength despite losing.

Reported P(A stronger): 0.08. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game B White shows substantially stronger opening and middlegame fundamentals: castling, coordinated development, 10.Re1+ forcing Black's king to f8, and sensible central exchanges. White later underestimated Black's attack with 30.Rd1 and especially 32.Qxd5, allowing decisive checks. Game A White instead played 3.Qf3 and repeated flank-pawn pushes while leaving the king and pieces undeveloped; 13.Nh3 allowed 13...Nc2+ and the loss of the a1 rook. A's eventual tactical success—33.Ra8 followed by 34.Rxc8 and a mating sequence—came against highly cooperative opposition and does not outweigh the much weaker earlier play.

Reported P(A stronger): 0.96. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B’s White showed more coherent opening and positional play: 1.e4, 7.O-O, 9.Bb2, 10.Re1+, and 12.Nxc6 develop with clear purpose. However, White ultimately missed the danger of Black’s attack, especially 30...Re2+ followed by 32...Qh3+ and 33...Qxg3+. Game A’s White found a strong tactical attack and mating net, but the opening was extremely loose (2.e3, 3.Qf3, 4.g4), and White allowed 13...Nc2+ and 14...Nxa1, losing a rook. The eventual win depended heavily on Black’s inaccurate defense, including 24...Rxc3+. Overall, B’s White appears more established despite losing.

Reported P(A stronger): 0.3. Consistent with choice.

GPT-6 Astra: chose B — correct

Game B White shows much stronger fundamentals: 7.O-O, 9.Bb2 and 10.Re1+ coordinate development, while 19.Rh1 sensibly exchanges Black’s attacking rook. The intermediate 27.Bxc5+ before 28.Qxc3 also shows tactical awareness. White later allows a decisive attack with 32.Qxd5 Qh3+. Game A White instead plays 3.Qf3 and repeated g- and h-pawn pushes while leaving almost everything undeveloped; 13.Nh3 allows 13...Nc2+ and the loss of the a1-rook. A’s later rook invasion and mating finish are creditable, but arise against highly cooperative opposition and do not outweigh the much weaker opening and king safety.

Reported P(A stronger): 0.96. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — correct

White in Game B plays a principled Sicilian/Taimanov setup, trades sensibly (11.Nxd4, 12.Nxc6), maneuvers well (21.cxd5, 22.Be3), and only loses after Black finds strong rook infiltrations (29...Re8, 30...Re2+); White's final blunder 32.Qxd5 walking into ...Qh3+ costs the game. White in Game A benefits from a catastrophically weak opponent (7...Rd8??, 13...Nc2+ giving a knight for nothing) and plays erratic moves himself (2...Bd7 response aside, 4.g4?!, 13.Nh3 with king marching). A's win is more about opponent collapse than skill; B shows more coherent technique despite losing.

Reported P(A stronger): 0.4. Consistent with choice.

Pair 01: 1 of 16 models wrong

A: Black, rated 1569 · B: White, rated 2668. Higher pre-game rating: B.

Game A: Black, rated 1569 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 d5 2. exd5 e6 3. dxe6 Bxe6 4. Nf3 Bc5 5. c3 Qe7 6. Be2 Bxf2+ 7. Kxf2 Nf6
8. Rf1 O-O 9. Kg1 Bd5 10. d4 Qd6 11. Bg5 Ng4 12. h3 Bxf3 13. hxg4 Bxe2 14. Qxe2
Nd7 15. Nd2 Rae8 16. Qf2 Nb6 17. Rae1 Rxe1 18. Rxe1 Nd5 19. Qf3 f6 20. Bh4 c5
21. Bg3 Qc6 22. Nb3 cxd4 23. Nxd4 Qc4 24. a3 a5 25. Qe2 Qxe2 26. Rxe2 Rd8 27.
Re6 Nc7 1-0
Game B: White, rated 2668 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 Nf6 2. c4 e6 3. Nc3 d5 4. cxd5 exd5 5. Bg5 c6 6. e3 Bf5 7. Qf3 Bg6 8.
Bxf6 Qxf6 9. Qxf6 gxf6 10. h4 h5 11. Nh3 Bd6 12. Kd2 Nd7 13. Bd3 b5 14. Rac1
Ke7 15. Ne2 Rac8 16. Nhf4 Rhg8 17. Ng3 Bxf4 18. exf4 Bxd3 19. Kxd3 Kd6 20. Rhe1
Rce8 21. Rxe8 Rxe8 22. Nxh5 c5 23. Ng3 c4+ 24. Kd2 Nf8 25. h5 Ne6 26. Nf5+ Kd7
27. g3 Rh8 28. h6 a5 29. b3 Nd8 30. bxc4 dxc4 31. d5 Nb7 32. Re1 Nd6 33. Re7+
Kd8 34. Ra7 Nxf5 35. Ra8+ Kd7 36. Rxh8 b4 37. h7 1-0

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A Black played a dubious gambit (2...e6) and then the unsound 6...Bxf2+, ending down material for pawns. 11...Ng4 12.h3 Bxf3 13.hxg4 Bxe2 only traded pieces without recovering the deficit. Black then drifted into a worse endgame and resigned after 27.Re6, showing little compensation or resourcefulness. Game B White handled a QGD Exchange competently: 8.Bxf6 led to a queen trade, 22.Nxh5 won a pawn, and 25.h5 with 28.h6 pushed the passed pawn. White then used the rook actively with 33.Re7+ and 34.Ra7, and punished 34...Nxf5 with 35.Ra8+ and 36.Rxh8. Game B's play is more coherent and technically sound.

Reported P(A stronger): 0.25. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A, Black: Black played a speculative 6...Bxf2+ sacrifice that gave up a piece for a pawn, then played reasonably (12...Bxf3, 17...Rxe1). The game ended abruptly after 27.Re6 Nc7, and Black was worse by then. The opening sacrifice suggests an intermediate-level player. Game B, White: White played a standard Queen's Gambit Exchange with Bg5, traded queens, and played a coherent plan (h4-h5, Rac1, Ne2-g3). Its 22.Nxh5 won a pawn, and it converted the endgame cleanly with 33.Re7+ and 36.Rxh8. Black in Game B also played reasonably but slipped. White in B looks somewhat more solid and technically sound than Black in A, though both are probably club-level players, so the evidence is modest.

Reported P(A stronger): 0.4. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game A Black played an unsound opening line and immediately sacrificed a full minor piece for zero compensation with 6...Bxf2+ 7.Kxf2, after which White smoothly consolidated with Rf1 and Kg1, leaving Black a piece down without counterplay. In contrast, Game B White demonstrated solid positional understanding and endgame technique. White strategically compromised Black's pawn structure (8.Bxf6 and 9.Qxf6), systematically exploited Black's weaknesses on the kingside (16.Nhf4, 22.Nxh5), created an unstoppable passed h-pawn, and executed a decisive tactical wrap-up (34.Ra7 and 35.Ra8+). Game B White shows vastly superior chess strength.

Reported P(A stronger): 0.03. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Black in Game A plays a highly dubious opening with 2... e6 and unprovokedly sacrifices a bishop for no compensation with 6... Bxf2+, showing a lack of basic tactical and positional understanding. In contrast, White in Game B demonstrates high-level positional play in a QGD Exchange, strategically damaging Black's pawn structure, maneuvering knights effectively (Nh3-f4-g3), and executing a precise endgame conversion with the tactical sequence 34. Ra7 and 35. Ra8+. White B is clearly the stronger, more advanced player.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

In Game A, Black plays a dubious opening gambit (2... e6) and an unsound piece sacrifice (6... Bxf2+?) that leaves them down a piece for no compensation. Black's play lacks tactical and positional foundation. In contrast, Game B (White) demonstrates high-level positional and endgame play. White expertly exploits Black's pawn weaknesses, maneuvers the knight efficiently (Nh3-f4-g3-f5), and executes a precise rook endgame. White secures the win with a neat tactical sequence starting with 34. Ra7 and 35. Ra8+, winning Black's rook. Target B is vastly more accomplished.

Reported P(A stronger): 0.03. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Target B (White in Game B) demonstrated high-level strategic and endgame skill. After doubling Black's pawns with 8. Bxf6 and trading queens, White accurately navigated the endgame with Kd2, Nh3-f4, exploited Black's kingside weaknesses to win pawns (22. Nxh5), and converted effortlessly with 32. Re1 and 35. Ra8+. In contrast, Target A (Black in Game A) played an unsound gambit and blundered a full piece with 6...Bxf2+? without any compensation, subsequently losing without mounting any real resistance.

Reported P(A stronger): 0.05. Consistent with choice.

Qwen3.8 Max Prime: chose A — wrong

Game A Black sacrifices the bishop unsoundly (6...Bxf2+), but recovers with sound positional moves: 12...Bxf3, 17...Rxe1, 22...cxd4, and the simplifying 25...Qxe2, reaching a roughly defensible rook ending before a late collapse. Game B White plays more passively and strategically shaky: 7.Qf3 invites trades, 9.Qxf6 gxf6 improves Black's structure, 10.h4 and 22.Nxh5 chase loose pawns while Black equalizes; the win owes more to opponent errors after 30...dxc4. A's middlegame resourcefulness under a bad opening suggests a slightly stronger established player.

Reported P(A stronger): 0.6. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A Black plays a speculative Scandinavian gambit with 3...Bxe6 and the unsound 6...Bxf2+ sacrifice, then 11...Ng4?? losing a piece outright (12.h3 Bxf3 13.hxg4), and resigns a pawn down in a lost ending after 27.Re6. That is a beginner-level piece blunder. Game B White conducts a solid Exchange QGD, trades sensibly (8.Bxf6, 9.Qxf6), centralizes the king for the queenless middlegame (12.Kd2, 19.Kxd3), wins a pawn with 22.Nxh5, and converts with the h-pawn and 33.Re7+ 34.Ra7 35.Ra8+. Technique is clean and purposeful; Black's defence in B was also reasonable. White in B shows clearly higher established strength.

Reported P(A stronger): 0.2. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Target B demonstrates a much higher level of positional understanding and endgame technique. In Game B, White navigates a Queen's Gambit Exchange structure expertly, utilizing the h-pawn and knight maneuvers (11. Nh3, 16. Nhf4) to exploit Black's doubled pawns. White converts the technical endgame efficiently with 32. Re1 and 37. h7. In contrast, Target A (Black) plays a dubious opening (2... e6) and an unsound piece sacrifice (6... Bxf2+). After the initial aggression fails, Black lacks positional compensation and eventually collapses, ending with the blunder 27... Nc7 which loses the knight on d5 to the rook. The disparity in strategic depth and tactical stability clearly favors Target B.

Reported P(A stronger): 0.1. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

In Game A, Black (Target A) played an unsound piece sacrifice with 6...Bxf2+? for minimal compensation and later blundered with 27...Nc7, hanging the knight. In contrast, Target B (White in Game B) demonstrated strong positional understanding in a standard QGD Carlsbad structure. White systematically exploited Black's weak f- and h-pawns through precise knight maneuvering (11.Nh3, 16.Nhf4, 22.Nxh5). White then cleanly converted the endgame with active piece play (33.Re7+, 34.Ra7) to create a decisive passed pawn (37.h7). Target B displayed much higher strategic depth and tactical soundness than Target A.

Reported P(A stronger): 0.1. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B’s White showed substantially stronger, coherent play: the queen trade with 8.Bxf6 and 9.Qxf6, king centralization, the maneuver Nh3-f4-g3-h5, and 22.Nxh5 followed by the passed h-pawn were purposeful. White also found the clean tactical finish 33.Re7+ and 35.Ra8+, winning the h8-rook. Game A’s Black chose the highly unsound 6...Bxf2+, leaving White materially ahead despite the resource 12...Bxf3 13.hxg4 Bxe2. Black then traded queens while a piece down and resigned after 27...Nc7. B is much more likely to have had the higher established rapid strength.

Reported P(A stronger): 0.07. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s White played a coherent endgame: 22.Nxh5 won a pawn, 25.h5 and 28.h6 created a dangerous passer, and 34.Ra7 followed by 35.Ra8+ exploited Black’s knight capture. Game A’s Black chose the unsound-looking 6...Bxf2+, then lost another minor piece in the sequence 11...Ng4 12.h3 Bxf3 13.hxg4 Bxe2 14.Qxe2, leaving White with a substantial material advantage. B’s play is stronger evidence of established skill, regardless of the results.

Reported P(A stronger): 0.04. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game A’s Black played the speculative 6...Bxf2+, sacrificing a bishop without a convincing continuation. The sequence 12...Bxf3 13.hxg4 Bxe2 left Black a minor piece down, and subsequent exchanges offered little compensation. Game B’s White showed substantially more coherent planning: 12.Kd2 activated the king after the queen trade, 22.Nxh5 won a pawn, and h5–h6 created a dangerous passer. White then combined central pressure with seventh-rank activity through 31.d5, 32.Re1 and 33.Re7+, followed by 35.Ra8+ and 36.Rxh8. Black’s cooperation in Game B limits certainty, but White’s sustained coordination is much stronger evidence of established playing strength.

Reported P(A stronger): 0.93. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B’s White showed substantially better practical play: after the unconventional 12.Kd2, White coordinated effectively with 20.Rhe1, exchanged favorably, created a dangerous h-pawn, and converted it with 33.Re7+, 34.Ra7, and 36.Rxh8. White also promptly exploited Black’s 34...Nxf5. In Game A, Black’s opening became strategically and materially dubious: 6...Bxf2+ sacrificed a bishop for a pawn, and 12...Bxf3 13.hxg4 further left Black’s position vulnerable. Black never generated adequate compensation and allowed 27.Re6. Although Game B’s opponent made larger errors, its target demonstrated much stronger conversion and coordination than Game A’s target.

Reported P(A stronger): 0.82. Contradicts choice; excluded from confidence diagnostics.

GPT-6 Astra: chose B — correct

Game B White showed coherent positional and endgame play: 8.Bxf6 followed by 9.Qxf6 damaged Black’s structure, 12.Kd2 activated the king, and 22.Nxh5 followed by the h-pawn advance created a dangerous passer. The rook invasion 33.Re7+ and 35.Ra8+ exploited Black’s exposed rook decisively. Game A Black developed actively, but 6...Bxf2+ was an unsupported sacrifice; 11...Ng4 and the ensuing exchanges left Black a bishop down without convincing compensation. The queen exchange with 25...Qxe2 further reduced any attacking chances. B’s sustained coordination is substantially stronger evidence of established playing strength.

Reported P(A stronger): 0.95. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — correct

Game B's White plays a clean, methodical game: timely queen trade (9.Qxf6), purposeful h4-h5 expansion, good rook placement (Rac1, Re1), and accurate endgame technique (Kd3, Re1-Ra7 winning the exchange, h7). Game A's Black plays erratically: the speculative 6...Bxf2+ gives up material without clear compensation, and 27...Re6 blunders the pinned rook with Nc7 while White threatens the e-file, losing immediately. Black's compensation attempt (17...Rxe1, 18...Nd5) shows some ideas but the overall play is less coherent and collapses faster. B's White shows stronger positional understanding and endgame technique, suggesting the higher rapid rating.

Reported P(A stronger): 0.3. Consistent with choice.

Pair 08: 1 of 16 models wrong

A: Black, rated 2410 · B: Black, rated 1903. Higher pre-game rating: A.

Game A: Black, rated 2410 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 e5 2. Nf3 Nc6 3. Bb5 a6 4. Ba4 Nf6 5. Nc3 b5 6. Bb3 Be7 7. O-O O-O 8. Nd5
d6 9. c3 Nxe4 10. d4 exd4 11. cxd4 Nf6 12. Nxf6+ Bxf6 13. Be3 Na5 14. Bc2 Nc4
15. Qd3 g6 16. Bh6 Re8 17. Qb3 Bb7 18. h4 Bxh4 19. Nxh4 Qxh4 20. Qh3 Qxh3 21.
gxh3 Re2 22. Bd3 Rxb2 23. a3 Rb3 24. Bc2 Rxh3 25. Bg5 Rh1# 0-1
Game B: Black, rated 1903 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 g6 2. Nc3 Bg7 3. d4 d6 4. Be3 Nf6 5. Qd2 O-O 6. f3 Re8 7. Bh6 Bh8 8.
O-O-O Nbd7 9. g4 c5 10. d5 a6 11. h4 b5 12. h5 b4 13. hxg6 fxg6 14. Nce2 Ne5
15. Ng3 Nf7 16. g5 Nd7 17. f4 Nxh6 18. Rxh6 Nf8 19. Bc4 a5 20. Nf3 Ba6 21. Bxa6
Rxa6 22. Rdh1 a4 23. Qh2 Bg7 24. f5 b3 25. a3 Kf7 26. Rxh7 Nxh7 27. Qxh7 Rh8
28. fxg6+ Kf8 29. Qxh8+ Bxh8 30. Rxh8+ Kg7 31. Rxd8 Kxg6 32. Rd7 Kf7 33. Nf5
bxc2 34. Rxe7+ Kf8 35. g6 Ra8 36. Ng5 c4 37. g7+ Kg8 38. Nh6# 1-0

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Game A Black punished errors cleanly. 9...Nxe4 took a free pawn after 8.Nd5, and 18...Bxh4 exploited 18.h4. After the queen trade, 21...Re2 led to 22...Rxb2, 24...Rxh3 and the mating 25...Rh1#, showing efficient, accurate conversion. Admittedly White there was weak. Game B Black played poorly against a standard kingside attack. 6...Re8 and 7...Bh8 wasted tempi and dropped the dark-squared bishop's defensive role. 13...fxg6 opened lines, and 17...Nxh6 handed White the h-file. Black's queenside play (b4, a4, b3) was far too slow, and the king collapsed after 26.Rxh7. Black B's opponent attacked competently, but B's opening and defensive understanding looks weaker than A's tactical alertness.

Reported P(A stronger): 0.68. Consistent with choice.

Claude Sonnet 5.5: chose A — correct

Game A, Black: plays the Open Spanish normally. After White's weak 8.Nd5, Black takes 9...Nxe4 and 10...exd4, then 14...Nc4 and 18...Bxh4 win material and tempo. Black then finishes cleanly with 21...Re2, 22...Rxb2 and 25...Rh1#. White blundered, but Black punished each error efficiently. Game B, Black: plays a King's Indian/Modern setup but loses a pawn structure battle. 7...Bh8 is odd, and 17...Nxh6 followed by 18.Rxh6 lets White's attack run. 24...b3 and 25...Kf7 are dubious, and 26...Nxh7 loses material. Black loses to a mating attack with lots of errors, including 28...Kf8. Black in A looks a bit stronger, though A's opponent was very weak, which tempers the evidence.

Reported P(A stronger): 0.66. Consistent with choice.

Gemini 3.8 Flash: chose A — correct

Game A's Black demonstrated clean, disciplined positional and tactical play throughout. In the Spanish opening, Black capitalized cleanly on White's pawn sacrifice with 9...Nxe4, consolidated comfortably with 13...Na5 and 15...g6, and methodically punished White's overextension on the kingside with 18...Bxh4 and 21...Re2, executing a swift finish. Game B's Black mishandled the defense against White's standard kingside storm in the Modern/Pirc, spending too many tempi shuffling knights (14...Ne5, 15...Nf7, 16...Nd7, 18...Nf8) while queenside counterplay lagged behind. This passive defense allowed White a decisive breakthrough via 23. Qh2 and 26. Rxh7.

Reported P(A stronger): 0.68. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — wrong

Black B demonstrates advanced opening knowledge (such as the thematic 7...Bh8 to preserve the bishop) and executes a coherent queenside counterattack in a highly complex position, ultimately falling to a brilliant combination by a strong opponent. In contrast, Black A plays a solid game but simply capitalizes on basic blunders from a much weaker White opponent, such as dropping a pawn with 9. c3 Nxe4 and walking into a simple mate-in-one with 25. Bg5. Black B shows a significantly higher level of strategic understanding.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.5 Flash: chose A — correct

In Game A, Black plays a highly logical Ruy Lopez, quickly punishing White's inaccuracy. Black coordinates pieces excellently with 13...Na5 and 14...Nc4, wins a pawn cleanly with 18...Bxh4, and executes a swift mating attack with 24...Rxh3 and 25...Rh1#. In Game B, Black's passive Modern Defense setup (such as 6...Re8) allows White to build a quick kingside storm. Black fails to generate effective counterplay on the queenside, defensive maneuvers like 18...Nf8 are too slow, and Black ultimately collapses to a straightforward sacrifice on h7. Target A exhibits superior tactical alertness and opening preparation.

Reported P(A stronger): 0.65. Consistent with choice.

Gemini 3.7 Flash: chose A — correct

Target A (Game A Black) demonstrated solid opening understanding and sharp tactical awareness. After White's inaccurate gambit play, Black smoothly collected material (9...Nxe4), activated pieces dynamically (13...Na5, 14...Nc4, 21...Re2), and delivered a flawless finish with 24...Rxh3 and 25...Rh1#. Conversely, Target B (Game B Black) defended passively against a standard Yugoslav-style kingside pawn storm, misplaying the kingside defense with 23...Bg7? and 25...Kf7?, allowing White a crushing breakthrough (26. Rxh7) and a clean tactical finish.

Reported P(A stronger): 0.78. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

Game A Black handled the Ruy Lopez solidly, exploiting White's weak Nd5 (Game A 9…Nxe4) and later capitalizing on White's flawed 18.h4 to win a pawn and deliver a clean mate (Game A 25…Rh1#). Game B Black, in the Pirc (Game B 1…g6, 7…Bh8), was repeatedly passive; 25…Kf7 and 27…Rh8 invited disaster, and Black failed to counter White's g-file attack, leading to mate on move 38. Game B Black showed weaker prophylaxis and middlegame judgment than Game A Black.

Reported P(A stronger): 0.68. Consistent with choice.

Claude Fable 5.1: chose A — correct

Game A Black played a sound Ruy Lopez, exploited 9...Nxe4 after White's inaccurate 8.Nd5/9.c3, then found the tactical sequence 18...Bxh4! 19...Qxh4 and the rook invasion 21...Re2 to 25...Rh1#, a crisp conversion against a weak White. Game B Black played a passive Pirc setup (7...Bh8, 15...Nf7, 16...Nd7) and allowed a known mating pattern; 25...Kf7?? and 26...Nxh7 walked into Rxh7/Qxh7 and total collapse, and the queenside counterplay (24...b3, 26...a4) was far too slow. Game B's White showed decent attacking technique, but Black's defensive understanding was weak. Game A Black looks the more competent player.

Reported P(A stronger): 0.62. Consistent with choice.

Gemini 3 Flash (Preview): chose A — correct

Player A (Black) played a precise and disciplined game, efficiently punishing White's inaccuracies. Moves like 9...Nxe4 and 14...Nc4 demonstrate strong tactical awareness, while the infiltration with 21...Re2 and 23...Rb3 showed high technique in converting an advantage. In contrast, Player B (Black) struggled to handle White’s aggressive kingside storm in a Modern Defense. Allowing the h-file to open with 17...Nxh6 and failing to generate sufficient counter-pressure led to a rapid collapse. While Player B faced a strong attack, Player A's clean execution and tactical alertness suggest a more established skill level.

Reported P(A stronger): 0.65. Consistent with choice.

Gemini 3.6 Flash: chose A — correct

Game A Black displayed excellent tactical precision and solid opening knowledge in the Ruy Lopez. Black accurately seized central control (14...Nc4), punished White's premature flank push 18.h4 with 18...Bxh4! 19.Nxh4 Qxh4, and executed a swift, decisive conversion ending in 25...Rh1#. In contrast, Game B Black defended poorly against White's kingside storm in a Pirc structure. Facing severe pressure on the h-file, Game B Black ignored White's threats with 24...b3? and collapsed to the blunder 25...Kf7??, allowing 26.Rxh7! which won White a queen and rook. Game A Black demonstrated significantly higher tactical awareness and positional understanding.

Reported P(A stronger): 0.88. Consistent with choice.

GPT-5.6 Sol: chose A — correct

Game A Black played a coherent tactical game: 9...Nxe4 exploited the center, 18...Bxh4! converted White’s kingside aggression into material and initiative, and 21...Re2 followed by ...Rxb2/...Rxh3 produced the precise 25...Rh1# finish. Game B Black’s queenside counterplay was too slow while the king came under a direct attack; 24...b3 and especially 25...Kf7 failed to address White’s threats, and 26...Nxh7 allowed the forcing sequence 27.Qxh7 Rh8 28.fxg6+ through 31.Rxd8, leaving Black decisively down. A showed stronger tactical awareness and defensive judgment.

Reported P(A stronger): 0.88. Consistent with choice.

GPT-6 Sol: chose A — correct

Game A’s Black played a coherent game: 18...Bxh4 met White’s advance, and 21...Re2 followed by 22...Rxb2 and 24...Rxh3 turned the open lines into a mating attack. Game B’s Black pursued queenside counterplay with 24...b3, but missed a decisive danger: after 27...Rh8 and 28...Kf8, White’s 29.Qxh8+ Bxh8 30.Rxh8+ led to 31.Rxd8, winning Black’s queen. A’s play looks substantially more controlled.

Reported P(A stronger): 0.98. Consistent with choice.

GPT-6.1 Sol: chose A — correct

Game A Black played more coherently: 9...Nxe4 exploited the center, 18...Bxh4 won another pawn through 19.Nxh4 Qxh4, and 21...Re2 converted the queen trade into active rook play. The finish 24...Rxh3 and 25...Rh1# correctly used the b7 bishop. Game B Black showed sensible queenside counterplay, but the repeated knight maneuvers consumed time while White built a clear kingside attack. Black's 25...Kf7 allowed the forcing sequence beginning 26.Rxh7, ultimately losing the queen on d8; 36...c4 also ignored the mating threat. A's opposition was weaker, which limits confidence, but A demonstrated better tactical control.

Reported P(A stronger): 0.76. Consistent with choice.

GPT-5.6 Luna: chose A — correct

Game A’s Black showed stronger tactical execution: 9...Nxe4 and 10...exd4 exploited White’s center, while 18...Bxh4! 19 Nxh4 Qxh4 removed the key defender. The follow-up 21...Re2 and 22...Rxb2 created a coherent rook attack, culminating in 25...Rh1#. Game B’s Black developed sensibly, but the kingside defense deteriorated: after White’s aggressive 24 f5, Black allowed 26 Rxh7 and 27 Qxh7, then was unable to stop the advancing g-pawn and was mated by 38 Nh6#. Although Game A’s opponent made larger errors, Black’s tactical recognition and conversion provide stronger evidence of established strength.

Reported P(A stronger): 0.8. Consistent with choice.

GPT-6 Astra: chose A — correct

Game A Black played more consistently: 9...Nxe4 exploited White’s center, 13...Na5–14...Nc4 improved the knight, and 18...Bxh4 won another pawn through a favorable exchange. After trading queens, 21...Re2 and 23...Rb3 showed effective rook activity, culminating in 25...Rh1#. Game B Black showed purposeful ideas with ...Nxh6, ...Nf8 and ...Ba6, against a more coherent attacking opponent. However, 25...Kf7 and 27...Rh8 allowed the forcing 28.fxg6+ through 31.Rxd8 sequence. A’s cleaner tactical judgment gives a moderate edge, tempered by White’s substantially weaker resistance in Game A.

Reported P(A stronger): 0.66. Consistent with choice.

GLM-5.3 Flash: chose A — correct

Game A's Black played a principled Ruy Lopez, grabbing a pawn with 9...Nxe4, regrouping (13...Na5, 14...Nc4), and executing a clean tactical sequence: 18...Bxh4! 19.Nxh4 Qxh4, then converting accurately to mate (21...Re2, 24...Rxh3, 25...Rh1#). Game B's Black chose a passive kingside setup (...Re8 6..., 7...Bh8) and drifted: 23...Bg7? allowed 24.Rxh7/26.Rxh7, and after 26...Nxh7 27.Qxh7 the defense collapsed, leading to a forced mating net (38.Nh6#). A's Black showed coherent strategy and precise calculation; B's Black was outplayed systematically.

Reported P(A stronger): 0.75. Consistent with choice.

Pair 10: 0 of 16 models wrong

A: Black, rated 564 · B: Black, rated 2177. Higher pre-game rating: B.

Game A: Black, rated 564 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 d5 2. Bf4 Nf6 3. h3 Nc6 4. e3 Bf5 5. Bb5 e6 6. Bg5 Be7 7. Bxf6 Bxf6 8.
Nf3 O-O 9. O-O a6 10. Bxc6 bxc6 11. Nbd2 e5 12. dxe5 Be7 13. Nd4 Bxh3 14. gxh3
f5 15. Nxf5 Bh4 16. Nxh4 Qxh4 17. f3 Qg5+ 18. Kh1 Qxe5 19. Re1 Qg3 20. Qe2
Qxh3+ 21. Qh2 Qe6 22. Qe5 Qg6 23. Rg1 Qh6+ 24. Qh2 1-0
Game B: Black, rated 2177 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. d4 Nf6 2. c4 c5 3. d5 d6 4. Nc3 g6 5. e4 Bg7 6. g3 O-O 7. Bg2 Na6 8. a3 Bd7
9. Nge2 Nc7 10. O-O b5 11. cxb5 Nxb5 12. Bd2 Rb8 13. Qc1 Ng4 14. h3 Ne5 15. Kh2
Nc4 16. Rb1 Nxd2 17. Qxd2 Qa5 18. Qc2 Rfc8 19. f4 Nd4 20. Qd3 Nxe2 21. Qxe2
Bxc3 22. bxc3 Qxa3 23. Ra1 Qxc3 24. Rfc1 Qd4 25. Rxa7 Rb2 26. Qf3 Qd2 27. Rg1
Rd8 28. e5 Bf5 29. g4 Bb1 30. f5 dxe5 31. fxg6 hxg6 32. Rxe7 Qf4+ 33. Qxf4 exf4
34. Kh1 Ba2 35. Rc1 Bxd5 36. Bxd5 Rxd5 37. Re8+ Kg7 38. Rc8 Rdd2 39. R8xc5 Rh2+
40. Kg1 Rbg2+ 41. Kf1 f3 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A Black blundered badly with 13...Bxh3?, an unsound sacrifice. After 14.gxh3 f5 15.Nxf5 they were simply down material, then wasted tempi with the queen (19...Qg3, 21...Qe6, 22...Qg6) and resigned early. Game B Black played a coherent Benoni/Benko-style plan with ...Na6-c7, ...b5, ...Rb8 and ...Ng4-e5-c4. They won material with 21...Bxc3 and 22...Qxa3, kept active rooks with 25...Rb2 and 26...Qd2, and converted cleanly with 35...Bxd5, ...Rdd2 and 41...f3 mating ideas. B shows much stronger understanding and technique.

Reported P(A stronger): 0.2. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A Black played the opening reasonably (4...Bf5, 8...O-O) but erred with 13...Bxh3 14.gxh3 f5 15.Nxf5, which dropped material; 15...Bh4 and 16...Qxh4 then lost a piece. Later 18...Qxe5 and 20...Qxh3+ grabbed pawns, but White had a strong attack, and Black lost quickly in 24 moves. Game B Black handled a King's Indian-style Benoni well: 11...Nxb5, 13...Ng4 and 15...Nc4 were active, 21...Bxc3 and 22...Qxa3 won material, and the tactical finish with 39...Rh2+ and 41...f3 was accurate. Black still allowed some complications (25.Rxa7), but overall played at a higher and more consistent level than A Black.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game B Black demonstrated strong strategic mastery in a Benoni setup, methodically executing the typical queenside expansion (...b5, ...Nc7, ...Rb8), winning a pawn cleanly with 21...Bxc3 and 22...Qxa3, and converting decisively with active rook play (...Rb2, ...Rdd2) to weave a mating net. In contrast, Game A Black blundered a pawn immediately with 11...e5?, followed up with an unsound piece sacrifice 13...Bxh3, and never achieved real compensation before losing quickly. Player B's understanding of opening plans, piece coordination, and technical execution reflects a considerably higher playing strength.

Reported P(A stronger): 0.12. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Target B demonstrates a sophisticated understanding of the Benoni structure, executing typical queenside expansion (10... b5) and excellent knight maneuvers (14... Ne5, 15... Nc4). Target B seamlessly converts a tactical advantage into a won endgame with precise technique (32... Qf4+). In contrast, Target A plays a chaotic and unsound game, sacrificing a piece dubiously with 13... Bxh3, and blundering another piece outright with 15... Bh4??. Target B's positional depth and tactical precision heavily outweigh Target A's poor tactical vision.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

In Game A, Black shows poor tactical and positional awareness, dropping a pawn with 11...e5? and then sacrificing a bishop unsoundly with 13...Bxh3?. In contrast, Game B's Black plays a highly coherent Benoni setup, executing a thematic queenside expansion (10...b5). Black shows great positional understanding, wins a pawn cleanly with 21...Bxc3 and 22...Qxa3, and converts the endgame flawlessly to create a mating net with 41...f3. Game B's Black is clearly the more established and stronger player.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Player B displays a sophisticated understanding of opening strategy and positional play in the Benoni/KID setup, executing a classic queenside expansion (...Na6-c7, ...b5) and active piece maneuvers (14...Ne5, 15...Nc4). Player B converted smoothly with 21...Bxc3, heavy-piece infiltration (...Rb2, ...Qd2), and precise technique in the endgame. In contrast, Player A demonstrated poor positional and tactical awareness, dropping a central pawn with 11...e5? and following up with an unsound piece sacrifice (13...Bxh3?) and weak defense that led to an early resignation after 24.Qh2.

Reported P(A stronger): 0.12. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

In Game A, Black plays a dubious 13...Bxh3 sacrifice that loses material after 14.gxh3 and then wastes moves with a wandering queen (17...Qg5+, 19...Qg3, 21...Qe6, 22...Qg6, 23...Qh6+) before resigning down a full piece. In Game B, Black executes a coherent Benko-style plan with 10...b5, 13...Ng4-e5-c4, and 19...Nd4, winning cleanly via 21...Bxc3 and a decisive attack ending with 39...Rdd2 and 41...f3. Game B's Black shows clearly superior strategic and tactical play.

Reported P(A stronger): 0.15. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A Black played a speculative, unsound 13...Bxh3 sacrifice after 12...Be7 (allowing the pawn loss), then dropped more material with 15...Bh4 and resigned by move 24 with a hopeless position — typical of a lower-rated player chasing cheap tactics. Game B Black handled a Benoni-style structure with purpose (7...Na6-c7, 10...b5, 13...Ng4-e5-c4 to win the bishop pair), then converted with precise tactics: 19...Nd4, 22...Qxa3, the 32...Qf4+ trade into a won ending, and a clean mating net with 38...Rdd2, 39...Rh2+, 41...f3. Game B's Black shows far superior strategic and tactical competence.

Reported P(A stronger): 0.12. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Target B demonstrates a clear understanding of Benoni structures, utilizing effective maneuvering (9... Nc7, 14... Ne5, 15... Nc4) and tactical alertness to secure a positional advantage. The conversion of the endgame was precise, culminating in a clean mating net (41... f3). In contrast, Target A displayed significant tactical weaknesses in Game A. They hung a pawn with 11... e5 and failed to capitalize on White's blunder 15. Nxf5, instead playing 15... Bh4?? which allowed White to save the knight. Target A's speculative sacrifice 13... Bxh3 was unsound and lacked follow-through, indicating a much lower level of tactical and positional awareness compared to Target B.

Reported P(A stronger): 0.1. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Player B demonstrates a far superior positional and tactical understanding. In Game B, Black plays a well-executed Benoni structure with 10...b5, coordinates pieces purposefully (13...Ng4, 15...Nc4, 19...Nd4), wins a pawn with 21...Bxc3, and executes a dominant endgame conversion with double rooks (38...Rdd2 and 41...f3 mate threat). In contrast, Player A plays poorly in Game A, hanging a pawn with 11...e5 and blundering a full bishop with 13...Bxh3 14.gxh3 without adequate compensation, leading to a swift defeat. Player B is clearly the stronger player.

Reported P(A stronger): 0.05. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B’s Black showed substantially stronger strategic and tactical control: 19...Nd4!, 20...Nxe2 and 21...Bxc3 dismantled White’s queenside, while 25...Rb2 and 32...Qf4+ transitioned cleanly into a winning rook invasion. Game A’s Black instead launched the unsound 13...Bxh3? and compounded it with 15...Bh4, surrendering both bishops for insufficient material. The ensuing queen checks never produced a decisive attack, leaving Black down a minor piece and resigning after 24.Qh2.

Reported P(A stronger): 0.03. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s Black played a coherent queenside plan with 10...b5 and 12...Rb8, then used 19...Nd4 and 21...Bxc3 to win material before converting with active rooks. Game A’s Black made a speculative 13...Bxh3 sacrifice and followed it with 15...Bh4, allowing 16.Nxh4 and leaving White with an extra knight. That large tactical oversight makes B the much likelier stronger player, despite the uncertainty of judging one game each.

Reported P(A stronger): 0.02. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game B Black shows coherent Benoni play: 10...b5, the knight maneuver ...Ng4–e5–c4, and 19...Nd4 followed by ...Nxe2 and ...Bxc3 simplify effectively and win queenside pawns. Black then coordinates the rooks with 25...Rb2 and finishes with 38...Rdd2, 39...Rh2+ and 40...Rbg2+, creating a mating net. Game A Black’s 13...Bxh3, 14...f5 and 15...Bh4 sacrifice substantial material without a convincing continuation; the subsequent queen checks produce little. B’s sustained positional planning and tactical execution strongly suggest the higher established rapid strength.

Reported P(A stronger): 0.98. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B’s Black shows substantially stronger strategic and tactical play: the queenside expansion with 10...b5, active maneuvering via 15...Nc4 and 19...Nd4, and the sequence 20...Nxe2 21...Bxc3 won material cleanly. Black then coordinated the rooks effectively with ...Rb2, ...Rd8, and ...Rd2, finishing with a forcing attack. Game A’s Black played 12...Be7 and then sacrificed both bishops with 13...Bxh3 and 15...Bh4, apparently without sufficient compensation; White simply took them with 14 gxh3 and 16 Nxh4. The ensuing queen attack was energetic but failed to produce a result, indicating much weaker judgment than Game B’s Black.

Reported P(A stronger): 0.04. Consistent with choice.

GPT-6 Astra: chose B — correct

Game B Black showed coherent Benoni counterplay with 10...b5 and 12...Rb8, purposeful knight maneuvering through ...Ng4–e5–c4, and 19...Nd4 followed by exchanges that exposed queenside pawns. The invasion 25...Rb2 and coordinated finish 38...Rdd2, 40...Rbg2+, 41...f3 demonstrate strong tactical and positional control. Game A Black developed reasonably but played the speculative 13...Bxh3 followed by 14...f5, leaving inadequate compensation after 15.Nxf5. Subsequent queen checks never established a convincing attack. B’s sustained coordination and conversion are substantially stronger evidence than A’s unsupported sacrifice.

Reported P(A stronger): 0.98. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — correct

Game A Black loses material cheaply: 13...Bxh3? 14.gxh3 then 14...f5? 15.Nxf5 Bxf5? loses a piece for little, and the attack fizzles while White simply consolidates. Game B Black plays a coherent Benko/Benoni-style game: timely ...b5 and knight redeployment, regains material (21...Bxc3, 22...Qxa3), grabs queenside pawns (23...Qxc3, 25...Rxa7), keeps coordination with ...Rb2 infiltration, and converts to a clean win despite complex defense. The difference in positional understanding and conversion skill favors B clearly.

Reported P(A stronger): 0.25. Consistent with choice.

Pair 21: 0 of 16 models wrong

A: White, rated 739 · B: White, rated 2157. Higher pre-game rating: B.

Game A: White, rated 739 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. d4 d5 2. Qd3 e5 3. dxe5 Qe7 4. f4 f6 5. Qxd5 fxe5 6. fxe5 Bf5 7. Qxb7 Qb4+
8. Qxb4 Bxb4+ 9. Bd2 Bxd2+ 10. Nxd2 Bxc2 11. Rc1 Ba4 12. Rxc7 Na6 13. Rc4 Bb5
14. Rd4 Rd8 15. Rxd8+ Kxd8 16. Nb3 Ba4 17. Nc1 Nb4 18. b3 Bb5 19. a3 Nc2+ 20.
Kd1 Ne3+ 21. Kd2 Nxf1+ 22. Ke1 Ne3 23. Kd2 Nxg2 24. a4 Nf4 25. e3 Bxa4 26. bxa4
Nd5 27. Nb3 Nh6 28. e4 Ng4 29. exd5 Nf2 30. Ke2 Nxh1 31. Nf3 Re8 32. Ng5 h6 33.
Nf7+ Ke7 34. Nd6 Rd8 35. Nb5 Rxd5 36. Nxa7 Rxe5+ 37. Kd3 Rh5 38. Nc6+ Kd7 39.
Ne5+ Rxe5 40. Kd4 Re2 41. h4 Rh2 42. Nc5+ Kc6 43. Ne4 Rxh4 44. a5 Kb7 45. Kd5
Ka6 46. Nc5+ Kxa5 47. Nd7 Rh2 48. Ne5 Rd2+ 49. Kc4 Ka4 50. Nf7 Rc2+ 51. Kd3 Kb3
52. Ke4 Rc3 53. Kf5 Re3 54. Kg6 h5 55. Kxg7 Rg3+ 56. Kh6 Rh3 57. Ne5 Rh2 58.
Nf3 Rh3 59. Ng5 Rh2 60. Nf3 Rf2 61. Ng5 Ng3 62. Nh3 Rh2 63. Nf4 h4 64. Kg5 h3
65. Kg4 Ne4 66. Nxh3 Nf2+ 67. Kg3 Rxh3+ 68. Kxf2 Rd3 69. Ke2 Kc2 0-1
Game B: White, rated 2157 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1/2-1/2"]
[TimeControl "600+0"]

1. d4 d5 2. c4 c6 3. Nc3 dxc4 4. e3 b5 5. a4 b4 6. Nb1 Nf6 7. Bxc4 e6 8. Nf3 a5
9. Nbd2 Be7 10. O-O O-O 11. Nb3 Ba6 12. Nfd2 Nbd7 13. Bxa6 Rxa6 14. Nc4 Nb6 15.
Ncxa5 Qd5 16. Nb7 Rxa4 17. Rxa4 Nxa4 18. N3a5 Nb6 19. Qc2 Rc8 20. Bd2 h6 21.
Rc1 Ne4 22. Be1 c5 23. dxc5 Nxc5 24. Rd1 Nxb7 25. Qxc8+ Nxc8 26. Rxd5 exd5 27.
Nxb7 Nb6 28. b3 Kf8 29. Na5 Nd7 30. Nc6 Nc5 31. Bxb4 Bd6 32. Kf1 Ke8 33. Ke2
Kd7 34. Bxc5 Bxc5 35. Ne5+ Ke6 36. Nd3 Bd6 37. g3 g5 38. b4 Kd7 39. Nc5+ Bxc5
40. bxc5 Kc6 41. Kd3 Kxc5 42. e4 d4 43. f4 f6 44. h4 gxh4 45. gxh4 h5 46. Kd2
Kc4 47. Ke2 d3+ 48. Kd2 Kd4 49. e5 fxe5 50. fxe5 Kxe5 51. Kxd3 Kf4 52. Ke2 Kg3
53. Kf1 Kxh4 54. Kg1 Kh3 55. Kh1 Kg3 56. Kg1 h4 57. Kh1 Kf2 58. Kh2 1/2-1/2

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A White plays an unsound opening (2.Qd3, 4.f4) and later blunders badly: 11.Rc1/12.Rxc7 lets Black's bishop roam, 19.a3? allows ...Nc2+ and ...Nxf1+, and 24.a4 drops more material. White did fight back, as Black blundered too (26...Nd5, 30...Nxh1), but later 39.Ne5+?? gives a knight away and 66.Nxh3?? loses to the fork ...Nf2+. Game B White plays a coherent Slav, wins the b-pawn and a piece via 24.Rd1/25.Qxc8+ tactics, and handles the minor-piece endgame reasonably. White then lost the king-and-pawn ending's winning chances but still held a correct draw (57.Kh1, 58.Kh2). Game B's play is clearly more competent.

Reported P(A stronger): 0.2. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A White played 2.Qd3, an early queen sortie, then 4.f4 and 5.Qxd5. Grabbing 7.Qxb7 allowed 7...Qb4+ and an immediate queen trade, with Black's rook and bishop active. It later lost the h1 rook to 30...Nxh1 and lost the endgame. These are weak, beginner-like choices. Game B White followed standard Queen's Gambit Accepted theory and developed sensibly. It kept the structure sound with 9.Nbd2, 10.O-O and 13.Bxa6. It traded into an even endgame with 25.Qxc8+ and 26.Rxd5. It held a draw accurately against a decent opponent with Black's active king play. B looks clearly stronger and more consistent, though the sample is small.

Reported P(A stronger): 0.25. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game A White demonstrates beginner-level play, starting with the awkward 2. Qd3 and repeatedly missing simple tactical combinations, allowing 20...Ne3+ winning a bishop on f1 and later blundering an entire rook via 28...Ng4 and 29...Nf2. In contrast, Game B White shows advanced positional and tactical understanding in a theoretical Slav Defense, executing a sharp queen-sac sequence (23. dxc5 Nxc5 24. Rd1 Nxb7 25. Qxc8+!) to reach a favorable endgame, which was then converted into a theoretically precise draw. Target B is far stronger.

Reported P(A stronger): 0.02. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Target A (White) played a highly irregular, beginner-level opening with 2. Qd3 and later completely collapsed tactically, blundering a knight fork (19. a3 Nc2+) and a full rook (29. exd5 Nf2). In contrast, Target B (White) navigated a theoretical Slav Defense (6. Nb1, 7. Bxc4) with deep positional understanding, solid piece coordination, and competent endgame technique to secure a draw in a pawn endgame. Target B's play demonstrates a vastly higher level of tactical and strategic maturity.

Reported P(A stronger): 0.02. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

Game A White shows low-level play, starting with the bizarre 2. Qd3, and subsequently blundering a full piece with 21. Kd2? allowing 21... Nxf1+. In contrast, Game B White exhibits sophisticated positional understanding in a main-line Slav Defense, managing complex queenless middlegame maneuvering (e.g., 14. Nc4, 18. N3a5) and demonstrating precise endgame technique to secure a draw. Game B White is vastly superior in tactical awareness and strategic planning.

Reported P(A stronger): 0.02. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Game A White opens with the dubious 2. Qd3 and demonstrates poor tactical awareness throughout the game, allowing Black to win a rook via ...Nf2 (moves 28-30) after previously missing multiple knight forks and hanging pawns. In contrast, Game B White exhibits sound positional understanding in the Slav Defense (4. e3, 5. a4), executes clean tactical simplifications (23. dxc5 through 27. Nxb7), and plays a technically solid knight and pawn endgame leading into a straightforward drawn king-and-pawn finish. Player B demonstrates much higher tactical and endgame competence.

Reported P(A stronger): 0.03. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

Game A White blunders the opening with 2.Qd3, loses material through 5.Qxd5/7.Qxb7, and never recovers. Game B White plays a solid Slav (5.a4, 7.Bxc4, 11.Nb3), reaches an equal middlegame, and navigates the complex knight maneuvers (15.Nxa5, 17.Rxa4, 25.Qxc8+) into a drawn king-and-pawn ending with accurate opposition technique. B's opening knowledge and endgame precision far exceed A's.

Reported P(A stronger): 0.22. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A White played 2.Qd3 and 5.Qxd5?? 6.fxe5?? 7.Qxb7?? grabbing pawns while undeveloped, then lost the whole queenside and later a rook to knight forks; endgame technique was aimless (Nf3/Ng5 shuffling) and the opponent also blundered badly (e.g. 8.Qxb4 trade allowed, missed wins), suggesting a low-rated pairing. Game B White handled a Slav structure competently, won material with 15.Ncxa5 and 16.Nb7, reached a favorable ending with 25.Qxc8+ 26.Rxd5 liquidation, and even when outplayed in the king-pawn ending found the drawing resource 49.e5 and held a defensive stalemate pattern. Game B White is clearly the stronger player.

Reported P(A stronger): 0.1. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Target A demonstrates a lack of basic opening principles and tactical awareness, playing 2. Qd3 and later hanging a piece to a simple sequence (20. Kd1 Ne3+ 21. Kd2 Nxf1+). The endgame play in Game A is erratic, with multiple blunders from both sides. In contrast, Target B shows a solid grasp of opening theory in a complex Slav line (3. Nc3 dxc4 4. e3 b5 5. a4). Target B handles a delicate positional struggle, accurately navigates a queen trade at move 25, and demonstrates precise knowledge of the drawn king-and-pawn endgame. The difference in technical proficiency between the two players is substantial.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Game A White demonstrates weak opening knowledge with 2. Qd3 and poor tactical vision, blundering material with 21. Kd2 (allowing 21...Nxf1+) and failing to see 29...Nf2 winning the rook on h1. In contrast, Game B White plays sound, standard opening theory in the Slav Defense (1. d4, 2. c4, 3. Nc3), shows good positional understanding, cleanly wins a pawn with 15. Ncxa5, and maintains strong tactical stability throughout the game into a classic drawn pawn endgame. Game B White exhibits vastly superior opening preparation, tactical awareness, and overall playing strength.

Reported P(A stronger): 0.05. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B White showed much stronger opening discipline and calculation: standard Slav development, then the forcing sequence 24.Rd1 Nxb7 25.Qxc8+ Nxc8 26.Rxd5 exd5 27.Nxb7, followed by purposeful simplification with 31.Bxb4 and 34.Bxc5. Game A White’s 2.Qd3 and repeated pawn-grabbing with 5.Qxd5 and 7.Qxb7 neglected development. More seriously, 19.a3? allowed 19...Nc2+ and the knight’s checking route to 21...Nxf1+, costing a rook and forcing prolonged king exposure. Although A resisted for many moves, B’s play was substantially more coherent and tactically controlled.

Reported P(A stronger): 0.06. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s White developed and castled normally, then handled the tactical sequence 24.Rd1 Nxb7 25.Qxc8+ Nxc8 26.Rxd5 to reach a manageable endgame. Game A’s White spent the opening moving the queen with 2.Qd3, 5.Qxd5 and 7.Qxb7 while leaving pieces undeveloped, then allowed 19...Nc2+ and 21...Nxf1+ to cost material. B’s play is much stronger evidence of established rapid strength, despite the different results.

Reported P(A stronger): 0.04. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game B White shows substantially more coherent development, castling and coordination. The sequence 24.Rd1, 25.Qxc8+ and 26.Rxd5 navigates the tactical exchanges effectively; later, 53.Kf1–55.Kh1 secures the drawing corner against Black’s rook pawn. Game A White opens with queen excursions and f-pawn moves, then allows Black’s knight raid with 19.a3 and 20.Kd1, losing the f1 bishop. White also permits 29...Nf2 followed by 30...Nxh1, and simply hangs a knight with 39.Ne5+ Rxe5. Despite imperfect play in Game B, the gap in tactical reliability strongly favors B.

Reported P(A stronger): 0.98. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B shows substantially more controlled play: White develops naturally, castles, coordinates rooks, and executes the concrete sequence 25 Qxc8+ Nxc8 26 Rxd5 exd5, simplifying into a sound minor-piece ending. White then handles the endgame actively with 35 Ne5+ and 39 Nc5+. In Game A, White’s early queen raid (2 Qd3, 5 Qxd5, 7 Qxb7) is speculative, and the resulting play is poorly coordinated: the rook is lost on 15...Kxd8, another rook is left vulnerable to 29...Nf2 and 30...Nxh1, and 39 Ne5+ allows 39...Rxe5. These concrete oversights strongly favor B.

Reported P(A stronger): 0.93. Contradicts choice; excluded from confidence diagnostics.

GPT-6 Astra: chose B — correct

B’s White shows much more coherent development and tactical calculation: 24.Rd1 followed by 25.Qxc8+ and 26.Rxd5 liquidates accurately into a balanced minor-piece ending. Later, 49.e5 helps eliminate Black’s central pawns, and White reaches a drawn rook-pawn ending. A’s White opens with premature queen adventures (2.Qd3, 5.Qxd5, 7.Qxb7), leaves the kingside undeveloped, and allows 19...Nc2+ followed by ...Ne3+ and ...Nxf1. Most tellingly, 39.Ne5+ simply hangs a knight to 39...Rxe5. Both opponents make mistakes, but B demonstrates substantially stronger coordination and tactical awareness.

Reported P(A stronger): 0.97. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — correct

Game B's White plays a coherent Semi-Slav setup (Nc3, Bxc4, regulated queenside maneuvers like 15.Nxa5 and 19.Qc2), trades into a safe rook endgame and holds a draw with accurate king play (46.Kd2, 57.Kh2). Game A's White opens with the weak 2.Qd3, loses material via 5.Qxd5 and the forced 7.Qxb7, and needs Black's own errors to reach an endgame, which White then misplayes (44.a4, 45.Kd5 letting the king through) and loses. B's technique and sound opening play indicate a clearly higher rating.

Reported P(A stronger): 0.15. Consistent with choice.

Pair 14: 0 of 16 models wrong

A: Black, rated 2641 · B: White, rated 1249. Higher pre-game rating: A.

Game A: Black, rated 2641 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 c5 2. Nf3 e6 3. b3 a6 4. g3 Nc6 5. Bg2 d5 6. exd5 exd5 7. O-O Nf6 8. d4
cxd4 9. Bb2 Bc5 10. Re1+ Kf8 11. Nxd4 h5 12. Nxc6 bxc6 13. Bd4 Bd6 14. c4 h4
15. Nc3 hxg3 16. hxg3 Bh3 17. Bxh3 Rxh3 18. Kg2 Rh6 19. Rh1 Rxh1 20. Qxh1 Ng8
21. cxd5 c5 22. Be3 Be5 23. Rc1 Bd4 24. Qe1 Nf6 25. f3 Qd7 26. Qd2 Bxc3 27.
Bxc5+ Kg8 28. Qxc3 Nxd5 29. Qd4 Re8 30. Rd1 Re2+ 31. Kf1 Rh2 32. Qxd5 Qh3+ 33.
Ke1 Qxg3+ 0-1
Game B: White, rated 1249 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 d6 2. Bc4 e6 3. Nf3 h6 4. e5 c6 5. exd6 Bxd6 6. O-O Be7 7. d4 Nf6 8. Nh4
O-O 9. Qf3 Nd5 10. Bxd5 Bxh4 11. Bc4 Bg5 12. Bxg5 Qxg5 13. Nc3 Re8 14. Ne4 Qf5
15. Qg3 Rd8 16. Bd3 Na6 17. Nd6 Qf6 18. Ne4 Qg6 19. Qe3 f5 20. Nc5 Nxc5 21.
dxc5 Rd5 22. c4 Rd8 23. Rae1 Bd7 24. f4 Re8 25. Rf3 Qf6 26. b3 Rad8 27. Rg3 Bc8
28. Qe5 Qxe5 29. fxe5 Rd4 30. Rg6 Rxd3 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Game A Black played a coherent game. After the slightly odd 10...Kf8, the plan of ...h5-h4 opened the h-file, and 15...hxg3 and 16...Bh3 traded off White's key bishop. Black finished tactically with 30...Re2+, 31...Rh2 and 33...Qxg3+. Game B White blundered badly. 8.Nh4 followed by 9.Qf3 let 9...Nd5 and 10...Bxh4 win a knight, so 10.Bxd5 simply lost a piece. Later 30.Rg6?? left the d3 bishop hanging to 30...Rxd3. These are basic tactical oversights, while Black in Game A showed better understanding and calculation.

Reported P(A stronger): 0.78. Consistent with choice.

Claude Sonnet 5.5: chose A — correct

Game A, Black: answered the Sicilian setup well with ...d5 and ...Nf6, and took the h-file with ...h5-h4 and ...hxg3. Black won with 26...Bxc3 and 28...Nxd5, and finished with ...Qh3+ and ...Qxg3+. ...Kf8 and ...h5 were a bit loose, but White's play was also weak (Bxc5+ and Qxd5 did not hold). Game B, White: opened with 2.Bc4 and 4.e5, and the 8.Nh4 and 9.Qf3 plan lost a piece after 10...Bxh4. Black's play was fairly accurate there, but White's blunders were basic. White also dropped a bishop with 30.Rg6 Rxd3. Both games show club-level play, but White in B made the larger errors, so target A looks somewhat stronger.

Reported P(A stronger): 0.66. Consistent with choice.

Gemini 3.8 Flash: chose A — correct

Player A (Black) conducted an energetic, purposeful kingside attack featuring early dynamic play with ...h5-h4. When White miscalculated with 30. Rd1, Black punished it crisply with 30...Re2+! followed by 31...Rh2! and 32...Qh3+, forcing resignation due to unstoppable mate. In contrast, Player B (White) displayed weak positional understanding with moves like 8. Nh4, outright hung a knight on e4 at move 15 (which Black missed), and eventually resigned after blatantly hanging the d3-bishop in one move with 30. Rg6??. Target A demonstrates far superior board vision and tactical sharpness.

Reported P(A stronger): 0.83. Consistent with choice.

Gemini 3.1 Pro (Preview): chose A — correct

Black in Game A demonstrated strong tactical awareness and attacking play, launching a successful kingside attack with moves like 11...h5 and finishing with the crushing sequence starting with 31...Rh2!. In contrast, White in Game B showed a lack of basic board awareness, hanging a piece in the opening with 10. Bxd5 Bxh4 (leaving the h4 knight unprotected) and blundering a bishop at the end with 30. Rg6 Rxd3. Target A is clearly the stronger player.

Reported P(A stronger): 0.95. Consistent with choice.

Gemini 3.5 Flash: chose A — correct

In Game A, Black demonstrated high-level play, employing a strong kingside attack (11...h5, 14...h4, 16...Bh3) and maintaining excellent piece activity despite losing castling rights. Black's tactical execution was sharp, culminating in a decisive mating net (29...Re8, 30...Re2+, and 32...Qh3+). In contrast, Game B White showed poor tactical awareness, misplacing a knight with 8. Nh4, which allowed 10...Bxh4, and later hanging a bishop with 30. Rg6?? Rxd3. Black in Game A is clearly the much stronger player.

Reported P(A stronger): 0.95. Consistent with choice.

Gemini 3.7 Flash: chose A — correct

Game A Black displayed strong tactical and positional awareness, conducting a purposeful kingside assault starting with 11...h5, coordinating pieces effectively, and executing a decisive mating sequence with 30...Re2+ and 31...Rh2. In contrast, Game B White played passively and aimlessly (e.g., 8. Nh4, allowing 15...Qxe4 which Black missed), before abruptly blundering a full minor piece with 30. Rg6?? Rxd3. Target A demonstrated much higher tactical vigilance and overall strategic cohesion.

Reported P(A stronger): 0.82. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

Game A Black shows coherent Sicilian plans: ...h5-hxg3 opening the h-file, the strong 20...Ng8 redeployment, and the clean finish 30...Re2+, 32...Qh3+, 33...Qxg3+. Errors are modest (10...Kf8). Game B White plays dubious moves: 8.Nh4 wasting time, the unnecessary 10.Bxd5 giving up the bishop pair, awkward 17.Nd6, passive 21.c4, and the losing tactical oversight allowing 30...Rxd3. White's positional drift and final collapse indicate noticeably weaker play than Black's purposeful, accurate attacking execution.

Reported P(A stronger): 0.72. Consistent with choice.

Claude Fable 5.1: chose A — correct

Game A Black played a reasonable Sicilian setup, handled 10.Re1+ Kf8 by going for a kingside attack (11...h5, 14...h4, 16...Bh3) and converted tactically with 29...Re8, 30...Re2+ and 32...Qh3+ winning material, though 20...Ng8 and 22...Be5 were loose. Game B White's play was weaker: 2.Bc4 and 4.e5 sideline, 8.Nh4?/9.Qf3 lost a piece to 9...Nd5 10.Bxd5 Bxh4, then aimless knight shuffling (17.Nd6 18.Ne4) and finally 30.Rg6?? dropping the bishop to 30...Rxd3. Opponents in both games were modest, but A's target showed better tactical awareness.

Reported P(A stronger): 0.6. Consistent with choice.

Gemini 3 Flash (Preview): chose A — correct

Player A (Black) demonstrates sophisticated tactical and attacking skills in a sharp Sicilian. Navigating a king-in-center position after 10...Kf8, they correctly prioritized a kingside initiative (11...h5, 14...h4). The concluding sequence (30...Re2+ to 33...Qxg3+) shows precise calculation. In contrast, Player B (White) played a more aimless middlegame, exemplified by the unnecessary 10.Bxd5. Crucially, Player B committed a primitive tactical error with 30.Rg6, simply leaving the d3-bishop en prise to 30...Rxd3. The difference in tactical alertness and strategic depth between the two players is significant.

Reported P(A stronger): 0.85. Consistent with choice.

Gemini 3.6 Flash: chose A — correct

Game A Black played a remarkably sharp and accurate tactical game despite losing castling rights early (10...Kf8). Black organized a powerful Kingside attack with 14...h4, 16...Bh3, and finished with a decisive mating sequence involving 30...Re2+, 31...Rh2, and 33...Qxg3+. In contrast, Game B White displayed poor opening understanding with 8. Nh4, allowing Black to win a piece for nothing with 9...Nd5 10. Bxd5 Bxh4. Game B White later blundered a full piece outright with 30. Rg6 Rxd3. Game A Black demonstrated far superior tactical alertness and calculation.

Reported P(A stronger): 0.85. Consistent with choice.

GPT-5.6 Sol: chose A — correct

Game A Black showed coherent opening play and strong tactical conversion: 29...Re8, 30...Re2+ and 31...Rh2 created a decisive attack, while 32...Qh3+ punished White’s greedy 32.Qxd5. Game B White used several inefficient moves, notably 8.Nh4 and repeated queen/knight maneuvers, then apparently resigned after 30...Rxd3 despite the immediate 31.Rxe8+, which wins Black’s hanging rook with check. Missing that elementary resource at the game’s conclusion is much stronger evidence of lower established strength than Game A Black’s occasional awkwardness such as 20...Ng8.

Reported P(A stronger): 0.9. Consistent with choice.

GPT-6 Sol: chose A — correct

Game A’s Black coordinated a kingside attack with 14...h4, 16...Bh3, and the decisive 30...Re2+ 31...Rh2 32...Qh3+. Game B’s White developed some active ideas, but after 29...Rd4 attacked the bishop on d3, 30.Rg6 allowed 30...Rxd3, losing it. A’s sustained tactical play is stronger evidence of established strength, despite the risk of 10...Kf8.

Reported P(A stronger): 0.91. Consistent with choice.

GPT-6.1 Sol: chose A — correct

Game A Black showed more coherent tactical play: 14...h4 and 16...Bh3 opened attacking lines, while 29...Re8, 30...Re2+, and 31...Rh2 coordinated the remaining heavy pieces against White’s king. After 32.Qxd5, Black continued with forcing checks rather than abandoning the attack. Game B White developed reasonably and gained central space, but the knight excursion 17.Nd6–18.Ne4 achieved little. More importantly, 30.Rg6 overlooked the direct capture 30...Rxd3, losing a bishop without evident compensation. A’s sustained attacking coordination is stronger evidence than B’s otherwise plausible buildup.

Reported P(A stronger): 0.8. Consistent with choice.

GPT-5.6 Luna: chose A — correct

Game A’s black showed the more convincing tactical and strategic play. After the awkward-looking 10...Kf8, Black built a coherent kingside attack with ...h5, ...h4, ...Bh3, and the rook maneuvers ...Re2+ and ...Rh2. The final 32...Qh3+ and 33...Qxg3+ demonstrated accurate attacking calculation. Game B’s white played actively but less consistently: 8 Nh4 and 9 Qf3 allowed Black comfortable development, while 24 f4 and 27 Rg3 left the position vulnerable. After 28 Qe5 Qxe5 29 fxe5, White failed to prevent 30...Rxd3, losing the bishop and ending the game. Thus A’s target more likely had the higher established rapid strength.

Reported P(A stronger): 0.74. Consistent with choice.

GPT-6 Astra: chose A — correct

Game A Black showed coherent attacking play: 11...h5 and 14...h4 opened the h-file, while 29...Re8, 30...Re2+ and 31...Rh2 coordinated the heavy pieces against White’s king. After 32.Qxd5, Black exploited the exposed king with ...Qh3+ and ...Qxg3+. Game B White developed actively and found useful moves such as 17.Nd6, but 8.Nh4 spent time without a clear payoff, and 30.Rg6 simply left the bishop on d3 hanging to ...Rxd3. A’s play was not flawless and benefited from White’s mistakes, but its sustained coordination suggests greater established strength than B’s attack and elementary final oversight.

Reported P(A stronger): 0.85. Consistent with choice.

GLM-5.3 Flash: chose A — correct

Game A's Black played forcefully and accurately: 11...h5!, 14...h4! gaining attack, clean rook doubling (16...Bh3, 19...Rxh1), and a precise tactical finish 26...Bxc3, 30...Re2+ 31...Rh2, 32...Qxd5! winning material decisively. Game B's White erred early with 8.Nh4?! (allowing ...g5 ideas) and 9...Nd5 10...Bxh4 regained a piece, later drifting passively (17.Nd6/19.Ne4 shuffling) before 29...Rd4 and 30...Rxd3 collapsed tactically. Black's sharp, coherent attacking play and conversion in A exceeds White's solid but blunder-prone effort in B.

Reported P(A stronger): 0.75. Consistent with choice.

Pair 04: 0 of 16 models wrong

A: Black, rated 557 · B: Black, rated 1791. Higher pre-game rating: B.

Game A: Black, rated 557 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 Nf6 2. Qf3 e5 3. g4 Nc6 4. g5 Nd4 5. Qd3 Ng4 6. h3 Nxf2 7. Kxf2 h6 8.
gxh6 Qf6+ 9. Ke3 Bc5 10. hxg7 Qxg7 11. c3 Nb5+ 12. Ke2 Qf6 13. Qxb5 Qf2+ 14.
Kd1 c6 15. Qc4 d5 16. exd5 cxd5 17. Qxc5 Bf5 18. Qxd5 Ke7 19. Bb5 Kf6 20. Ne2
Qf4 21. Qxb7 Qg5 22. Rg1 Qxg1+ 23. Nxg1 1-0
Game B: Black, rated 1791 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. Nc3 e5 2. d3 d5 3. e3 Bb4 4. Bd2 Nf6 5. g3 O-O 6. Bg2 c6 7. Nf3 Bg4 8. O-O
Bxc3 9. Bxc3 e4 10. dxe4 Nxe4 11. Bb4 Re8 12. Qd4 Be6 13. Rad1 Nd7 14. Rfe1 c5
15. Bxc5 Ndxc5 16. Ne5 Qb6 17. Bxe4 Nxe4 18. Qxb6 axb6 19. a3 f6 20. Nf3 Bg4
21. Kg2 Ng5 22. Rd4 Bxf3+ 23. Kg1 Be4 24. h4 Nf3+ 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A Black played poorly: 6...Nxf2?! was a dubious sacrifice, and 11...Nb5+? simply hung the knight to 13.Qxb5. Black then dropped more material and finished with 22...Qxg1+??, giving away the queen. The opponent's 2.Qf3 and 3.g4 suggest low-level play. Game B Black played a sound opening (...e5, ...d5, ...Bb4, ...O-O) and won with tactics: 15...Ndxc5 recovered material, and 22...Bxf3+ followed by 24...Nf3+ won the exchange and more. The opponent did blunder with 15.Bxc5 and 22.Rd4, but Black's play was coherent and accurate. B looks clearly stronger.

Reported P(A stronger): 0.15. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A Black met 2.Qf3 with decent development, but 5...Ng4 and 6...Nxf2 was an unsound sacrifice. Black followed with 9...Bc5 and 13...Qf2+, yet lost material. Black then blundered with 20...Qf4 and 22...Qxg1+, which gave up the queen, so the game was messy and error-prone. Game B Black played a solid opening with 3...Bb4, 5...O-O and 7...Bg4. The play was coherent: 9...e4 and 10...Nxe4 won a pawn, 14...c5 was a good central break, and 22...Bxf3+ followed by 24...Nf3+ finished cleanly. White in Game B was passive. Black in B appears stronger and more consistent.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game B's Black played sound, disciplined chess, exploiting the pin on f3 with 9...e4 and winning material cleanly with the 14...c5 fork. Black then executed a completely winning endgame sequence with 20...Bg4 and 21...Ng5 to force White's resignation. In contrast, Game A's Black played an unsound, chaotic attack (6...Nxf2), missed capturing White's queen on move 17 (17...Bf5 instead of 17...Qxc5), and blundered away their queen multiple times (20...Qf4 and 22...Qxg1+), showing very low tactical awareness.

Reported P(A stronger): 0.08. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Black in Game B demonstrates solid, principled play, controlling the center, developing logically, and executing strong tactics such as the 14... c5 fork and the game-winning 24... Nf3+. In stark contrast, Black in Game A plays recklessly and blunders multiple pieces outright, such as hanging a knight with 11... Nb5+ (13. Qxb5) and a bishop with 16... cxd5 (17. Qxc5), before eventually blundering their queen as well. Target B is clearly a much more experienced and competent player.

Reported P(A stronger): 0.02. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

Game B Black demonstrates solid positional play, sound development, and sharp tactical awareness. Black exploits a pin with 7... Bg4, spots the tactical piece win 14... c5, and cleanly converts the advantage, finishing with the fork 24... Nf3+. In contrast, Game A Black plays an chaotic game, starting with an unsound sacrifice 6... Nxf2, followed by numerous positional errors, and concluding with a catastrophic blunder of the queen with 22... Qxg1+.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Target A plays a chaotic, low-accuracy game characterized by unsound piece sacrifices (6...Nxf2), huge tactical oversights (failing to capture White's Queen on move 17 with 17...Bf5 instead of 17...Qxc5), and ultimately blundering the Queen entirely (22...Qxg1+). In contrast, Target B demonstrates solid opening principles, spots a clean tactical fork to win material (14...c5!), and smoothly converts the advantage with accurate tactical strikes (21...Ng5 pinning the knight and 24...Nf3+ winning the rook). Target B shows vastly superior tactical vision and positional understanding.

Reported P(A stronger): 0.05. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

Game A Black (target) collapses tactically: after 5...Ng4 6.h3 Nxf2 7.Kxf2, Black plays 8...Qf6+ 9.Ke3, then later loses material with 11...Nb5+ 12.Ke2 Qf6 13.Qxb5, dropping a knight for nothing. Black's king safety and coordination are poor throughout, and White wins comfortably. Game B Black plays solidly: ...e4 9...Nxe4 gains a central foothold, ...c5 15...Ndxc5 regains the pawn, and the endgame technique is clean—21...Ng5, 22...Bxf3+ 23...Be4, 24...Nf3+ forcing mate threats. Black's play is coherent and decisive, indicating a higher established strength than Game A Black.

Reported P(A stronger): 0.3. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A Black played a speculative Alekhine against a weak Qf3/g4 setup but then sacrificed unsoundly (6...Nxf2, 11...Nb5+) and hung the bishop and knight, then blundered the queen with 22...Qxg1+ into a hopeless position; the whole game shows poor calculation and piece-hanging. Game B Black played coherent developing moves (3...Bb4, 8...Bxc3, 9...e4), kept pieces coordinated, exploited White's errors (15...Ndxc5, 17...Nxe4) and finished with a clean tactical sequence 21...Ng5, 22...Bxf3+, 24...Nf3+ winning material. Game B's opponent was also weak, but Black B's play is markedly more sound than Black A's.

Reported P(A stronger): 0.2. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Game A Black displays extremely low-level play, losing two minor pieces to simple captures (13. Qxb5 and 17. Qxc5) and hanging their queen with 22... Qxg1+. In contrast, Game B Black follows sound opening principles (2... d5, 5... O-O) and demonstrates tactical competence. Black exploits the pin on the f3 knight with 9... e4 and 21... Ng5, wins a piece with the pawn fork 14... c5, and finishes the game with the fork 24... Nf3+. Game B Black's play is consistent with a much higher rating, whereas Game A Black lacks basic board vision and coordination.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Game A Black played wildly and inaccurately, sacrificing material unsoundly with 6...Nxf2, missing basic tactical defenses, and ultimately blundering a queen outright with 22...Qxg1+. In contrast, Game B Black demonstrated solid positional understanding, tactical awareness, and smooth execution. Game B Black effectively capitalized on White's passivity, finding the winning tactic 14...c5 to pick up a piece, followed by a crisp sequence with 20...Bg4, 21...Ng5, and 24...Nf3+ to force resignation. Game B Black clearly displays a much higher established rating.

Reported P(A stronger): 0.05. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B’s Black played a coherent, controlled game: 5...O-O completed king safety, 9...e4 and 14...c5 challenged the center, and 15...Ndxc5 correctly recovered the bishop. After the queen trade, 20...Bg4, 21...Ng5, 22...Bxf3+, and 24...Nf3+ formed a purposeful winning sequence. Game A’s Black instead chose the highly speculative 6...Nxf2, then lost the knight after 11...Nb5+ 13.Qxb5 and a bishop after 15...d5 17.Qxc5. The final 22...Qxg1+ simply allowed 23.Nxg1. B showed substantially stronger coordination, calculation, and risk control.

Reported P(A stronger): 0.04. Consistent with choice.

GPT-6 Sol: chose B — correct

Game A’s Black played ambitiously, but 6...Nxf2 sacrificed a knight without sufficient compensation. After 11...Nb5+ 12.Ke2 Qf6, White simply took that knight with 13.Qxb5; Black then also allowed 17.Qxc5. Game B’s Black played much more coherently: 14...c5 challenged White’s center, 15...Ndxc5 recaptured the bishop, and 20...Bg4 followed by 22...Bxf3+ and 24...Nf3+ showed coordinated tactics.

Reported P(A stronger): 0.01. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game B Black develops coherently with ...O-O, ...Re8 and ...Nd7, then exploits White’s mistakes: 14...c5 15.Bxc5 Ndxc5 wins a bishop, while 22...Bxf3+ wins another piece and 24...Nf3+ forks the king and both rooks. Game A Black attacks an exposed king but repeatedly sacrifices material without sufficient compensation: 6...Nxf2, 11...Nb5+ allowing Qxb5, and ultimately 22...Qxg1+ allowing Nxg1, exchanging the queen for a rook. Although both opponents offer opportunities, B shows much stronger development, tactical execution and material judgment.

Reported P(A stronger): 0.97. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B’s Black played a coherent, sound game: rapid development with ...O-O, ...Re8, and ...c5, then accurately recovered the sacrificed bishop after 15.Bxc5 with 15...Ndxc5. The sequence 16...Qb6 and 18...axb6 led to a favorable endgame, and 20...Bg4, 22...Bxf3+, and 24...Nf3+ showed consistent tactical control. Game A’s Black chose a dubious tactical line with 2...e5 and 3...Nc6, allowed the king to be driven around after 7...h6 and 8...Qf6+, and then lost substantial material to 13.Qxb5, 17.Qxc5, and 18.Qxd5. Although White in Game A also played poorly, Black’s handling was much less reliable.

Reported P(A stronger): 0.08. Consistent with choice.

GPT-6 Astra: chose B — correct

Game B Black shows coherent development and tactical awareness: 14...c5 attacks White’s queen and bishop, 15...Ndxc5 collects the bishop, and 19...f6 gains time against the knight. The finish 22...Bxf3+ followed by 23...Be4 and 24...Nf3+ creates a king-and-rooks fork. Game A Black attacks White’s exposed king but repeatedly sacrifices material without adequate compensation: 6...Nxf2, 11...Nb5+ allowing Qxb5, and ultimately 22...Qxg1+ surrendering the queen for a rook. Despite Game A White’s highly vulnerable opening, Black cannot exploit it and displays much weaker material judgment.

Reported P(A stronger): 0.99. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — correct

Game A's Black found the strong 5...Ng4 and 6...Nxf2 sacrifice, but then erred badly: 12...Qf6 allowed 13.Qxb5, and after winning material Black drifted (19...Kf6, 20...Qf4, 21...Qg5) and lost the queen with 21...Qg5 22.Rg1 Qxg1+??, resigning from a winning position. Game B's Black played coherently throughout: logical development, 9...e4 seizing the center, accurate tactics 16...Qb6 forcing exchanges, and a clean conversion ending with 22...Bxf3+ 23...Be4 and 24...Nf3+ winning the knight. Consistent sound play and a won-game finish suggest B is the stronger player.

Reported P(A stronger): 0.3. Consistent with choice.

Pair 18: 0 of 16 models wrong

A: Black, rated 765 · B: White, rated 1877. Higher pre-game rating: B.

Game A: Black, rated 765 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 e5 2. Nf3 d6 3. Nc3 c6 4. Bc4 b5 5. Bb3 a5 6. a4 b4 7. Ne2 f6 8. Ng3 h5
9. d3 g5 10. Nd2 h4 11. Nh5 c5 12. Nc4 Nc6 13. O-O Nd4 14. Ba2 Be6 15. g3 Qd7
16. gxh4 Bg4 17. Nxf6+ Nxf6 18. Qe1 gxh4 19. Bg5 Nh7 20. Nb6 Qc6 21. Nxa8 Qxa8
22. Bd5 Qd8 23. Bxd8 1-0
Game B: White, rated 1877 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 g6 2. Nf3 Bg7 3. e3 d6 4. Bd3 Nc6 5. a3 Bg4 6. Nbd2 e5 7. h3 Bxf3 8. Nxf3
exd4 9. exd4 Qe7+ 10. Be3 Nxd4 11. Nxd4 Bxd4 12. Bb5+ c6 13. Qxd4 Nf6 14. Bd3
O-O-O 15. O-O c5 16. Qa4 Kb8 17. b4 Nd5 18. Bd2 Nb6 19. Qb3 f5 20. bxc5 dxc5
21. a4 c4 22. Bf4+ Ka8 23. Bxc4 Rd4 24. Rfe1 Qc5 25. Be5 Rd2 26. Re3 Nxc4 27.
Bxh8 Nxe3 28. Qg8+ 1-0

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A Black plays a chaotic opening with many flank pawn moves (4...b5, 5...a5, 8...h5, 9...g5), which leaves the kingside weak. Black then allows 17.Nxf6+ and 20.Nb6 forking the rook. The decisive error is 22...Qd8??, which simply loses the queen to 23.Bxd8. Game B White is imprecise early: 10.Be3 lets Black win the d4 pawn. Afterward, though, White plays coherently. 12.Bb5+ forces 12...c6, then 13.Qxd4 regains the piece, followed by queenside pressure with 17.b4 and 20.bxc5. White finishes with active tactics (22.Bf4+, 25.Be5, 27.Bxh8) and 28.Qg8+. Game B White looks stronger.

Reported P(A stronger): 0.25. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A Black played weakly: 2...d6 and 3...c6 with the pawn pushes 4...b5, 5...a5, 6...b4, 8...h5 and 9...g5 ignored development and weakened the king. 11...c5 and 13...Nd4 left material hanging, and 16...Bg4 allowed 17.Nxf6+. 20...Qc6 then 21...Qxa8 lost to 22.Bd5 and 23.Bxd8, a queen trap-style blunder. White in A was also loose but won tactically. Game B White played a sound opening (Bd3, Nbd2, Be3) and punished 7...Bxf3 and 10...Nxd4 with solid moves like 12.Bb5+, 13.Qxd4 and 17.b4. It then found 22.Bf4+, 23.Bxc4 and 25.Be5, picking up material and finishing with 28.Qg8+. B's opponent was also inaccurate, but White in B played cleanly, while A's Black blundered repeatedly. Black in A looks weaker than White in B.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Player B demonstrates far superior tactical vision and positional understanding compared to Player A. In Game B, White cleverly won material with 12. Bb5+ followed by 13. Qxd4, then converted with active piece play (22. Bf4+, 25. Be5, 28. Qg8#). In contrast, Player A pushed pawns recklessly on the flanks leaving their king completely vulnerable (7...f6, 8...h5, 9...g5), and finally blundered their queen outright in one move with 22...Qd8?? allowing 23. Bxd8.

Reported P(A stronger): 0.08. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Target A (Black in Game A) demonstrates extremely poor fundamentals, making numerous weakening pawn moves in the opening (7...f6, 8...h5, 9...g5) while neglecting development, and ultimately blunders their queen with 22...Qd8??. Target B (White in Game B) plays logically and demonstrates strong tactical awareness, such as recovering a piece with 12. Bb5+ and 13. Qxd4, creating attacking chances with 16. Qa4, exploiting a fork with 25. Be5, and finding a decisive back-rank combination with 28. Qg8+. Target B's play is vastly superior.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

Target B (Game B, White) demonstrated solid tactical awareness. They quickly capitalized on Black's mistake with the combination 12. Bb5+ and 13. Qxd4 to win a piece, and converted the game cleanly, finding the mating sequence starting with 28. Qg8+. Conversely, Target A (Game A, Black) exhibited poor board vision. They missed a simple, winning royal fork with 18...Nf3+ (which would have won White's queen) and later blundered their own queen in a single move with 22...Qd8??. Target B is clearly the stronger, more established player.

Reported P(A stronger): 0.1. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Target B (White in Game B) demonstrated solid tactical vision and strategic competence. After Black grabbed a central pawn with 10...Nxd4 and 11...Bxd4, White immediately punished the tactic with the intermediate check 12.Bb5+! winning a piece (13.Qxd4). White then converted the advantage cleanly, concluding with 28.Qg8+. In contrast, Target A (Black in Game A) played an opening with excessive weakening pawn pushes (7...f6, 8...h5, 9...g5, 11...c5) leaving the king vulnerable, missed multiple tactics, and ultimately hung the queen outright with 22...Qd8?? allowing 23.Bxd8.

Reported P(A stronger): 0.05. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

Game A target (Black) played bizarrely with 3...c6, 4...b5, 7...f6, and 8...h5, weakening the kingside and allowing White a winning attack; after 21.Nxa8 Black resigned at 23.Bxd8, showing weak strategic understanding. Game B target (White) played solidly with d4/Nf3/e3, handled Black's tactics calmly after 10...Nxd4, and converted with accurate moves like 25.Be5, 27.Bxh8, and the mating finish 28.Qg8+, demonstrating clearly superior technique and composure.

Reported P(A stronger): 0.12. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A Black played a chaotic, non-developing opening (3...c6, 4...b5, 5...a5, 7...f6, 8...h5, 9...g5) and then hung a knight with 16...Bg4 (17.Nxf6+), lost a rook to 20.Nb6 and finally 22...Qd8?? dropping the queen — very weak, beginner-level play. Game B White showed decent fundamentals: 7.h3 provoking Bxf3, recognizing 10.Be3 was fine after Nxd4 since 12.Bb5+ regained material, then exploited Black's loose king with 20.bxc5, 22.Bf4+, 25.Be5 and finished with 28.Qg8+ winning. White in B made some inaccuracies but clearly played coherent chess, far above A's target.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Target B (White) demonstrates superior tactical awareness and positional stability. In Game B, White calculates a favorable exchange sequence starting with 12. Bb5+ and finishes the game with a clean back-rank mate (28. Qg8+). Target A (Black), conversely, plays an extremely loose and weakening opening with 7...f6 and 9...g5, creating numerous holes in their position. Furthermore, Target A blunders their queen in a single move with 22...Qd8, allowing 23. Bxd8. While Target A shows some tactical aggression (16...Bg4), their lack of basic safety and the catastrophic blunder at the end suggest a lower established rating compared to Target B's coordinated play.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Game A's target (Black) played an excessively dubious opening, severely weakening the kingside with 7...f6, 8...h5, and 9...g5 while leaving the king uncastled. Black capped off a chaotic performance by hanging their queen to a simple bishop attack with 22...Qd8 23.Bxd8. Conversely, Game B's target (White) displayed strong positional awareness and sharp tactical execution. White utilized neat tactical finesses such as 12.Bb5+ c6 13.Qxd4 to regain material, launched a decisive queenside expansion with 17.b4 and 21.a4, and spotted the powerful tactical strike 25.Be5, converting smoothly into a back-rank mate with 28.Qg8+.

Reported P(A stronger): 0.08. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game A Black showed repeated strategic and tactical weakness: 7...f6, 8...h5, and 9...g5 badly loosened the position, while 20...Qc6 allowed 21.Nxa8 to win the rook. Most decisively, 22...Qd8 simply placed the queen where White’s bishop could take it with 23.Bxd8. Game B White played much more coherently and tactically: 12.Bb5+ was a strong zwischenzug enabling 13.Qxd4, the queenside expansion with 17.b4 was purposeful, and 25.Be5 followed by 28.Qg8+ demonstrated awareness of the mating attack despite material complications.

Reported P(A stronger): 0.05. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s White played a coherent opening and found tactical resources: 12.Bb5+ followed by 13.Qxd4 recovered the bishop, while 22.Bf4+ and 27.Bxh8 led to a decisive attack. Game A’s Black pushed several kingside pawns without developing or securing the king, then allowed 20.Nb6 and 21.Nxa8 to lose a rook. Black also missed 22.Bd5, which attacked the queen and led to 23.Bxd8. Despite some imperfect play by B’s White, B shows substantially stronger tactical awareness.

Reported P(A stronger): 0.02. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game A Black neglects development and king safety with 7...f6, 8...h5 and 9...g5. Although 13...Nd4 shows some positional awareness, Black allows the rook fork 20.Nb6 and then simply hangs the queen with 22...Qd8 23.Bxd8. Game B White demonstrates substantially better tactical coordination: 12.Bb5+ followed by 13.Qxd4 recovers material, and 22.Bf4+ gains time before 23.Bxc4. Most impressively, 27.Bxh8 allows 27...Nxe3 because 28.Qg8+ forces mate after any interposition. B's coherent development and concrete calculation strongly outweigh A's play.

Reported P(A stronger): 0.03. Consistent with choice.

GPT-5.6 Luna: chose B — correct

Game B’s White showed substantially better tactical awareness and coordination. White correctly exploited the sequence 10...Nxd4 11 Nxd4 Bxd4 13 Qxd4, emerging with material advantage, then developed a coherent attack with 17 b4, 20 bxc5, and 22 Bf4+. The decisive continuation 23 Bxc4, 24 Rfe1, 25 Be5, and 27 Bxh8 demonstrated effective calculation. Game A’s Black played an unsound-looking kingside expansion with 7...f6, 8...h5, and 9...g5, then missed the simple tactical threat after 20 Nb6: 21 Nxa8 won the rook, and 22...Qd8 allowed 23 Bxd8 to take the queen. Thus B is much more likely the stronger established player.

Reported P(A stronger): 0.08. Consistent with choice.

GPT-6 Astra: chose B — correct

Game A Black spends many opening moves on pawns (7...f6, 8...h5, 9...g5), leaving the king exposed and development neglected. Black then allows 20.Nb6, forking queen and rook, and outright hangs the queen with 22...Qd8 23.Bxd8. Game B White develops coherently and finds 12.Bb5+ followed by 13.Qxd4, exploiting Black’s capture sequence to win material. White also coordinates an attack against the castled king with 17.b4 and 22.Bf4+, finishing with the decisive 28.Qg8+. Despite some loose play, B shows substantially stronger tactical awareness and positional organization.

Reported P(A stronger): 0.98. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — correct

Game B's White plays a clean strategic-tactical game: punishing ...Nd5? (17...Nb6 retreat, 26.Re3-xe3 fork trick winning material), building pressure with a4-b4, and finishing with 28.Qg8+ mate. Black's errors were punished efficiently. Game A's Black played weakly throughout: the bizarre 4...b5/6...b4, 9...g5 and 10...h4 pawn storms against his own king, allowing 17.Nxf6+ and losing material (18...gxh4, then the queen-trap 22...Qd8 losing the queen to 23.Bxd8). Black A's play was unsound beginner-level aggression; White B showed solid technique. Player B is clearly the stronger.

Reported P(A stronger): 0.12. Consistent with choice.

Pair 15: 0 of 16 models wrong

A: White, rated 1120 · B: White, rated 1932. Higher pre-game rating: B.

Game A: White, rated 1120 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 c5 2. Bc4 e6 3. Nf3 a6 4. c3 b5 5. Bd3 c4 6. Be2 Bb7 7. d3 cxd3 8. Bxd3
Nf6 9. Nbd2 Be7 10. e5 Nd5 11. Be4 f5 12. Bd3 O-O 13. b3 Nc6 14. Bb2 Qc7 15. c4
Nf4 16. Be4 Nxg2+ 17. Kf1 fxe4 18. Nxe4 Nf4 19. Nfg5 h6 20. Nf3 Nb4 21. a3 Nbd3
22. Bc3 Bxe4 23. Nd4 Bxh1 24. Kg1 Qb7 25. f3 Nh3+ 26. Kf1 Bxf3 27. Qxd3 Be4+
28. Nf3 Bxd3+ 29. Kg2 Qxf3# 0-1
Game B: White, rated 1932 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 g6 2. Nc3 Bg7 3. d4 d6 4. Be3 Nf6 5. Qd2 O-O 6. f3 Re8 7. Bh6 Bh8 8.
O-O-O Nbd7 9. g4 c5 10. d5 a6 11. h4 b5 12. h5 b4 13. hxg6 fxg6 14. Nce2 Ne5
15. Ng3 Nf7 16. g5 Nd7 17. f4 Nxh6 18. Rxh6 Nf8 19. Bc4 a5 20. Nf3 Ba6 21. Bxa6
Rxa6 22. Rdh1 a4 23. Qh2 Bg7 24. f5 b3 25. a3 Kf7 26. Rxh7 Nxh7 27. Qxh7 Rh8
28. fxg6+ Kf8 29. Qxh8+ Bxh8 30. Rxh8+ Kg7 31. Rxd8 Kxg6 32. Rd7 Kf7 33. Nf5
bxc2 34. Rxe7+ Kf8 35. g6 Ra8 36. Ng5 c4 37. g7+ Kg8 38. Nh6# 1-0

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A White played a wandering opening (2.Bc4, 5.Bd3, 6.Be2, 11.Be4, 12.Bd3) and then blundered badly: 16.Be4?? allowed Nxg2+, and 22.Bc3/23.Nd4 dropped the h1 rook. This is a collapse to simple tactics. Game B White ran a coherent Pirc/150 attack: Be3, Qd2, f3, O-O-O, g4, h4, h5 and hxg6, then Rxh6. The 26.Rxh7 sacrifice and the precise finish ending in 38.Nh6# show real attacking skill. Black in B did misplay with 7...Bh8 and 23...Bg7, but B's play is still clearly more competent.

Reported P(A stronger): 0.22. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A White played a Bc4/c3 setup against the Sicilian, then went wrong with 10.e5 and 11.Be4, which allowed 11...f5. After 15.c4 Nf4 and 16.Be4?, Black's 16...Nxg2+ won material with a strong attack, and White never recovered. Overall that was a loss of control in the opening and middlegame, with White missing the tactics. Game B White played a coherent Pirc attack with Be3, Qd2, Bh6 and a pawn storm (g4, h4-h5). White found the 26.Rxh7 combination, then 29.Qxh8+ and the 33.Nf5 and 38.Nh6# finish, which converted accurately. Black's defence was weak, but White's play was more purposeful and tactically sound, so B looks stronger.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Game A White shows poor opening understanding, moving the light-squared bishop five times in the first twelve moves (Bc4-d3-e2-d3-e4-d3) before blundering tactically with 16. Be4?? (allowing 16...Nxg2+ and dropping decisive material). In contrast, Game B White executes a textbook Yugoslav-style kingside storm against the Pirc/Modern. Game B White methodically opens the h-file with 11. h4, 12. h5, and 22. Rdh1, followed by a crisp tactical breakthrough with 26. Rxh7! and 28. fxg6+, efficiently winning Black's queen and converting the ending directly into a forced checkmate.

Reported P(A stronger): 0.03. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Target B (White in Game B) demonstrated a much higher level of play than Target A. White in Game B executed a thematic and well-calculated kingside attack against the Pirc/Modern defense, complete with long castling, a pawn storm (h4-h5), and brilliant sacrifices like 26. Rxh7 and the queen sacrifice 29. Qxh8+, leading to a beautiful checkmate. In contrast, White in Game A struggled with basic opening principles (2. Bc4, 5. Bd3) and blundered a piece to a simple tactic (16. Be4?? Nxg2+), getting completely crushed. The difference in tactical and positional understanding is vast.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

Game A White shows a low level of play, making severe tactical blunders such as 16. Be4? (allowing 16... Nxg2+ and losing central control), 23. Nd4? (hanging the h1-rook), and 27. Qxd3?? (hanging the queen to a simple discovery). In contrast, Game B White exhibits advanced positional and attacking understanding in a Pirc/Modern structure, orchestrating a high-level kingside assault (7. Bh6, 11. h4, 22. Qh2). Game B White executes a beautiful, calculation-heavy combination starting with 25. Rxh7! and the queen sacrifice 28. Qxh8+! to transition into a winning endgame, finishing with an elegant mating net.

Reported P(A stronger): 0.01. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Player B demonstrates a high level of strategic and tactical competence against the Modern Defense. White in Game B executes a cohesive kingside attack (g4, h4-h5, Qh2, f4-f5), culminating in a precise queen sacrifice and checkmate (26. Rxh7! through 38. Nh6#). In contrast, Player A shows poor opening handling and low tactical awareness, repeatedly shuffling the light-squared bishop (Bc4-d3-e2-d3-e4), blundering a piece with 16. Be4? fxe4, and collapsing rapidly to a decisive mating attack on f3.

Reported P(A stronger): 0.04. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

In Game A, White plays passively and blunders: 16.Kf1 allows 16...Nxg2+ losing the h1-rook and king safety, and White's pieces are quickly overwhelmed (23...Bxh1). In Game B, White executes a coherent, aggressive plan (7.Bh6, 11.h5, 16.g5), keeps pressure, and delivers a precise mating finish (31.Rxd8, 37.g7+, 38.Nh6#). Game A's target is outplayed from the opening and loses badly, while Game B's target dominates with strong attacking technique. The evidence strongly favors B as the higher-rated player.

Reported P(A stronger): 0.08. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A White played an incoherent opening (2.Bc4, 5.Bd3, 6.Be2, 7.d3 losing tempi), then 15.c4?? and 16.Be4?? allowed 16...Nxg2+ and a collapsing position, finishing with 27.Qxd3 walking into mate by 29...Qxf3#. Game B White executed a thematic Pirc/150 Attack plan: 7.Bh6, 9.g4, 11.h4, 12.h5, 13.hxg6, then 17.f4 Nxh6 18.Rxh6, doubled rooks 22.Rdh1, and converted cleanly with 26.Rxh7 Nxh7 27.Qxh7 and a precise mating net ending in 38.Nh6#. Game B White showed clear opening knowledge and tactical calculation; Game A White showed neither.

Reported P(A stronger): 0.12. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Player A demonstrates poor opening fundamentals, wasting tempi with 2. Bc4 and 5. Bd3, and then commits a massive tactical blunder with 16. Be4?, losing a piece to 16... Nxg2+ and 17... fxe4. Player A's defense is non-existent, leading to a quick mate. In contrast, Player B plays a sophisticated, thematic kingside attack against the Pirc Defense. White correctly executes the h-pawn storm (11. h4), coordinates pieces well (14. Nce2), and finds a decisive exchange sacrifice (26. Rxh7!). Player B's ability to simplify into a won endgame (29. Qxh8+) and finish with a clean mate (38. Nh6#) indicates a significantly higher established playing strength.

Reported P(A stronger): 0.05. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Game A White played very weakly, opening with the passive 2. Bc4, misplacing pieces with 5. Bd3, and collapsing rapidly after tactical blunders like 13. b3 (hanging the c3 pawn) and 16. Be4?? (allowing 16...Nxg2+ and losing a piece). White in Game A offered no resistance and was mated on move 29. In contrast, Game B White demonstrated clear strategic understanding of the Yugoslav-style attack against the Modern Defense with 8. O-O-O, 11. h4, and 15. g5. Game B White conducted a powerful kingside storm, brilliantly sacrificed with 25. Rxh7! to dismantle Black's defense, and executed a clean tactical win ending in checkmate with 38. Nh6#.

Reported P(A stronger): 0.05. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B White showed a coherent attacking plan: opposite-side castling followed by g4, h4-h5, Rdh1 and Qh2, then calculated 26.Rxh7! Nxh7 27.Qxh7, converting into a decisive material attack and finishing accurately with 38.Nh6#. Game A White spent many tempi retreating the same bishop and overlooked elementary tactics: 16.Be4? allowed 16...Nxg2+ 17.Kf1 fxe4, losing the bishop, while 22.Bc3? Bxe4 and 23.Nd4? Bxh1 led to collapse. B’s target displayed substantially stronger planning and tactical awareness.

Reported P(A stronger): 0.06. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s White built a coherent kingside attack with 8.O-O-O, 11.h4 and 22.Rdh1, then calculated 26.Rxh7 through 31.Rxd8, winning Black’s queen. Game A’s White repeatedly retreated the bishop and lost it after 16.Be4 Nxg2+ 17.Kf1 fxe4; later, 27.Qxd3 allowed 27...Be4+ and the queen was lost. Despite the limited evidence of one game each, B’s play shows substantially stronger planning and tactical awareness.

Reported P(A stronger): 0.01. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game B White showed a coherent attacking plan: queenside castling, h-pawn pressure, 22.Rdh1 and 23.Qh2, followed by the forcing 26.Rxh7–31.Rxd8 sequence. White then coordinated both knights and the passed pawn with 33.Nf5, 36.Ng5 and 38.Nh6#. Game A White spent several opening tempi relocating the bishop (2.Bc4, 5.Bd3, 6.Be2), then 16.Be4 allowed ...Nxg2+ and ...fxe4, losing a bishop for pawns. Later, 22.Bc3 and 23.Nd4 permitted ...Bxe4 and ...Bxh1. Although A faced stronger resistance, B’s sustained coordination and tactical execution provide substantially stronger evidence of established playing strength.

Reported P(A stronger): 0.95. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B’s White showed a coherent attacking plan and strong tactical execution: the kingside expansion with 9 g4, 11 h4, and 12 h5 was followed by 18 Rxh6, 22 Rdh1, and the combination 26 Rxh7 Nxh7 27 Qxh7, winning decisive material. The finish 29 Qxh8+ Bxh8 30 Rxh8+ and 38 Nh6# was accurate. Game A’s White played an unfocused opening and missed major tactical threats: after 16...Nxg2+ and 17...fxe4, the attack collapsed, and 23 Nd4?? allowed ...Bxh1, losing a rook. White then permitted the forced mating sequence ending 29...Qxf3#.

Reported P(A stronger): 0.9. Contradicts choice; excluded from confidence diagnostics.

GPT-6 Astra: chose B — correct

Game B White shows coherent opening preparation and attacking coordination: queenside castling, the h-pawn advance, and 22.Rdh1 followed by 23.Qh2. The forcing sequence 26.Rxh7 through 31.Rxd8 converts the attack into material gain, while 35.g6 and 36.Ng5 create a precise mating finish. Game A White repeatedly spends tempi moving the same bishop, then allows 15...Nf4 and 16...Nxg2+, losing a bishop after ...fxe4. Later, 23.Nd4 permits ...Bxh1, and 27.Qxd3 allows the discovered check ...Be4+. Although A faced more effective opposition, B demonstrates substantially stronger planning and tactical awareness.

Reported P(A stronger): 0.98. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — correct

Game A's White plays weakly: 15.Be4 allows 15...Nxg2+ winning a piece (16.Kf1 fxe4 traps the bishop and 18.Nxe4 Nf4 keeps material), then 23.Nd4?? loses the exchange to 23...Bxh1. Black methodically converts, and White resigns mated — White looks ~1400-1600 level. Game B's White handles a sharp Saemisch KID: solid setup, timely 17.f4 and 13.hxg6, accurate attack culminating in 26.Rxh7! 27.Qxh7 Rh8 28.fxg6+ and clean queen sacrifice 29.Qxh8+, finishing with a model mate 38.Nh6#. B is clearly the stronger player.

Reported P(A stronger): 0.12. Consistent with choice.

Pair 06: 0 of 16 models wrong

A: Black, rated 1639 · B: Black, rated 838. Higher pre-game rating: A.

Game A: Black, rated 1639 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 e5 2. Bc4 Nf6 3. Nf3 Bc5 4. d3 O-O 5. O-O Nc6 6. a3 d5 7. exd5 Nxd5 8.
Nbd2 Bg4 9. Ne4 Bb6 10. h3 Bh5 11. g4 Bg6 12. Neg5 h6 13. Ne4 Nf4 14. Bxf4 exf4
15. Kg2 Qe7 16. Re1 Bxe4 17. Rxe4 Qf6 18. c3 g5 19. Bb3 Rfe8 20. Bc2 Rxe4 21.
dxe4 Rd8 22. Qe2 Ne5 23. Rd1 Rd6 24. Rd5 c6 1-0
Game B: Black, rated 838 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. d4 d5 2. e3 Bd7 3. Qf3 c6 4. g4 b5 5. h3 Na6 6. h4 Qc7 7. g5 Rd8 8. h5 c5 9.
c3 b4 10. b3 c4 11. bxc4 dxc4 12. cxb4 Nxb4 13. Nh3 Nc2+ 14. Kd2 Nxa1 15. Bb2
Qa5+ 16. Nc3 Bxh3 17. Rxh3 Qxg5 18. Bxa1 Qg1 19. Rh1 Qg5 20. Bxc4 Rxd4+ 21. Kc2
Rxc4 22. Rd1 Qc5 23. Qa8+ Qc8 24. Qd5 Rxc3+ 25. Bxc3 f6 26. f4 e6 27. Qb5+ Kf7
28. e4 e5 29. f5 Nh6 30. Rd7+ Be7 31. Qd5+ Ke8 32. Rxa7 Nxf5 33. Ra8 Bd8 34.
Rxc8 Ke7 35. Qb7+ Kd6 36. Qc6+ Ke7 37. Bb4+ Kf7 38. Qd5+ Ke8 39. Rxd8# 1-0

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Game A Black plays a coherent Italian with sensible developing moves (…Bc5, …O-O, …d5, …Bg4). The resignation at move 24 likely followed a pawn loss or tactical issue after 24.Rd5, but the play up to that point is solid and positional. Game B Black plays a chaotic opening: 2…Bd7, 3…c6, 4…b5, 5…Na6. Black does win the a1 rook, but then blunders. After 25.Bxc3, Black is down material, and Black later allows mate with 39.Rxd8#. Game B's opponent was also weak (3.Qf3, 4.g4). On balance, A's target shows more competent play.

Reported P(A stronger): 0.72. Consistent with choice.

Claude Sonnet 5.5: chose A — correct

Game A Black played a standard Italian setup with sensible development (4...O-O, 6...d5, 8...Bg4, 9...Bb6). Black traded pieces and kept a fairly solid position. The 12...h6 and 13...Nf4 moves were reasonable, though 18...g5 weakened the king and Black eventually lost slowly. Game B Black played a bizarre opening (2...Bd7, 3...c6, 4...b5, 5...Na6) and left the king exposed. Black did win material with tactics (13...Nc2+, 14...Nxa1, 16...Bxh3), but later blundered with 20...Rxd4+ and 21...Rxc4, then dropped the queen with 23...Qc8 and 24...Rxc3+. Overall Game B Black looks weaker and more erratic, so A is more likely stronger.

Reported P(A stronger): 0.66. Consistent with choice.

Gemini 3.8 Flash: chose A — correct

Player A demonstrates sound opening principles and structured positional understanding in a Giuoco Pianissimo, coordinating pieces logically (5...Nc6, 6...d5, 8...Bg4) and managing trades cleanly until 24...c6. In contrast, Player B plays an extremely low-level game characterized by bizarre early pawn thrusts and awkward setups (2...Bd7, 4...b5, 7...Rd8). Despite winning a rook with 13...Nc2+, Black in Game B displayed severe tactical blindness, needlessly sacrificing material with 24...Rxc3+ and passively walking into back-rank and pin disasters (30...Be7?, 32...Nxf5?) culminating in 39. Rxd8#.

Reported P(A stronger): 0.88. Consistent with choice.

Gemini 3.1 Pro (Preview): chose A — correct

Target A (Black in Game A) played a standard Italian Game, demonstrating solid opening principles, logical piece development (...Bg4, ...Nf4), and a good understanding of pawn structure. The game ended abruptly, likely due to a timeout or disconnect, but Black's play was positionally sound. In contrast, Target B (Black in Game B) played a chaotic, beginner-level opening (...Bd7, ...b5, ...Na6, ...Rd8) against an opponent who recklessly pushed kingside pawns early on. Although Black B found some basic tactical tricks like 20...Rxd4+, their overall strategic understanding was poor, culminating in severe blunders and getting checkmated. Target A's solid positional play strongly suggests a higher rating.

Reported P(A stronger): 0.85. Consistent with choice.

Gemini 3.5 Flash: chose A — correct

Black in Game A plays standard, principled opening moves, castling early and maintaining a solid positional structure before a final blunder with 24...c6, which hangs the e5-knight. In contrast, Black in Game B plays extremely erratically from the start, choosing bizarre moves like 2...Bd7 and 7...Rd8. Despite capturing a rook after 13...Nc2+, Black displays poor board vision, blundering back material with 24...Rxc3+ and 32...Nxf5, leading to a quick checkmate. Black in Game A shows much higher baseline chess competence.

Reported P(A stronger): 0.85. Consistent with choice.

Gemini 3.7 Flash: chose A — correct

Game A Black demonstrates solid fundamental chess understanding, navigating an open game with sound opening principles (6...d5, 8...Bg4), active piece placement (13...Nf4), and classical central play (21...Rd8, 22...Ne5). Although Black resigned or flagged after 24...c6, their position remained completely playable and logical throughout. In contrast, Game B Black plays an erratic, passive opening (2...Bd7, 7...Rd8) and, despite winning a trapped rook on a1, displays terrible king safety and coordination. Game B Black completely mishandles White's uncoordinated attack, missing major tactical threats (22...Qc5 allowing 23.Qa8+) and collapsing rapidly into checkmate (39.Rxd8#).

Reported P(A stronger): 0.88. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

Game A Black played solid, standard rapid lines: …Bc5, …d5 counter-strike, …Nxd5, …Rfe8, and only resigned after 24.Rd5. Game B Black made bizarre, non-developing moves: 2…Bd7, 4…b5, 6…Qc7, 7…Rd8, then won material but collapsed tactically after 20…Rxd4+, allowing the forced mate 39.Rxd8#. B's opening was incoherent and defense hopeless; A showed ordinary club competence.

Reported P(A stronger): 0.87. Consistent with choice.

Claude Fable 5.1: chose A — correct

Game A Black played a coherent Italian: ...d5 break, ...Bg4, ...Nf4 exchange, ...Rxe4 sac for play; the game ended early (likely resignation/flag) without gross blunders shown. Game B Black played bizarre, incoherent moves from the start (2...Bd7, 4...b5, 5...Na6, 7...Rd8), then after winning material with ...Nc2+ and ...Nxa1 repeatedly misplayed: 19...Qg5 wasted tempi, 24...Rxc3+ gave back a rook, 25...f6/28...e5 weakened the king, and allowed a mating net. Game B White was also very weak (2.e3 Qf3 g4 h3), indicating a low-rated pairing. Black in Game A shows clearly more structured understanding.

Reported P(A stronger): 0.8. Consistent with choice.

Gemini 3 Flash (Preview): chose A — correct

Game A Black demonstrates sound opening principles (3... Bc5, 6... d5) and maintains a balanced, coordinated position throughout the middlegame. In contrast, Game B Black plays an erratic opening (2... Bd7, 5... Na6) and shows poor tactical awareness. Despite winning a rook early, Game B Black collapses with unnecessary exchanges (24... Rxc3+) and fails to defend a simple mating threat, ultimately allowing 39. Rxd8#. Player A's play shows a significantly higher level of foundational chess knowledge and calculation compared to Player B's chaotic performance, which is characterized by uncoordinated development and severe blunders under pressure.

Reported P(A stronger): 0.9. Consistent with choice.

Gemini 3.6 Flash: chose A — correct

Game A Black demonstrates solid opening knowledge and positional principles in an Italian Game (6...d5, active piece placement). Black maintained a balanced position until a single tactical oversight on move 24 (24...c6, allowing 25.Rxe5). In contrast, Game B Black plays unprovoked passive/erratic moves early on (2...Bd7, 7...Rd8). Despite winning a full rook via 13...Nc2+, Game B Black showed poor board vision, voluntarily sacrificing material back (24...Rxc3+) and allowing White's rook and queen to infiltrate for a decisive mate on d8. Game A Black shows far higher fundamental chess competence.

Reported P(A stronger): 0.85. Consistent with choice.

GPT-5.6 Sol: chose A — correct

Game A Black played a coherent, principled game: 6...d5 challenged the center, 13...Nf4 created activity, and 16...Bxe4 followed by 19...Rfe8 and 21...Rd8 showed purposeful coordination. The unexplained ending after 24...c6 does not reflect an obvious collapse. Game B Black found the strong fork 13...Nc2+ and tactical 20...Rxd4+, but the opening setup (...Bd7, ...b5, ...Na6, ...Rd8) was awkward, and 24...Rxc3+ gave up a rook without sufficient return. Most decisively, 32...Nxf5 allowed 33.Ra8 and 34.Rxc8, losing the queen and leading to mate. A appears more consistently skilled.

Reported P(A stronger): 0.84. Consistent with choice.

GPT-6 Sol: chose A — correct

Game A’s Black played a coherent opening with 6...d5, then coordinated exchanges and central pressure with 20...Rxe4, 21...Rd8, and 24...c6. Game B’s Black spotted tactics against White’s exposed king, notably 13...Nc2+ and 20...Rxd4+, but developed awkwardly and ultimately left the queen on c8 to be taken after 33...Bd8 34.Rxc8. A’s play looks substantially more reliable, regardless of the recorded results.

Reported P(A stronger): 0.97. Consistent with choice.

GPT-6.1 Sol: chose A — correct

Game A Black showed coherent development and central play with 6...d5, 8...Bg4, and 13...Nf4, then coordinated the rooks through 19...Rfe8 and 21...Rd8. The final position does not indicate an obvious decisive blunder. Game B Black found tactics such as 13...Nc2+ and 20...Rxd4+, but against highly unconventional opposition. Black’s early ...Bd7, ...Na6, and ...Rd8 neglected kingside development, while 32...Nxf5 and 33...Bd8 allowed 34.Rxc8, losing the queen. A’s play is substantially more consistent with established strength.

Reported P(A stronger): 0.96. Consistent with choice.

GPT-5.6 Luna: chose A — correct

Game A’s Black played a coherent, principled opening: 6...d5, castling, active piece placement with ...Nf4, and pressure on the e-file with ...Rfe8 and ...Rxe4. The later 18...g5 and 24...c6 may be inaccurate, but the game shows substantially more positional discipline. Game B’s Black chose a highly dubious setup with 2...Bd7, 3...c6, and 4...b5, then allowed major tactical complications. Although 13...Nc2+ and 14...Nxa1 won material, Black’s subsequent 20...Rxd4+ and 21...Rxc4 led to tactical exchanges, and the king was eventually mated after 39.Rxd8#. Overall, A’s Black more likely had the higher established rapid strength.

Reported P(A stronger): 0.8. Consistent with choice.

GPT-6 Astra: chose A — correct

Game A Black shows coherent development and positional play: 6...d5 challenges the center, 13...Nf4 establishes an active knight, and 19...Rfe8 followed by 21...Rd8 activates the rooks. The final position after 24...c6 does not explain the recorded loss. Game B Black exploits White’s weak play with 13...Nc2+ and the rook-winning fork, but 2...Bd7 and 7...Rd8 reflect awkward development while the kingside remains undeveloped. Later, 32...Nxf5 permits Ra8, and 33...Bd8 allows 34.Rxc8, losing the queen for a rook. A’s consistently purposeful play is substantially stronger evidence of established strength.

Reported P(A stronger): 0.95. Consistent with choice.

GLM-5.3 Flash: chose A — correct

Game A's Black plays a coherent Italian: 8...Bg4 active development, 13...Nf4 hitting the bishop, 16...Bxe4/17...Qf6 clean trades, then 22...Ne5 and 23...Rd6 with a solid, equal game; White resigned at move 24 with 24.Rd5? meeting 24...c6. Game B's Black plays erratically: 2...Bd7 and 4...b5/5...Na6 are aimless, 13...Nc2+ 14...Nxa1 trades a knight for a rook-corner trap but abandons queenside, then loses material (20...Rxd4+ 21...Rxc4 losing the rook for bishop) and gets mated. Black B's planless aggressive pawn pushes (g4?!) by White aside, Black A shows far better positional understanding and technique, suggesting the higher rating.

Reported P(A stronger): 0.75. Consistent with choice.

Pair 17: 0 of 16 models wrong

A: White, rated 1126 · B: White, rated 1752. Higher pre-game rating: B.

Game A: White, rated 1126 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. d4 e6 2. Nf3 c5 3. dxc5 Qa5+ 4. Bd2 Qxc5 5. Be3 Qb5 6. Nc3 Qxb2 7. Rb1 Qa3
8. Rb3 Qa5 9. Bd2 Bb4 10. Qa1 Ne7 11. Nb1 Bxd2+ 12. Nfxd2 f6 13. Ra3 Qc7 14.
Rc3 Nbc6 15. e3 Nd5 16. Rc5 Ndb4 17. c3 b6 18. Qb2 bxc5 19. cxb4 cxb4 20. Ne4
O-O 21. Bb5 Qe5 22. Qxe5 Nxe5 23. Nd6 a6 24. Ba4 Nc6 25. O-O f5 26. Re1 Rf6 27.
Rc1 e5 28. Ne8 Re6 29. Nc7 Ra7 30. Nxe6 dxe6 31. Rxc6 Bd7 32. Rc4 Bxa4 33. Rxb4
Bb5 34. Nc3 Rc7 35. Nxb5 Rc1# 0-1
Game B: White, rated 1752 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "1-0"]
[TimeControl "600+0"]

1. e4 d5 2. exd5 e6 3. dxe6 Bxe6 4. Nf3 Bc5 5. c3 Qe7 6. Be2 Bxf2+ 7. Kxf2 Nf6
8. Rf1 O-O 9. Kg1 Bd5 10. d4 Qd6 11. Bg5 Ng4 12. h3 Bxf3 13. hxg4 Bxe2 14. Qxe2
Nd7 15. Nd2 Rae8 16. Qf2 Nb6 17. Rae1 Rxe1 18. Rxe1 Nd5 19. Qf3 f6 20. Bh4 c5
21. Bg3 Qc6 22. Nb3 cxd4 23. Nxd4 Qc4 24. a3 a5 25. Qe2 Qxe2 26. Rxe2 Rd8 27.
Re6 Nc7 1-0

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose B — correct

Game A White played oddly throughout: 5.Be3 then 9.Bd2, 10.Qa1 and 11.Nb1 undeveloping, and 16.Rc5?? dropping the exchange to ...b6. Despite later gaining material after Black's errors (28.Ne8 forking/winning the exchange), White ignored back-rank danger and allowed 35...Rc1#. Game B White handled a sloppy opponent sensibly: accepted 6...Bxf2+ (a bad sacrifice), 13.hxg4 kept the extra piece, and the follow-up was solid with central control (10.d4, 17.Rae1). White also traded queens with 25.Qe2 to convert. Black's 27...Nc7?? hangs f6 or leads to resignation-level loss. B's play shows cleaner technique, so B is likely stronger.

Reported P(A stronger): 0.3. Consistent with choice.

Claude Sonnet 5.5: chose B — correct

Game A White played an odd, passive opening: 3.dxc5 Qa5+ 4.Bd2 Qxc5 5.Be3 Qb5 6.Nc3 Qxb2 lost a pawn and allowed the queen raid. Later 7.Rb1 Qa3 8.Rb3 and 16.Rc5 were dubious rook moves. 20.Ne4 and 23.Nd6 were also shaky, and 28.Ne8 and 29.Nc7 lost material. White then missed the final back-rank mate (35.Nxb5 Rc1#). Game B White took a pawn after 2.exd5, and Black's 6...Bxf2+ was an unsound sacrifice. White handled it cleanly with 7.Kxf2, 10.d4, 11.Bg5 and 13.hxg4, then traded into a won endgame. Black's play was also weak, but White was accurate and reasonable. Game B White looks somewhat stronger, though both are modest in level.

Reported P(A stronger): 0.3. Consistent with choice.

Gemini 3.8 Flash: chose B — correct

Player B exhibits solid fundamentals, comfortably defending against Black's unsound sacrifice 6...Bxf2+ by safely repositioning the king with 8.Rf1 and 9.Kg1. From there, B smoothly coordinates, develops pieces, controls central files, and converts the material advantage cleanly without any tactical oversights. In contrast, Player A plays haphazardly, needlessly giving up pawn structure and falling into a cheap exchange tactic with 17.c3? b6, and later blunders a simple back-rank checkmate in one move with 35.Nxb5?? Rc1#.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.1 Pro (Preview): chose B — correct

Game A White demonstrates very low-level play, with bizarre maneuvers like 10. Qa1 and 11. Nb1, dropping an exchange with 16. Rc5 Ndb4 17. c3 b6, and eventually blundering a simple back-rank mate with 35. Nxb5 Rc1#. Game B White, facing an early speculative sacrifice (6...Bxf2+), calmly defends by castling by hand (7. Kxf2, 8. Rf1, 9. Kg1) and efficiently trades down to consolidate the extra piece into a winning endgame. White B's play is fundamentally sound and coordinated, whereas White A plays erratically and blunders severely.

Reported P(A stronger): 0.1. Consistent with choice.

Gemini 3.5 Flash: chose B — correct

White in Game B demonstrated significantly better tactical and positional awareness than White in Game A. In Game A, White committed several large tactical errors, including 17. c3? (which allowed a knight fork) and the final blunder 35. Nxb5??, which completely overlooked a simple back-rank checkmate. In contrast, White in Game B responded calmly to an early sacrifice on f2, consolidated their material advantage, and converted the game smoothly without giving Black any counterplay.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.7 Flash: chose B — correct

Player B shows far superior tactical and strategic discipline. Against Black's speculative piece sacrifice (6...Bxf2+), Player B cleanly executed artificial castling (8.Rf1, 9.Kg1), consolidated the extra piece smoothly, and converted the advantage without any concessions. In contrast, Player A in Game A played awkwardly in the opening (10.Qa1), permitted tactical vulnerabilities like 17.c3 (allowing 17...Nc2+), and ultimately blundered a basic back-rank mate with 35.Nxb5?? Rc1#.

Reported P(A stronger): 0.12. Consistent with choice.

Qwen3.8 Max Prime: chose B — correct

Game A White played several dubious moves: 10.Qa1 was awkward, 18.Qb2 dropped the c5 rook to 18...bxc5, and 28.Ne8/29.Nc7 were poorly timed, losing material and being checkmated. Game B White, though winning partly due to Black's blunder 27...Nc7 (allowing the winning tactic), generally kept control after absorbing the Bxf2+ sac (7.Kxf2), coordinated well with 11.Bg5 and 23.Nxd4, and maintained a solid edge into the endgame. White B showed better overall piece play; White A made more elementary errors and lost convincingly.

Reported P(A stronger): 0.34. Consistent with choice.

Claude Fable 5.1: chose B — correct

Game A White played very oddly: 3.dxc5 then 4.Bd2 5.Be3 wasting tempi, allowed 6...Qxb2, then 8.Rb3, 10.Qa1, 10.Nb1 and 13.Ra3/14.Rc3/16.Rc5?? dropping an exchange, and finally allowed 35...Rc1# back-rank mate. Black in A was also weak (early queen hunting). Game B White handled the Bxf2+ sacrifice calmly (7.Kxf2, 8.Rf1, 9.Kg1), won material cleanly with 12.h3/13.hxg4, consolidated with Qf2/Rae1 and converted via 25.Qe2 trade and 27.Re6, with the opponent resigning. Game B White showed solid, error-free play; Game A White showed basic tactical blindness.

Reported P(A stronger): 0.25. Consistent with choice.

Gemini 3 Flash (Preview): chose B — correct

Target B shows significantly better tactical awareness and fundamental coordination. In Game A, White loses a pawn early (6... Qxb2), allows unnecessary pressure, and ultimately blunders a simple back-rank mate with 35. Nxb5??. In contrast, Target B handles an unsound sacrifice (6. Be2 Bxf2+) with composure, maintains the material advantage through clean development (17. Rae1, 26. Rxe2), and forces a resignation through solid positional play. Target A's failure to see the back-rank threat is a strong indicator of a lower rating compared to Target B's steady conversion.

Reported P(A stronger): 0.15. Consistent with choice.

Gemini 3.6 Flash: chose B — correct

Game B White played cleanly and displayed solid tactical awareness. After Black's unsound piece sacrifice with 6...Bxf2+ 7.Kxf2, Game B White consolidated comfortably, castled by hand, smoothly centralized forces, simplified the position, and maintained a clear winning advantage throughout without any major flaws. In contrast, Game A White showed weak tactical oversight, hanging the b2-pawn early with 6.Nc3, allowing tactical vulnerabilities like 17.c3? (permitting 17...Nc2+ fork), and ultimately blundering a back-rank checkmate in one with 35.Nxb5?? Rc1#. Game B White demonstrates significantly higher established playing strength.

Reported P(A stronger): 0.15. Consistent with choice.

GPT-5.6 Sol: chose B — correct

Game B White responded calmly to Black’s dubious 6...Bxf2+: 7.Kxf2, 8.Rf1 and 9.Kg1 restored king safety. The tactical 12.h3 correctly met ...Bxf3 with 13.hxg4, winning the g4-knight, and White then simplified sensibly with 25.Qe2 Qxe2 26.Rxe2. Game A White showed some tactical awareness with 28.Ne8–30.Nxe6 and 31.Rxc6, but the opening rook excursions (8.Rb3, 13.Ra3, 14.Rc3, 16.Rc5) culminated in losing that rook after 18.Qb2? bxc5. Most decisively, 35.Nxb5?? overlooked the immediate 35...Rc1#.

Reported P(A stronger): 0.18. Consistent with choice.

GPT-6 Sol: chose B — correct

Game B’s White handled Black’s unsound 6...Bxf2+ calmly with 7.Kxf2, 8.Rf1 and 9.Kg1, then developed and simplified with 25.Qe2 while ahead. Game A’s White found useful tactics with 30.Nxe6 and 31.Rxc6, but spent many opening moves shuffling the rook, lost it to 18...bxc5, and missed the mating threat before 35...Rc1#. B’s play looks more consistently controlled.

Reported P(A stronger): 0.08. Consistent with choice.

GPT-6.1 Sol: chose B — correct

Game A White repeatedly spent tempi on queen and rook excursions (10.Qa1, 13.Ra3, 14.Rc3), then left the attacked rook hanging with 18.Qb2 bxc5. Despite later tactical recovery, 35.Nxb5 allowed immediate mate with ...Rc1#. Game B White handled Black’s speculative sacrifice coherently: 7.Kxf2 followed by 8.Rf1 and 9.Kg1 secured the king, while 12.h3 and 14.Qxe2 emerged with extra material. Subsequent play—17.Rae1, 22.Nb3, 23.Nxd4, and 25.Qe2—showed purposeful coordination and simplification. B’s opponent played unsoundly, but B’s much cleaner execution strongly favors the higher established rating.

Reported P(A stronger): 0.95. Contradicts choice; excluded from confidence diagnostics.

GPT-5.6 Luna: chose B — correct

Game B’s White showed stronger tactical and positional control: after 6...Bxf2+ 7.Kxf2, White stabilized the exposed king, then used 12.h3 and 13.hxg4 to eliminate Black’s active knight. White coordinated the rooks with 17.Rae1 and 18.Rxe1, improved the position, and invaded with 27.Re6. In Game A, White’s 18.c3?? allowed 18...bxc5, losing the rook on c5 outright. Although White later found 30.Nxe6 and 31.Rxc6, the game ended with the basic mating oversight 35...Rc1#. These games favor B as the more established player.

Reported P(A stronger): 0.15. Consistent with choice.

GPT-6 Astra: chose B — correct

Game B White showed coherent development and king safety: 7.Kxf2 accepted Black’s speculative sacrifice, followed by 8.Rf1, 9.Kg1 and 10.d4. White then coordinated with 17.Rae1 and simplified through 25.Qe2, retaining the extra piece. Game A White spent many opening tempi moving the same rook (Rb1–b3–a3–c3–c5), and 18.Qb2 allowed ...bxc5, conceding the exchange. Although 28.Ne8 and 29.Nc7 showed tactical awareness, 35.Nxb5 overlooked the immediate ...Rc1#. B’s opponent supplied an early advantage, but B’s subsequent handling was substantially more controlled.

Reported P(A stronger): 0.95. Contradicts choice; excluded from confidence diagnostics.

GLM-5.3 Flash: chose B — correct

White in Game A shows poor technique: grabbing ...Qxb2 (6...), then after luck (Black's 18...bxc5?? dropping a rook) still collapses with 23.Nd6??/24.Ba4 weakening, allowing Black's smooth conversion and mate (35...Rc1#). White in Game B handles a tricky Scandinavian exchange calmly: 7.Kxf2 safely, solid development, 12.hxg4 winning a pawn, careful king safety (Kg1, Rf1), and converts cleanly with 25.Qe2 Qxe2 26.Rxe2 and 27.Re6 exploiting the pin. Black in B was slightly more solid than Black in A, making B's win more convincing. B's target likely had the higher rating.

Reported P(A stronger): 0.75. Contradicts choice; excluded from confidence diagnostics.

Pair 03: 0 of 16 models wrong

A: White, rated 1363 · B: White, rated 1112. Higher pre-game rating: A.

Game A: White, rated 1363 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. e4 e5 2. d3 d5 3. Nc3 dxe4 4. Nxe4 f5 5. Nc3 Nc6 6. Nf3 Bd6 7. Bg5 Qd7 8.
Qe2 h6 9. Bd2 g5 10. O-O-O Nf6 11. h4 g4 12. Nh2 h5 13. g3 b6 14. Bg5 Nh7 15.
Nd5 Nxg5 16. Nf6+ Ke7 17. Nxd7 Kxd7 18. hxg5 Bb7 19. f3 Nd4 20. Qf2 gxf3 21.
Nxf3 Nxf3 22. Bg2 Rab8 23. Bxf3 Bxf3 24. Qxf3 Ke6 25. Rdf1 Rbf8 26. Qc6 a5 27.
Re1 e4 28. Qc4+ Kd7 29. Qa4+ c6 30. dxe4 Bxg3 31. e5 Bxe1 32. Rxe1 h4 33. Qd4+
Kc7 34. Qd6+ Kb7 35. Qd7+ Ka6 36. Qd3+ b5 37. Qd6 Kb6 38. e6 h3 39. e7 Re8 40.
Qb8+ Rxb8 41. e8=Q Rbxe8 42. Rxe8 Rxe8 0-1
Game B: White, rated 1112 — anonymized PGN

Replay on Lichess

[White "White"]
[Black "Black"]
[Result "0-1"]
[TimeControl "600+0"]

1. c4 d6 2. Nc3 Nf6 3. e3 Bd7 4. Bd3 e5 5. a3 Be7 6. Bb1 O-O 7. Qc2 g6 8. h4 h5
9. b3 Bf5 10. e4 Bd7 11. Bb2 Nc6 12. Nf3 Bg4 13. Ng5 Nd4 14. Qd3 Nd7 15. f3 Be6
16. f4 Bxg5 17. hxg5 Nc5 18. Qe3 Ncxb3 19. f5 Nxa1 20. Bxa1 Bxc4 21. fxg6 fxg6
22. Nd5 Ne6 23. Qc3 Ba6 24. Nf6+ Kg7 25. Rf1 Bxf1 26. Kxf1 Nxg5 27. Ke1 Qxf6
28. Qxc7+ Kh6 29. Bd3 Rac8 30. Qxb7 Rc1+ 31. Ke2 Qf2# 0-1

Model explanations

These are the models’ stated reasons, not engine-verified annotations.

Claude Opus 5.5: chose A — correct

Both targets look weak. Game A White found 15.Nd5 and 16.Nf6+, forking king and queen to win the queen and reach a winning position. Its later collapse was dramatic: 40.Qb8+?? gave away the queen, and 42.Rxe8 left a lost ending. Game B White played an aimless opening (6.Bb1, 8.h4, 9.b3) and allowed 18...Ncxb3, losing the exchange. 25.Rf1?? dropped another exchange, and 27.Ke1?? hung the f6 knight to Qxf6. White was then quickly mated. Game B White shows more repeated basic tactical oversights, while Game A White showed some tactical vision before blundering. I lean slightly to A.

Reported P(A stronger): 0.58. Consistent with choice.

Claude Sonnet 5.5: chose A — correct

Both White players lost. In Game A, White lasted 42 moves in a messy game. Opening with 2.d3 and 3.Nc3 was passive, but 10.O-O-O and 11.h4 showed some attacking intent. 15.Nd5 and 17.Nxd7 were reasonable, and White kept pace until Black's passed e-pawn and 30...Bxg3 decided it. Black also played loosely, so White's resistance was fairly normal for the level. In Game B, White's play was much weaker: 8.h4 and 9.b3 loosened the position, 13.Ng5 and 14.Qd3 drifted, and 18.Qe3 allowed 18...Ncxb3 and 19...Nxa1. After 24.Nf6+ Kg7 25.Rf1, White hung material and was mated by 31...Qf2#, having ignored threats with 30.Qxb7. Game B's White therefore looks weaker.

Reported P(A stronger): 0.6. Consistent with choice.

Gemini 3.8 Flash: chose A — correct

Target A demonstrated good tactical alertness with 16. Nf6+, winning Black's queen, and built a completely winning position before suffering an apparent mouse-slip or massive lapse with 40. Qb8+. Up to that point, White in Game A navigated tactical complications and pushed pawns purposefully. In contrast, Target B played a very passive and awkward opening (4. Bd3, 6. Bb1) and was outplayed step-by-step. Black quickly dominated the board with active knights and bishops (18... Ncxb3), while Game B White shed material easily (20. Bxa1, 25. Rf1) and succumbed swiftly to 31... Qf2#.

Reported P(A stronger): 0.78. Consistent with choice.

Gemini 3.1 Pro (Preview): chose A — correct

Target A played a somewhat standard opening and spotted a nice tactical sequence (15. Nd5 followed by the 16. Nf6+ royal fork) to win the queen, even though they later self-destructed with a horrific endgame blunder (40. Qb8+??). Target B played an extremely bizarre and unprincipled opening setup (4. Bd3, 6. Bb1, 7. Qc2) and was easily outplayed tactically by Black's knights, dropping an exchange to a simple fork (18... Ncxb3) and getting checkmated. A's grasp of opening fundamentals and early tactical awareness suggest a higher rating than B's beginner-like maneuvering.

Reported P(A stronger): 0.7. Consistent with choice.

Gemini 3.5 Flash: chose A — correct

Game A White showed reasonable tactical awareness, capitalizing on Black's mistake with the fork 16. Nf6+ to win the queen. Despite throwing away a completely winning position with the colossal blunder 40. Qb8+?? (likely a mouse slip or time-trouble hallucination), their play prior was competent. In contrast, Game B White showed poor fundamental understanding, wasting tempi with passive moves like 4. Bd3 and 6. Bb1. Game B White failed to develop effectively, lost material easily, and was quickly mated on f2 with no counterplay, suggesting a lower playing strength.

Reported P(A stronger): 0.75. Consistent with choice.

Gemini 3.7 Flash: chose A — correct

White in Game A demonstrated solid tactical awareness by spotting the intermediate fork 16.Nf6+, winning Black's queen. White then navigated the complications well and pushed a dangerous passed pawn to e7 before suffering a catastrophic, single-move blunder with 40.Qb8+??. In contrast, White in Game B played an awkward opening setup (4.Bd3 and 6.Bb1), lost control of the dark squares and key outposts (13...Nd4, 18...Ncxb3), and showed little resistance as Black quickly broke through to deliver checkmate on move 31. Despite the endgame blunder, Player A displayed superior tactical vision and positional competence.

Reported P(A stronger): 0.62. Consistent with choice.

Qwen3.8 Max Prime: chose A — correct

Both whites lost, but Game B's collapse was far more one-sided. B's White played passively (6.Bb1, 7.Qc2), then after 17...Nc5 and 18...Ncxb3 lost material with no compensation; 24.Nf6+ Kg7 simply hung into 25...Bxf1, and Black mated with 31...Qf2#. Game A's White built a playable kingside attack (11.h4, 14.Bg5, 16.Nf6+), creating real tactical problems, and only lost after a long, complex endgame where 30.dxe4 allowed 30...Bxg3. A showed more competence in a richer middlegame.

Reported P(A stronger): 0.62. Consistent with choice.

Claude Fable 5.1: chose A — correct

Game A White played a coherent if passive setup, found the tactical shot 15.Nd5/16.Nf6+ winning the queen for two pieces, then blundered a rook with 27.Re1?? allowing 27...e4 and later 40.Qb8+?? hallucinating a promotion trick; still, the earlier play showed genuine tactical awareness. Game B White drifted badly: 6.Bb1 and 7.Qc2 wasted tempi, 9.b3 and 13.Ng5 allowed ...Nd4 and ...Ncxb3 forking queen and rook, dropping the a1 rook and then c4, Rf1 for free, and finally walked into 31...Qf2# after grabbing pawns. Black in A also played weakly, making both opponents comparable. A's White seems slightly stronger.

Reported P(A stronger): 0.58. Consistent with choice.

Gemini 3 Flash (Preview): chose A — correct

Player A shows superior tactical awareness and opening knowledge compared to Player B. In Game A, White executes a strong tactical sequence beginning with 15. Nd5 and 16. Nf6+ to win the Queen for two pieces. While Player A ultimately collapses in the endgame, blundering the Queen back with 40. Qb8+, their play was coherent until the time pressure phase. Player B, by contrast, plays a nonsensical opening (4. Bd3, 6. Bb1) and shows poor tactical vision, allowing a simple knight fork on a1 (19. f5) and failing to defend against a basic mating sequence. Player A’s peak level within the game is significantly higher.

Reported P(A stronger): 0.7. Consistent with choice.

Gemini 3.6 Flash: chose A — correct

Game A White demonstrated good tactical vision by finding 16. Nf6+ to win Black's queen, subsequently building a completely dominant position before suffering a catastrophic late blunder (40. Qb8+). Game B White showed poor opening planning (4. Bd3, 6. Bb1), lost central control, allowed Black's knights to infiltrate unpunished (13...Nd4, 17...Nc5), and quickly collapsed under pressure (25. Rf1, 26...Nxg5), leading to mate on move 31. Despite Game A White's single-move blunder, their overall tactical awareness and positional play were significantly superior to Game B White.

Reported P(A stronger): 0.78. Consistent with choice.

GPT-5.6 Sol: chose A — correct

Game A White showed substantially better tactical awareness and coordination: 15.Nd5 followed by 16.Nf6+ and 17.Nxd7 won Black’s queen, while 26.Qc6 and the later checks kept pressure. The conversion was poor—especially 40.Qb8+?, which allowed ...Rxb8 and liquidation—but the preceding play displayed real calculation. Game B White spent many opening tempi on Bd3-b1, a3, h4, and b3, then overextended with 15.f3 and 16.f4. More seriously, White allowed 18...Ncxb3 and 19...Nxa1, played 25.Rf1? Bxf1, and failed to address the mating attack ending in 31...Qf2#. A therefore looks more established despite also losing.

Reported P(A stronger): 0.86. Consistent with choice.

GPT-6 Sol: chose A — correct

Game A’s White spotted a concrete tactic with 15.Nd5 and 16.Nf6+, winning Black’s queen with 17.Nxd7. Although 40.Qb8+ was a decisive blunder, White had pursued a dangerous passed pawn with 38.e6 and 39.e7. Game B’s White showed less consistent awareness of threats: 19.f5 allowed ...Nxa1 to take a rook, 25.Rf1 allowed ...Bxf1 to take the other, and 30.Qxb7 overlooked the mating attack. The contrast favors A despite both losses.

Reported P(A stronger): 0.88. Consistent with choice.

GPT-6.1 Sol: chose A — correct

Game A White showed more coherent development, castled early, and exploited Black’s queen placement with 15.Nd5 and 16.Nf6+, winning the queen on d7. White later made serious errors: 31.e5 allowed ...Bxe1, and 40.Qb8+ overlooked Black’s second rook in the promotion combination. Game B White spent opening tempi on Bd3–b1 and f3–f4 while leaving the king uncastled. After 18...Ncxb3, 19.f5 allowed ...Nxa1; later 25.Rf1 simply permitted ...Bxf1, followed by losing the knight to ...Qxf6. Despite facing a more coherent opponent, B’s repeated development and material-safety failures suggest lower established strength.

Reported P(A stronger): 0.79. Consistent with choice.

GPT-5.6 Luna: chose A — correct

Game A’s White showed a significant tactical resource with 16.Nf6+ followed by 17.Nxd7, winning Black’s queen, and later advanced a dangerous passed e-pawn. However, White then overlooked 30...Bxg3 and 31...Bxe1, losing a rook, and the eventual promotion tactic failed. Game B’s White made more fundamental oversights: 19.f5 allowed ...Nxa1 to win the rook, 25.Rf1 allowed ...Bxf1, and 27.Ke1 permitted ...Qxf6 before the forced mate 31...Qf2#. Thus A’s play contained stronger tactical ideas despite serious later inaccuracies.

Reported P(A stronger): 0.72. Consistent with choice.

GPT-6 Astra: chose A — correct

Game A White shows better tactical awareness and coordination: 15.Nd5 followed by 16.Nf6+ and 17.Nxd7 exploits Black’s queen placement, while castling and centralizing the rooks provide a coherent foundation. A nevertheless makes serious errors, notably 31.e5 allowing ...Bxe1 and the losing liquidation beginning 40.Qb8+. Game B White spends several opening tempi on Bd3–b1 and pawn moves, then allows ...Nc5–b3–a1 to win a rook. Later, 25.Rf1 simply exposes another rook to ...Bxf1, and 27.Ke1 leaves the knight on f6 undefended. Despite both players’ blunders, A demonstrates the stronger overall baseline.

Reported P(A stronger): 0.81. Consistent with choice.

GLM-5.3 Flash: chose A — correct

Game A's White plays a coherent strategic game: regrouping Nf6+/Nxd7 tricks (15-17), trading into a defensible endgame, and only slipping late (30.dxe4? Bxg3 dropping a rook after 31...Bxe1). Game B's White self-destructs: 13.Ng5? allows ...Nd4 pressuring f3, then 16.f4? Bxg5 loses material, 18.Qe3? runs into 18...Ncxb3, and 19.f5? Nxa1 simply drops a rook; Black converts with 24...Qxf6 and a clean 31...Qf2#. White's handling in B (open king, aimless wing play 8.h4) is far weaker than A's, who fought a sound attacking plan.

Reported P(A stronger): 0.9. Consistent with choice.