GR-Ben - Analogical Reasoning: leaderboard
GR-Ben Analogical Reasoning is the subset of GR-Ben (2026) built from the analogical-reasoning problems (AnaloBench sentence-level): a model reads a problem and a step-by-step solution written by one of 15 LLMs and must name the earliest erroneous step or state that every step is correct. The score is F1 in percent, the harmonic mean of accuracy on flawed and on correct solutions, higher is better; process reward models are scored from their step predictions and LLMs are prompted as critics with thinking disabled.
Metric: F1 (%) of first-error identification: the harmonic mean of the accuracy on flawed solutions (the earliest erroneous step named) and the accuracy on fully correct solutions (judged correct), on the 402 analogical-reasoning problems (AnaloBench sentence-level) of GR-Ben (3,595 human-annotated solutions written by 15 LLMs); process reward models by their step predictions (scalar-score PRMs thresholded on a 40-item dev split held out of their scoring), LLMs prompted as critics with thinking disabled (mean of three runs for open models, one run for API models); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 22 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | Gemini 3 Flash | Minimal | 60.8 | #175 |
| 2 | VersaPRM (8B) | 54.2 | #1572 | |
| 3 | Qwen 3 32B | Non-reasoning | 53.5 | #972 |
| 4 | Qwen 3 8B | Non-reasoning | 52.1 | #1873 |
| 5 | GPT-5.2 (2025-12-11) | Non-reasoning | 52.1 | |
| 6 | Qwen 3 14B | Non-reasoning | 50 | #1269 |
| 7 | Qwen2.5-Math-PRM-72B | 49.4 | ||
| 8 | Gemma 3 27B (IT) | 49.1 | #1324 | |
| 9 | Kimi K2 0905 | 45.7 | #570 | |
| 10 | DeepSeek V3.2 | Non-reasoning | 42.2 | #419 |
| 11 | Qwen2.5-Math-PRM-7B | 40.7 | ||
| 12 | Qwen 3 4B | Non-reasoning | 37.9 | #2348 |
| 13 | Universal-PRM-7B | 34.5 | ||
| 14 | Qwen2.5-Math-SCAN-Pro-7B | 34.2 | #5247 | |
| 15 | Llama 3.3 70B Instruct | 26.9 | #1385 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=gr-ben-analogical-reasoning · How It Works · Data refreshed daily, snapshot 2026-11-21.