A2RBench - Symbolic Tasks (Remapped Symbols): leaderboard
The same symbolic rule tasks after every token is remapped to unfamiliar symbols, which tests reliance on familiar symbols, from A2RBench (2026), an automatically generated abstract reasoning benchmark in which a model infers a bijective transformation rule over sequences, grids or voxels from examples and applies it to a query; answers are judged against code-executed ground truth. Accuracy in percent; higher is better.
Metric: Accuracy (%) on the same 351 symbolic rule tasks after every token is remapped to unfamiliar symbols (P1), the solver states its inferred rule and reasoning, then its answer at near-zero temperature (1e-7) is judged against the code-executed ground truth by a GPT-5-mini judge; higher is better. Source: arxiv.org. Saturation forecast: Around 2029. 13 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3 Pro | 34.8 | #145 |
| 2 | Claude Sonnet 4.5 | 32.5 | #261 |
| 3 | GPT-5.2 | 28.5 | #185 |
| 4 | GPT-5 | 23.6 | #178 |
| 5 | Gemini 2.5 Flash | 23.4 | #463 |
| 6 | GPT-5 Mini | 22.8 | #364 |
| 7 | Qwen 3 32B | 21.9 | #972 |
| 8 | GLM-4.6 | 21.4 | #498 |
| 9 | O4 Mini | 20.2 | #368 |
| 10 | DeepSeek V3.2 | 18.2 | #419 |
| 11 | Qwen 3 14B | 17.1 | #1269 |
| 12 | GPT-4o Mini | 15.1 | #1667 |
| 13 | Gemini 3 Flash | 10.3 | #175 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=a2rbench-symbolic-tasks-remapped-symbols · How It Works · Data refreshed daily, snapshot 2026-11-21.