A2RBench - Symbolic Tasks (Remapped Symbols): leaderboard

The same symbolic rule tasks after every token is remapped to unfamiliar symbols, which tests reliance on familiar symbols, from A2RBench (2026), an automatically generated abstract reasoning benchmark in which a model infers a bijective transformation rule over sequences, grids or voxels from examples and applies it to a query; answers are judged against code-executed ground truth. Accuracy in percent; higher is better.

Metric: Accuracy (%) on the same 351 symbolic rule tasks after every token is remapped to unfamiliar symbols (P1), the solver states its inferred rule and reasoning, then its answer at near-zero temperature (1e-7) is judged against the code-executed ground truth by a GPT-5-mini judge; higher is better. Source: arxiv.org. Saturation forecast: Around 2029. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3 Pro34.8#145
2Claude Sonnet 4.532.5#261
3GPT-5.228.5#185
4GPT-523.6#178
5Gemini 2.5 Flash23.4#463
6GPT-5 Mini22.8#364
7Qwen 3 32B21.9#972
8GLM-4.621.4#498
9O4 Mini20.2#368
10DeepSeek V3.218.2#419
11Qwen 3 14B17.1#1269
12GPT-4o Mini15.1#1667
13Gemini 3 Flash10.3#175

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=a2rbench-symbolic-tasks-remapped-symbols · How It Works · Data refreshed daily, snapshot 2026-11-21.