A2RBench - Semantic Tasks: leaderboard

Semantic, knowledge-dependent rule tasks of A2RBench (2026), an automatically generated abstract reasoning benchmark in which a model infers a bijective transformation rule over sequences, grids or voxels from examples and applies it to a query; answers are judged against code-executed ground truth. Accuracy in percent; higher is better.

Metric: Accuracy (%) on the 352 semantic (knowledge-dependent) rule tasks, the solver states its inferred rule and reasoning, then its answer at near-zero temperature (1e-7) is judged against the code-executed ground truth by a GPT-5-mini judge; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-552#178
2Gemini 3 Pro48.6#145
3GPT-5 Mini47.7#364
4O4 Mini44.3#368
5Gemini 2.5 Flash30.4#463
6Claude Sonnet 4.525.3#261
7GPT-5.223.9#185
8DeepSeek V3.220.7#419
9GLM-4.616.8#498
10Gemini 3 Flash13.4#175
11Qwen 3 32B12.8#972
12GPT-4o Mini9.4#1667
13Qwen 3 14B8.8#1269

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=a2rbench-semantic-tasks · How It Works · Data refreshed daily, snapshot 2026-11-21.