Gakucho Benchmark - Mathematics Open-Ended: leaderboard

The Gakucho Benchmark (2026) is built from the officially released Grade 9 items of the 2022 Japanese National Assessment of Academic Ability, keeping the original layouts, diagrams and nationwide student response statistics. This board scores the answers to the open-ended Mathematics items by character-level F1 against the official answers, on a 0-100 scale; higher is better.

Metric: Character-level F1 (0-100, the paper's 0-1 score times 100) of the answers to the 10 open-ended Mathematics items against the official correct answers, of the officially released Grade 9 items of Japan's 2022 National Assessment of Academic Ability (original exam layouts and diagrams, Japanese prompts, nationwide student response statistics), zero-shot with the item images; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 11 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude Sonnet 4.569.5#261
2Claude 3.7 Sonnet (20250219)66.3#421
3GPT-562.3#178
4Gemini 2.5 Pro55.8#276
5Claude Sonnet 4 (20250514)54.8#405
6Gemini 2.5 Flash54.2#463
7GPT-4o49.2#751
8GPT-4o Mini46.1#1667
9Gemini 3.1 Pro (Preview)35.8#99
10Gemma 3 12B (IT)24.6#1887
11Qwen 2.5 VL 7B Instruct19.5#1902

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=gakucho-benchmark-mathematics-open-ended · How It Works · Data refreshed daily, snapshot 2026-11-21.