IntentGrasp - Gem Set: leaderboard

The Gem Set of IntentGrasp (2026), 470 intent-understanding items from 12 domains that every open-weight model failed in its first run, balanced across domains. Each item asks for the intended meaning among options; the score is instance F1 in percent, averaged over four option orders; higher is better.

Metric: Instance F1 (%) on the 470-item Gem Set, mean of four option-order runs (no shuffle and shuffle seeds 7, 42 and 365), zero-shot, API default settings; higher is better. Source: arxiv.org. Saturation forecast: Around December 2027. 9 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3 Flash (Preview)24.67#152
2Gemini 3.1 Pro (Preview)21.51#99
3Claude Opus 4.716.55#85
4Gemini 3.1 Flash Lite (Preview)11.77#417
5GPT-5.411.71#135
6Claude Sonnet 4.610.91#173
7GPT-5.4 Nano8.68#651
8GPT-5.4 Mini7.08#406
9Claude Haiku 4.55.76#583

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=intentgrasp-gem-set · How It Works · Data refreshed daily, snapshot 2026-11-21.