RoboAbstention - Subjective Intent: leaderboard

The Subjective Intent category of RoboAbstention (2026, The Yes-Man Syndrome): 242 instructions that depend on unstated personal taste, given with an image to a vision-language robot planner that should abstain or ask. The score is the abstention rate in percent; higher is better.

Metric: Abstention rate (%) on the 242 Subjective Intent instructions, image plus instruction to an embodied VLM planner, every instruction warranting abstention, act or abstain judged by GPT-5.4 mini (97.5 percent agreement with human labels), API default sampling; higher is better. Source: arxiv.org. Saturation forecast: Around August 2027. 8 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini Robotics-ER 1.660.3#170
2Gemini 2.5 Flash31.8#463
3Claude Sonnet 4.630.6#173
4GPT-5.416.9#135
5GPT-5.4 MiniNon-reasoning16.1#406
6Qwen 3.5 27B5.4#375
7Llama 4 Maverick5#1110
8GPT-5.4 Nano3.7#651

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=roboabstention-subjective-intent · How It Works · Data refreshed daily, snapshot 2026-11-21.