AutoGUI-v2 - Element Captioning: leaderboard

AutoGUI-v2 (2026) tests functionality understanding of GUI regions and elements on web, desktop and mobile screenshots. This board asks which of several options states the outcome of an interaction with an element, with easy and hard distractors. Accuracy in percent; higher is better.

Metric: Element interaction-outcome captioning on 788 tasks, multiple-choice accuracy (%) with easy and hard (same-group) distractors; the authors' tuned prompts; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 8 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 2.5 Pro70.3#276
2Claude Sonnet 4.566.9#261
3GPT-566.5#178
4O365.4#237
5GLM-4.5V64.8#754
6Qwen 3 VL 32B Instruct61.3#606
7Qwen 3 VL 8B Instruct57.1#1029
8Step-352.8#734

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=autogui-v2-element-captioning · How It Works · Data refreshed daily, snapshot 2026-11-21.