MUIAnno: leaderboard

MUIAnno (2026) tests multimodal LLMs on extracting every visible UI element, with its type and bounding box, from screenshots of 1,000 expert-annotated iOS app screens. An element counts as correct when its box overlaps the annotation at IoU 0.5 or more and its type matches, and the score is F1 in percent; higher is better.

Metric: F1-score (%) of UI element extraction on all 1,000 annotated iOS screens: a predicted element counts only if its box reaches IoU 0.5 with a ground-truth element under one-to-one matching and its type label matches; fixed prompt with one reference example, JSON-schema constrained output, deterministic API generation; times 100; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 5 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-5.470#135
2Claude Opus 4.667#113
3Gemini 3.1 Pro (Preview)55#99
4Llama 4 Scout44.5#1818
5Gemma 4 31B (IT)41.6#350

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=muianno · How It Works · Data refreshed daily, snapshot 2026-11-21.