ROSE - Local Count: leaderboard
ROSE (2026) measures the gap between seeing and acting in multimodal models: grid images hold repeated majority elements and a few exceptions from five controlled visual sources, and models must count or click the exceptions exactly under changing region constraints. This board is strict pass rate (%) for counting the exceptions inside a numerically specified region, averaged over the visual sources; higher is better.
Metric: Strict PASS (%) on T2 local counting: the number of exceptions inside a numerically specified row, column or rectangle region, macro-averaged over the five visual sources, on the 5,560-instance test split of ROSE v0.1: grid images of repeated majority elements with a few exceptions, from five controlled visual sources (confusable Chinese glyphs, emoji styles and identities, pixel-art edits and assets); strict PASS requires an exact count or an exact set of clicked grid coordinates in the prescribed answer grammar; temperature 0, reasoning or thinking disabled where supported; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.5 (Non-reasoning) | 97 | #49 (GPT-5.5) |
| 2 | Gemini 3.1 Pro (Preview) | 93.9 | #94 |
| 3 | Qwen 3.6 Plus (Non-reasoning) | 65.3 | #222 (Qwen 3.6 Plus) |
| 4 | GLM-4.6V (Non-reasoning) | 30.8 | #701 (GLM-4.6V) |
| 5 | Claude Sonnet 4.6 | 21.6 | #161 |
| 6 | Claude Opus 4.8 | 21.2 | #56 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=rose-local-count · How It Works · Data refreshed daily, snapshot 2026-11-13.