DiffCap-Bench - Attribute: leaderboard

DiffCap-Bench (2026) image difference captioning restricted to attribute modifications such as size or colour. Macro-averaged F1* in percent, combining the share of true differences described and the share of described differences that are correct, as judged by Gemini 2.5 Pro; higher is better.

Metric: Macro-averaged F1* (%) on attribute modifications such as size or colour: harmonic mean of recall* (share of key differences described) and precision* (share of described differences that are correct) on the 1,075 image pairs of DiffCap-Bench (6,713 human-verified difference items), one fixed captioning prompt, open models at temperature 0.1 and proprietary models at their defaults, a Gemini 2.5 Pro judge matching each described difference against the key difference list; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 14 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Qwen 3 VL Plus82.4#514
2Gemini 3 Pro80.8#146
3Kimi K2.5Thinking80#272
4Seed1.6-VLThinking79.2#389
5Qwen 3 VL 32B (Thinking)78.6#577
6GPT-5.277.1#190
7MiMo-VL-7B-RL-250874.9#829
8Qwen 3 VL 8B (Thinking)73.3#924
9Step3 VL 10B70.9#1058
10Qwen 3 VL 8B Instruct70.4#1043
11Grok 4.2070.3#307
12Qwen 3 VL 32B Instruct68.8#607
13InternVL3.5-38B63.5#1026
14InternVL3.5-8B62#1618

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=diffcap-bench-attribute · How It Works · Data refreshed daily, snapshot 2026-11-22.