HAERAE-Vision: leaderboard
Korean visual question answering on real user queries (HAERAE-Vision, What Users Leave Unsaid, Choi et al., ACL 2026 Findings): 653 image questions from Korean online communities in 13 domains, kept in the users' original informal, under-specified wording. Each answer is checked against a 1-5 item checklist derived from the community-accepted answer: a GPT-5-mini judge marks every item met (1), partly met (0.5) or not met (0), and the score is the mean over questions of each question's mean credit, in percent, averaged over three runs; higher is better.
Metric: Checklist Score (%; original under-specified queries, GPT-5-mini judge, mean of 3 runs). Source: arxiv.org. Saturation forecast: Around February 2027. 45 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 2.5 Pro | 48.54 | #262 |
| 2 | GPT-5 | 48.01 | #163 |
| 3 | GPT-5 Mini | 45.21 | #349 |
| 4 | Gemini 2.5 Flash | 41.05 | #452 |
| 5 | Qwen 3 VL 235B A22B Instruct | 38.41 | #521 |
| 6 | Grok 4 | 36.08 | #306 |
| 7 | Qwen 3 VL 32B Instruct | 36.08 | #582 |
| 8 | Qwen 3 VL 32B (Thinking) | 35.49 | #557 |
| 9 | Qwen 3 VL 235B A22B (Thinking) | 35.47 | #451 |
| 10 | Qwen 3 VL 30B A3B (Thinking) | 35.41 | #710 |
| 11 | Qwen 3 VL 30B A3B Instruct | 30.92 | #870 |
| 12 | Gemini 2.5 Flash Lite | 30.29 | #958 |
| 13 | Qwen 3 VL 8B (Thinking) | 28.01 | #887 |
| 14 | Qwen 3 VL 4B (Thinking) | 26.18 | #1303 |
| 15 | Mistral Medium 3.1 | 24.86 | #852 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=haerae-vision · How It Works · Data refreshed daily, snapshot 2026-11-12.