MMOOC - Multimodal Ambiguity (Multiple Choice): leaderboard
MMOOC tests whether multimodal models decline questions their image and context cannot answer, and explain why. This board covers multiple-choice questions where the image and question leave the answer genuinely ambiguous, and averages the share of correct refusals with a judge rating of the abstention reasoning.
Metric: Out-of-context score (%; mean of the refusal rate, the share of the Multimodal Ambiguity Multiple Choice questions, which the image and context cannot answer, that the model correctly declines, and refusal rationality, the mean 0-1 judge score of its abstention reasoning in steps of 0.25; judge scores averaged over GPT-5.6, Claude Opus 5 and DeepSeek-V4-Pro). Source: arxiv.org. 18 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 4 26B | 70.25 | #612 |
| 2 | Gemma 4 31B | 52.25 | #394 |
| 3 | GPT-4o | 51.25 | #741 |
| 4 | Ministral 3 14B | 50.25 | #1845 |
| 5 | O1 | 45.5 | #393 |
| 6 | Ministral 3 8B | 41.5 | #2023 |
| 7 | O3 | 37.75 | #228 |
| 8 | Qwen 3.5 27B | 32.5 | #359 |
| 9 | Llama 4 Maverick | 32.25 | #1099 |
| 10 | InternVL3-8B | 29.75 | #1688 |
| 11 | Qwen 3.5 122B A10B | 28.25 | #331 |
| 12 | Claude Opus 4.6 | 14.25 | #102 |
| 13 | Gemini 3.1 Pro (Preview) | 4.5 | #94 |
Interactive version: theaggregate.ai/benchmark?slug=mmooc-multimodal-ambiguity-multiple-choice · How It Works · Data refreshed daily, snapshot 2026-11-13.