VERHallu - Counterfactual QA: leaderboard
VERHallu (Zhang et al., 2026) probes event-relation hallucination in video LLMs with clips from Mr. Bean films and episodes, whose counterintuitive event chains defeat language and vision-language priors. The 967 counterfactual questions ask about an event that does not occur in the clip, and the correct choice among seven states that the video does not settle it; accuracy in percent, higher is better.
Metric: Accuracy (%; 967 seven-option questions about an event absent from the clip, correct only when the model answers that the video information is incomplete; random 14.3; each model's own frame sampling and default settings, answer given as the option number). Source: arxiv.org. Saturation forecast: Around February 2027. 18 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | ChatGPT 4o | 68.5 | #610 |
| 2 | Qwen 2.5 VL 72B Instruct | 67.6 | #845 |
| 3 | Qwen 2.5 VL 7B Instruct | 50.6 | #1842 |
| 4 | Gemini 3 Pro | 45 | #135 |
| 5 | Qwen 2.5 VL 32B Instruct | 44.9 | #1055 |
| 6 | MiniCPM-V-2.6 | 36.4 | #3095 |
| 7 | InternVL3-8B | 15.6 | #1695 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=verhallu-counterfactual-qa · How It Works · Data refreshed daily, snapshot 2026-11-12.