OVIBench (Open-Ended) - Intent Fulfillment: leaderboard
OVIBench (2026) evaluates how vision-language models handle user interruptions while answering questions about streaming video, using an offline simulation of interruptions during answer generation. Share of open-ended post-interruption responses that satisfy the user intent, judged by Qwen3-VL-235B-A22B, in percent; higher is better.
Metric: Intent fulfillment accuracy (%; share of post-interruption responses that satisfy the user actual intent, judged 0/1 by Qwen3-VL-235B-A22B; open-ended subset of 13,145 samples; offline simulation of an interruption arriving while the model is generating its answer about a streaming video (3,200 videos from ActivityNet, MovieChat, QVHighlights, UCF-Crime and YouCook2)). Source: arxiv.org. Saturation forecast: Estimated already saturated. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Seed-1.6 | 79.42 | #447 |
| 2 | Gemini 2.5 Flash | 64.58 | #452 |
| 3 | Qwen 2.5 VL 7B | 50.11 | #1796 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=ovibench-open-ended-intent-fulfillment · How It Works · Data refreshed daily, snapshot 2026-11-12.