O-IN-Bench: leaderboard

Omni-modal spoken intention understanding benchmark (arXiv:2611.24706v1) evaluating 30 native audio-video models on 1,000 video-audio requests across five categories and 16 subcategories, including silence-required items. Scored on a 0-100 scale by unweighted mean rubric accuracy (language, intent, and content) judged by GPT-6-Luna.

Metric: Overall score (0-100, unweighted mean across 1,000 audio-video samples). Source: arxiv.org. Saturation forecast: Around July 2027. 30 models tracked. Updated 20 Nov 2026.

Top models

#ModelScoreOverall rank
1Gemini 3.7 Flash59.6#42
2Gemini 3.8 Flash58.8#43
3Gemini 4 Argon58.2#18
4Gemini 3.5 Flash58#108
5Qwen3.8 Omni Flash57.3#124
6Gemini 3.1 Pro (Preview)56.9#101
7Gemini 3 Pro (Preview)56.7#120
8Gemini 3.6 Flash55.5#79
9Gemini 3 Flash (Preview)53.3#156
10Muse Spark Open52.4#82
11MiMo-V2.5-310B-A15B50.2
12MiMo-V2.6-Pro-1T-A42B50
13Doubao-Seed-2.0-Mini-26042849.4
14Gemini 2.5 Flash47.7#469
15doubao-seed-2.0-lite-26042847.6#221

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=o-in-bench · How It Works · Data refreshed daily, snapshot 2026-11-20.