O-IN-Bench: leaderboard
Omni-modal spoken intention understanding benchmark (arXiv:2611.24706v1) evaluating 30 native audio-video models on 1,000 video-audio requests across five categories and 16 subcategories, including silence-required items. Scored on a 0-100 scale by unweighted mean rubric accuracy (language, intent, and content) judged by GPT-6-Luna.
Metric: Overall score (0-100, unweighted mean across 1,000 audio-video samples). Source: arxiv.org. Saturation forecast: Around July 2027. 30 models tracked. Updated 20 Nov 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.7 Flash | 59.6 | #42 |
| 2 | Gemini 3.8 Flash | 58.8 | #43 |
| 3 | Gemini 4 Argon | 58.2 | #18 |
| 4 | Gemini 3.5 Flash | 58 | #108 |
| 5 | Qwen3.8 Omni Flash | 57.3 | #124 |
| 6 | Gemini 3.1 Pro (Preview) | 56.9 | #101 |
| 7 | Gemini 3 Pro (Preview) | 56.7 | #120 |
| 8 | Gemini 3.6 Flash | 55.5 | #79 |
| 9 | Gemini 3 Flash (Preview) | 53.3 | #156 |
| 10 | Muse Spark Open | 52.4 | #82 |
| 11 | MiMo-V2.5-310B-A15B | 50.2 | |
| 12 | MiMo-V2.6-Pro-1T-A42B | 50 | |
| 13 | Doubao-Seed-2.0-Mini-260428 | 49.4 | |
| 14 | Gemini 2.5 Flash | 47.7 | #469 |
| 15 | doubao-seed-2.0-lite-260428 | 47.6 | #221 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=o-in-bench · How It Works · Data refreshed daily, snapshot 2026-11-20.