DiaDemBench - Utterance transcription F1 (ASR): leaderboard
DiaDemBench Overall: dialogue-centric videos with curated dialogue annotations; Gemini-2.5-Pro extracts the dialogue list from each model's audiovisual caption, predicted and reference (speaker, utterance) tuples are aligned by utterance, and ASR is the F1 of correctly transcribed utterances. The human reference row (three authors) is omitted. ARC-Qwen-Video-7B-Narrator is scored on its thinking content because its final answer omits speakers and utterances. DiaDem is the paper's own 7B captioner. Metric: ASR F1. Higher is better. Source: https://arxiv.org/html/2601.19267v1
Metric: ASR F1. Source: arxiv.org. Saturation forecast: Around January 2027. 14 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 2.5 Pro | 74.8 | #151 |
| 2 | Gemini 2.5 Flash | 73.3 | #242 |
| 3 | Gemini 3 Pro | 71 | #78 |
| 4 | Qwen3 Omni 30B A3B Instruct | 47.5 | #379 |
| 5 | Qwen2.5-Omni-7B | 37.1 | #622 |
No result here: #4 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=diadembench-utterance-transcription-f1-asr · How It Works · Data refreshed daily, snapshot 2026-10-18.