DiaDemBench - Speaker attribution F1 (REF): leaderboard
DiaDemBench Overall: dialogue-centric videos with curated dialogue annotations; Gemini-2.5-Pro extracts the dialogue list from each model's audiovisual caption, predicted and reference (speaker, utterance) tuples are aligned by utterance, and REF is the F1 of correct speaker attribution among matched tuples. The human reference row (three authors) is omitted. ARC-Qwen-Video-7B-Narrator is scored on its thinking content because its final answer omits speakers and utterances. DiaDem is the paper's own 7B captioner. Metric: REF F1. Higher is better. Source: https://arxiv.org/html/2601.19267v1
Metric: REF F1. Source: arxiv.org. Saturation forecast: Around January 2027. 14 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 2.5 Pro | 63.6 | #151 |
| 2 | Gemini 3 Pro | 63.1 | #78 |
| 3 | Gemini 2.5 Flash | 58.5 | #242 |
| 4 | Qwen3 Omni 30B A3B Instruct | 36.8 | #379 |
| 5 | Qwen2.5-Omni-7B | 26.1 | #622 |
No result here: #4 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=diadembench-speaker-attribution-f1-ref · How It Works · Data refreshed daily, snapshot 2026-10-18.