SpeakerSleuth - Detection: leaderboard
Speaker-consistency detection task of SpeakerSleuth (Lee, Pyo, Seo and Jo, Seoul National University, ACL 2026): an audio-language model hears the five turns of one speaker in a dialogue plus a reference clip and must say whether every turn comes from that speaker, where the inconsistent versions swap one turn to an opposite-gender or an acoustically similar voice by voice conversion. The score is balanced accuracy in percent (half the accuracy on the consistent version plus half the mean accuracy on the two inconsistent versions, 606 dialogues each), so answering one class every time scores 50; higher is better.
Metric: Balanced accuracy (%; 0.5 x (S1 + mean(S2, S3)) over 606 dialogues per scenario). Source: arxiv.org. Saturation forecast: Estimated already saturated. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 2.5 Pro | 64.7 | #262 |
| 2 | Gemini 2.5 Flash | 63.1 | #452 |
| 3 | Gemini 2.5 Flash Lite | 55.3 | #958 |
| 4 | Phi-4 Multimodal Instruct | 51.7 | #3914 |
| 5 | Qwen2.5-Omni-7B | 51.5 | #1797 |
| 6 | Gemma 3n E4B | 50.4 | #3330 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=speakersleuth-detection · How It Works · Data refreshed daily, snapshot 2026-11-12.