MSI-Bench - Mandarin Selective Disclosure: leaderboard
The selective disclosure pattern of MSI-Bench (Xiong et al., 2026), which tests voice assistants on synthesized multi-party audio scenes that end in a request to the assistant, Mandarin split: earlier speech tells the assistant which facts to keep from which listeners, and those listeners later ask for them. All-pass rate in percent over 96 cases, where a case passes only when all its atomic rubrics, judged by DeepSeek V4 Pro, and any required tool call are correct; higher is better.
Metric: All-pass rate (%; the 96 selective-disclosure cases, where earlier speech says which facts to withhold from which listeners and those listeners later ask for them, Mandarin split; a case passes when all its atomic rubrics pass, judged by DeepSeek V4 Pro on the dialogue transcript, with a deterministic tool-call validator as one more rubric for cases that need a tool call; one prediction per model and case from the audio of a multi-party, multi-turn scene ending in an assistant-directed handoff; open-weight models served with JSON-schema constrained decoding). Source: arxiv.org. Saturation forecast: Around April 2028. 15 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 53.1 | #91 |
| 2 | Phi-4 Multimodal Instruct | 36.5 | #3914 |
| 3 | Gemini 3.5 Flash | 35.4 | #98 |
| 4 | Gemma 4 12B (Non-reasoning) | 27.1 | #927 (Gemma 4 12B) |
| 5 | Qwen2.5-Omni-7B | 16.7 | #1797 |
| 6 | Gemma 4 12B (Reasoning) | 13.5 | #927 (Gemma 4 12B) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=msi-bench-mandarin-selective-disclosure · How It Works · Data refreshed daily, snapshot 2026-11-12.