MuPPET - One-on-One: leaderboard
MuPPET (Multi-Party Privacy Exposure Testing, 2026) tests whether LLM assistants that hold memories of private user information leak it in multi-party workplace group conversations, judged by an LLM under contextual integrity. This board is the leakage rate (%) when the same items are recast as private one-on-one replies with a note that the information must not be discussed; lower is better.
Metric: Privacy leakage rate (%) in the one-on-one control setting: the model receives the full MuPPET item (multi-party workplace group conversation plus memory summaries with private information about the user it acts for) but is told that only the sender of the last message will see its reply, with a note asking it not to discuss the highly sensitive information with that recipient; a response leaks when a contextual-integrity LLM judge finds that it reveals the private detail directly or by strong implication; lower is better. Source: arxiv.org. Saturation forecast: Around February 2027. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 2.5 Pro | 9.88 | #266 |
| 2 | GPT-5.5 | 14.22 | #49 |
| 3 | Qwen 3 14B | 40.99 | #1233 |
| 4 | Llama 3 8B Instruct | 45.68 | #4612 |
| 5 | Qwen 3 4B | 53.09 | #2281 |
| 6 | Llama 3.1 8B Instruct | 62.5 | #4246 |
| 7 | Qwen 3 8B | 63.58 | #1830 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=muppet-one-on-one · How It Works · Data refreshed daily, snapshot 2026-11-13.