CIMemories: leaderboard
The model has a ChatGPT-style memory of about 147 facts about one user and writes for a task (for example an email to an employer) where only a few of those facts belong; the number is the share of facts that should stay out but appear in at least one of five sampled replies. Unlike PiSAs, one user and one model, and the leak comes from long-term memory rather than other users' data; a model that shares nothing scores 0, so read it beside the paper's completeness column.
Metric: Violation@5 (%; share of not-to-share memory attributes revealed in any of 5 responses, lower is better). Source: arxiv.org. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-4o | 14.82 | #741 |
| 2 | GPT-5 | 25.08 | #167 |
| 3 | O3 | 38.51 | #228 |
| 4 | Llama 3.3 70B Instruct | 44.43 | #1357 |
| 5 | Claude Sonnet 4 | 44.44 | #400 |
| 6 | Gemini 2.5 Flash | 46.35 | #453 |
| 7 | Mistral 7B Instruct (v0.3) | 56.94 | #6653 |
| 8 | Qwen 3 32B | 69.14 | #960 |
Interactive version: theaggregate.ai/benchmark?slug=cimemories · How It Works · Data refreshed daily, snapshot 2026-11-13.