PiSAs (Single Agent): leaderboard
One assistant serves several co-workers and holds all their personal details while making an organisational decision such as ticket allocation or a discharge check; the number is the share of details inappropriate for that task that it passes on, under a High privacy instruction. Unlike CIMemories, the information belongs to several users of one shared system, not to the one user it is serving.
Metric: Appropriateness violation rate (%; share of inappropriate personal attributes exposed, lower is better). Source: arxiv.org. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-6 Astra | 19.3 | #6 |
| 2 | GPT-5.6 Sol | 24.8 | #30 |
| 3 | Kimi K3 | 36.5 | #54 |
| 4 | Claude Sonnet 5 | 39.1 | #111 |
| 5 | Claude Fable 5.1 | 40.3 | #15 |
| 6 | Qwen 3.6 27B | 42 | #341 |
| 7 | Qwen 3.8 Max | 43 | #55 |
| 8 | Kimi K2.5 | 49.5 | #256 |
Interactive version: theaggregate.ai/benchmark?slug=pisas-single-agent · How It Works · Data refreshed daily, snapshot 2026-11-13.