POLAR-Bench - Privacy: leaderboard

The privacy component of POLAR-Bench (2026): across 10 domains, an agent holding a document, a privacy policy and a task is questioned by a third-party model using five attack strategies, and the score is the share of protected attributes it keeps hidden. Percent; higher is better.

Metric: Privacy score (%): share of the protected attributes the model kept hidden, trusted agent holding a source document, a privacy policy and a task, questioned single-turn or multi-turn by a fixed Llama-3.3-70B-Instruct third party under five attack strategies, attributes revealed in the transcript extracted by normalization and regex, mean over the 10 domains; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 22 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-5.499.86#138
2Gemma 4 31B99.64#390
3Kimi K2.599.42#272
4GLM-5.199.23#200
5DeepSeek V3.199.17#511
6GLM-4.7 Flash98.9#1349
7Llama 3.3 70B Instruct98.22#1394
8Gemma 4 E4B95.45#2199
9OLMo-3.1-32B93.87
10GPT-OSS-20B89#1188
11GPT-OSS-120B88.19#712
12DeepSeek R1 Distill Qwen 32B84.79#1844
13Gemma 3 27B81.61#1672
14Qwen 3 32B77.3#983
15Gemma 4 E2B71.92#4202

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=polar-bench-privacy · How It Works · Data refreshed daily, snapshot 2026-11-22.