POLAR-Bench - Utility: leaderboard

The utility component of POLAR-Bench (2026): across 10 domains, an agent holding a document, a privacy policy and a task must still share the attributes the task needs with a questioning third-party model; the score is the share of task-relevant attributes shared. Percent; higher is better.

Metric: Utility score (%): share of the task-relevant attributes the model shared to complete the task, trusted agent holding a source document, a privacy policy and a task, questioned single-turn or multi-turn by a fixed Llama-3.3-70B-Instruct third party under five attack strategies, attributes revealed in the transcript extracted by normalization and regex, mean over the 10 domains; higher is better. Source: arxiv.org. Saturation forecast: Not forecast. 22 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GLM-5.189.46#198
2Gemma 4 E2B87.56#4317
3Gemma 4 E4B86.29#2184
4GPT-5.484.79#135
5Gemma 3 27B84.71#1663
6Ministral 3 14B84.48#1870
7Qwen 3 32B84.43#972
8Ministral 3 8B83.7#2131
9DeepSeek R1 Distill Llama 70B82.93#1669
10Gemma 4 31B82.77#382
11DeepSeek V3.182.44#511
12Apertus-70B-Instruct-250981.8#2632
13Ministral 3 3B81.46#3504
14OLMo-3.1-32B80.02
15SmolLM3-3B77.66#6555

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=polar-bench-utility · How It Works · Data refreshed daily, snapshot 2026-11-21.