tau-Rec - Policy Compliance: leaderboard
tau-Rec (2026) tests LLM agents as conversational movie recommenders: through dialogue and catalog tools the agent must elicit the constraints of a simulated user, some stated only on asking or never, and recommend one title that satisfies typed catalog predicates and recommendation policies, over 60 tasks with four trials each. This board is the mean programmatic policy-compliance score in percent over seven policies such as availability, watch history and transparency; higher is better.
Metric: Mean policy sub-score (0-1 scaled to %), programmatic compliance with the seven recommendation policies (recommend tool, watch history, availability, age restriction, sponsorship disclosure, transparency, single recommendation); the agent recommends one movie from a 153-title TMDB catalog to a GPT-5 mini user simulator through dialogue and catalog tools on 60 tasks (constraints revealed up front, only on asking, or never), 4 trials per task, agent temperature 0 where supported; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5 Mini | 96.3 | #348 |
| 2 | GPT-5.4 (Medium) | 95.8 | #123 (GPT-5.4) |
| 3 | DeepSeek V4 Flash (Max) | 94.2 | #258 (DeepSeek V4 Flash) |
| 4 | DeepSeek V4 Flash (High) | 93.7 | #258 (DeepSeek V4 Flash) |
| 5 | DeepSeek V4 Flash (Non-reasoning) | 92.1 | #258 (DeepSeek V4 Flash) |
| 6 | Gemini 2.5 Flash | 92 | #453 |
| 7 | GPT-5.4 (Non-reasoning) | 90.8 | #123 (GPT-5.4) |
| 8 | Claude Sonnet 4.6 | 89.6 | #161 |
| 9 | Qwen 3 32B | 75.6 | #960 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=tau-rec-policy-compliance · How It Works · Data refreshed daily, snapshot 2026-11-13.