tau-Rec - Constraint Score: leaderboard
tau-Rec (2026) tests LLM agents as conversational movie recommenders: through dialogue and catalog tools the agent must elicit the constraints of a simulated user, some stated only on asking or never, and recommend one title that satisfies typed catalog predicates and recommendation policies, over 60 tasks with four trials each. This board is the mean partial-credit constraint score in percent; higher is better.
Metric: Mean constraint sub-score (0-1 scaled to %), the partial-credit share of typed constraints the recommendation satisfies; the agent recommends one movie from a 153-title TMDB catalog to a GPT-5 mini user simulator through dialogue and catalog tools on 60 tasks (constraints revealed up front, only on asking, or never), 4 trials per task, agent temperature 0 where supported; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V4 Flash (Max) | 59.6 | #258 (DeepSeek V4 Flash) |
| 2 | DeepSeek V4 Flash (High) | 58.6 | #258 (DeepSeek V4 Flash) |
| 3 | DeepSeek V4 Flash (Non-reasoning) | 58.3 | #258 (DeepSeek V4 Flash) |
| 4 | Claude Sonnet 4.6 | 57.9 | #161 |
| 5 | GPT-5.4 (Medium) | 56.5 | #123 (GPT-5.4) |
| 6 | GPT-5.4 (Non-reasoning) | 49.6 | #123 (GPT-5.4) |
| 7 | GPT-5 Mini | 42.9 | #348 |
| 8 | Qwen 3 32B | 36.1 | #960 |
| 9 | Gemini 2.5 Flash | 31.1 | #453 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=tau-rec-constraint-score · How It Works · Data refreshed daily, snapshot 2026-11-13.