tau-Rec: leaderboard
tau-Rec (2026) tests LLM agents as conversational movie recommenders: through dialogue and catalog tools the agent must elicit the constraints of a simulated user, some stated only on asking or never, and recommend one title that satisfies typed catalog predicates and recommendation policies, over 60 tasks with four trials each. The board is pass^1 in percent, the strict per-trial success rate; higher is better.
Metric: pass^1 (0-1 scaled to %), the mean strict success rate per trial: a recommendation succeeds only when every required typed constraint holds against the catalog and every active policy is respected; the agent recommends one movie from a 153-title TMDB catalog to a GPT-5 mini user simulator through dialogue and catalog tools on 60 tasks (constraints revealed up front, only on asking, or never), 4 trials per task, agent temperature 0 where supported; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V4 Flash (Max) | 57.1 | #258 (DeepSeek V4 Flash) |
| 2 | DeepSeek V4 Flash (High) | 56 | #258 (DeepSeek V4 Flash) |
| 3 | GPT-5.4 (Medium) | 55.1 | #123 (GPT-5.4) |
| 4 | DeepSeek V4 Flash (Non-reasoning) | 54.6 | #258 (DeepSeek V4 Flash) |
| 5 | Claude Sonnet 4.6 | 53.7 | #161 |
| 6 | GPT-5.4 (Non-reasoning) | 47.1 | #123 (GPT-5.4) |
| 7 | GPT-5 Mini | 41.7 | #348 |
| 8 | Gemini 2.5 Flash | 27.5 | #453 |
| 9 | Qwen 3 32B | 27.1 | #960 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=tau-rec · How It Works · Data refreshed daily, snapshot 2026-11-13.