tau-Rec - Pass^4: leaderboard
tau-Rec (2026) tests LLM agents as conversational movie recommenders: through dialogue and catalog tools the agent must elicit the constraints of a simulated user, some stated only on asking or never, and recommend one title that satisfies typed catalog predicates and recommendation policies, over 60 tasks with four trials each. This board is pass^4 in percent, the probability that the agent succeeds on all four trials of a task; higher is better.
Metric: pass^4 (0-1 scaled to %), the probability that the agent succeeds on all four independent trials of a task under the strict constraint and policy check; the agent recommends one movie from a 153-title TMDB catalog to a GPT-5 mini user simulator through dialogue and catalog tools on 60 tasks (constraints revealed up front, only on asking, or never), 4 trials per task, agent temperature 0 where supported; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V4 Flash (High) | 38.3 | #258 (DeepSeek V4 Flash) |
| 2 | Claude Sonnet 4.6 | 35 | #161 |
| 3 | DeepSeek V4 Flash (Max) | 35 | #258 (DeepSeek V4 Flash) |
| 4 | GPT-5.4 (Medium) | 35 | #123 (GPT-5.4) |
| 5 | DeepSeek V4 Flash (Non-reasoning) | 33.3 | #258 (DeepSeek V4 Flash) |
| 6 | GPT-5.4 (Non-reasoning) | 28.3 | #123 (GPT-5.4) |
| 7 | GPT-5 Mini | 25 | #348 |
| 8 | Gemini 2.5 Flash | 13.8 | #453 |
| 9 | Qwen 3 32B | 11.7 | #960 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=tau-rec-pass-pow-4 · How It Works · Data refreshed daily, snapshot 2026-11-13.