VitaBench 2.0 (RAG Memory): leaderboard
VitaBench 2.0 (2026) personalized agent tasks with retrieval memory: past user interactions sit in a vector-retrieval memory bank queried for each task, and the model must infer and apply the relevant preferences while solving the task with tools. Mean success over four runs in percent, judged by a GPT-4.1 user simulator; higher is better.
Metric: Avg@4 (%, times 100): mean task success over four runs at temperature 0 of VitaBench 2.0 personalized agentic tasks (temporally ordered per-user task sequences across domains, tool-calling agent in an executable environment, preferences scattered over fragmented interaction histories, gpt-4.1-2025-04-14 as user simulator and rubric evaluator), with the interaction history kept in a vector-retrieval (RAG) memory bank queried per task instead of the full context; higher is better. Source: arxiv.org. Saturation forecast: Around March 2028. 20 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | DeepSeek V4 Pro | 43 | #196 | |
| 2 | Claude Opus 4.6 | Thinking | 43 | #113 |
| 3 | GPT-5 | 41 | #178 | |
| 4 | Seed 2.0 Pro | Non-reasoning | 40.6 | #222 |
| 5 | DeepSeek R1 0528 | 39 | #428 | |
| 6 | Kimi K2.6 | Non-reasoning | 38.3 | #192 |
| 7 | GLM-5.1 | Non-reasoning | 38.3 | #198 |
| 8 | Seed-1.6 | 37.5 | #469 | |
| 9 | Claude Sonnet 4.5 | Thinking | 37.4 | #261 |
| 10 | O3 | 36.2 | #237 | |
| 11 | GLM-4.6 | 33.6 | #498 | |
| 12 | GLM-4.5 | 33.6 | #542 | |
| 13 | Gemini 2.5 Pro | 32 | #276 | |
| 14 | Qwen 3 Max | Thinking | 31.5 | #415 |
| 15 | MiniMax-M2.7 | 31.4 | #539 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=vitabench-2-0-rag-memory · How It Works · Data refreshed daily, snapshot 2026-11-21.