MultiSessionCollab: leaderboard
MultiSessionCollab (Mehri, Kargupta, August and Hakkani-Tür, 2026) pairs a conversational agent with 100 simulated users, each with a persona and interaction preferences, over 20 problem-solving sessions per task; the user updates a hidden draft answer only when the agent's help respects those preferences and otherwise corrects the agent. Task success is the percentage accuracy of the final draft answer, averaged over MATH-500, MATH-Hard, LogiQA, MMLU and MedQA; higher is better. Agents run without memory across sessions, and Llama-3.3-70B-Instruct simulates the users.
Metric: Task success (%; accuracy of the simulated user's final draft answer, mean of MATH-500, MATH-Hard, LogiQA, MMLU and MedQA, 100 simulated users x 20 sessions, agent without memory). Source: arxiv.org. Saturation forecast: Around February 2027. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-OSS-20B | 42.35 | #1150 |
| 2 | Llama 3.3 70B Instruct | 41.78 | #1353 |
| 3 | Qwen 2.5 7B Instruct | 36.21 | #2583 |
| 4 | Llama 3.1 8B Instruct | 34.04 | #4277 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=multisessioncollab · How It Works · Data refreshed daily, snapshot 2026-11-12.