MMShopBench (No Verification) - Judge@3: leaderboard
MMShopBench (2026) is a real-log benchmark for multimodal, multi-turn shopping agents that must recommend catalog products meeting intents and requirements expressed through images and dialogue. Share of cases where any of the first three products satisfies every requirement per a GPT-5.5 judge, in percent, without the verification step; higher is better.
Metric: Judge success at 3 (%; share of cases in which any of the first three recommended products satisfies the annotated purchase intent and every mandatory requirement according to a GPT-5.5 multimodal judge given the product evidence; multimodal multi-turn shopping requests from real assistant logs, answered by a tool-using agent over a frozen offline catalog with text and region-aware image retrieval (at most eight tool steps), without the evidence-grounded verification step). Source: arxiv.org. Saturation forecast: Around February 2027. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 69.2 | #94 |
| 2 | Claude Opus 4.8 (Thinking) | 65.4 | #56 (Claude Opus 4.8) |
| 3 | Kimi K2.6 (Thinking) | 43.6 | #178 (Kimi K2.6) |
| 4 | MiniMax-M2.7 | 24.9 | #518 |
| 5 | Qwen 3.5 9B (Non-reasoning) | 11.1 | #803 (Qwen 3.5 9B) |
| 6 | Qwen 3.5 27B (Non-reasoning) | 10 | #359 (Qwen 3.5 27B) |
| 7 | Qwen 3.5 122B A10B (Non-reasoning) | 5.2 | #331 (Qwen 3.5 122B A10B) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=mmshopbench-no-verification-judge-3 · How It Works · Data refreshed daily, snapshot 2026-11-13.