CARTBench Shopping - Context Excellence: leaderboard
Shopping advice under complex constraints, ambiguous needs and changing timelines;169 cases across17 retail domains. Expert-confirmed pass rate (%). CARTBench Shopping v1 Table2, mean expert-confirmed rate over three complete169-case evaluations. Fixed product evidence and evidence-linked per-case rubrics; no invented snapshot or reasoning setting. This is not CArtBench CuratorQA.
Metric: Expert-confirmed pass rate (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 5 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 40.9 | |
| 2 | Qwen 3.8 Max (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 37.9 | |
| 3 | Kimi K3 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 37.3 | |
| 4 | DeepSeek V4 Pro (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 37 | |
| 5 | MiMo-V2.6-Pro (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 34.1 | |
| 6 | GPT-6.1 Sol (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 32 | |
| 7 | MiniMax-M3 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 27.8 | |
| 8 | Qwen 3.5 122B A10B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 27.7 | |
| 9 | Qwen 3.5 35B A3B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 22.4 | |
| 10 | Qwen 3.5 397B A17B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 19.9 |
Interactive version: theaggregate.ai/benchmark?slug=cartbench-shopping-context-excellence · How It Works · Data refreshed daily, snapshot 2026-11-06.