CARTBench Shopping - Core: leaderboard
Shopping advice under complex constraints, ambiguous needs and changing timelines;169 cases across17 retail domains. Expert-confirmed pass rate (%). CARTBench Shopping v1 Table2, mean expert-confirmed rate over three complete169-case evaluations. Fixed product evidence and evidence-linked per-case rubrics; no invented snapshot or reasoning setting. This is not CArtBench CuratorQA.
Metric: Expert-confirmed pass rate (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-6.1 Sol (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 95.9 | |
| 2 | Qwen 3.8 Max (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 95.4 | |
| 3 | DeepSeek V4 Pro (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 91 | |
| 4 | Kimi K3 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 89.3 | |
| 5 | Claude Opus 5 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 89 | |
| 6 | MiniMax-M3 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 88.8 | |
| 7 | MiMo-V2.6-Pro (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 87 | |
| 8 | Qwen 3.5 397B A17B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 80.3 | |
| 9 | Qwen 3.5 122B A10B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 71.3 | |
| 10 | Qwen 3.5 35B A3B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 68.9 |
Interactive version: theaggregate.ai/benchmark?slug=cartbench-shopping-core · How It Works · Data refreshed daily, snapshot 2026-11-06.