CARTBench Shopping - Constraints Excellence: leaderboard
Shopping advice under complex constraints, ambiguous needs and changing timelines;169 cases across17 retail domains. Expert-confirmed pass rate (%). CARTBench Shopping v1 Table2, mean expert-confirmed rate over three complete169-case evaluations. Fixed product evidence and evidence-linked per-case rubrics; no invented snapshot or reasoning setting. This is not CArtBench CuratorQA.
Metric: Expert-confirmed pass rate (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 5 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 37.1 | |
| 2 | Kimi K3 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 35.3 | |
| 3 | Qwen 3.8 Max (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 35.1 | |
| 4 | DeepSeek V4 Pro (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 34.8 | |
| 5 | MiMo-V2.6-Pro (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 30.1 | |
| 6 | GPT-6.1 Sol (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 26.8 | |
| 7 | MiniMax-M3 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 23.5 | |
| 8 | Qwen 3.5 122B A10B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 22.7 | |
| 9 | Qwen 3.5 35B A3B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 19.9 | |
| 10 | Qwen 3.5 397B A17B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 17.6 |
Interactive version: theaggregate.ai/benchmark?slug=cartbench-shopping-constraints-excellence · How It Works · Data refreshed daily, snapshot 2026-11-06.