CARTBench Shopping - Safety: leaderboard
Shopping advice under complex constraints, ambiguous needs and changing timelines;169 cases across17 retail domains. Expert-confirmed pass rate (%). CARTBench Shopping v1 Table2, mean expert-confirmed rate over three complete169-case evaluations. Fixed product evidence and evidence-linked per-case rubrics; no invented snapshot or reasoning setting. This is not CArtBench CuratorQA.
Metric: Expert-confirmed pass rate (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-6.1 Sol (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 100 | |
| 2 | MiMo-V2.6-Pro (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 95.6 | |
| 3 | DeepSeek V4 Pro (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 95.6 | |
| 4 | Qwen 3.8 Max (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 93.3 | |
| 5 | Kimi K3 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 91.1 | |
| 6 | Claude Opus 5 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 91.1 | |
| 7 | Qwen 3.5 397B A17B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 84.4 | |
| 8 | MiniMax-M3 (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 82.2 | |
| 9 | Qwen 3.5 122B A10B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 71.1 | |
| 10 | Qwen 3.5 35B A3B (Benchmark Sweep Run: source=CARTBenchShopping; native paper configuration; three169-case evaluations; effort and exact snapshots unreported) | 60 |
Interactive version: theaggregate.ai/benchmark?slug=cartbench-shopping-safety · How It Works · Data refreshed daily, snapshot 2026-11-06.