Visa Agentic Commerce Benchmark - All-two Reliability: leaderboard
Visa Agentic Commerce Benchmark (Visa Research, October 2026 results snapshot, released November 5) evaluates whether a shopping assistant can uncover a customer's hidden requirements and select a suitable offer in a fixed simulated market, or correctly decline when nothing fits. Twenty-two models each complete 15 episodes three times, for 45 scored runs per model and 990 total. This series reports All-two reliability (%): mean over 15 episodes of choose(successes,2)/choose(3,2); both sampled runs must pass; not at-least-one Pass@2. Reasoning effort, the exact third-party access service, customer/judge models and harness version are not reported; native model IDs preserve dated and preview variants. Announced tool/format limits count as failure. Nine unannounced-limit truncations were each replaced once; every replacement outcome counts. The separate public demo replays are not part of the scored pool.
Metric: All-two reliability (%): mean over 15 episodes of choose(successes,2)/choose(3,2); both sampled runs must pass; not at-least-one Pass@2; October 2026 snapshot, higher is better. Source: www.visapaymentsfrontier.io. Saturation forecast: Rough model projection: around 2026. 22 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Muse Wonder (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=meta%2Fmuse-wonder; effort=unreported; service=third-party) | 71.11 | |
| 2 | Gemini 4 Argon (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=google%2Fgemini-4-argon; effort=unreported; service=third-party) | 68.89 | |
| 3 | Claude Fable 5.1 (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=anthropic%2Fclaude-fable-5.1; effort=unreported; service=third-party) | 66.67 | |
| 4 | GPT-6.1 Sol (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=openai%2Fgpt-6.1-sol; effort=unreported; service=third-party) | 51.11 | |
| 5 | GLM-5.3 (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=z-ai%2Fglm-5.3; effort=unreported; service=third-party) | 48.89 | |
| 6 | Claude Sonnet 5.5 (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=anthropic%2Fclaude-sonnet-5.5; effort=unreported; service=third-party) | 48.89 | |
| 7 | Claude Opus 5.5 (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=anthropic%2Fclaude-opus-5.5; effort=unreported; service=third-party) | 48.89 | |
| 8 | Grok 4.7 (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=x-ai%2Fgrok-4.7; effort=unreported; service=third-party) | 48.89 | |
| 9 | GPT-6 Astra (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=openai%2Fgpt-6-astra; effort=unreported; service=third-party) | 46.67 | |
| 10 | Qwen 3.8 Max (0902) (Benchmark Sweep Run: Visa Commerce; snapshot=results-2026-11-03; source-model=qwen%2Fqwen3.8-max-0902; effort=unreported; service=third-party) | 46.67 |
Interactive version: theaggregate.ai/benchmark?slug=visa-agentic-commerce-benchmark-all-two-reliability · How It Works · Data refreshed daily, snapshot 2026-11-06.