CRAB-Bench (Generic User) - Concrete-State Verification: leaderboard
CRAB-Bench (Constraint-based Realistic Agent Benchmark, 2026) gives a tool-using agent 200 trip-booking tasks built from constraint graphs, where flights, a hotel and an attraction must jointly satisfy the user among thousands of distractor options. This board is the share of tasks, in percent, whose final databases satisfy every task verifier, with a generic cooperative user simulator. Higher is better.
Metric: Concrete-state verification pass rate (%) with the generic tau2-bench-style user simulator without persona or behaviors; 200 trip-booking tasks (departure and return flight, hotel, attraction) generated from constraint graphs with node and edge distractors, tau-bench framework, pass@1, default inference and reasoning settings, Claude Sonnet 4.6 simulates the user; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V3.2 | 88 | #406 |
| 2 | Claude Sonnet 4.6 | 76 | #161 |
| 3 | GLM-5 | 76 | #260 |
| 4 | Qwen3 Coder Next | 47 | #714 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=crab-bench-generic-user-concrete-state-verification · How It Works · Data refreshed daily, snapshot 2026-11-13.