CRAB-Bench - Concrete-State Verification: leaderboard
CRAB-Bench (Constraint-based Realistic Agent Benchmark, 2026) gives a tool-using agent 200 trip-booking tasks built from constraint graphs, where flights, a hotel and an attraction must jointly satisfy the user among thousands of distractor options. This board is the share of tasks, in percent, whose final booking and wallet databases satisfy every task verifier, with the RUSE simulator, which plays users with personas and human-grounded behaviors in communication style, information disclosure, clarification and error reaction. Higher is better.
Metric: Concrete-state verification pass rate (%): share of tasks whose final booking and wallet databases satisfy every task-specific verifier, with the RUSE human-aligned user simulator (personas sampled uniformly, all four behavior dimensions); 200 trip-booking tasks (departure and return flight, hotel, attraction) generated from constraint graphs with node and edge distractors, tau-bench framework, pass@1, default inference and reasoning settings, Claude Sonnet 4.6 simulates the user; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V3.2 | 73 | #406 |
| 2 | Claude Sonnet 4.6 | 62 | #161 |
| 3 | GLM-5 | 56 | #260 |
| 4 | Qwen3 Coder Next | 28 | #714 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=crab-bench-concrete-state-verification · How It Works · Data refreshed daily, snapshot 2026-11-13.