CRAB-Bench (Generic User) - Abstract-State Verification: leaderboard
CRAB-Bench (Constraint-based Realistic Agent Benchmark, 2026) gives a tool-using agent 200 trip-booking tasks built from constraint graphs, where flights, a hotel and an attraction must jointly satisfy the user among thousands of distractor options. This board is the share of tasks, in percent, whose conversation passes the factuality and completion checks, with a generic cooperative user simulator. Higher is better.
Metric: Abstract-state verification pass rate (%) with the generic tau2-bench-style user simulator without persona or behaviors; 200 trip-booking tasks (departure and return flight, hotel, attraction) generated from constraint graphs with node and edge distractors, tau-bench framework, pass@1, default inference and reasoning settings, Claude Sonnet 4.6 simulates the user; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V3.2 | 79 | #406 |
| 2 | GLM-5 | 70 | #260 |
| 3 | Qwen3 Coder Next | 63 | #714 |
| 4 | Claude Sonnet 4.6 | 62 | #161 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=crab-bench-generic-user-abstract-state-verification · How It Works · Data refreshed daily, snapshot 2026-11-13.