CRAB-Bench - Abstract-State Verification: leaderboard
CRAB-Bench (Constraint-based Realistic Agent Benchmark, 2026) gives a tool-using agent 200 trip-booking tasks built from constraint graphs, where flights, a hotel and an attraction must jointly satisfy the user among thousands of distractor options. This board is the share of tasks, in percent, whose conversation passes factuality (bookings match what the agent stated) and completion checks, with the RUSE simulator, which plays users with personas and human-grounded behaviors in communication style, information disclosure, clarification and error reaction. Higher is better.
Metric: Abstract-state verification pass rate (%): share of tasks whose conversation passes factuality (actions match the stated plan) and completion (interaction ends correctly), with the RUSE human-aligned user simulator (personas sampled uniformly, all four behavior dimensions); 200 trip-booking tasks (departure and return flight, hotel, attraction) generated from constraint graphs with node and edge distractors, tau-bench framework, pass@1, default inference and reasoning settings, Claude Sonnet 4.6 simulates the user; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V3.2 | 79 | #406 |
| 2 | GLM-5 | 65 | #260 |
| 3 | Qwen3 Coder Next | 63 | #714 |
| 4 | Claude Sonnet 4.6 | 60 | #161 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=crab-bench-abstract-state-verification · How It Works · Data refreshed daily, snapshot 2026-11-13.