KC-Bench - Retail: leaderboard
The input-inconsistency domain of KC-Bench (Lyu et al., 2026): 93 retail customer-service tasks in which the user credentials match a database record on email but not on name, and the agent must clarify before disclosing or changing account data. One trial per task under a user simulator and stateful tools; a task succeeds only if the agent detects the conflict without exposing or changing protected state, as verified by human annotators; percent, higher is better.
Metric: Success rate (%; input inconsistency: 93 retail customer-service tasks in which the user credentials match a database record on email but not name, and the agent must clarify before disclosing or changing account state; one trial per task, greedy decoding, at most 30 interaction steps, LLM user simulator, stateful tools, success only if the conflict is detected and no protected state is exposed or modified; human-verified label (automatic assertion verdicts adjudicated by annotators)). Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Haiku 4.5 (20251001) | 73.12 | #529 |
| 2 | DeepSeek V4 Flash | 65.22 | #254 |
| 3 | GLM-5.2 | 61.96 | #137 |
| 4 | GPT-5.1 | 50.54 | #225 |
| 5 | Gemini 3 Flash (Preview) | 50.54 | #141 |
| 6 | GLM 4.5 Air | 46.23 | #863 |
| 7 | MiniMax-M3 | 38.04 | #265 |
| 8 | Qwen 3.5 35B A3B | 19.35 | #480 |
| 9 | GPT-OSS-120B | 11.82 | #699 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=kc-bench-retail · How It Works · Data refreshed daily, snapshot 2026-11-12.