ToolFailBench: leaderboard
ToolFailBench (2026) checks whether LLM agents use tools faithfully on single-turn tasks in finance, medicine, law, cybersecurity and real estate, where mock tool returns contradict plausible remembered values. The clean tool-use rate is the percent of the 750 tool-required tasks in which the model calls the tool and reports its result without skipping, ignoring or fabricating, labelled by a rule classifier and two LLM judges; higher is better.
Metric: Clean tool-use rate (%; share of the 750 tool-required single-turn tasks whose trace calls the needed tool and answers with the returned value, without skipping the tool, ignoring its result or fabricating output; labels by majority vote of a rule classifier and two LLM judges; mock tool returns contradict plausible prior values; temperature 0, 1,024-token output limit; higher is better). Source: arxiv.org. Saturation forecast: Around February 2027. 19 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Grok 4.3 | 86.33 | #333 |
| 2 | Grok 4.1 Fast (Reasoning) | 84.11 | #430 (Grok 4.1 Fast) |
| 3 | Qwen 2.5 32B Instruct | 82.68 | #1215 |
| 4 | Qwen 3.6 27B | 79.33 | #341 |
| 5 | Claude Sonnet 4.5 | 79.28 | #250 |
| 6 | GPT-5.4 Mini | 79.14 | #389 |
| 7 | QwQ-32B | 79.04 | #910 |
| 8 | Qwen 2.5 72B Instruct | 79 | #1092 |
| 9 | Qwen 3.6 35B A3B | 78.47 | #462 |
| 10 | Gemma 4 31B (IT) | 78.12 | #342 |
| 11 | Qwen 3.5 27B | 77.38 | #359 |
| 12 | Claude Haiku 4.5 | 76.47 | #568 |
| 13 | DeepSeek V4 Flash | 75.84 | #258 |
| 14 | GLM-4.7 Flash | 71.49 | #1259 |
| 15 | Qwen 3.5 9B | 70.03 | #803 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=toolfailbench · How It Works · Data refreshed daily, snapshot 2026-11-13.