When2Tool (Single-Hop): leaderboard
Tool-use decisions from When2Tool (Sun et al., 2026): 2,250 test tasks in 15 simulated offline environments that probe computation, knowledge and procedure tracing at easy, medium and hard difficulty, where only the hard tasks truly need the tool. Accuracy in percent of exact-form answers checked by a deterministic evaluator, with tools offered under a neutral default prompt, mean of three seeds; higher is better.
Metric: Accuracy (%; exact-form answers in a boxed final response, checked by a deterministic evaluator; simulated offline tool environments at easy, medium and hard difficulty; 2,250 single-hop test tasks in 15 environments; tools offered under the neutral default prompt; mean of three seeds). Source: arxiv.org. Saturation forecast: Estimated already saturated. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 32B | 94.1 | #960 |
| 2 | Qwen 3 14B | 93.7 | #1235 |
| 3 | Qwen 3 4B 2507 Instruct | 89.2 | #2174 |
| 4 | Qwen 3 1.7B | 88.2 | #5551 |
| 5 | Llama 3.3 70B Instruct | 83.1 | #1353 |
| 6 | Llama 3.1 8B | 79.5 | #4900 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=when2tool-single-hop · How It Works · Data refreshed daily, snapshot 2026-11-12.