When2Tool (No Tool Access): leaderboard
The closed-book baseline of When2Tool (Sun et al., 2026): all 2,700 test tasks of its 18 environments answered directly with no tool available, which checks that easy tasks are solvable unaided and hard ones are not. Accuracy in percent of exact-form answers checked by a deterministic evaluator; higher is better.
Metric: Accuracy (%; exact-form answers in a boxed final response, checked by a deterministic evaluator; simulated offline tool environments at easy, medium and hard difficulty; all 2,700 test tasks of the 18 environments answered directly with no tool available). Source: arxiv.org. Saturation forecast: Estimated already saturated. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 32B | 57.7 | #960 |
| 2 | Llama 3.3 70B Instruct | 53 | #1353 |
| 3 | Qwen 3 4B 2507 Instruct | 52 | #2174 |
| 4 | Qwen 3 14B | 50.4 | #1235 |
| 5 | Llama 3.1 8B | 42.7 | #4900 |
| 6 | Qwen 3 1.7B | 22.9 | #5551 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=when2tool-no-tool-access · How It Works · Data refreshed daily, snapshot 2026-11-12.