Audio2Tool (Text Oracle) - Implicit Reasoning: leaderboard
Tool calling from the ground-truth text of Audio2Tool (2026) spoken requests, the oracle setting that removes speech recognition errors, on the implicit reasoning tier (requests whose intended action is only implied). Exact match of the tool calls and all arguments, in percent; higher is better.
Metric: Exact match (%): share of queries whose predicted tool or ordered tool sequence and all arguments exactly match the reference after deterministic normalization, on the 4,560 Tier 4 implicit-reasoning queries (the intended action is only implied) of Audio2Tool, with the ground-truth text of the query given to a text LLM (the oracle setting that removes ASR errors), zero-shot; higher is better. Source: arxiv.org. Saturation forecast: Around July 2027. 4 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 3 12B | 14.5 | #2059 |
| 2 | Qwen 3 8B | 12.6 | #1873 |
| 3 | Qwen 3 4B | 10.1 | #2348 |
| 4 | Qwen 3 1.7B | 8.9 | #5860 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=audio2tool-text-oracle-implicit-reasoning · How It Works · Data refreshed daily, snapshot 2026-11-21.