Audio2Tool (Text Oracle) - Implicit Reasoning: leaderboard

Tool calling from the ground-truth text of Audio2Tool (2026) spoken requests, the oracle setting that removes speech recognition errors, on the implicit reasoning tier (requests whose intended action is only implied). Exact match of the tool calls and all arguments, in percent; higher is better.

Metric: Exact match (%): share of queries whose predicted tool or ordered tool sequence and all arguments exactly match the reference after deterministic normalization, on the 4,560 Tier 4 implicit-reasoning queries (the intended action is only implied) of Audio2Tool, with the ground-truth text of the query given to a text LLM (the oracle setting that removes ASR errors), zero-shot; higher is better. Source: arxiv.org. Saturation forecast: Around July 2027. 4 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemma 3 12B14.5#2059
2Qwen 3 8B12.6#1873
3Qwen 3 4B10.1#2348
4Qwen 3 1.7B8.9#5860

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=audio2tool-text-oracle-implicit-reasoning · How It Works · Data refreshed daily, snapshot 2026-11-21.