KAware: leaderboard
KAware (2026) tests the self-awareness of tool-using LLM agents: whether an agent knows which external tools a task needs and then calls exactly those, across tasks that need tools, mix tools with parametric knowledge, or need no tools at all. The score is the task-weighted knowing-acting score, the harmonic mean of the two tool-set overlaps, in percent; higher is better.
Metric: Average knowing-acting score (%, KAS): harmonic mean of knowing accuracy (Jaccard overlap between the tools the agent says the task needs and the gold tool set) and acting accuracy (Jaccard overlap between the tools it actually calls and the gold set), task-weighted over the three settings, on the KAware tool-use tasks, where tools are always available but the task may need them (external function, 310 tasks), mix them with parametric knowledge (hybrid composition, 368 tasks) or be solvable from knowledge alone (internal function, 398 tasks); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 18 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5 | 69.73 | #167 |
| 2 | GPT-5.5 | 65.55 | #49 |
| 3 | O4 Mini (2025-04-16) | 64.15 | #365 |
| 4 | Gemini 3 Pro (Preview) | 62.9 | #113 |
| 5 | Qwen 3.5 397B A17B | 56.92 | #246 |
| 6 | Qwen 3.5 397B A17B (Non-reasoning) | 55.72 | #246 (Qwen 3.5 397B A17B) |
| 7 | GPT-4.1 | 54.75 | #495 |
| 8 | Gemini 3 Flash (Preview) | 53.61 | #144 |
| 9 | DeepSeek V4 Flash (Non-reasoning) | 53.4 | #258 (DeepSeek V4 Flash) |
| 10 | Claude Sonnet 4.5 | 53.34 | #250 |
| 11 | GPT-4o (2024-11-20) | 53.1 | #866 |
| 12 | Qwen 3 Max (Preview) | 51.99 | #447 |
| 13 | DeepSeek R1 0528 | 49.8 | #428 |
| 14 | DeepSeek V3.2 (Non-reasoning) | 49.14 | #406 (DeepSeek V3.2) |
| 15 | Qwen 3 235B A22B 2507 Instruct | 42.43 | #619 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=kaware · How It Works · Data refreshed daily, snapshot 2026-11-13.