KaliBench: leaderboard

Cybersecurity tool use on Kali Linux: from an analyst's natural-language request, the model writes the exact command for the right Kali tool, with no tool list or documentation given (5,000 verified queries). Scored by exact-correct rate after canonicalization, which allows documented flag aliases and reordered optional flags but enforces positional order; blocked or missing answers count as wrong. Higher is better (KaliBench, 2026).

Metric: Exact Correct (%), Unrestricted mode: share of the 5,000 verified queries whose generated Kali Linux command matches the reference after canonicalization (documented aliases and reordered optional flags allowed, positional order enforced), with only the query given; higher is better. Source: arxiv.org. Saturation forecast: Around October 2027. 22 models tracked. Updated 4 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1GPT-5.6 Sol61.68#36
2GPT-5.5xHigh51.68#54
3Claude Opus 544.02#26
4GLM-5.241.3#155
5DeepSeek V3.2Thinking33#423
6Qwen3 Coder Next26.2#728
7Qwen 2.5 72B Instruct25.4#1123
8GPT-OSS-120B24#708
9Qwen 3 32BThinking24#972
10Llama 3.3 70B Instruct22.6#1390
11Qwen 3 8BThinking20.9#1872
12Mistral Small 3.220.7#1650
13RedSage-8B-Ins19.7
14GPT-OSS-20B19.6#1166
15Gemma 3 27B (IT)18.9#1339

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=kalibench · How It Works · Data refreshed daily, snapshot 2026-11-20.