KaliBench: leaderboard
Cybersecurity tool use on Kali Linux: from an analyst's natural-language request, the model writes the exact command for the right Kali tool, with no tool list or documentation given (5,000 verified queries). Scored by exact-correct rate after canonicalization, which allows documented flag aliases and reordered optional flags but enforces positional order; blocked or missing answers count as wrong. Higher is better (KaliBench, 2026).
Metric: Exact Correct (%), Unrestricted mode: share of the 5,000 verified queries whose generated Kali Linux command matches the reference after canonicalization (documented aliases and reordered optional flags allowed, positional order enforced), with only the query given; higher is better. Source: arxiv.org. Saturation forecast: Around October 2027. 22 models tracked. Updated 4 Oct 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | GPT-5.6 Sol | 61.68 | #36 | |
| 2 | GPT-5.5 | xHigh | 51.68 | #54 |
| 3 | Claude Opus 5 | 44.02 | #26 | |
| 4 | GLM-5.2 | 41.3 | #155 | |
| 5 | DeepSeek V3.2 | Thinking | 33 | #423 |
| 6 | Qwen3 Coder Next | 26.2 | #728 | |
| 7 | Qwen 2.5 72B Instruct | 25.4 | #1123 | |
| 8 | GPT-OSS-120B | 24 | #708 | |
| 9 | Qwen 3 32B | Thinking | 24 | #972 |
| 10 | Llama 3.3 70B Instruct | 22.6 | #1390 | |
| 11 | Qwen 3 8B | Thinking | 20.9 | #1872 |
| 12 | Mistral Small 3.2 | 20.7 | #1650 | |
| 13 | RedSage-8B-Ins | 19.7 | ||
| 14 | GPT-OSS-20B | 19.6 | #1166 | |
| 15 | Gemma 3 27B (IT) | 18.9 | #1339 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=kalibench · How It Works · Data refreshed daily, snapshot 2026-11-20.