RealVuln: leaderboard

RealVuln (2026) benchmarks security scanners on 26 intentionally vulnerable Python repositories with 796 hand-labeled entries, including false-positive traps; LLMs scan in an agentic loop with file, search and shell tools under one shared prompt. This board is the strict F3 score in percent, which weights recall nine times over precision; higher is better.

Metric: Strict F3 score (recall weighted nine times over precision) of vulnerability findings against 796 hand-labeled entries (676 vulnerabilities, 120 false-positive traps) in 26 intentionally vulnerable Python repositories, agentic scanning with file, search and shell tools; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 10 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Sonnet 4.651.7#173
2Gemini 3.1 Pro (Preview)51#99
3Claude Opus 4.647.7#113
4Kimi K2.546.6#266
5GLM-545.8#269
6MiniMax-M2.739#539
7Qwen 3.5 397B A17B38.1#264
8Claude Haiku 4.537.2#583
9Grok 4.20Reasoning28.4#301
10Grok 321.3#611

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=realvuln · How It Works · Data refreshed daily, snapshot 2026-11-21.