RealVuln - F1: leaderboard

RealVuln (2026) benchmarks security scanners on 26 intentionally vulnerable Python repositories with 796 hand-labeled entries, including false-positive traps; LLMs scan in an agentic loop with file, search and shell tools under one shared prompt. This board is the strict F1 score of the findings in percent; higher is better.

Metric: Strict F1 score of vulnerability findings against 796 hand-labeled entries in 26 intentionally vulnerable Python repositories, agentic scanning; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 10 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Sonnet 4.660.9#173
2Gemini 3.1 Pro (Preview)60.1#99
3Claude Opus 4.657.8#113
4GLM-555.5#269
5Kimi K2.554.3#266
6MiniMax-M2.748.7#539
7Claude Haiku 4.547.9#583
8Qwen 3.5 397B A17B46#264
9Grok 4.20Reasoning41#301
10Grok 331.9#611

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=realvuln-f1 · How It Works · Data refreshed daily, snapshot 2026-11-21.