RealVuln: leaderboard
RealVuln (2026) benchmarks security scanners on 26 intentionally vulnerable Python repositories with 796 hand-labeled entries, including false-positive traps; LLMs scan in an agentic loop with file, search and shell tools under one shared prompt. This board is the strict F3 score in percent, which weights recall nine times over precision; higher is better.
Metric: Strict F3 score (recall weighted nine times over precision) of vulnerability findings against 796 hand-labeled entries (676 vulnerabilities, 120 false-positive traps) in 26 intentionally vulnerable Python repositories, agentic scanning with file, search and shell tools; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 10 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | Claude Sonnet 4.6 | 51.7 | #173 | |
| 2 | Gemini 3.1 Pro (Preview) | 51 | #99 | |
| 3 | Claude Opus 4.6 | 47.7 | #113 | |
| 4 | Kimi K2.5 | 46.6 | #266 | |
| 5 | GLM-5 | 45.8 | #269 | |
| 6 | MiniMax-M2.7 | 39 | #539 | |
| 7 | Qwen 3.5 397B A17B | 38.1 | #264 | |
| 8 | Claude Haiku 4.5 | 37.2 | #583 | |
| 9 | Grok 4.20 | Reasoning | 28.4 | #301 |
| 10 | Grok 3 | 21.3 | #611 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=realvuln · How It Works · Data refreshed daily, snapshot 2026-11-21.