VulnGym (Hard): leaderboard

VulnGym (Tencent, CUHK, Fudan University, HKU and Peking University, 2026) pairs reviewed GitHub security advisories with their vulnerable repository versions and annotates each vulnerability entry with its entry point, critical operation and line-level trace. On the Hard level, 31 advisories whose vulnerability traces have nine or more steps, a coding agent reads the advisory, explores the repository with read-only commands and reports vulnerability entries; advisory recall is the percentage of advisories for which it localizes at least one entry point and critical operation pair, higher is better.

Metric: Advisory recall (%; share of the 31 advisories with at least one reference vulnerability entry whose entry point and critical operation the agent both localizes, end-to-end detection from the advisory title and description; Hard level of VulnGym (reference traces of 9 or more steps: 31 GitHub advisories, 46 vulnerability entries); read-only shell commands; locations match within 5 lines of the same repository-relative path). Source: arxiv.org. Saturation forecast: Around July 2028. 9 models tracked.

Top models

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=vulngym-hard · How It Works · Data refreshed daily, snapshot 2026-11-13.