VulnGym (Hard): leaderboard

Metric: Advisory recall (%; share of the 31 advisories with at least one reference vulnerability entry whose entry point and critical operation the agent both localizes, end-to-end detection from the advisory title and description; Hard level of VulnGym (reference traces of 9 or more steps: 31 GitHub advisories, 46 vulnerability entries); read-only shell commands; locations match within 5 lines of the same repository-relative path). Source: arxiv.org. Saturation forecast: Around 2029. 9 models tracked.

Top models

Interactive version: theaggregate.ai/benchmark?slug=vulngym-hard · How It Works · Data refreshed daily, snapshot 2026-09-29.