SecureVibeBench v0.1 - CLI Agents - Correct and Secure: leaderboard
SecureVibeBench v0.1 (ACL 2026): 105 C/C++ coding tasks from 41 OSS-Fuzz projects, each rebuilt at the commit where a human developer introduced a real, reproducible vulnerability (from ARVO), so the agent implements the same requirement in the same codebase. Unlike SWE-bench-style boards, which only check that tests pass, every patch is also run against the original proof-of-vulnerability input and scanned with Semgrep 1.137.0 (more than 2,400 community and Pro rules), and each lands in one outcome: correct and secure (C-SEC), correct but flagged by the scan (C-SUS), correct but reproducing the vulnerability (C-VUL), or incorrect. Agents: two command-line coding agents, Claude Code with Claude Sonnet 4.5 and Codex with GPT-5 (Table 4); their versions and settings are not stated. Score: C-SEC rate (%), the share of the 105 tasks whose patch passes the functional tests and shows neither the vulnerability nor a new scan finding. Scores are Table 4's rates as whole numbers of the 105 tasks; the text gives the same correctness rates (48.6% and 46.7%). Version 0.2 (2026-10-29) moved to Harbor tasks and mini-swe-agent and may score differently. Higher is better. Source: https://arxiv.org/abs/2509.22097v5
Metric: C-SEC rate (%): functionally correct and secure patches. Source: arxiv.org. Saturation forecast: Around September 2028. 2 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5 | 19.05 | #105 |
| 2 | Claude Sonnet 4.5 | 17.14 | #148 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=securevibebench-v0-1-cli-agents-correct-and-secure · How It Works · Data refreshed daily, snapshot 2026-11-05.