SecureVibeBench v0.1 - CLI Agents - Functionally Correct: leaderboard
SecureVibeBench v0.1 (ACL 2026): 105 C/C++ coding tasks from 41 OSS-Fuzz projects, each rebuilt at the commit where a human developer introduced a real, reproducible vulnerability (from ARVO), so the agent implements the same requirement in the same codebase. Unlike SWE-bench-style boards, which only check that tests pass, every patch is also run against the original proof-of-vulnerability input and scanned with Semgrep 1.137.0 (more than 2,400 community and Pro rules), and each lands in one outcome: correct and secure (C-SEC), correct but flagged by the scan (C-SUS), correct but reproducing the vulnerability (C-VUL), or incorrect. Agents: two command-line coding agents, Claude Code with Claude Sonnet 4.5 and Codex with GPT-5 (Table 4); their versions and settings are not stated. Score: functional correctness (%), the share of the 105 tasks whose patch passes the functional tests, secure or not (C-SEC plus C-SUS plus C-VUL). Scores are Table 4's rates as whole numbers of the 105 tasks; the text gives the same correctness rates (48.6% and 46.7%). Version 0.2 (2026-10-29) moved to Harbor tasks and mini-swe-agent and may score differently. Higher is better. Source: https://arxiv.org/abs/2509.22097v5
Metric: Functional correctness (%): patches passing the tests. Source: arxiv.org. Saturation forecast: Around March 2027. 2 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 4.5 | 48.57 | #148 |
| 2 | GPT-5 | 46.67 | #105 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=securevibebench-v0-1-cli-agents-functionally-correct · How It Works · Data refreshed daily, snapshot 2026-11-05.