SecureVibeBench v0.1 - Aider - Correct and Secure: leaderboard
SecureVibeBench v0.1 (ACL 2026): 105 C/C++ coding tasks from 41 OSS-Fuzz projects, each rebuilt at the commit where a human developer introduced a real, reproducible vulnerability (from ARVO), so the agent implements the same requirement in the same codebase. Unlike SWE-bench-style boards, which only check that tests pass, every patch is also run against the original proof-of-vulnerability input and scanned with Semgrep 1.137.0 (more than 2,400 community and Pro rules), and each lands in one outcome: correct and secure (C-SEC), correct but flagged by the scan (C-SUS), correct but reproducing the vulnerability (C-VUL), or incorrect. Agent: Aider 0.86.1 at temperature 0.0 (it has no iteration or cost limit), browsing disabled, five backbone models through their official APIs, one run per combination. Score: C-SEC rate (%), the share of the 105 tasks whose patch passes the functional tests and shows neither the vulnerability nor a new scan finding. The paper prints these combinations only as a stacked bar chart (Figure 4); the counts are read from its vector data, where every segment is a whole number of the 105 tasks, and they reproduce the paper's Table 3 averages. Version 0.2 (2026-10-29) moved to Harbor tasks and mini-swe-agent and may score differently. Higher is better. Source: https://arxiv.org/abs/2509.22097v5
Metric: C-SEC rate (%): functionally correct and secure patches. Source: arxiv.org. Saturation forecast: Around 2029. 5 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V3.1 | 13.33 | #275 |
| 2 | Claude Sonnet 4.5 | 8.57 | #148 |
| 3 | GPT-5 | 5.71 | #105 |
| 4 | Claude 3.7 Sonnet | 3.81 | #259 |
| 5 | GPT-4.1 | 1.9 | #258 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=securevibebench-v0-1-aider-correct-and-secure · How It Works · Data refreshed daily, snapshot 2026-11-05.