SecureVibeBench v0.1 - SWE-agent - Functionally Correct: leaderboard
SecureVibeBench v0.1 (ACL 2026): 105 C/C++ coding tasks from 41 OSS-Fuzz projects, each rebuilt at the commit where a human developer introduced a real, reproducible vulnerability (from ARVO), so the agent implements the same requirement in the same codebase. Unlike SWE-bench-style boards, which only check that tests pass, every patch is also run against the original proof-of-vulnerability input and scanned with Semgrep 1.137.0 (more than 2,400 community and Pro rules), and each lands in one outcome: correct and secure (C-SEC), correct but flagged by the scan (C-SUS), correct but reproducing the vulnerability (C-VUL), or incorrect. Agent: SWE-agent 1.1.0 at temperature 0.0 with at most 75 iterations and a cost limit of 2 per task, five backbone models through their official APIs, one run per combination. Score: functional correctness (%), the share of the 105 tasks whose patch passes the functional tests, secure or not (C-SEC plus C-SUS plus C-VUL). The paper prints these combinations only as a stacked bar chart (Figure 4); the counts are read from its vector data, where every segment is a whole number of the 105 tasks, and they reproduce the paper's Table 3 averages. Version 0.2 (2026-10-29) moved to Harbor tasks and mini-swe-agent and may score differently. Higher is better. Source: https://arxiv.org/abs/2509.22097v5
Metric: Functional correctness (%): patches passing the tests. Source: arxiv.org. Saturation forecast: Around December 2027. 5 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 4.5 | 46.67 | #148 |
| 2 | Claude 3.7 Sonnet | 41.9 | #259 |
| 3 | DeepSeek V3.1 | 41.9 | #275 |
| 4 | GPT-5 | 37.14 | #105 |
| 5 | GPT-4.1 | 29.52 | #258 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=securevibebench-v0-1-swe-agent-functionally-correct · How It Works · Data refreshed daily, snapshot 2026-11-05.