SecureVibeBench v0.1 - SWE-agent - Correct and Secure: leaderboard

SecureVibeBench v0.1 (ACL 2026): 105 C/C++ coding tasks from 41 OSS-Fuzz projects, each rebuilt at the commit where a human developer introduced a real, reproducible vulnerability (from ARVO), so the agent implements the same requirement in the same codebase. Unlike SWE-bench-style boards, which only check that tests pass, every patch is also run against the original proof-of-vulnerability input and scanned with Semgrep 1.137.0 (more than 2,400 community and Pro rules), and each lands in one outcome: correct and secure (C-SEC), correct but flagged by the scan (C-SUS), correct but reproducing the vulnerability (C-VUL), or incorrect. Agent: SWE-agent 1.1.0 at temperature 0.0 with at most 75 iterations and a cost limit of 2 per task, five backbone models through their official APIs, one run per combination. Score: C-SEC rate (%), the share of the 105 tasks whose patch passes the functional tests and shows neither the vulnerability nor a new scan finding. The paper prints these combinations only as a stacked bar chart (Figure 4); the counts are read from its vector data, where every segment is a whole number of the 105 tasks, and they reproduce the paper's Table 3 averages. Version 0.2 (2026-10-29) moved to Harbor tasks and mini-swe-agent and may score differently. Higher is better. Source: https://arxiv.org/abs/2509.22097v5

Metric: C-SEC rate (%): functionally correct and secure patches. Source: arxiv.org. Saturation forecast: Around December 2027. 5 models tracked.

Top models

No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.

Interactive version: theaggregate.ai/benchmark?slug=securevibebench-v0-1-swe-agent-correct-and-secure · How It Works · Data refreshed daily, snapshot 2026-11-05.