ALPHA CWE Penalty (SVEN): leaderboard
Hierarchy-aware CWE prediction on the 342 Python functions of SVEN (ALPHA, Adnan and Kuhn, University of Canberra, 2026): the model names the single most critical CWE in a vulnerable function, and each wrong answer costs its shortest-path distance to the true CWE in the MITRE CWE graph times a multiplier for the error type (generalising 2.0, lateral 1.8, over-specifying 1.1-1.8), with 17.5 for a CWE outside the graph. The score is the penalty summed over all functions, averaged over three runs at temperature 0.8; lower is better and 0 is perfect.
Metric: ALPHA penalty (sum over 342 vulnerable Python functions of hierarchy distance x direction multiplier between the predicted and true CWE; 0 is perfect; mean of three runs at temperature 0.8). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | qwen2.5-coder:32b [Q4_K_M] | 574.7 | |
| 2 | phi4:14b [Q4_K_M] | 646.4 | |
| 3 | devstral:24b [Q4_K_M] | 695.1 | |
| 4 | qwen2.5-coder:7b [Q4_K_M] | 753.2 | |
| 5 | llama3.1:8b [Q4_K_M] | 813.5 | |
| 6 | mistral:7b [Q4_K_M] | 923.3 | |
| 7 | deepseek-coder:6.7b [Q4_0] | 2557 |
Interactive version: theaggregate.ai/benchmark?slug=alpha-cwe-penalty-sven · How It Works · Data refreshed daily, snapshot 2026-11-12.