ALPHA CWE Penalty (SecurityEval): leaderboard
Hierarchy-aware CWE prediction on the 121 vulnerable Python functions of SecurityEval, whose labels span pillar, class, base and variant weaknesses (ALPHA, Adnan and Kuhn, University of Canberra, 2026): each wrong single-CWE answer costs its shortest-path distance to the true CWE in the MITRE CWE graph times an error-type multiplier, with 17.5 for a CWE outside the graph. The score is the penalty summed over all functions, averaged over three runs at temperature 0.8; lower is better and 0 is perfect.
Metric: ALPHA penalty (sum over 121 vulnerable Python functions of hierarchy distance x direction multiplier between the predicted and true CWE; 0 is perfect; mean of three runs at temperature 0.8). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | devstral:24b [Q4_K_M] | 578.5 | |
| 2 | qwen2.5-coder:32b [Q4_K_M] | 617.8 | |
| 3 | phi4:14b [Q4_K_M] | 797.4 | |
| 4 | llama3.1:8b [Q4_K_M] | 856.5 | |
| 5 | qwen2.5-coder:7b [Q4_K_M] | 921.5 | |
| 6 | mistral:7b [Q4_K_M] | 1156.9 | |
| 7 | deepseek-coder:6.7b [Q4_0] | 1228.5 |
Interactive version: theaggregate.ai/benchmark?slug=alpha-cwe-penalty-securityeval · How It Works · Data refreshed daily, snapshot 2026-11-12.