JavaVulBench: leaderboard
JavaVulBench (University of Szeged, 2026) tests method-level Java vulnerability detection under realistic split strategies; each model classifies a fixed, stratified 200-sample probe of the project-disjoint test split. The score is F1 for the vulnerable class in percent; higher is better.
Metric: Detection F1 (%; vulnerable methods as the positive class) of zero-shot method-level Java vulnerability classification on the fixed 200-sample stratified probe (seed 42) of the project-disjoint test split, API-served LLMs through OpenRouter; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-4o | 41.9 | #741 |
| 2 | Claude Sonnet 4 | 41.6 | #400 |
| 3 | GPT-4.1 Mini | 40 | #779 |
| 4 | DeepSeek V3 | 30 | #716 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=javavulbench · How It Works · Data refreshed daily, snapshot 2026-11-13.