TOSSS (Java, Security Hint): leaderboard
Metric: TOSSS security score (0-1, times 100): the share of 500 Java function pairs (the function before and after a CVE fix, mined from MegaVul) for which the model picks the secure version when shown both in random order with a prompt that asks for the most secure implementation; a constant or random choice scores about 50; higher is better. Source: arxiv.org. Saturation forecast: Around January 2027. 14 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 | 85.2 | #76 |
| 2 | Claude Opus 4.6 | 84.4 | #60 |
| 3 | GLM-5 | 83.8 | #137 |
| 4 | Kimi K2.5 | 83.6 | #139 |
| 5 | Claude 3.5 Sonnet | 83 | #337 |
| 6 | Gemini 3 Flash (Preview) | 82 | #78 |
| 7 | MiniMax-M2.5 | 81.6 | #295 |
| 8 | Claude Sonnet 4.6 | 81.2 | #85 |
| 9 | Gemini 3.1 Flash Lite (Preview) | 79.6 | #186 |
| 10 | Llama 3 70B Instruct | 77.3 | #624 |
| 11 | Qwen3 Coder Next | 69.6 | #321 |
| 12 | DeepSeek V3.2 | 69.4 | #198 |
| 13 | Mistral Large 3 | 58.8 | #388 |
| 14 | codestral-2508 | 58 | #518 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #8 Claude Fable 5.1.
Interactive version: theaggregate.ai/benchmark?slug=tosss-java-security-hint · How It Works · Data refreshed daily, snapshot 2026-10-11.