Olmo 3 Safety Evaluation - StrongREJECT: leaderboard
The model gets about 2,300 forbidden requests, each wrapped in one of 37 jailbreak techniques or none; the StrongREJECT judge rates harmfulness 1-5 and the score reverses it to 0-100, higher meaning safer. Unlike HarmBench, it scores how useful the harmful answer would be, not only whether the model refused.
Metric: Safety score (reversed StrongREJECT judge score, 0-100, 2,294 jailbreak prompts, higher is better). Source: arxiv.org. 36 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Olmo-3.1-32B-Instruct-SFT | 95.5 | #3407 |
| 2 | OLMo 3.1 32B Instruct | 91.7 | #2384 |
| 3 | OLMo-3.1 32B Think | 90.5 | #1443 |
| 4 | Qwen 3 VL 32B Instruct | 90.2 | #585 |
| 5 | OLMo-2-1124-7B-Instruct | 89.4 | #7343 |
| 6 | Olmo-3.1-32B-Instruct-DPO | 89.3 | #3174 |
| 7 | Qwen 3 VL 32B (Thinking) | 88.5 | #559 |
| 8 | Gemma 2 27B (IT) | 88.1 | #1782 |
| 9 | OLMo 3 7B Instruct | 88.1 | #6860 |
| 10 | OLMo-2-0325-32B-Instruct | 87.4 | #2570 |
| 11 | Qwen 3 32B (Non-reasoning) | 86.9 | #960 (Qwen 3 32B) |
| 12 | Qwen 2.5 32B Instruct | 85.6 | #1215 |
| 13 | NVIDIA Nemotron Nano 9B V2 | 85.6 | #2270 |
| 14 | Qwen 3 VL 8B Instruct | 85.3 | #981 |
| 15 | Gemma 3 27B (IT) | 84.6 | #1289 |
Interactive version: theaggregate.ai/benchmark?slug=olmo-3-safety-evaluation-strongreject · How It Works · Data refreshed daily, snapshot 2026-11-13.