Olmo 3 Safety Evaluation - WildJailbreak (Benign): leaderboard
The model gets 250 adversarial prompts that look like jailbreaks but are harmless; the score is the share it answers rather than refuses. It is the over-refusal counterpart of WildJailbreak (Harmful): K2-V2-Instruct refuses almost everything and is last here, first there.
Metric: Compliance rate on adversarial benign prompts (%, 250 prompts, higher is better). Source: arxiv.org. 36 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 32B (Thinking) | 100 | #960 (Qwen 3 32B) |
| 2 | Qwen 3 8B (Thinking) | 99.7 | #1830 (Qwen 3 8B) |
| 3 | OLMo 3 32B (Thinking) | 99.7 | #1884 (OLMo 3 32B) |
| 4 | Qwen 2.5 32B Instruct | 99.6 | #1215 |
| 5 | Olmo-3.1-32B-Instruct-DPO | 99.6 | #3174 |
| 6 | Gemma 3 27B (IT) | 99.5 | #1289 |
| 7 | Qwen 2.5 7B Instruct | 99.3 | #2564 |
| 8 | Qwen 3 8B (Non-reasoning) | 99.3 | #1830 (Qwen 3 8B) |
| 9 | Qwen 3 32B (Non-reasoning) | 99.3 | #960 (Qwen 3 32B) |
| 10 | DeepSeek R1 Distill Qwen 32B | 99.2 | #1788 |
| 11 | OLMo-2-0325-32B-Instruct | 99.1 | #2570 |
| 12 | OLMo 3.1 32B Instruct | 98.8 | #2384 |
| 13 | OLMo 3 7B (Thinking) | 98.8 | #3899 (OLMo 3 7B) |
| 14 | Qwen 3 VL 32B Instruct | 98.7 | #585 |
| 15 | Gemma 2 27B (IT) | 98.5 | #1782 |
Interactive version: theaggregate.ai/benchmark?slug=olmo-3-safety-evaluation-wildjailbreak-benign · How It Works · Data refreshed daily, snapshot 2026-11-13.