SciHazard - Oversafety: leaderboard
SciHazard (Shanghai AI Laboratory, 2026) measures scientific misuse risk with 2,400 hazardous questions grounded in regulated substances and documented misuse scenarios across 12 disciplines, plus 600 benign questions that reuse the same trigger words. Over-refusal is the share of the 600 benign defence, detection and emergency-response questions that are refused, in percent; lower is better; this board covers standard language models answering directly.
Metric: Over-refusal rate (%; share of 600 benign defence, detection and emergency-response questions that reuse hazardous trigger words and are refused; lower is better; standard LLMs at temperature 0). Source: arxiv.org. 19 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Llama 3.1 70B Instruct | 6.8 | #1546 |
| 2 | Claude Haiku 4.5 | 8.8 | #568 |
| 3 | Llama 4 Scout | 10.7 | #1776 |
| 4 | Kimi K2.5 | 15.3 | #256 |
| 5 | Qwen 3.5 35B A3B | 16.8 | #482 |
| 6 | O3 | 16.8 | #228 |
| 7 | Gemini 3 Flash | 18.3 | #164 |
| 8 | Grok 4.1 Fast | 18.3 | #430 |
| 9 | GLM-5.1 | 20.3 | #179 |
| 10 | Qwen 3.5 397B A17B | 21.8 | #246 |
| 11 | Gemini 3.1 Pro (Preview) | 22.8 | #94 |
| 12 | GPT-5.4 | 26 | #123 |
| 13 | GPT-5.2 | 26.5 | #176 |
| 14 | O3 Mini | 34.7 | #533 |
| 15 | Step 3.5 Flash | 44.5 | #511 |
Interactive version: theaggregate.ai/benchmark?slug=scihazard-oversafety · How It Works · Data refreshed daily, snapshot 2026-11-13.