SSP-Bench - Over-Refusal: leaderboard
The over-refusal service of SSP-Bench (Deniz, Boshmaf and Khalil, QCRI, 2026), a dynamic benchmark that generates items from external sources and selects them for difficulty and separability. Each model answers 684 generated benign questions phrased near the safety boundary, and a gpt-oss-safeguard-120b judge labels each response substantive compliance or refusal. Compliance rate in percent; higher is better.
Metric: Compliance rate (%; share of 684 generated benign boundary questions answered substantively rather than refused, judged by gpt-oss-safeguard-120b; SSP-Bench dynamic benchmark instance scored on the 24-model testing panel, held out from item generation and selection; proprietary models through OpenRouter, March-April 2026). Source: arxiv.org. 24 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3 Flash (Preview) | 100 | #141 |
| 2 | Gemma 3 27B | 99.7 | #1638 |
| 3 | Gemma 3 12B | 99.6 | #2013 |
| 4 | Gemma 3 4B | 99.6 | #5104 |
| 5 | Grok 4 Fast | 99.6 | #438 |
| 6 | Grok 4.1 Fast | 99.4 | #426 |
| 7 | Llama 3.3 70B | 98.7 | #1511 |
| 8 | Gemma 4 26B A4B | 98.5 | #617 |
| 9 | Grok 3 Mini | 98.4 | #535 |
| 10 | Gemma 4 31B | 98 | #393 |
| 11 | DeepSeek R1 0528 | 97.7 | #425 |
| 12 | Claude Haiku 4.5 | 96.8 | #568 |
| 13 | Gemma 3 1B | 96.3 | #9004 |
| 14 | GPT-4o | 96.1 | #741 |
| 15 | GPT-5 Mini | 92.8 | #349 |
Interactive version: theaggregate.ai/benchmark?slug=ssp-bench-over-refusal · How It Works · Data refreshed daily, snapshot 2026-11-12.