Cohere Command A Report - Internal Over-Refusal: leaderboard
Same measure on Cohere's internal set of should-answer prompts about safety topics and ambiguous words; the number is the share refused. Lower is better; harder than XSTest for most models in the table.
Metric: Over-refusal rate (%, lower is better). Source: arxiv.org. 14 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 2.5 72B Instruct | 1.2 | #1092 |
| 2 | DeepSeek V3 | 1.2 | #716 |
| 3 | Llama 3.1 405B Instruct | 2.4 | #1115 |
| 4 | Mistral Large 2 (Jul) | 2.4 | #1155 |
| 5 | c4ai-command-r-08-2024 | 2.4 | #3374 |
| 6 | Llama 3.3 70B Instruct | 3.5 | #1357 |
| 7 | Claude 3.5 Sonnet | 4.1 | #746 |
| 8 | Command-R+ (08-2024) | 5.3 | #2551 |
| 9 | GPT-4o | 7.1 | #741 |
| 10 | Command A | 7.1 | #1274 |
| 11 | Llama 3.1 8B Instruct | 8.8 | #4246 |
| 12 | Qwen 2.5 7B Instruct | 9.4 | #2564 |
| 13 | Command R7B | 11.8 | #7513 |
| 14 | Gemma 2 9B (IT) | 24.1 | #2262 |
Interactive version: theaggregate.ai/benchmark?slug=cohere-command-a-report-internal-over-refusal · How It Works · Data refreshed daily, snapshot 2026-11-13.