Cohere Command A Report - XSTest Over-Refusal: leaderboard
Cohere's run of the XSTest safe prompts (questions that only sound harmful); the number is the share the model fully or partly refuses, as judged by an LLM. Lower is better; the other half of the refusal trade-off measured by the default-safety boards.
Metric: Over-refusal rate (%, lower is better). Source: arxiv.org. 14 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 2.5 72B Instruct | 0.4 | #1092 |
| 2 | Qwen 2.5 7B Instruct | 0.8 | #2564 |
| 3 | DeepSeek V3 | 0.8 | #716 |
| 4 | Mistral Large 2 (Jul) | 0.8 | #1155 |
| 5 | Command A | 1.1 | #1274 |
| 6 | Llama 3.3 70B Instruct | 1.2 | #1357 |
| 7 | Llama 3.1 405B Instruct | 1.2 | #1115 |
| 8 | c4ai-command-r-08-2024 | 3.2 | #3374 |
| 9 | Claude 3.5 Sonnet | 3.6 | #746 |
| 10 | Command-R+ (08-2024) | 4.8 | #2551 |
| 11 | Command R7B | 4.8 | #7513 |
| 12 | GPT-4o | 5.6 | #741 |
| 13 | Llama 3.1 8B Instruct | 6.4 | #4246 |
| 14 | Gemma 2 9B (IT) | 6.8 | #2262 |
Interactive version: theaggregate.ai/benchmark?slug=cohere-command-a-report-xstest-over-refusal · How It Works · Data refreshed daily, snapshot 2026-11-13.