CoCoNot - Contrast Set: leaderboard
The model receives 379 benign requests built to look like CoCoNot's noncompliance prompts (unsupported, incomplete and unsafe look-alikes), and the score is the share it complies with, so higher is better. It is the over-refusal side of CoCoNot; AbstentionBench reuses CoCoNot's noncompliance subsets but not this contrast set.
Metric: Compliance rate on benign contrast requests (%, higher is better). Source: arxiv.org. 19 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-4o (2024-05-13) | 98.4 | #735 |
| 2 | GPT-4 (0613) | 97.4 | #1214 |
| 3 | GPT-4 Preview (1106) | 97.4 | #953 |
| 4 | Mixtral 8x22B Instruct | 96.8 | #2039 |
| 5 | GPT-3.5 Turbo (0125) | 95.3 | #3500 |
| 6 | tulu-2-13B | 93.7 | #7711 |
| 7 | tulu-2-7B | 92.4 | #8164 |
| 8 | Mistral 7B Instruct (v0.2) | 88.4 | #7007 |
| 9 | tulu-2-dpo-13B | 87.3 | #6621 |
| 10 | Llama 3 70B Instruct | 86.5 | #1676 |
| 11 | tulu-2-dpo-7B | 86.3 | #7492 |
| 12 | Llama 3 8B Instruct | 84.2 | #4612 |
| 13 | tulu-2-dpo-70B | 84.2 | #3944 |
| 14 | Claude 3 Sonnet | 80.16 | #2309 |
| 15 | Llama 2 7B Chat | 73.6 | #8805 |
Interactive version: theaggregate.ai/benchmark?slug=coconot-contrast-set · How It Works · Data refreshed daily, snapshot 2026-11-13.