CoCoNot - Unsupported Requests: leaderboard
The model receives requests it cannot honestly fulfil (other modalities, events after its knowledge cutoff, outputs beyond its length limits), and the score is the share it attempts anyway instead of stating the limitation.
Metric: Compliance rate on unsupported requests (%, lower is better). Source: arxiv.org. 19 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-4 Preview (1106) | 7.6 | #953 |
| 2 | tulu-2-dpo-70B | 7.6 | #3944 |
| 3 | GPT-4 (0613) | 11.5 | #1214 |
| 4 | tulu-2-dpo-13B | 14 | #6621 |
| 5 | Claude 3 Sonnet | 16.8 | #2309 |
| 6 | tulu-2-dpo-7B | 17.8 | #7492 |
| 7 | GPT-4o (2024-05-13) | 19.1 | #735 |
| 8 | GPT-3.5 Turbo (0125) | 21 | #3500 |
| 9 | tulu-2-7B | 21 | #8164 |
| 10 | tulu-2-13B | 21.7 | #7711 |
| 11 | Mixtral 8x22B Instruct | 22.3 | #2039 |
| 12 | Mistral 7B Instruct (v0.2) | 23.6 | #7007 |
| 13 | Llama 3 70B Instruct | 29.9 | #1676 |
| 14 | Llama 3 8B Instruct | 32.5 | #4612 |
| 15 | Llama 2 70B Chat | 40.8 | #6800 |
Interactive version: theaggregate.ai/benchmark?slug=coconot-unsupported-requests · How It Works · Data refreshed daily, snapshot 2026-11-13.