CoCoNot - Incomplete Requests: leaderboard
The model receives underspecified, incomprehensible or false-premise requests, and the score is the share it answers as asked instead of asking for clarification or correcting the premise. AbstentionBench scores some of the same CoCoNot subsets on newer models with an abstention F1; this is the paper's own compliance rate.
Metric: Compliance rate on incomplete requests (%, lower is better). Source: arxiv.org. 19 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Mixtral 8x22B Instruct | 7.6 | #2039 |
| 2 | GPT-4o (2024-05-13) | 8.9 | #735 |
| 3 | Llama 2 70B Chat | 10.1 | #6800 |
| 4 | Claude 3 Sonnet | 10.2 | #2309 |
| 5 | Mistral 7B Instruct (v0.2) | 11.1 | #7007 |
| 6 | tulu-2-dpo-70B | 12 | #3944 |
| 7 | tulu-2-dpo-7B | 17.3 | #7492 |
| 8 | tulu-2-dpo-13B | 17.3 | #6621 |
| 9 | Llama 3 70B Instruct | 17.5 | #1676 |
| 10 | tulu-2-13B | 21.3 | #7711 |
| 11 | Llama 2 13B Chat | 22.2 | #8186 |
| 12 | GPT-4 Preview (1106) | 22.7 | #953 |
| 13 | Llama 2 7B Chat | 24.4 | #8805 |
| 14 | tulu-2-7B | 25.8 | #8164 |
| 15 | Llama 3 8B Instruct | 28.4 | #4612 |
Interactive version: theaggregate.ai/benchmark?slug=coconot-incomplete-requests · How It Works · Data refreshed daily, snapshot 2026-11-13.