CoCoNot - Indeterminate Requests: leaderboard
The model receives questions it cannot determine the answer to (universally unknown facts, information it cannot have, and subjective questions with no single answer), and the score is the share it answers as asked instead of pointing out that the answer is unknown or contested.
Metric: Compliance rate on indeterminate requests (%, lower is better). Source: arxiv.org. 19 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | tulu-2-dpo-13B | 0.7 | #6621 |
| 2 | tulu-2-13B | 0.7 | #7711 |
| 3 | Claude 3 Sonnet | 1.4 | #2309 |
| 4 | tulu-2-dpo-70B | 1.4 | #3944 |
| 5 | Llama 2 70B Chat | 2.1 | #6800 |
| 6 | Mistral 7B Instruct (v0.2) | 2.1 | #7007 |
| 7 | GPT-4 Preview (1106) | 2.1 | #953 |
| 8 | tulu-2-dpo-7B | 2.1 | #7492 |
| 9 | Mixtral 8x22B Instruct | 2.8 | #2039 |
| 10 | Llama 2 13B Chat | 3.5 | #8186 |
| 11 | GPT-4o (2024-05-13) | 4.2 | #735 |
| 12 | tulu-2-7B | 4.2 | #8164 |
| 13 | Llama 3 70B Instruct | 4.9 | #1676 |
| 14 | Llama 2 7B Chat | 7.8 | #8805 |
| 15 | Llama 3 8B Instruct | 9.9 | #4612 |
Interactive version: theaggregate.ai/benchmark?slug=coconot-indeterminate-requests · How It Works · Data refreshed daily, snapshot 2026-11-13.