FailSafeQA Context Grounding - Irrelevant Context: leaderboard
The finance question comes with a 10-K filing that does not contain the answer, and the model should notice the mismatch and decline or flag it; the score is the share of such queries handled that way. Unlike the missing-context slice, the model has a plausible-looking document it could wrongly cite.
Metric: Share of queries over an irrelevant document declined or flagged (0-1, higher is better). Source: arxiv.org. 23 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 2.5 32B Instruct | 0.89 | #1215 |
| 2 | Gemini 2.0 Flash (Preview) | 0.81 | #736 |
| 3 | Qwen2.5-14B-Instruct-1M | 0.78 | #1414 |
| 4 | Qwen 2.5 7B Instruct | 0.75 | #2564 |
| 5 | Qwen 2.5 14B Instruct | 0.75 | #1732 |
| 6 | Gemini 1.5 Pro (002) | 0.74 | #838 |
| 7 | Qwen 2.5 72B Instruct | 0.69 | #1092 |
| 8 | Llama 3.1 8B Instruct | 0.67 | #4246 |
| 9 | O3 Mini | 0.67 | #533 |
| 10 | Qwen2.5-7B-Instruct-1M | 0.63 | #2407 |
| 11 | O1 | 0.56 | #393 |
| 12 | DeepSeek R1 Distill Qwen 32B | 0.54 | #1788 |
| 13 | Phi-3 Mini 128K Instruct | 0.54 | #6912 |
| 14 | GPT-4o | 0.52 | #741 |
| 15 | Llama 3.1 Nemotron 70B Instruct | 0.52 | #1444 |
Interactive version: theaggregate.ai/benchmark?slug=failsafeqa-context-grounding-irrelevant-context · How It Works · Data refreshed daily, snapshot 2026-11-13.