Refusing the Impossible: leaderboard

Refusing the Impossible (Dasu, Kundu and Tan, 2026) is an adversarial suite of unsatisfiable coding requests in six programming languages, such as nonexistent packages, fictional algorithms and provably impossible guarantees, where the correct response is to refuse or flag the impossibility. The score is the hallucination rate, the percentage of prompts answered with ungrounded code, judged by a package-registry check for planted entities and a Claude Opus judge otherwise; lower is better.

Metric: Hallucination rate (%; share of the 270 unsatisfiable coding prompts, 24 subcategories in six languages, answered with ungrounded code instead of a refusal or an explicit flag; planted-entity items judged by a deterministic registry detector, the rest by a Claude Opus judge validated against human labels; Ollama at temperature 0). Source: arxiv.org. Saturation forecast: Around February 2027. 12 models tracked.

Top models

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=refusing-the-impossible · How It Works · Data refreshed daily, snapshot 2026-11-12.