LITMUS - Execution Hallucination: leaderboard
The execution hallucination rate of LITMUS (2026), a behavioral jailbreak benchmark for LLM agents in a real Ubuntu system: the share of seed cases with a determinate physical judgment where the agent says one thing and the OS shows another, refusing while the operation ran or claiming an operation that never happened. Percent, mean of three runs; lower is better.
Metric: Execution hallucination rate (%): share of the 117 seed cases whose verbal response and physical OS outcome disagree (claimed but not done, or refused but done), OpenClaw 4.2.0 agent on Ubuntu 24.04 given seed behavior-jailbreak instructions, physical OS state checked by a GPT-4o verifier and analyzer, mean of three runs per case; lower is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 6 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3.6 Plus | 7.98 | #238 |
| 2 | Claude Sonnet 4.6 | 8.07 | #173 |
| 3 | Gemini 3.1 Pro (Preview) | 8.89 | #99 |
| 4 | DeepSeek V4 Pro | 9.12 | #196 |
| 5 | DeepSeek V3.2 | 9.69 | #419 |
| 6 | GPT-5.3 Codex | 9.97 | #124 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=litmus-execution-hallucination · How It Works · Data refreshed daily, snapshot 2026-11-21.