QuoteBench: leaderboard
QuoteBench (Li et al., 2026) checks whether one-shot LLM-generated Bash commands reach an exact final state across 56 tasks in 14 quoting-hazard families, such as heredocs, hostile filenames and nested shells. Here the model is told about the command boundary and its reply is replayed through an added nested shell parser, each model at its best-observed effort setting. Task success in percent; higher is better.
Metric: Task success (%; exact final-state validation over 56 one-shot Bash tasks in 14 operation families, 14 benign controls and 42 hostile payloads; disclosed-boundary generation contract with the reply replayed through one added nested shell parser; each model at its best-observed effort setting, one stored trial). Source: arxiv.org. Saturation forecast: Estimated already saturated. 13 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.5 (xHigh) | 100 | #48 (GPT-5.5) |
| 2 | Claude Fable 5 (Max) | 100 | #26 (Claude Fable 5) |
| 3 | Claude Opus 5 (xHigh) | 100 | #20 (Claude Opus 5) |
| 4 | GPT-5.6 Sol (High) | 98.2 | #29 (GPT-5.6 Sol) |
| 5 | Claude Opus 4.8 (Max) | 94.6 | #56 (Claude Opus 4.8) |
| 6 | Gemini 3.1 Pro (Preview) (Low) | 91.1 | #91 (Gemini 3.1 Pro (Preview)) |
| 7 | Claude Sonnet 4.6 (High) | 64.3 | #158 (Claude Sonnet 4.6) |
| 8 | Gemini 3.5 Flash (Medium) | 64.3 | #98 (Gemini 3.5 Flash) |
| 9 | Qwen 3.5 27B (Non-reasoning) | 30.4 | #358 (Qwen 3.5 27B) |
| 10 | Qwen 3.5 4B | 21.4 | #1180 |
| 11 | Qwen 3.5 9B (Non-reasoning) | 17.9 | #803 (Qwen 3.5 9B) |
| 12 | Gemini 3.1 Flash Lite (Preview) | 14.3 | #413 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=quotebench · How It Works · Data refreshed daily, snapshot 2026-11-12.