STT-Arena - Impossible: leaderboard

Impossible tasks, where no valid completion exists, of STT-Arena (2026), a tool-use benchmark in executable stateful environments whose conditions change in time and space during the task, with a simulated user. Pass@1 in percent (a judge must find that the agent recognized the infeasibility and told the user), mean of three runs; higher is better.

Metric: Pass@1 (%) on the 30 impossible tasks (no valid completion path exists); a task passes when a Qwen3.5-397B judge finds that the model both recognized the infeasibility and told the user; the model acts in an executable stateful environment whose state changes under scheduled spatio-temporal triggers, with a passive Qwen3.5-397B user simulator and at most 50 turns; mean of three runs at temperature 0.7; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 22 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1STT-Agent-4B (SFT+RL)64.44#324
2DeepSeek V3.263.33#425
3STT-Agent-4B (SFT)61.11#424
4Claude Sonnet 4.658.89#179
5Qwen 3.6 Plus54.44#245
6Gemini 3.1 Pro (Preview)52.22#102
7GPT-5.452.22#138
8Claude Opus 4.652.22#114
9DeepSeek V4 Pro45.56#201
10GLM-543.33#276
11MiniMax-M2.542.22#600
12GLM-5.141.11#200
13Kimi K2.537.78#272
14MiniMax-M2.736.67#539
15Qwen 3.5 9B32.22#838

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=stt-arena-impossible · How It Works · Data refreshed daily, snapshot 2026-11-22.