STT-Arena - Hard: leaderboard

Hard tasks, long-horizon cascading constraints that need global replanning, of STT-Arena (2026), a tool-use benchmark in executable stateful environments whose conditions change in time and space during the task, with a simulated user. Pass@1 in percent (every rule-based check must hold in the final state), mean of three runs; higher is better.

Metric: Pass@1 (%) on the 59 hard tasks (long-horizon cascading constraints requiring global replanning); a task passes when every rule-based check function holds in the final state; the model acts in an executable stateful environment whose state changes under scheduled spatio-temporal triggers, with a passive Qwen3.5-397B user simulator and at most 50 turns; mean of three runs at temperature 0.7; higher is better. Source: arxiv.org. Saturation forecast: Around October 2028. 21 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-5.424.29#135
2Claude Opus 4.623.73#113
3Qwen 3.6 Plus23.16#238
4Gemini 3.1 Pro (Preview)21.47#99
5Claude Sonnet 4.619.77#173
6DeepSeek V3.215.82#419
7DeepSeek V4 Pro15.25#196
8GLM-5.115.25#198
9Kimi K2.514.12#266
10STT-Agent-4B (SFT+RL)14.12#316
11MiniMax-M2.713.56#539
12STT-Agent-4B (SFT)13.56#418
13Gemini 2.5 Flash12.43#463
14MiniMax-M2.512.43#600
15Qwen 3.5 9B11.3#828

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=stt-arena-hard · How It Works · Data refreshed daily, snapshot 2026-11-21.