STT-Arena - Medium: leaderboard

Medium-difficulty tasks of STT-Arena (2026), a tool-use benchmark in executable stateful environments whose conditions change in time and space during the task, with a simulated user. Pass@1 in percent (every rule-based check must hold in the final state), mean of three runs; higher is better.

Metric: Pass@1 (%) on the 67 medium tasks; a task passes when every rule-based check function holds in the final state; the model acts in an executable stateful environment whose state changes under scheduled spatio-temporal triggers, with a passive Qwen3.5-397B user simulator and at most 50 turns; mean of three runs at temperature 0.7; higher is better. Source: arxiv.org. Saturation forecast: Around January 2028. 21 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude Opus 4.631.34#114
2Gemini 3.1 Pro (Preview)30.35#102
3Claude Sonnet 4.629.85#179
4GPT-5.429.85#138
5DeepSeek V3.227.86#425
6Qwen 3.6 Plus24.38#245
7GLM-5.123.38#200
8GLM-521.39#276
9DeepSeek V4 Pro20.4#201
10STT-Agent-4B (SFT+RL)19.9#324
11STT-Agent-4B (SFT)17.41#424
12MiniMax-M2.715.92#539
13MiniMax-M2.515.92#600
14Gemini 2.5 Flash13.43#466
15Kimi K2.513.43#272

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=stt-arena-medium · How It Works · Data refreshed daily, snapshot 2026-11-22.