EnvSimBench - Medium: leaderboard

The Medium tier of EnvSimBench (2026): 200 tool calls whose correct outcome changes 3 to 6 changed state fields of the environment state. The model sees the state, the action and its code and must predict the resulting state; the score is Config Match, the percent of samples reproduced exactly, with non-thinking mode throughout; higher is better.

Metric: Config Match (%): share of the 200 Medium state-changing samples (3 to 6 changed state fields) whose predicted state changes, applied to the before-state, reproduce the executor after-state exactly; MDP prompt with the state, the action and its code, non-thinking mode; higher is better. Source: arxiv.org. Saturation forecast: Around 2030. 10 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3.1 Pro (Preview)17.5#102
2GPT-5.4Non-reasoning17.5#138
3Qwen 3.5 397B A17BNon-reasoning17#270
4MiniMax-M2.716#539
5GLM-5Non-reasoning14#276
6Claude Sonnet 4.612#179
7Qwen 3.5 9BNon-reasoning11.6#838
8Qwen 3.5 4BNon-reasoning10.5#1270
9DeepSeek V3.2Non-reasoning8.5#425
10Qwen 3.5 27BNon-reasoning2#380

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=envsimbench-medium · How It Works · Data refreshed daily, snapshot 2026-11-22.