SMH-Bench (Direct Reasoning) - State Query: leaderboard
SMH-Bench (2026) category on read-only questions that inspect and aggregate the current home state, scored in the executable HomeEnv smart-home simulator. In the Direct Reasoning setting the model sees the full serialized home state and answers once with structured actions that are replayed in the simulator. The board is the success rate in percent; higher is better.
Metric: Success rate (%) on Environment-Grounded Query (TC7, 100 non-mutating inspection and aggregation questions, GPT-5 judged); Direct Reasoning: the full serialized home state (rooms, devices, current values, service schemas) and the request in one prompt, one structured JSON answer replayed in the HomeEnv simulator; final home state checked against the goal with untouched devices preserved; clarification and query answers judged by GPT-5; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 13 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 | 95 | #123 |
| 2 | Gemini 3.1 Pro (Preview) | 94 | #94 |
| 3 | GPT-5.4 Mini | 89 | #389 |
| 4 | DeepSeek V3.2 (Thinking) | 89 | #406 (DeepSeek V3.2) |
| 5 | Claude Sonnet 4.6 | 85 | #161 |
| 6 | Qwen 3.5 397B A17B | 84 | #246 |
| 7 | Claude Haiku 4.5 | 79 | #568 |
| 8 | GLM-5 | 78 | #260 |
| 9 | Qwen 3.5 9B | 77 | #803 |
| 10 | Qwen 3.5 4B | 75 | #1175 |
| 11 | Qwen 3.5 Plus | 69 | #232 |
| 12 | MiniMax-M2.7 | 63 | #518 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=smh-bench-direct-reasoning-state-query · How It Works · Data refreshed daily, snapshot 2026-11-13.