SMH-Bench (Direct Reasoning): leaderboard
SMH-Bench (2026) tests LLM smart-home agents on 1,100 human-audited tasks in HomeEnv, an executable simulator of homes from small apartments to a 31-room, 135-device house; success means the final device state meets the goal without touching unrelated devices, with GPT-5 judging clarifications and query answers. In the Direct Reasoning setting the model sees the full serialized home state and answers once with structured actions that are replayed in the simulator. The board is the success rate in percent over all tasks; higher is better.
Metric: Success rate (%), instance-weighted over 1,100 tasks in 7 capability categories; Direct Reasoning: the full serialized home state (rooms, devices, current values, service schemas) and the request in one prompt, one structured JSON answer replayed in the HomeEnv simulator; final home state checked against the goal with untouched devices preserved; clarification and query answers judged by GPT-5; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 13 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 83.5 | #94 |
| 2 | GPT-5.4 | 80.9 | #123 |
| 3 | Qwen 3.5 397B A17B | 77.6 | #246 |
| 4 | Claude Sonnet 4.6 | 76.7 | #161 |
| 5 | Claude Haiku 4.5 | 76.6 | #568 |
| 6 | DeepSeek V3.2 (Thinking) | 69.5 | #406 (DeepSeek V3.2) |
| 7 | GPT-5.4 Mini | 68.6 | #389 |
| 8 | Qwen 3.5 Plus | 62.6 | #232 |
| 9 | GLM-5 | 59.7 | #260 |
| 10 | MiniMax-M2.7 | 51.8 | #518 |
| 11 | Qwen 3.5 9B | 49.2 | #803 |
| 12 | Qwen 3.5 4B | 45.9 | #1175 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=smh-bench-direct-reasoning · How It Works · Data refreshed daily, snapshot 2026-11-13.