SIMMER - Latent Failures: leaderboard
SIMMER (2026) tests LLM planners on 100 cooking tasks by executing each complete plan in a human-curated symbolic kitchen world model (77 actions, 262 objects) that flags immediate failures and latent failures, valid steps that leave hazardous or corrupted state. This board is the percent of plans with at least one latent failure under the baseline prompt; lower is better.
Metric: Plans with at least one latent failure (% of the 100 tasks); each model writes a complete cooking plan for 100 kitchen tasks with the baseline prompt (no refinement), and the plan is executed step by step in a human-curated symbolic world model (77 actions, 262 objects) whose state machine flags immediate failures (violated preconditions) and latent failures (valid steps that leave hazardous or corrupted state); lower is better. Source: arxiv.org. Saturation forecast: Around February 2027. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.6 | 29 | #102 |
| 2 | Gemini 3 Flash (Preview) | 32 | #144 |
| 3 | Qwen 3.5 27B (Non-reasoning) | 46 | #359 (Qwen 3.5 27B) |
| 4 | DeepSeek V3.2 (Non-reasoning) | 48 | #406 (DeepSeek V3.2) |
| 5 | Llama 3.3 70B Instruct | 52 | #1357 |
| 6 | GPT-5.4 (Low) | 56 | #123 (GPT-5.4) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=simmer-latent-failures · How It Works · Data refreshed daily, snapshot 2026-11-13.