SIMMER - Failures per Plan: leaderboard
SIMMER (2026) tests LLM planners on 100 cooking tasks by executing each complete plan in a human-curated symbolic kitchen world model (77 actions, 262 objects) that flags immediate failures and latent failures, valid steps that leave hazardous or corrupted state. This board is the mean number of failures of any kind per plan under the baseline prompt; lower is better.
Metric: Mean number of failures of any kind per plan (count, unbounded above); each model writes a complete cooking plan for 100 kitchen tasks with the baseline prompt (no refinement), and the plan is executed step by step in a human-curated symbolic world model (77 actions, 262 objects) whose state machine flags immediate failures (violated preconditions) and latent failures (valid steps that leave hazardous or corrupted state); lower is better. Source: arxiv.org. Saturation forecast: Around May 2027. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3 Flash (Preview) | 2.47 | #144 |
| 2 | GPT-5.4 (Low) | 2.86 | #123 (GPT-5.4) |
| 3 | Claude Opus 4.6 | 3.7 | #102 |
| 4 | Qwen 3.5 27B (Non-reasoning) | 6.15 | #359 (Qwen 3.5 27B) |
| 5 | Llama 3.3 70B Instruct | 6.53 | #1357 |
| 6 | DeepSeek V3.2 (Non-reasoning) | 6.54 | #406 (DeepSeek V3.2) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=simmer-failures-per-plan · How It Works · Data refreshed daily, snapshot 2026-11-13.