AsyncPlan Online Robo Challenge (CP-SAT Formalizer): leaderboard
Online Robo Challenge from Robust Asynchronous Planning via Auto-Formalization (Zhang et al., 2026): the 140 Robotouille-based kitchen tasks with execution-time events such as new deliveries, deadlines and resource changes, so completed and running actions must be preserved while the rest is replanned. Here the LLM writes a CP-SAT constraint program that OR-Tools solves, zero-shot at temperature 0; plan accuracy in percent, higher is better.
Metric: Plan accuracy (%; share of the 140 Online Robo Challenge episodes of the kitchen tasks (seven splits of 20 built on Robotouille, with object states, resources, stations and multiple agents) whose plan is valid and reaches the optimal makespan; execution-time events (new deliveries, deadlines, resource changes) require replanning or one-shot re-formalization; the LLM writes a CP-SAT constraint program that OR-Tools solves (CP-SAT Formalizer); zero-shot, temperature 0). Source: arxiv.org. Saturation forecast: Around February 2027. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3 Flash | 83.6 | #159 |
| 2 | Qwen 3.6 35B A3B | 72.1 | #461 |
| 3 | GPT-5 Mini | 25 | #349 |
| 4 | DeepSeek V4 Flash | 3.6 | #254 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=asyncplan-online-robo-challenge-cp-sat-formalizer · How It Works · Data refreshed daily, snapshot 2026-11-12.