Software World Model Baselines - odoo18 (held-out version) - clean - Irreversible Changes: leaderboard
Native Phase0 zero-shot app-control tasks; 30-step budget and one run per task. GPT-6 Luna API uses accessibility tree+screenshot/element IDs; UI-TARS-1.5-7B uses screenshots/native coordinates via adapter and vLLM A100. Nine skills, nominal eight tasks each; UI-TARS other apps four per skill. Exact native n per row retained.12 InvoiceNinja Luna episodes affected by fallback bug are excluded; other affected episodes were rerun, originals superseded. This is source failure policy, not complete-cohort success. App train/held-out labels refer to the separate world-model track, not LLM training. GPT-6 Luna n=72; UI-TARS-1.5-7B n=36. Success is evaluator-verified task state; clean success additionally forbids collateral changes. Timeouts and wrong actions remain failures. No cost, latency, or latent transition-model scores. Metric: Native episode rate (%). Source: https://raw.githubusercontent.com/shubhamjoshipromail-svg/software-world-model/7fa10cf4f7d294ba80e8d6c357f77d3a6340c3ce/reports/phase0_results.md
Metric: Native episode rate (%). Source: raw.githubusercontent.com. Saturation forecast: Around 2029. 2 models tracked. Updated 17 Nov 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-6 Luna | 10 | #105 |
| 2 | uitars-1.5-7B | 8 | #7798 |
No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.
Interactive version: theaggregate.ai/benchmark?slug=software-world-model-baselines-odoo18-held-out-version-clean-irreversible-changes · How It Works · Data refreshed daily, snapshot 2026-11-17.