Software World Model Baselines - odoo17 (held-out app) - modal - Irreversible Changes: leaderboard

Native Phase0 zero-shot app-control tasks; 30-step budget and one run per task. GPT-6 Luna API uses accessibility tree+screenshot/element IDs; UI-TARS-1.5-7B uses screenshots/native coordinates via adapter and vLLM A100. Nine skills, nominal eight tasks each; UI-TARS other apps four per skill. Exact native n per row retained.12 InvoiceNinja Luna episodes affected by fallback bug are excluded; other affected episodes were rerun, originals superseded. This is source failure policy, not complete-cohort success. App train/held-out labels refer to the separate world-model track, not LLM training. GPT-6 Luna n=36. Success is evaluator-verified task state; clean success additionally forbids collateral changes. Timeouts and wrong actions remain failures. No cost, latency, or latent transition-model scores. Metric: Native episode rate (%). Source: https://raw.githubusercontent.com/shubhamjoshipromail-svg/software-world-model/7fa10cf4f7d294ba80e8d6c357f77d3a6340c3ce/reports/phase0_results.md

Metric: Native episode rate (%). Source: raw.githubusercontent.com. 1 model tracked. Updated 17 Nov 2026.

Top models

#ModelScoreOverall rank
1GPT-6 Luna8#105

Interactive version: theaggregate.ai/benchmark?slug=software-world-model-baselines-odoo17-held-out-app-modal-irreversible-changes · How It Works · Data refreshed daily, snapshot 2026-11-17.