WindowsWorld (Set-of-Marks): leaderboard

Metric: Final task success rate (%) over all 181 tasks of WindowsWorld (persona-driven Windows desktop tasks across 17 applications, executed with a unified PyAutoGUI action space and level-specific step budgets of 15, 25, 40 and 20), the model observing Set-of-Marks annotated screenshots, states judged by a human-validated Qwen3-VL-Plus judge; infeasible Level 4 tasks count as solved when the agent correctly rejects them; higher is better. Source: arxiv.org. Saturation forecast: Around September 2028. 5 models tracked.

Top models

Interactive version: theaggregate.ai/benchmark?slug=windowsworld-set-of-marks · How It Works · Data refreshed daily, snapshot 2026-10-07.