CLI-Advantage Suite (CLI Agents): leaderboard
Android tasks that touchscreen apps serve poorly (bulk operations, multi-condition filtering, aggregation and top-K, cross-app workflows, hidden device state), 45 templates with three seeds each, from the CLI-Advantage Suite of Gu et al. (2026). Coding-agent harnesses (Claude Code, mini-swe-agent, Terminus-2) control the phone only through adb shell commands, without extra tools. Success rate in percent from rule-based verifiers of the final device state or answer, weighted by task; higher is better.
Metric: Success rate (%; 45 task templates x 3 seeds = 135 instances, rule-based verifier on the final device state or reported answer; mean of three seeds, weighted by task); CLI coding agents over adb, bash only. Source: arxiv.org. Saturation forecast: Around February 2027. 3 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.7 | 68.9 | #75 |
| 2 | MiniMax-M2.7 | 66.7 | #519 |
| 3 | GPT-5.3 Codex | 60.7 | #114 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=cli-advantage-suite-cli-agents · How It Works · Data refreshed daily, snapshot 2026-11-12.