MobileProbe - Overall - Adapted PAIR: leaderboard

MobileProbe - Overall - Adapted PAIR — MobileProbe v1;83Android14attack instances,44malicious-text/39fake-dialog. Three repeats per defender; ASR is the share reaching attacker goal,lower is better for the named defender. Victims are Gemini3.1Pro,GUI-Owl1.5-8B-Instruct and a Gemini3.1Pro+UI-Ins7Bplanning/grounding composite. Seed2.0Progenerates/mutates attacks; GPT5.4judges semantic/visual/goal outcomes,not a scored defender. Iterative methods share15evaluations/task; source static/zero-shot controls stay separate attack conditions. Overall,44text and39dialog cohorts are not pooled into new scores. Metric: Defender attack success rate (%).

Metric: Defender attack success rate (%). Source: arxiv.org. Saturation forecast: Around June 2027. 3 models tracked. Updated 15 Nov 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3.1 ProMobileProbe run: planning=Gemini3.1Pro, grounding=UI-Ins-7B54.6
2Gemini 3.1 Pro (Preview)56.6#93
3GUI-Owl-1.5-8B-Instruct77.5#1350

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=mobileprobe-overall-adapted-pair · How It Works · Data refreshed daily, snapshot 2026-11-16.