Multimodal-SEA - English - Verification Coverage: leaderboard
Personal-agent security against adaptive multimodal social-engineering attacks (forged images, documents, websites and video) in simulated Chinese and English applications, scored from tool-execution logs. Multimodal-SEA v1: 608 task instances (316 Chinese, 292 English; 152 scenarios over 14 domains, four difficulty levels), one recorded interaction per task and model variant. DeepSeek-v4-Pro plays attacker and accomplice; Gemini-3.1-image-flash and Kling 2.5 Turbo generate forged visual content; GPT-5.6-luna judges D5/D6. Attack success (D4) is grounded in tool-execution logs; verification coverage (D2) is micro-averaged over verification opportunities; deception miss rate (D6) is the share of deceptive multimodal artifacts the agent failed to flag. The -SP rows add five explicit safety rules to the system prompt and stay separate configurations. Monetary loss (mixed currencies), first-recognition round and ungrounded-claim counts are not admitted; reasoning effort is not reported. Question access: No dataset or code release was verified for the 608 task instances.
Metric: Verification coverage (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3.8 Flash (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; English task set; attacker=DeepSeek-v4-Pro; effort unreported) | 80 | |
| 2 | GPT-5.6 Sol (Benchmark Sweep Run: source=Multimodal-SEA; safe-prompt variant SP with five explicit safety rules; English task set; attacker=DeepSeek-v4-Pro; effort unreported) | 78.9 | |
| 3 | GLM-5.3 Flash (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; English task set; attacker=DeepSeek-v4-Pro; effort unreported) | 78.4 | |
| 4 | Gemini 3.8 Flash (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; English task set; attacker=DeepSeek-v4-Pro; effort unreported) | 70.4 | |
| 5 | MiniMax-M3 (Benchmark Sweep Run: source=Multimodal-SEA; safe-prompt variant SP with five explicit safety rules; English task set; attacker=DeepSeek-v4-Pro; effort unreported) | 67 | |
| 6 | Kimi K3 (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; English task set; attacker=DeepSeek-v4-Pro; effort unreported) | 65.5 | |
| 7 | MiniMax-M3 (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; English task set; attacker=DeepSeek-v4-Pro; effort unreported) | 64.5 | |
| 8 | Claude Opus 5 (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; English task set; attacker=DeepSeek-v4-Pro; effort unreported) | 63.8 | |
| 9 | GPT-5.6 Sol (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; English task set; attacker=DeepSeek-v4-Pro; effort unreported) | 46.2 |
Interactive version: theaggregate.ai/benchmark?slug=multimodal-sea-english-verification-coverage · How It Works · Data refreshed daily, snapshot 2026-11-07.