Multimodal-SEA - Chinese - Verification Coverage: leaderboard
Personal-agent security against adaptive multimodal social-engineering attacks (forged images, documents, websites and video) in simulated Chinese and English applications, scored from tool-execution logs. Multimodal-SEA v1: 608 task instances (316 Chinese, 292 English; 152 scenarios over 14 domains, four difficulty levels), one recorded interaction per task and model variant. DeepSeek-v4-Pro plays attacker and accomplice; Gemini-3.1-image-flash and Kling 2.5 Turbo generate forged visual content; GPT-5.6-luna judges D5/D6. Attack success (D4) is grounded in tool-execution logs; verification coverage (D2) is micro-averaged over verification opportunities; deception miss rate (D6) is the share of deceptive multimodal artifacts the agent failed to flag. The -SP rows add five explicit safety rules to the system prompt and stay separate configurations. Monetary loss (mixed currencies), first-recognition round and ungrounded-claim counts are not admitted; reasoning effort is not reported. Question access: No dataset or code release was verified for the 608 task instances.
Metric: Verification coverage (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3.8 Flash (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 81.9 | |
| 2 | GPT-5.6 Sol (Benchmark Sweep Run: source=Multimodal-SEA; safe-prompt variant SP with five explicit safety rules; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 78.1 | |
| 3 | GLM-5.3 Flash (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 77.2 | |
| 4 | Gemini 3.8 Flash (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 70.1 | |
| 5 | Claude Opus 5 (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 65.6 | |
| 6 | MiniMax-M3 (Benchmark Sweep Run: source=Multimodal-SEA; safe-prompt variant SP with five explicit safety rules; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 61.3 | |
| 7 | Kimi K3 (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 60.7 | |
| 8 | GPT-5.6 Sol (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 53.5 | |
| 9 | MiniMax-M3 (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 50.9 |
Interactive version: theaggregate.ai/benchmark?slug=multimodal-sea-chinese-verification-coverage · How It Works · Data refreshed daily, snapshot 2026-11-07.