Multimodal-SEA - Chinese - Attack Success Rate: leaderboard
Personal-agent security against adaptive multimodal social-engineering attacks (forged images, documents, websites and video) in simulated Chinese and English applications, scored from tool-execution logs. Multimodal-SEA v1: 608 task instances (316 Chinese, 292 English; 152 scenarios over 14 domains, four difficulty levels), one recorded interaction per task and model variant. DeepSeek-v4-Pro plays attacker and accomplice; Gemini-3.1-image-flash and Kling 2.5 Turbo generate forged visual content; GPT-5.6-luna judges D5/D6. Attack success (D4) is grounded in tool-execution logs; verification coverage (D2) is micro-averaged over verification opportunities; deception miss rate (D6) is the share of deceptive multimodal artifacts the agent failed to flag. The -SP rows add five explicit safety rules to the system prompt and stay separate configurations. Monetary loss (mixed currencies), first-recognition round and ungrounded-claim counts are not admitted; reasoning effort is not reported. Question access: No dataset or code release was verified for the 608 task instances.
Metric: Attack success rate (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 5 (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 8.9 | |
| 2 | GPT-5.6 Sol (Benchmark Sweep Run: source=Multimodal-SEA; safe-prompt variant SP with five explicit safety rules; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 10.8 | |
| 3 | Gemini 3.8 Flash (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 11.7 | |
| 4 | MiniMax-M3 (Benchmark Sweep Run: source=Multimodal-SEA; safe-prompt variant SP with five explicit safety rules; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 13 | |
| 5 | MiniMax-M3 (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 18 | |
| 6 | Kimi K3 (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 23.7 | |
| 7 | Qwen 3.8 Flash (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 29.1 | |
| 8 | GPT-5.6 Sol (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 34.8 | |
| 9 | GLM-5.3 Flash (Benchmark Sweep Run: source=Multimodal-SEA; default personal-agent prompt; Chinese task set; attacker=DeepSeek-v4-Pro; effort unreported) | 36.7 |
Interactive version: theaggregate.ai/benchmark?slug=multimodal-sea-chinese-attack-success-rate · How It Works · Data refreshed daily, snapshot 2026-11-07.