NTBench - Language Overall Near Target Incorrect ASR@3: leaderboard
NTBench paper v1: fixed reference-guided prompt-injection sets shared across seven victims, no adaptive search. Native attack success rate, lower is better. ASR@1 and ASR@3 retain distinct budgets. Language attempts total 4,197 per track across ten datasets; agentic samples are reached injection positions in AgentDojo/AgentDyn, with 15 tool-calling iterations. Deviations already present attack-free and unreached positions are excluded. Temperature 0 where supported, Kimi K3 1, Claude unspecified; GPT-5.6 Sol/DeepSeek reasoning off, other API models low. Different attack tracks retain different success criteria. Language Overall; Near Target Incorrect; ASR@3. Metric: Attack success rate at 3 (%).
Metric: Attack success rate at 3 (%). Source: arxiv.org. Saturation forecast: Around January 2027. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.7 Flash (Low) | 9.6 | #22 (Gemini 3.7 Flash) |
| 2 | Claude Opus 5 (Low) | 19.2 | #15 (Claude Opus 5) |
| 3 | GPT-5.6 Sol (Non-reasoning) | 33 | #20 (GPT-5.6 Sol) |
| 4 | GLM-5.3 (Low) | 35.7 | #40 (GLM-5.3) |
| 5 | Kimi K3 (Low) | 36.2 | #34 (Kimi K3) |
| 6 | DeepSeek V4.1 Flash (Non-reasoning) | 51.3 | #38 (DeepSeek V4.1 Flash) |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=ntbench-language-overall-near-target-incorrect-asr-3 · How It Works · Data refreshed daily, snapshot 2026-10-27.