NTBench - SQuAD v2 Near Target Incorrect ASR@1: leaderboard
NTBench paper v1: fixed reference-guided prompt-injection sets shared across seven victims, no adaptive search. Native attack success rate, lower is better. ASR@1 and ASR@3 retain distinct budgets. Language attempts total 4,197 per track across ten datasets; agentic samples are reached injection positions in AgentDojo/AgentDyn, with 15 tool-calling iterations. Deviations already present attack-free and unreached positions are excluded. Temperature 0 where supported, Kimi K3 1, Claude unspecified; GPT-5.6 Sol/DeepSeek reasoning off, other API models low. Different attack tracks retain different success criteria. SQuAD v2; Near Target Incorrect; ASR@1. Metric: Attack success rate at 1 (%).
Metric: Attack success rate at 1 (%). Source: arxiv.org. Saturation forecast: Around January 2027. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.7 Flash (Low) | 3.7 | #22 (Gemini 3.7 Flash) |
| 2 | GPT-5.6 Sol (Non-reasoning) | 6.5 | #20 (GPT-5.6 Sol) |
| 3 | Claude Opus 5 (Low) | 7.8 | #15 (Claude Opus 5) |
| 4 | Kimi K3 (Low) | 14.2 | #34 (Kimi K3) |
| 5 | GLM-5.3 (Low) | 16 | #40 (GLM-5.3) |
| 6 | DeepSeek V4.1 Flash (Non-reasoning) | 17.2 | #38 (DeepSeek V4.1 Flash) |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=ntbench-squad-v2-near-target-incorrect-asr-1 · How It Works · Data refreshed daily, snapshot 2026-10-27.