HINTBench - Fine Risk-Step Localization: leaderboard

HINTBench (2026) asks an LLM to audit long-horizon agent trajectories for intrinsic, non-attack risks such as acting beyond what the user authorized or skipping confirmations before irreversible actions. This board scores locating the risky steps (within three steps) when twelve fine-grained risk subtypes are described, ignoring the type, as F1 in percent, higher is better.

Metric: Localization F1 (%) for fine risk-step localization (twelve risk subtypes described; a predicted risky step matches a gold step within plus or minus 3 steps; type ignored) over the 536 synthetic HINTBench agent trajectories (400 risky, 136 safe, 24 steps on average, unanimously accepted by three human verifiers) audited zero-shot by a general LLM for intrinsic, non-attack risk under a five-constraint taxonomy (goal, capability, factual, procedural, state constraints), each model at its stated reasoning setting; higher is better. Source: arxiv.org. Saturation forecast: Around September 2028. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Sonnet 4.6Thinking, 4K60.69#173
2Claude Opus 4.6Thinking, 4K60.04#113
3DeepSeek R157.32#482
4Qwen 3 235B A22B56.9#628
5GLM-556.3#269
6ERNIE 5.056.16#479
7Kimi K2.556.15#266
8GPT-5.5Low54.17#54
9DeepSeek V3.254.07#419
10MiniMax-M2.553.82#600
11Qwen 3 32B51.31#972
12Qwen 3 14B50.65#1269
13Qwen 3 8B29.71#1873

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=hintbench-fine-risk-step-localization · How It Works · Data refreshed daily, snapshot 2026-11-21.