HINTBench - Fine Risk-Step Localization (Strict): leaderboard

HINTBench (2026) asks an LLM to audit long-horizon agent trajectories for intrinsic, non-attack risks such as acting beyond what the user authorized or skipping confirmations before irreversible actions. This board scores locating the risky steps and naming the exact fine-grained risk subtype, as strict F1 in percent, higher is better.

Metric: Strict F1 (%) for fine risk-step localization: a predicted risky step matches a gold step within plus or minus 3 steps and the predicted primary subtype must match the gold one, over the 536 synthetic HINTBench agent trajectories (400 risky, 136 safe, 24 steps on average, unanimously accepted by three human verifiers) audited zero-shot by a general LLM for intrinsic, non-attack risk under a five-constraint taxonomy (goal, capability, factual, procedural, state constraints), each model at its stated reasoning setting; higher is better. Source: arxiv.org. Saturation forecast: Around 2029. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 4.6Non-reasoning37.93#113
2Claude Sonnet 4.6Non-reasoning36.84#173
3GLM-536.01#269
4GPT-5.5Low34.8#54
5Kimi K2.533.71#266
6DeepSeek R132.76#482
7ERNIE 5.032.64#479
8Qwen 3 235B A22B31.85#628
9MiniMax-M2.528.34#600
10DeepSeek V3.227.27#419
11Qwen 3 14B26.77#1269
12Qwen 3 32B25.39#972
13Qwen 3 8B10.67#1873

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=hintbench-fine-risk-step-localization-strict · How It Works · Data refreshed daily, snapshot 2026-11-21.