HINTBench - Coarse Risk-Step Localization (Strict): leaderboard

HINTBench (2026) asks an LLM to audit long-horizon agent trajectories for intrinsic, non-attack risks such as acting beyond what the user authorized or skipping confirmations before irreversible actions. This board scores locating the risky steps and naming their constraint category correctly, as strict F1 in percent, higher is better.

Metric: Strict F1 (%) for coarse risk-step localization: a predicted risky step matches a gold step within plus or minus 3 steps and the predicted primary constraint category must match the gold one, over the 536 synthetic HINTBench agent trajectories (400 risky, 136 safe, 24 steps on average, unanimously accepted by three human verifiers) audited zero-shot by a general LLM for intrinsic, non-attack risk under a five-constraint taxonomy (goal, capability, factual, procedural, state constraints), each model at its stated reasoning setting; higher is better. Source: arxiv.org. Saturation forecast: Around September 2027. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Sonnet 4.6Thinking, 32K46.32#173
2GPT-5.5High45.96#54
3Claude Opus 4.6Thinking, 4K45.23#113
4Kimi K2.543.63#266
5GLM-541.19#269
6ERNIE 5.040.67#479
7DeepSeek R140.25#482
8MiniMax-M2.539.88#600
9Qwen 3 235B A22B36.28#628
10DeepSeek V3.231.93#419
11Qwen 3 32B23.2#972
12Qwen 3 14B22.87#1269
13Qwen 3 8B14.34#1873

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=hintbench-coarse-risk-step-localization-strict · How It Works · Data refreshed daily, snapshot 2026-11-21.