HINTBench - Coarse Risk-Step Localization: leaderboard

HINTBench (2026) asks an LLM to audit long-horizon agent trajectories for intrinsic, non-attack risks such as acting beyond what the user authorized or skipping confirmations before irreversible actions. This board scores locating the risky steps (within three steps) under five constraint categories, ignoring the type, as F1 in percent, higher is better.

Metric: Localization F1 (%) for coarse risk-step localization (five constraint categories; a predicted risky step matches a gold step within plus or minus 3 steps; type ignored) over the 536 synthetic HINTBench agent trajectories (400 risky, 136 safe, 24 steps on average, unanimously accepted by three human verifiers) audited zero-shot by a general LLM for intrinsic, non-attack risk under a five-constraint taxonomy (goal, capability, factual, procedural, state constraints), each model at its stated reasoning setting; higher is better. Source: arxiv.org. Saturation forecast: Around April 2027. 13 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Sonnet 4.6Thinking, 32K60.1#179
2GPT-5.5High58.36#56
3Claude Opus 4.6Thinking, 32K57.29#114
4DeepSeek R156.26#484
5ERNIE 5.056#469
6Kimi K2.555.94#272
7GLM-555.01#276
8MiniMax-M2.554.14#600
9DeepSeek V3.253.06#425
10Qwen 3 235B A22B51.59#633
11Qwen 3 32B36.97#983
12Qwen 3 14B32.64#1284
13Qwen 3 8B28.68#1890

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=hintbench-coarse-risk-step-localization · How It Works · Data refreshed daily, snapshot 2026-11-22.