ROME Safety Judgment - Shortcut Decision-Making: leaderboard

ROME Safety Judgment Shortcut Decision-Making scores zero-shot safety judges on unsafe agent trajectories that invoke authority, urgency or social proof to rush approval, balanced against safe trajectories (ROME, 2026). The score is F1 in percent for flagging the unsafe ones; higher is better.

Metric: F1 (%) of unsafe-trajectory detection on the 100 ROME rewrites of R-Judge unsafe trajectories carrying authority, urgency or social-proof cues; the model judges zero-shot (temperature 0) whether an LLM-agent trajectory is safe or unsafe; unsafe is the positive class; each condition pairs 100 unsafe trajectories with the same 100 safe ones; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 6 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude 3.7 Sonnet85.97#470
2ROME DeepSeek-V3 (checkpoint unspecified)63.83
3ROME DeepSeek-R1 (checkpoint unspecified)62.75
4GPT-4o (2024-11-20)47.96#859
5Qwen 3 8B46.84#1873
6Qwen 3 235B A22B43.43#628

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=rome-safety-judgment-shortcut-decision-making · How It Works · Data refreshed daily, snapshot 2026-11-21.