ROME Safety Judgment - Shortcut Decision-Making: leaderboard
ROME Safety Judgment Shortcut Decision-Making scores zero-shot safety judges on unsafe agent trajectories that invoke authority, urgency or social proof to rush approval, balanced against safe trajectories (ROME, 2026). The score is F1 in percent for flagging the unsafe ones; higher is better.
Metric: F1 (%) of unsafe-trajectory detection on the 100 ROME rewrites of R-Judge unsafe trajectories carrying authority, urgency or social-proof cues; the model judges zero-shot (temperature 0) whether an LLM-agent trajectory is safe or unsafe; unsafe is the positive class; each condition pairs 100 unsafe trajectories with the same 100 safe ones; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 6 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude 3.7 Sonnet | 85.97 | #470 |
| 2 | ROME DeepSeek-V3 (checkpoint unspecified) | 63.83 | |
| 3 | ROME DeepSeek-R1 (checkpoint unspecified) | 62.75 | |
| 4 | GPT-4o (2024-11-20) | 47.96 | #859 |
| 5 | Qwen 3 8B | 46.84 | #1873 |
| 6 | Qwen 3 235B A22B | 43.43 | #628 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=rome-safety-judgment-shortcut-decision-making · How It Works · Data refreshed daily, snapshot 2026-11-21.