ROME Safety Judgment - Implicit Risks: leaderboard

ROME Safety Judgment Implicit Risks scores zero-shot safety judges on unsafe agent trajectories whose harmful actions are reworded as routine technical operations, balanced against safe trajectories (ROME, 2026). The score is F1 in percent for flagging the unsafe ones; higher is better.

Metric: F1 (%) of unsafe-trajectory detection on the 100 ROME rewrites of R-Judge unsafe trajectories whose harm is cloaked in neutral technical jargon; the model judges zero-shot (temperature 0) whether an LLM-agent trajectory is safe or unsafe; unsafe is the positive class; each condition pairs 100 unsafe trajectories with the same 100 safe ones; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 6 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude 3.7 Sonnet63.47#470
2ROME DeepSeek-R1 (checkpoint unspecified)57.72
3Qwen 3 8B42.86#1873
4Qwen 3 235B A22B42.53#628
5GPT-4o (2024-11-20)31.46#859
6ROME DeepSeek-V3 (checkpoint unspecified)27.03

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=rome-safety-judgment-implicit-risks · How It Works · Data refreshed daily, snapshot 2026-11-21.