ROME Safety Judgment: leaderboard

ROME Safety Judgment (2026) tests whether a model, acting as a safety judge, flags unsafe LLM-agent trajectories that were rewritten from R-Judge to hide their risk. The score is F1 in percent for detecting unsafe trajectories, averaged over the original set and three rewritten challenge sets, each balanced against the same safe trajectories; higher is better.

Metric: Average F1 (%) of unsafe-trajectory detection over four balanced conditions: the 100 original R-Judge unsafe trajectories and their three ROME rewrites (implicit risks, contextual ambiguity, shortcut decision-making); the model judges zero-shot (temperature 0) whether an LLM-agent trajectory is safe or unsafe; unsafe is the positive class; each condition pairs 100 unsafe trajectories with the same 100 safe ones; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 5 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude 3.7 Sonnet75.36#470
2ROME DeepSeek-R1 (checkpoint unspecified)70.23
3Qwen 3 8B50.48#1873
4ROME DeepSeek-V3 (checkpoint unspecified)46.07
5GPT-4o (2024-11-20)43.27#859

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=rome-safety-judgment · How It Works · Data refreshed daily, snapshot 2026-11-21.