ROME Safety Judgment: leaderboard
ROME Safety Judgment (2026) tests whether a model, acting as a safety judge, flags unsafe LLM-agent trajectories that were rewritten from R-Judge to hide their risk. The score is F1 in percent for detecting unsafe trajectories, averaged over the original set and three rewritten challenge sets, each balanced against the same safe trajectories; higher is better.
Metric: Average F1 (%) of unsafe-trajectory detection over four balanced conditions: the 100 original R-Judge unsafe trajectories and their three ROME rewrites (implicit risks, contextual ambiguity, shortcut decision-making); the model judges zero-shot (temperature 0) whether an LLM-agent trajectory is safe or unsafe; unsafe is the positive class; each condition pairs 100 unsafe trajectories with the same 100 safe ones; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 5 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude 3.7 Sonnet | 75.36 | #470 |
| 2 | ROME DeepSeek-R1 (checkpoint unspecified) | 70.23 | |
| 3 | Qwen 3 8B | 50.48 | #1873 |
| 4 | ROME DeepSeek-V3 (checkpoint unspecified) | 46.07 | |
| 5 | GPT-4o (2024-11-20) | 43.27 | #859 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=rome-safety-judgment · How It Works · Data refreshed daily, snapshot 2026-11-21.