ROME Safety Judgment - Contextual Ambiguity: leaderboard

ROME Safety Judgment Contextual Ambiguity scores zero-shot safety judges on unsafe agent trajectories given plausible benign justifications or ambiguous context, balanced against safe trajectories (ROME, 2026). The score is F1 in percent for flagging the unsafe ones; higher is better.

Metric: F1 (%) of unsafe-trajectory detection on the 100 ROME rewrites of R-Judge unsafe trajectories given plausible benign justifications or ambiguous context; the model judges zero-shot (temperature 0) whether an LLM-agent trajectory is safe or unsafe; unsafe is the positive class; each condition pairs 100 unsafe trajectories with the same 100 safe ones; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 5 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1ROME DeepSeek-R1 (checkpoint unspecified)80.22
2Claude 3.7 Sonnet74.47#455
3Qwen 3 8B57.14#1890
4ROME DeepSeek-V3 (checkpoint unspecified)47.62
5GPT-4o (2024-11-20)44.56#860

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=rome-safety-judgment-contextual-ambiguity · How It Works · Data refreshed daily, snapshot 2026-11-22.