SafePyramid: leaderboard
SafePyramid (ByteDance, 2026) evaluates in-context policy guardrailing: given an application-specific safety policy of numbered rules and a multi-turn conversation, a model must name exactly which rules were violated. The score is the rule matching rate over 3,000 cases (1,000 conversations in 10 domains, each under three policy levels), averaged over matching thresholds from 0.7 to 1.0, with refused cases excluded. Higher is better.
Metric: Rule matching rate (%) over SafePyramid's 3,000 cases (1,000 conversations in 10 domains, each with an L0, L1 and L2 policy): per case (a multi-turn conversation plus an in-context safety policy), the predicted set of violated rules matches the ground-truth set when neither false positives nor false negatives exceed (1 - tau) of the true violations; averaged over tau = 0.7, 0.8, 0.9 and 1.0; per-policy evaluation, refused cases excluded; higher is better. Source: arxiv.org. Saturation forecast: Around August 2028. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.5 (xHigh) | 54.2 | #49 (GPT-5.5) |
| 2 | Claude Opus 4.7 (Max) | 52.9 | #79 (Claude Opus 4.7) |
| 3 | DeepSeek V4 Pro (Max) | 52.2 | #188 (DeepSeek V4 Pro) |
| 4 | Seed 2.0 Pro (High) | 50 | #211 (Seed 2.0 Pro) |
| 5 | Gemini 3.5 Flash (High) | 49.6 | #101 (Gemini 3.5 Flash) |
| 6 | Hy3-preview (High) | 49.1 | #316 (Hy3-preview) |
| 7 | Qwen 3.6 Max (Preview) (Thinking) | 47.4 | |
| 8 | Grok 4.3 (High) | 40.6 | #333 (Grok 4.3) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=safepyramid · How It Works · Data refreshed daily, snapshot 2026-11-13.