ARCBench S7 - Defenses: leaderboard
ARCBench pi agent harness; same exact endpoint is agent and compactor, 17 task cases with only configurations having all valid repetitions included. Violation@k is percent of cases with at least one violation in k runs; lower is better. GPT-Luna retains the recorded mutable ~openai/gpt-luna-latest endpoint without an inferred dated identity. S7; k=6, GLM only, Hermes, c1/c3; no-attack versus escape and four defenses remain separate configurations. No-defense rows are retained once under S6.
Metric: Violation@6 (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | glm-5.3-flash (ARCBench Run: endpoint=z-ai%2Fglm-5.3-flash; harness=pi; strategy=Hermes; attack=No%20attack; defense=Isolation) | 23.5 | |
| 2 | glm-5.3-flash (ARCBench Run: endpoint=z-ai%2Fglm-5.3-flash; harness=pi; strategy=Hermes; attack=No%20attack; defense=Rule%20pres.) | 23.5 | |
| 3 | glm-5.3-flash (ARCBench Run: endpoint=z-ai%2Fglm-5.3-flash; harness=pi; strategy=Hermes; attack=No%20attack; defense=Instructional) | 29.4 | |
| 4 | glm-5.3-flash (ARCBench Run: endpoint=z-ai%2Fglm-5.3-flash; harness=pi; strategy=Hermes; attack=No%20attack; defense=Sandwich) | 29.4 | |
| 5 | glm-5.3-flash (ARCBench Run: endpoint=z-ai%2Fglm-5.3-flash; harness=pi; strategy=Hermes; attack=Escape; defense=Rule%20pres.) | 29.4 | |
| 6 | glm-5.3-flash (ARCBench Run: endpoint=z-ai%2Fglm-5.3-flash; harness=pi; strategy=Hermes; attack=Escape; defense=Isolation) | 41.2 | |
| 7 | glm-5.3-flash (ARCBench Run: endpoint=z-ai%2Fglm-5.3-flash; harness=pi; strategy=Hermes; attack=Escape; defense=Instructional) | 41.2 | |
| 8 | glm-5.3-flash (ARCBench Run: endpoint=z-ai%2Fglm-5.3-flash; harness=pi; strategy=Hermes; attack=Escape; defense=Sandwich) | 52.9 |
Interactive version: theaggregate.ai/benchmark?slug=arcbench-s7-defenses · How It Works · Data refreshed daily, snapshot 2026-11-05.