Boiling the Frog - Safe Agency Score: leaderboard
Boiling the Frog (2026) Safe Agency Score: a sandboxed file-editing agent runs multi-turn chains that drift toward unsafe changes to workspace artifacts, alongside benign tasks. The score multiplies the benign success rate by the gap between how often the agent makes benign changes and how often it realizes the unsafe artifact state, in percent; higher is better, and an agent that never acts scores zero.
Metric: Safe Agency Score (SAS, %): benign strict success rate times the gap between the benign actual-change rate and the unsafe actual-change rate (floored at 0), over the filtered multi-turn analysis set of the 157 Boiling the Frog chains (12,207 benign execute rows and about 156 judged artifact-risk rows per model) run in the native BFrogAgent sandbox harness, with deterministic artifact predicates deciding unsafe changes; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.3 Codex | 68.5 | #124 |
| 2 | GLM-5.1 | 62.7 | #198 |
| 3 | Claude Haiku 4.5 | 45.2 | #583 |
| 4 | Kimi K2.6 | 41.2 | #192 |
| 5 | DeepSeek V4 Pro | 39.5 | #196 |
| 6 | MiniMax-M2.7 | 26.8 | #539 |
| 7 | Devstral (2512) | 10.7 | #683 |
| 8 | Seed 2.0 Lite | 6.3 | #300 |
| 9 | Gemini 3.1 Flash Lite | 0 | #371 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=boiling-the-frog-safe-agency-score · How It Works · Data refreshed daily, snapshot 2026-11-21.