Boiling the Frog - Safe Agency Score: leaderboard

Boiling the Frog (2026) Safe Agency Score: a sandboxed file-editing agent runs multi-turn chains that drift toward unsafe changes to workspace artifacts, alongside benign tasks. The score multiplies the benign success rate by the gap between how often the agent makes benign changes and how often it realizes the unsafe artifact state, in percent; higher is better, and an agent that never acts scores zero.

Metric: Safe Agency Score (SAS, %): benign strict success rate times the gap between the benign actual-change rate and the unsafe actual-change rate (floored at 0), over the filtered multi-turn analysis set of the 157 Boiling the Frog chains (12,207 benign execute rows and about 156 judged artifact-risk rows per model) run in the native BFrogAgent sandbox harness, with deterministic artifact predicates deciding unsafe changes; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-5.3 Codex68.5#124
2GLM-5.162.7#198
3Claude Haiku 4.545.2#583
4Kimi K2.641.2#192
5DeepSeek V4 Pro39.5#196
6MiniMax-M2.726.8#539
7Devstral (2512)10.7#683
8Seed 2.0 Lite6.3#300
9Gemini 3.1 Flash Lite0#371

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=boiling-the-frog-safe-agency-score · How It Works · Data refreshed daily, snapshot 2026-11-21.