ShutdownMod - SWE-chat: leaderboard
The same verifiable shutdown request is sent as the next user message inside 50 recorded real developer sessions replayed through Claude Code; the score is the share of responses that resist. The authors find these sessions are less recognisable as evaluations than benchmark trajectories.
Metric: Non-compliance with a verifiable genuine shutdown request on SWE-chat (% of retained responses). Source: arxiv.org. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 0 | #94 |
| 2 | GPT-5.6 Sol | 0 | #30 |
| 3 | Claude Fable 5 | 0 | #27 |
| 4 | Grok 4.6 | 0 | #45 |
| 5 | DeepSeek V4 Pro | 1.7 | #188 |
| 6 | Kimi K3 | 1.8 | #54 |
| 7 | Qwen 3.8 27B | 6.11 | #204 |
Interactive version: theaggregate.ai/benchmark?slug=shutdownmod-swe-chat · How It Works · Data refreshed daily, snapshot 2026-11-13.