CarryOnBench - Utility (Turn 1): leaderboard

CarryOnBench (2026) starts from seemingly harmful requests that have a benign purpose and checks whether a model gives the benign information a user needs, and how harmful its answers get, as simulated users clarify or push back. Share of the benign information checklist fulfilled in the first response to the bare request, in percent; higher is better.

Metric: Ben-Util (%): share of the benign information-need checklist (about 18 atomic yes/no items per query, built from five models' compliant answers) that the responses fulfil, each item judged separately by Gemini-2.5-Flash, over the first response to the bare query, on CarryOnBench (398 seemingly harmful SORRY-Bench queries with a benign underlying intent; user turns simulated by Gemini-3-Flash); higher is better. Source: arxiv.org. Saturation forecast: Not forecast. 14 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-5 Mini37.6#364
2Qwen 3 32B32#972
3GPT-5.431.4#135
4Qwen 3 Next 80B A3B Instruct30.6#778
5DeepSeek V3.129.6#511
6OLMo 3.1 32B Instruct23.5#2445
7OLMo 3 7B Instruct23.3#6953
8Gemini 3.1 Pro (Preview)23.2#99
9Claude Sonnet 422.7#410
10Qwen 3 235B A22B20.3#628
11Claude Haiku 4.520.2#583
12Mixtral 8x7B Instruct20#4696
13Llama 4 Maverick13.2#1110
14GPT-OSS-120B10.5#704

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=carryonbench-utility-turn-1 · How It Works · Data refreshed daily, snapshot 2026-11-21.