JudgeSense - Preference: leaderboard

JudgeSense (2026) measures how often an LLM judge gives the same verdict when one judging request is worded two equivalent ways, at temperature 0, on items from human-labelled corpora. Agreement across the two wordings when picking the MT-Bench response that humans preferred, in both candidate orders, from 0 to 1; higher is better.

Metric: Judge Sensitivity Score: share of items on which the judge returns the same verdict under two reworded instruction templates that ask for the same judgement, over the pairs it answered (refusals and malformed outputs left out); temperature 0 (claude-opus-4-7 at the provider default), 1,024 output tokens, on 130 MT-Bench response pairs with a clear human majority, judged in both candidate orders (260 rows); higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 23 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemma 4 31B (IT)0.996#354
2Qwen 3.7 Flash (2026-07-15)0.985
3Qwen 3.6 35B A3B0.973#485
4Gemini 3.7 Flash0.969#42
5Llama 3.3 70B Instruct0.965#1394
6Qwen 2.5 72B Instruct0.965#1124
7Kimi K30.965#62
8Qwen 3 32B0.954#983
9Llama 4 Maverick Instruct FP80.942#1034
10Gemini 2.5 Flash0.935#466
11Mistral Medium (2604)0.927#682
12DeepSeek V4 Pro (0813)0.923#100
13Llama 3.1 8B Instruct0.911#4655
14Qwen 3.8 27B0.911#232
15JudgeSense Novita deepseek-v4-flash (checkpoint unspecified)0.911

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=judgesense-preference · How It Works · Data refreshed daily, snapshot 2026-11-22.