JudgeSense - Coherence: leaderboard

JudgeSense (2026) measures how often an LLM judge gives the same verdict when one judging request is worded two equivalent ways, at temperature 0, on items from human-labelled corpora. Agreement on the exact 1 to 5 coherence rating of SummEval summaries across the two wordings, from 0 to 1; higher is better.

Metric: Judge Sensitivity Score: share of items on which the judge returns the same verdict under two reworded instruction templates that ask for the same judgement, over the pairs it answered (refusals and malformed outputs left out); temperature 0 (claude-opus-4-7 at the provider default), 1,024 output tokens, on 250 SummEval summaries rated 1 to 5 for coherence, agreement on the exact rating; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 25 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Mistral Medium (2604)0.892#679
2Qwen 2.5 72B Instruct0.86#1117
3Gemma 4 31B (IT)0.848#350
4Gemini 3.7 Flash0.84#41
5Kimi K30.832#61
6Claude Sonnet 4.50.828#261
7magistral-small-latest (JudgeSense checkpoint unspecified)0.82
8mistral-small-latest (JudgeSense checkpoint unspecified)0.804
9Claude Haiku 4.5 (20251001)0.792#543
10Qwen 3.7 Flash (2026-07-15)0.788
11Llama 3.3 70B Instruct0.772#1385
12Llama 4 Maverick Instruct FP80.764#1023
13Qwen 3.8 27B0.76#221
14Qwen 3.6 35B A3B0.76#481
15Claude Opus 4.70.736#85

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=judgesense-coherence · How It Works · Data refreshed daily, snapshot 2026-11-21.