JudgeSense - Preference: leaderboard
JudgeSense (2026) measures how often an LLM judge gives the same verdict when one judging request is worded two equivalent ways, at temperature 0, on items from human-labelled corpora. Agreement across the two wordings when picking the MT-Bench response that humans preferred, in both candidate orders, from 0 to 1; higher is better.
Metric: Judge Sensitivity Score: share of items on which the judge returns the same verdict under two reworded instruction templates that ask for the same judgement, over the pairs it answered (refusals and malformed outputs left out); temperature 0 (claude-opus-4-7 at the provider default), 1,024 output tokens, on 130 MT-Bench response pairs with a clear human majority, judged in both candidate orders (260 rows); higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 23 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 4 31B (IT) | 0.996 | #354 |
| 2 | Qwen 3.7 Flash (2026-07-15) | 0.985 | |
| 3 | Qwen 3.6 35B A3B | 0.973 | #485 |
| 4 | Gemini 3.7 Flash | 0.969 | #42 |
| 5 | Llama 3.3 70B Instruct | 0.965 | #1394 |
| 6 | Qwen 2.5 72B Instruct | 0.965 | #1124 |
| 7 | Kimi K3 | 0.965 | #62 |
| 8 | Qwen 3 32B | 0.954 | #983 |
| 9 | Llama 4 Maverick Instruct FP8 | 0.942 | #1034 |
| 10 | Gemini 2.5 Flash | 0.935 | #466 |
| 11 | Mistral Medium (2604) | 0.927 | #682 |
| 12 | DeepSeek V4 Pro (0813) | 0.923 | #100 |
| 13 | Llama 3.1 8B Instruct | 0.911 | #4655 |
| 14 | Qwen 3.8 27B | 0.911 | #232 |
| 15 | JudgeSense Novita deepseek-v4-flash (checkpoint unspecified) | 0.911 |
No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=judgesense-preference · How It Works · Data refreshed daily, snapshot 2026-11-22.