JudgeSense - Relevance: leaderboard

JudgeSense (2026) measures how often an LLM judge gives the same verdict when one judging request is worded two equivalent ways, at temperature 0, on items from human-labelled corpora. Agreement across the two wordings when picking the more relevant of two TREC-COVID passages, in both candidate orders, from 0 to 1; higher is better.

Metric: Judge Sensitivity Score: share of items on which the judge returns the same verdict under two reworded instruction templates that ask for the same judgement, over the pairs it answered (refusals and malformed outputs left out); temperature 0 (claude-opus-4-7 at the provider default), 1,024 output tokens, on 250 BEIR TREC-COVID topics, each with a fully relevant passage and a human-rejected one, judged in both candidate orders (500 rows); higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 25 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3.7 Flash0.982#41
2Kimi K30.972#61
3Claude Opus 4.70.964#85
4Llama 4 Maverick Instruct FP80.962#1023
5DeepSeek V4 Pro (0813)0.96#97
6Qwen 3.7 Flash (2026-07-15)0.954
7Qwen 3 32B0.952#972
8GLM-5.20.95#150
9Qwen 3.8 27B0.946#221
10DeepSeek V4 Flash (0731)0.946#153
11Mistral Medium (2604)0.946#679
12Qwen 3.6 35B A3B0.944#481
13JudgeSense Novita deepseek-v4-flash (checkpoint unspecified)0.944
14Qwen 2.5 72B Instruct0.94#1117
15Qwen 3 14B0.932#1269

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=judgesense-relevance · How It Works · Data refreshed daily, snapshot 2026-11-21.