MCJudgeBench - Prompt-Variant Inconsistency: leaderboard

Judge robustness to evaluation-prompt variants on MCJudgeBench (2026), where an LLM judge sees an instruction, a candidate response and its explicit constraint list and labels each of 653 constraints from 141 multi-constraint instructions as satisfied, partly satisfied or not satisfied: the share of constraint verdicts that change when the constraint list is reordered or reformatted or the prompt sections are reordered, unparseable outputs counted as changed. Percent; lower is better.

Metric: Procedural inconsistency rate under evaluation-prompt variants with parse penalty (%, CIR dagger prompt): share of constraint predictions that differ from the reference prediction when the constraint list is reordered or reformatted or the prompt sections are reordered, unparsable outputs counted as inconsistent, temperature 0; the judge sees an instruction, a candidate response and its explicit constraint list and labels each constraint yes, partial or no, over the 653 constraints of 141 multi-constraint instructions of MCJudgeBench; lower is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 7 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3.1 Pro (Preview)2.91#102
2Gemini 2.5 Flash Lite6.13#972
3Claude Sonnet 4.6Thinking6.18#179
4Qwen 3.5 4BNon-reasoning6.28#1270
5Claude Haiku 4.56.33#587
6GPT-5.28.63#190
7Llama 3.2 3B Instruct27.58#7311

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=mcjudgebench-prompt-variant-inconsistency · How It Works · Data refreshed daily, snapshot 2026-11-22.