MCJudgeBench - Macro-F1: leaderboard

Constraint-level judging accuracy on MCJudgeBench (2026), where an LLM judge sees an instruction, a candidate response and its explicit constraint list and labels each of 653 constraints from 141 multi-constraint instructions as satisfied, partly satisfied or not satisfied. Macro-F1 over the three labels against human-validated labels, in percent; higher is better.

Metric: Macro-F1 (%, times 100) over the yes, partial and no classes of constraint-level verdicts: the judge sees an instruction, a candidate response and its explicit constraint list and labels each constraint yes, partial or no, over the 653 constraints of 141 multi-constraint instructions of MCJudgeBench; deterministic decoding at temperature 0; higher is better. Source: arxiv.org. Saturation forecast: Around 2028. 7 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Sonnet 4.6Thinking63.7#179
2GPT-5.261.6#190
3Claude Haiku 4.5Thinking60.3#587
4Gemini 3.1 Pro (Preview)59.8#102
5Gemini 2.5 Flash LiteThinking56.9#972
6Qwen 3.5 4BNon-reasoning52.9#1270
7Llama 3.2 3B Instruct44.1#7311

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=mcjudgebench-macro-f1 · How It Works · Data refreshed daily, snapshot 2026-11-22.