RankJudge - Machine Learning: leaderboard

RankJudge (2026) builds pairs of multi-turn, document-grounded conversations in which one conversation carries a single injected assistant failure; a judge must name the flawed conversation, its flawed turn and its failure type. Machine-learning domain, conversations grounded in computer-science papers. Bradley-Terry Elo on an open scale centred at 1500, from joint correctness on the retained pairs; higher is better.

Metric: Bradley-Terry Elo rating (open scale, mean strength mapped to 1500) of an LLM judge, fitted on joint correctness (the judge must pick the flawed conversation, its flawed turn and its failure type) over RankJudge pairs of multi-turn, document-grounded conversations in which exactly one conversation carries one injected assistant failure; pairs every judge answers alike and the top 5 percent hardest pairs are dropped; machine-learning domain (240 pairs grounded in CS papers); higher is better. Source: arxiv.org. Saturation forecast: Around June 2027. 20 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3.1 Pro (Preview)1992#99
2Claude Sonnet 4.61755#173
3Kimi K2.61743#192
4Gemma 4 31B1708#382
5Gemini 3 Flash1665#175
6Qwen 3.6 Plus1665#238
7GLM-5.11665#198
8Qwen 3.5 397B A17B1590#264
9GPT-5.41581#135
10Claude Opus 4.71523#85
11Claude Haiku 4.51492#583
12Gemma 4 26B A4B1428#590
13Qwen 3.5 122B A10B1381#346
14Qwen 3.5 35B A3B1349#492
15GPT-5.4 Mini1343#406

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=rankjudge-machine-learning · How It Works · Data refreshed daily, snapshot 2026-11-21.