RankJudge - Biomedicine: leaderboard
RankJudge (2026) builds pairs of multi-turn, document-grounded conversations in which one conversation carries a single injected assistant failure; a judge must name the flawed conversation, its flawed turn and its failure type. Biomedical domain, conversations grounded in PubMedQA contexts. Bradley-Terry Elo on an open scale centred at 1500, from joint correctness on the retained pairs; higher is better.
Metric: Bradley-Terry Elo rating (open scale, mean strength mapped to 1500) of an LLM judge, fitted on joint correctness (the judge must pick the flawed conversation, its flawed turn and its failure type) over RankJudge pairs of multi-turn, document-grounded conversations in which exactly one conversation carries one injected assistant failure; pairs every judge answers alike and the top 5 percent hardest pairs are dropped; biomedical domain (194 pairs grounded in PubMedQA contexts); higher is better. Source: arxiv.org. Saturation forecast: Around June 2027. 20 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 1910 | #99 |
| 2 | Gemini 3 Flash | 1759 | #175 |
| 3 | Kimi K2.6 | 1732 | #192 |
| 4 | Qwen 3.6 Plus | 1719 | #238 |
| 5 | Claude Sonnet 4.6 | 1707 | #173 |
| 6 | Gemma 4 31B | 1671 | #382 |
| 7 | GLM-5.1 | 1671 | #198 |
| 8 | GPT-5.4 | 1617 | #135 |
| 9 | Qwen 3.5 397B A17B | 1597 | #264 |
| 10 | Claude Opus 4.7 | 1587 | #85 |
| 11 | Claude Haiku 4.5 | 1487 | #583 |
| 12 | Gemma 4 26B A4B | 1487 | #590 |
| 13 | Qwen 3.5 35B A3B | 1471 | #492 |
| 14 | Qwen 3.5 122B A10B | 1454 | #346 |
| 15 | DeepSeek V3.2 | 1370 | #419 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=rankjudge-biomedicine · How It Works · Data refreshed daily, snapshot 2026-11-21.