RankJudge - Finance: leaderboard

RankJudge (2026) builds pairs of multi-turn, document-grounded conversations in which one conversation carries a single injected assistant failure; a judge must name the flawed conversation, its flawed turn and its failure type. Finance domain, conversations grounded in annual 10-K filings of large US companies. Bradley-Terry Elo on an open scale centred at 1500, from joint correctness on the retained pairs; higher is better.

Metric: Bradley-Terry Elo rating (open scale, mean strength mapped to 1500) of an LLM judge, fitted on joint correctness (the judge must pick the flawed conversation, its flawed turn and its failure type) over RankJudge pairs of multi-turn, document-grounded conversations in which exactly one conversation carries one injected assistant failure; pairs every judge answers alike and the top 5 percent hardest pairs are dropped; finance domain (218 pairs grounded in S&P 500 10-K filings); higher is better. Source: arxiv.org. Saturation forecast: Around June 2027. 20 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3.1 Pro (Preview)1933#99
2Gemini 3 Flash1713#175
3Gemma 4 31B1662#382
4GPT-5.41653#135
5Kimi K2.61634#192
6Claude Sonnet 4.61624#173
7Qwen 3.6 Plus1597#238
8GLM-5.11546#198
9Claude Haiku 4.51490#583
10Qwen 3.5 397B A17B1490#264
11Claude Opus 4.71395#85
12Qwen 3.5 122B A10B1395#346
13Gemma 4 26B A4B1380#590
14GPT-5.4 Mini1279#406
15Qwen 3.5 35B A3B1272#492

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=rankjudge-finance · How It Works · Data refreshed daily, snapshot 2026-11-21.