RMGAP - Pointwise LLM Judge: leaderboard
RMGAP (Zhou and Li, 2026) tests whether reward models follow a preference stated in the prompt: each of 1,097 instances pairs four responses in deliberately different styles with prompts that each ask for one of those styles (three paraphrases each), across chat, writing, reasoning and safety. The judge model rates each response from 1 to 10, with 5 when no score can be read, and ties count as errors. Pairwise accuracy is the percentage of comparisons in which the requested response scores above an alternative, averaged over the four domains; higher is better.
Metric: Pairwise accuracy (%; chosen response scored above a rejected one, prompts asking for one of four styles, mean of chat, writing, reasoning and safety; the model rates each response 1-10, 5 when no score can be parsed, ties count as errors). Source: arxiv.org. Saturation forecast: Around February 2027. 5 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 235B A22B 2507 | 58.08 | #613 |
| 2 | DeepSeek V3.2 | 52.51 | #403 |
| 3 | Qwen 3 8B | 44.95 | #1838 |
| 4 | Qwen 2.5 7B Instruct | 43.77 | #2583 |
| 5 | Llama 3.1 8B Instruct | 30 | #4277 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=rmgap-pointwise-llm-judge · How It Works · Data refreshed daily, snapshot 2026-11-12.