NVIDIA Nemotron Report JudgeBench - Knowledge: leaderboard
JudgeBench: the model acts as a judge and must tell the higher-quality response from the lower-quality one in each pair; the number is the percentage of pairs judged correctly on the knowledge pairs. From Table 6 of NVIDIA's Llama-Nemotron technical report, comparing LN-Super and LN-Ultra with o1-preview, o1-mini, o3-mini at low, medium and high effort, and DeepSeek-R1; the report does not state the reasoning mode used for its own models in this test.
Metric: Preference discrimination accuracy (%). Source: arxiv.org. Saturation forecast: Estimated already saturated. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Llama 3.1 Nemotron Ultra 253B v1 | 70.13 | #1027 |
| 2 | O3 Mini (High) | 67.53 | #536 (O3 Mini) |
| 3 | O1 Preview | 66.23 | #483 |
| 4 | O3 Mini (Low) | 62.99 | #536 (O3 Mini) |
| 5 | O3 Mini (Medium) | 62.34 | #536 (O3 Mini) |
| 6 | DeepSeek R1 | 59.09 | #467 |
| 7 | O1 Mini | 58.44 | #761 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=nvidia-nemotron-report-judgebench-knowledge · How It Works · Data refreshed daily, snapshot 2026-11-12.