NVIDIA Nemotron Report JudgeBench - Coding: leaderboard
JudgeBench: the model acts as a judge and must tell the higher-quality response from the lower-quality one in each pair; the number is the percentage of pairs judged correctly on the coding pairs. From Table 6 of NVIDIA's Llama-Nemotron technical report, comparing LN-Super and LN-Ultra with o1-preview, o1-mini, o3-mini at low, medium and high effort, and DeepSeek-R1; the report does not state the reasoning mode used for its own models in this test.
Metric: Preference discrimination accuracy (%). Source: arxiv.org. Saturation forecast: Estimated already saturated. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | O3 Mini (High) | 100 | #536 (O3 Mini) |
| 2 | DeepSeek R1 | 92.86 | #467 |
| 3 | O3 Mini (Medium) | 92.86 | #536 (O3 Mini) |
| 4 | Llama 3.1 Nemotron Ultra 253B v1 | 92.86 | #1027 |
| 5 | O1 Preview | 85.71 | #483 |
| 6 | O3 Mini (Low) | 83.33 | #536 (O3 Mini) |
| 7 | O1 Mini | 78.57 | #761 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=nvidia-nemotron-report-judgebench-coding · How It Works · Data refreshed daily, snapshot 2026-11-12.