Arabic Dialect SME Rubrics - Judge Deviation: leaderboard
How closely an LLM judge reproduces native-speaker expert grades on the Egyptian and Iraqi Arabic rubric set of the same 2026 study. Mean absolute deviation from the expert rubric score in percentage points, over the responses the judge may grade when it never grades its own provider; lower is better.
Metric: Judge mean absolute deviation (percentage points) between the judge model rubric score and the native-speaker expert score on the Egyptian and Iraqi Arabic rubric set, over the responses the judge may grade under a provider-level self-evaluation guard; temperature 0; lower is better. Source: arxiv.org. Saturation forecast: Around September 2027. 5 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 (2026-03-05) | 10.21 | #87 |
| 2 | Gemini 3.1 Pro (Preview) | 10.61 | #94 |
| 3 | Claude Opus 4.7 | 11.35 | #79 |
| 4 | Qwen 3.7 Plus | 12.42 | #207 |
| 5 | Grok 4.3 | 13.03 | #333 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=arabic-dialect-sme-rubrics-judge-deviation · How It Works · Data refreshed daily, snapshot 2026-11-13.