Multilingual Judges - SSA-MTE Trans-Eval: leaderboard
Multilingual Judges - SSA-MTE Trans-Eval. This multilingual judge study measures instance-level Spearman correlation with human quality scores on SSA-MTE, which contains 12,565 translations. Each prompting strategy is a separate condition. Judges are Aya Expanse 32B, Gemma 3 27B IT, Qwen2.5 32B Instruct, and Qwen2.5 72B Instruct AWQ; the quantized 72B identity is preserved. Inference uses vLLM on A100/H100 GPUs, temperature 0, top-p 1, and seed 42. Invalid holistic scores outside the required 1–5 response schema are excluded by the paper. Table 8’s final Mean column is a cross-dataset aggregate, not another judge. Mean ratings, language score gaps, and coefficients of variation are not treated as correctness. Reported metric: Spearman correlation.
Metric: Spearman correlation. Source: arxiv.org. Saturation forecast: Estimated already saturated. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 3 27B (IT) | 0.33 | #529 |
| 2 | Qwen 2.5 32B Instruct | 0.27 | #507 |
| 3 | aya-expanse-32B | 0.24 | #690 |
No result here: #5 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=multilingual-judges-ssa-mte-trans-eval · How It Works · Data refreshed daily, snapshot 2026-10-25.