CanMT: leaderboard

CanMT (2026) is a culture-aware machine translation benchmark of novel sentences containing culture-specific items, scored by an LLM judge with references on five translation-quality dimensions. This board is the mean translation score on a 1 to 7 scale over twelve translation directions; higher is better.

Metric: Mean translation score (1-7) over the 12 CanMT directions (unweighted mean of the direction scores), GPT-5-nano judge with the reference translation scoring contextual accuracy, cultural adaptation, functional equivalence, fidelity and naturalness on 7-point Likert scales, averaged per sentence; default translation prompt; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 22 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Grok 4.15.18#462
2GPT-45.14#970
3DeepSeek V3.25.08#419
4DeepSeek R15.02#482
5Gemini 2.5 Flash Lite5#964
6GPT-4o4.96#751
7Qwen 2.5 72B Instruct4.91#1117
8Llama 3.3 70B Instruct4.78#1385
9Seed-X-PPO-7B4.76
10Qwen 3 32BNon-reasoning4.74#972
11Qwen 3 14BNon-reasoning4.73#1269
12Qwen 2.5 32B Instruct4.69#1247
13Qwen 2.5 14B Instruct4.51#1775
14Google Translate4.5
15Qwen 3 8BNon-reasoning4.48#1873

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=canmt · How It Works · Data refreshed daily, snapshot 2026-11-21.