LiT (Lost in Translation) Robustness - Polysemy and Lexical Ambiguity: leaderboard

LiT, Lost in Translation (2026), measures multilingual proficiency by round-trip translation: English passages pass through a sequence of four languages and back, and an LLM judge scores the result against the source with MQM error weights. Share of robustness sentences with polysemous words whose round trips score at least 80, averaged over eight translation sequences, in percent; higher is better.

Metric: MQM>=80 rate (%): share of round-trip translations whose back-translation to English scores at least 80 on an MQM error scale (minor -1, major -5, critical -25) from a Grok 4.1 Fast judge comparing it with the English source; each passage is translated serially through a four-language sequence and back, averaged over eight sequences (high, medium and low-resource languages); robustness subset: polysemous words; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 17 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3 FlashNon-reasoning68.8#180
2Qwen 3.5 397B A17B59.4#270
3Kimi K2 (Thinking)55.2
4Gemma 4 31B (IT)Thinking54.2#354
5GLM-550#276
6GLM-4.747.9#383
7Qwen 3 235B A22B 2507 (Thinking)45.8#471
8DeepSeek V3.2 ExpThinking43.8#440
9MiniMax-M2.541.7#600
10Qwen 3.5 35B A3B38.5#494
11Gemma 3 27B (IT)34.4#1341
12Qwen 3 235B A22B 2507 Instruct33.3#625
13Kimi K230.5#448
14GPT-OSS-120BHigh30.2#712
15MiMo-V2-FlashNon-reasoning22.9#691

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=lit-lost-in-translation-robustness-polysemy-and-lexical-ambiguity · How It Works · Data refreshed daily, snapshot 2026-11-22.