LiT (Lost in Translation) - Informal Text: leaderboard

LiT, Lost in Translation (2026), measures multilingual proficiency by round-trip translation: English passages pass through a sequence of four languages and back, and an LLM judge scores the result against the source with MQM error weights. Share of informal passages with slang and idioms whose round trip scores at least 80, in percent; higher is better.

Metric: MQM>=80 rate (%): share of round-trip translations whose back-translation to English scores at least 80 on an MQM error scale (minor -1, major -5, critical -25) from a Grok 4.1 Fast judge comparing it with the English source; each passage is translated serially through a four-language sequence and back, averaged over eight sequences (high, medium and low-resource languages); informal passages (slang, idioms, colloquial register); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 17 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3 FlashNon-reasoning80.3#175
2Gemma 4 31B (IT)56.9#350
3Kimi K2 (Thinking)47.2
4Qwen 3.5 397B A17BNon-reasoning46.9#264
5GLM-545.9#269
6DeepSeek V3.2 ExpNon-reasoning36.6#439
7Gemma 3 27B (IT)32.8#1324
8Kimi K232.1#443
9GLM-4.731.6#380
10Qwen 3 235B A22B 2507 Instruct25.3#616
11Qwen 3 235B A22B 2507 (Thinking)22.8#466
12Qwen 3.5 35B A3B15.9#492
13MiMo-V2-FlashNon-reasoning10.3#684
14GPT-OSS-120BHigh7.8#704
15MiniMax-M2.57.2#600

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=lit-lost-in-translation-informal-text · How It Works · Data refreshed daily, snapshot 2026-11-21.