LiT (Lost in Translation) - Discourse Coherence: leaderboard

LiT, Lost in Translation (2026), measures multilingual proficiency by round-trip translation: English passages pass through a sequence of four languages and back, and an LLM judge scores the result against the source with MQM error weights. Share of pragmatics passages on reference chains and connectives whose round trip scores at least 80, in percent; higher is better.

Metric: MQM>=80 rate (%): share of round-trip translations whose back-translation to English scores at least 80 on an MQM error scale (minor -1, major -5, critical -25) from a Grok 4.1 Fast judge comparing it with the English source; each passage is translated serially through a four-language sequence and back, averaged over eight sequences (high, medium and low-resource languages); pragmatics passages on referential chains and connectives; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 17 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3 FlashNon-reasoning89.3#175
2GLM-577.4#269
3Qwen 3.5 397B A17B75.6#264
4Gemma 4 31B (IT)75#350
5Kimi K2 (Thinking)70.8
6GLM-4.770.2#380
7Qwen 3 235B A22B 2507 (Thinking)66.7#466
8DeepSeek V3.2 ExpThinking61.9#439
9Qwen 3.5 35B A3B60.1#492
10Kimi K255.6#443
11Gemma 3 27B (IT)55.4#1324
12GPT-OSS-120BHigh54.8#704
13Qwen 3 235B A22B 2507 Instruct52.4#616
14MiniMax-M2.551.8#600
15MiMo-V2-FlashNon-reasoning41.7#684

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=lit-lost-in-translation-discourse-coherence · How It Works · Data refreshed daily, snapshot 2026-11-21.