LiT (Lost in Translation) - STEM Abstracts: leaderboard

LiT, Lost in Translation (2026), measures multilingual proficiency by round-trip translation: English passages pass through a sequence of four languages and back, and an LLM judge scores the result against the source with MQM error weights. Share of STEM paper abstracts whose round trip scores at least 80, in percent; higher is better.

Metric: MQM>=80 rate (%): share of round-trip translations whose back-translation to English scores at least 80 on an MQM error scale (minor -1, major -5, critical -25) from a Grok 4.1 Fast judge comparing it with the English source; each passage is translated serially through a four-language sequence and back, averaged over eight sequences (high, medium and low-resource languages); STEM paper abstracts; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 17 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Qwen 3.5 397B A17B88.1#264
2Gemini 3 FlashNon-reasoning85.6#175
3Gemma 4 31B (IT)76.2#350
4GLM-574.4#269
5GLM-4.760.6#380
6Kimi K2 (Thinking)58.1
7Qwen 3.5 35B A3B52.5#492
8Qwen 3 235B A22B 2507 (Thinking)51.9#466
9Kimi K243.3#443
10DeepSeek V3.2 ExpNon-reasoning42.5#439
11GPT-OSS-120BHigh41.2#704
12MiniMax-M2.536.2#600
13Qwen 3 235B A22B 2507 Instruct35#616
14Gemma 3 27B (IT)28.1#1324
15LiT Qwen3-30B-A3B Instruct (checkpoint unspecified)19.4#2113

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=lit-lost-in-translation-stem-abstracts · How It Works · Data refreshed daily, snapshot 2026-11-21.