JP-TL-Bench: leaderboard
Japanese-English translation quality from JP-TL-Bench (Lin and Lensenmayer, Shisa.AI, 2026): 70 source texts in both directions, split into Easy and Hard, that stress register and keigo, implicature, ellipsis and cultural adaptation. gemini-2.5-flash compares translations pairwise and reference-free at temperature 0, and a Bradley-Terry fit is mapped to a 0-10 LT score; this board is the paper's overall slice for the frozen Base Set v1.0 anchor models, fitted among themselves. Higher is better; the judge model is also one of the anchors.
Metric: LT Score (0-10; both directions, anchor-set Bradley-Terry fit, gemini-2.5-flash judge). Source: arxiv.org. Saturation forecast: Estimated already saturated. 20 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 2.5 Pro | 9.94 | #262 |
| 2 | Gemini 2.5 Flash | 9.89 | #452 |
| 3 | Qwen 3 30B A3B 2507 Instruct | 9.63 | #1078 |
| 4 | GPT-4o | 9.12 | #741 |
| 5 | Llama-3.1-Swallow-8B-Instruct-v0.5 | 7.42 | #2711 |
| 6 | Nemotron Nano 12B V2 | 7.05 | #1753 |
| 7 | Llama 3.3 70B Instruct | 6.72 | #1353 |
| 8 | Phi-4 | 5.12 | #1920 |
| 9 | Qwen 3 4B | 4.1 | #2297 |
| 10 | LFM2 2.6B | 3.91 | #7560 |
| 11 | Llama 3.1 8B Instruct | 2.94 | #4277 |
| 12 | Phi-4 Mini Instruct | 0.99 | #6452 |
| 13 | Llama 3.2 3B Instruct | 0.54 | #7185 |
| 14 | LFM2-350M | 0.14 | #9549 |
| 15 | TinySwallow-1.5B | 0.04 | #7366 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=jp-tl-bench · How It Works · Data refreshed daily, snapshot 2026-11-12.