GaelEval - Gaelic to English Translation (chrF): leaderboard

GaelEval (2026) is a Scottish Gaelic benchmark from the DR-LIB project. This board is chrF, a character n-gram F-score (0 to 100), for translating 908 An Litir Bheag podcast transcripts from Gaelic into English, scored against the professional parallel translations; higher is better.

Metric: chrF character n-gram F-score of Gaelic-to-English translation against gold references; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 18 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3 Pro (Preview)78.2#119
2DeepSeek R177.73#482
3Gemini 2.5 Flash77.58#463
4Claude Opus 4.677.38#113
5Gemini 3 Flash (Preview)77.38#152
6GPT-5.276.18#185
7GPT-575.82#178
8Llama 4 Maverick75.74#1110
9GPT-4.1 Mini75.48#789
10GPT-4o74.8#751
11Claude Haiku 4.573.92#583
12GPT-5 Mini73.34#364
13GPT-4o Mini73.29#1667
14GPT-4.172.77#505
15GPT-4.1 Nano69.04#2056

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=gaeleval-gaelic-to-english-translation-chrf · How It Works · Data refreshed daily, snapshot 2026-11-21.