IFMTBench - Single-Constraint: leaderboard

Instruction-following score (%) on the 4,506 single-constraint translation items in IFMTBench (2026), a benchmark of translation requests into seven languages that carry constraints such as schema preservation, glossaries, code spans and register. Hard constraints are checked by deterministic verifiers and gate the item, soft ones are rated by an LLM judge; higher is better.

Metric: IF_Score (%): hard constraints checked by deterministic verifiers gate the item, soft constraints are scored by a gpt-oss-120b rubric judge, mean over items, on the 4,506 single-constraint items, translation requests into seven languages with instructions paraphrased in all seven, officially recommended decoding with non-thinking mode for open models; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3.1 Pro (Preview)91.95#102
2Hy-MT2-30B-A3B91.94
3Hy-MT2-7B89.73
4Gemma 4 31B84#390
5Gemma 4 26B A4B83.17#602
6Qwen 3.5 27BNon-reasoning82.48#380
7Qwen 3.6 35B A3BNon-reasoning77.95#485
8Hy-MT2-1.8B76.76
9Qwen 3.5 35B A3BNon-reasoning76.46#494
10Qwen 3.5 9BNon-reasoning75.59#838
11Gemma 4 E4B74.64#2199
12Qwen 3.5 4BNon-reasoning72.58#1270
13Gemma 4 E2B63.71#4202
14Qwen 3.5 2BNon-reasoning54.77#5380
15Qwen 3.5 0.8BNon-reasoning40.92#8198

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=ifmtbench-single-constraint · How It Works · Data refreshed daily, snapshot 2026-11-22.