Lara Mini Base Report (base-model protocol) - XNLI, All Languages (0-shot): leaderboard

Harness: lm-evaluation-harness 0.4.14.dev0; task configs unchanged except shots (A4.T17); no chat template; identical prompts/settings/answer extraction for every model (§5.1; Appendix D). Backends: Log-likelihood tasks: HF Transformers backend, fp32 log-softmax. Generative tasks: vLLM, bf16, greedy decoding, 4,096-token context; backends agree within 0.6 points on checked tasks (§5.1; Appendix D). Shots and metrics: Per A4.T17 (e.g. HellaSwag 10-shot acc_norm, MMLU 5-shot, MMLU-Pro 5-shot CoT exact match, GSM8K 5-shot EM, MATH 4-shot Math-Verify, MGSM 8-shot CoT over 11 languages, MBPP/MBPP+ 3-shot pass@1, CRUXEval 2-shot pass@1) (A4.T17). Answer extraction: Stricter parsers for GSM8K, MGSM (native CoT answer phrase) and MBPP+ (must pass MBPP and EvalPlus tests); MATH Minerva prompts with Math-Verify over 5,000 problems; CRUXEval direct prediction with official parser (Appendix D Answer extraction). Evaluated Lara checkpoint: Released checkpoint after the 100B-token long-context stage (translated/Lara-4-Mini-Base @ c998c0e1251a734eeff364d532330790a4f93c65 as captured; revision used by the authors not stated) (§5 introduction; §7 Release). Long context: RULER (13 tasks), ONERULER (4 retrieval tasks, 26 languages), HELMET (4 categories) at 32k and 128k with base-model modifications in A4.T24 (§5.7; Appendix D.3; A4.T24). Runs / seeds: not stated; scoring is log-likelihood or greedy generation (not stated). Benchmark availability: T-HellaSwag is the authors' translation, announced as "will release" at v1 (§5.2; §7). This board: All languages or variants of the benchmark (count not printed; T-HellaSwag has 110 variants of 200 documents each). The A4.T17 Lang. value (12) counts only Lara-supported languages; Unweighted mean over languages as printed; single evaluation per model. lm-evaluation-harness 0.4.14.dev0, 0-shot. Metric: Accuracy (%) averaged over all languages/variants of the benchmark, including those Lara Mini base does not support Higher is better.

Metric: Accuracy (%) averaged over all languages/variants of the benchmark, including those Lara Mini base does not support. Source: arxiv.org. Saturation forecast: Around 2031. 55 models tracked. Updated 14 Nov 2026.

Top models

#ModelScoreOverall rank
1tiny-aya-base45.6#6632
2Gemma 4 E4B45.1#2412
3Llama 3.1 8B44.9#5129
4Apertus-v1.5-8B-base44.6#4427
5Llama 3 8B44.4#5867
6Gemma 4 E2B43.5#4669
7Apertus-v1.1-1.5B43.5#9114
8Lara-4-Mini-Base43.4#5152
9gemma-7B42.9#7067
10Teuken-7B-base-v0.642.8#8722
11Ministral-3-3B-Base-251242.7#3138
12Qwen 2.5 7B42.6#2829
13Qwen3-4B-Base42.4#2469
14granite-4.0-h-tiny-base41.8#5994
15Mistral-7B-v0.341.5#7188

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=lara-mini-base-report-base-model-protocol-xnli-all-languages-0-shot · How It Works · Data refreshed daily, snapshot 2026-11-16.