Lara Mini Base Report (base-model protocol) - GSM8K (5-shot): leaderboard
Harness: lm-evaluation-harness 0.4.14.dev0; task configs unchanged except shots (A4.T17); no chat template; identical prompts/settings/answer extraction for every model (§5.1; Appendix D). Backends: Log-likelihood tasks: HF Transformers backend, fp32 log-softmax. Generative tasks: vLLM, bf16, greedy decoding, 4,096-token context; backends agree within 0.6 points on checked tasks (§5.1; Appendix D). Shots and metrics: Per A4.T17 (e.g. HellaSwag 10-shot acc_norm, MMLU 5-shot, MMLU-Pro 5-shot CoT exact match, GSM8K 5-shot EM, MATH 4-shot Math-Verify, MGSM 8-shot CoT over 11 languages, MBPP/MBPP+ 3-shot pass@1, CRUXEval 2-shot pass@1) (A4.T17). Answer extraction: Stricter parsers for GSM8K, MGSM (native CoT answer phrase) and MBPP+ (must pass MBPP and EvalPlus tests); MATH Minerva prompts with Math-Verify over 5,000 problems; CRUXEval direct prediction with official parser (Appendix D Answer extraction). Evaluated Lara checkpoint: Released checkpoint after the 100B-token long-context stage (translated/Lara-4-Mini-Base @ c998c0e1251a734eeff364d532330790a4f93c65 as captured; revision used by the authors not stated) (§5 introduction; §7 Release). Long context: RULER (13 tasks), ONERULER (4 retrieval tasks, 26 languages), HELMET (4 categories) at 32k and 128k with base-model modifications in A4.T24 (§5.7; Appendix D.3; A4.T24). Runs / seeds: not stated; scoring is log-likelihood or greedy generation (not stated). Benchmark availability: T-HellaSwag is the authors' translation, announced as "will release" at v1 (§5.2; §7). This board: Full benchmark (MATH: 5,000 problems; others not printed); Single greedy generation per item (vLLM bf16, 4,096-token context); run count not stated. lm-evaluation-harness 0.4.14.dev0, 5-shot (A4.T17); answer-extraction overrides in Appendix D. Metric: Exact match (strict extraction: first "#### N", else first \boxed{N}, else first "the answer is N"; never the last number) (%) Higher is better.
Metric: Exact match (strict extraction: first "#### N", else first \boxed{N}, else first "the answer is N"; never the last number) (%). Source: arxiv.org. Saturation forecast: Around February 2027. 55 models tracked. Updated 14 Nov 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen3-4B-Base | 87.8 | #2469 |
| 2 | Qwen3-8B-Base | 86.7 | #1774 |
| 3 | Qwen 2.5 7B | 83.9 | #2829 |
| 4 | Qwen3.5-4B-Base | 82.4 | #2478 |
| 5 | limite-1B-base | 78.6 | #8819 |
| 6 | Hunyuan-4B-Pretrain | 78 | #2455 |
| 7 | limite-1B-base-soup | 76.6 | |
| 8 | Domyn-Edge-Base | 74.6 | #4368 |
| 9 | Falcon-H1-1.5B-Deep-Base | 72.3 | #5677 |
| 10 | granite-4.1-3B-base | 72.2 | #6109 |
| 11 | Qwen3-1.7B-Base | 71.8 | #5246 |
| 12 | Hunyuan-1.8B-Pretrain | 71.6 | #5939 |
| 13 | SmolLM3-3B-Base | 68.7 | #6586 |
| 14 | LFM2.5-8B-A1B-Base | 67.6 | #6243 |
| 15 | Lara-4-Mini-Base | 67.2 | #5152 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=lara-mini-base-report-base-model-protocol-gsm8k-5-shot · How It Works · Data refreshed daily, snapshot 2026-11-16.