Open SLM Leaderboard - ArithMark-3: leaderboard
The Open SLM Leaderboard (Axiomic Labs, Hugging Face Space) ranks open-weight language models under 400M parameters that their submitters pretrained from scratch, with zero-shot scores from LM-eval harness and Axiomic Labs' ArithMark-3 script that the leaderboard team checks before merging. Unlike the Hugging Face Open LLM Leaderboard, which evaluated every submitted model on its own cluster, it lists zero-shot results the submitters ran and admits only small from-scratch checkpoints (no merges). The page does not state the harness version, the accuracy variant (acc or acc_norm) or the evaluated splits. Rows are bound to their Hugging Face model card; a row without one (Mystery-5M-Preview) is withheld, and a row missing a score is absent from that board only. ArithMark-3.0, Axiomic Labs' synthetic arithmetic set: 1,000 four-choice word problems (grades 1-5) scored by continuation log-likelihood, accuracy (%, chance 25).
Metric: Zero-shot accuracy (%, chance 25). Source: huggingface.co. Saturation forecast: Rough model projection: around 2026. 256 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Inferno-Exp-1.7 (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=AlgoDriveAI/Inferno-Exp-1.7) | 85.7 | |
| 2 | Inferno-Exp-1.6 (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=AlgoDriveAI/Inferno-Exp-1.6) | 83.1 | |
| 3 | MobileLLM-R1-360M-base (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=facebook/MobileLLM-R1-360M-base) | 80.4 | |
| 4 | Inferno-Exp-1.5 (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=AlgoDriveAI/Inferno-Exp-1.5) | 71 | |
| 5 | MobileLLM-R1-140M-base (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=facebook/MobileLLM-R1-140M-base) | 65.7 | |
| 6 | SmolLM2-360M (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=HuggingFaceTB/SmolLM2-360M) | 59.6 | |
| 7 | KLM-1.6 (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=MinimaLabs/KLM-1.6) | 55.9 | |
| 8 | Inferno-Exp-1.0 (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=AlgoDriveAI/Inferno-Exp-1.0) | 54.2 | |
| 9 | GPT-X3-150M (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=AxiomicLabs/GPT-X3-150M) | 50.9 | |
| 10 | KLM-1.5 (Benchmark Sweep Run: source=Open SLM Leaderboard; zero-shot; card=MinimaLabs/KLM-1.5) | 48.2 |
Interactive version: theaggregate.ai/benchmark?slug=open-slm-leaderboard-arithmark-3 · How It Works · Data refreshed daily, snapshot 2026-11-12.