pythia-12B: benchmark results
Provider: EleutherAI. Released 2023-02-28. Access: Open.
Unified ELO 1281 ± 1, rank #1364 of 1392 rated models, from 102 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| HELM Classic - Dyck | 79.8 | Exact Match (%) | 97.1 |
| HELM Classic - MATH Chain-of-Thought | 6.8 | Equivalent (%) | 63.2 |
| HELM Classic - bAbI | 49.12 | Exact Match (%) | 56.5 |
| MMLU-by-task - Machine Learning | 33.04 | Accuracy (%) | 53.1 |
| HELM Classic - MATH | 10.05 | Equivalent (%) | 51.5 |
| MMLU-by-task - High School Mathematics | 27.04 | Accuracy (%) | 50.5 |
| MMLU-by-task - High School Statistics | 36.57 | Accuracy (%) | 45.6 |
| HELM Classic - Entity Matching | 77.98 | Exact Match (%) | 45.5 |
| MMLU-by-task - College Mathematics | 31 | Accuracy (%) | 45.4 |
| ToolBench - WebShop Long | 0 | Task Score | 44.2 |
| HELM Classic - IMDB | 93.1 | Exact Match (%) | 42.4 |
| MMLU-by-task - Elementary Mathematics | 26.98 | Accuracy (%) | 39 |
Interactive version: theaggregate.ai/model?slug=pythia-12b · How It Works · Data refreshed daily, snapshot 2026-09-05.