pythia-12B: benchmark results
EleutherAI's Pythia-12B, the largest model in the eight-size Pythia Scaling Suite built for interpretability research, trained on the Pile with 154 intermediate checkpoints released for analysis. Provider: EleutherAI. Released 2023-02-28. Access: Open.
Aggregate Rating 1154 ± 11, rank #1621 of 1656 rated models, from 115 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| HELM Classic - Dyck | 79.8 | Exact Match (%) | 97.1 |
| HELM Classic - MATH Chain-of-Thought | 6.8 | Equivalent (%) | 63.2 |
| HELM Classic - bAbI | 49.12 | Exact Match (%) | 56.5 |
| MMLU-by-task - Machine Learning | 33.04 | Accuracy (%) | 53.4 |
| HELM Classic - MATH | 10.05 | Equivalent (%) | 51.5 |
| MMLU-by-task - High School Mathematics | 27.04 | Accuracy (%) | 50.6 |
| HELM Classic - Entity Matching | 77.98 | Exact Match (%) | 45.5 |
| MMLU-by-task - College Mathematics | 31 | Accuracy (%) | 45.5 |
| MMLU-by-task - High School Statistics | 36.57 | Accuracy (%) | 45.4 |
| ToolBench - WebShop Long | 0 | Task Score | 44.2 |
| HELM Classic - IMDB | 93.1 | Exact Match (%) | 42.4 |
| MMLU-by-task - Elementary Mathematics | 26.98 | Accuracy (%) | 39.2 |
Interactive version: theaggregate.ai/model?slug=pythia-12b · How It Works · Data refreshed daily, snapshot 2026-10-20.