Smaug-72B-v0.1: benchmark results
Abacus.AI's Qwen-72B-derived fine-tune using DPO-Positive, the first open-weights model to average over 80 on the Open LLM Leaderboard (February 2024). Provider: Abacus.AI. Released 2024-02-02. Access: Open.
Aggregate Rating 1551 ± 23, rank #611 of 1656 rated models, from 25 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| Open LLM Leaderboard v1 - ARC Challenge | 76.02 | Normalized accuracy (%) (25-shot) | 99.5 |
| Open LLM Leaderboard v1 - GSM8K | 78.7 | Accuracy (%) (5-shot) | 99.5 |
| Open LLM Leaderboard v1 - HellaSwag | 89.27 | Normalized accuracy (%) (10-shot) | 98.9 |
| Open Chinese LLM Leaderboard | 73.16 | Official Nonzero-Task Mean (%) | 98.8 |
| Open LLM Leaderboard v1 - MMLU | 77.15 | Accuracy (%) (5-shot) | 98.3 |
| Open Chinese LLM - ARC Challenge | 67.66 | Accuracy (%) | 98.2 |
| Open Chinese LLM - TruthfulQA MC | 67.02 | Accuracy (%) | 98.2 |
| Open Chinese LLM - HellaSwag | 75 | Accuracy (%) | 97.9 |
| Open LLM Leaderboard v1 - WinoGrande | 85.08 | Accuracy (%) (5-shot) | 97.1 |
| Open LLM Leaderboard v1 - TruthfulQA MC2 | 76.67 | MC2 (%) (0-shot) | 96.2 |
| Open Chinese LLM - CMMLU | 72.32 | Accuracy (%) | 95.9 |
| Open Chinese LLM - GSM8K | 70.13 | Accuracy (%) | 95.7 |
Interactive version: theaggregate.ai/model?slug=smaug-72b-v0-1 · How It Works · Data refreshed daily, snapshot 2026-10-20.