Smaug-Llama-3-70B-Instruct: benchmark results
Abacus.AI's Smaug-Llama-3-70B-Instruct, fine-tuned from Llama 3 70B Instruct using Abacus.AI's Smaug recipe aimed at improving real-world multi-turn conversation quality. Provider: Abacus.AI. Released 2024-05-17. Access: Open.
Aggregate Rating 1521 ± 34, rank #716 of 1656 rated models, from 19 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| Open Chinese LLM - CMMLU | 80.7 | Accuracy (%) | 100 |
| Enkrypt AI - Toxicity Risk | 0 | Risk Score | 98.5 |
| Open Portuguese LLM - OAB Exams | 63.64 | Accuracy (%) | 96.2 |
| Open Chinese LLM - GSM8K | 70.05 | Accuracy (%) | 95.3 |
| Open Chinese LLM Leaderboard | 69.11 | Official Nonzero-Task Mean (%) | 92 |
| Open Portuguese LLM - BLUEX | 69.54 | Accuracy (%) | 91.6 |
| Open Portuguese LLM - ENEM | 77.89 | Accuracy (%) | 91.5 |
| Open Portuguese LLM - ASSIN2 RTE | 93.62 | Macro F1 (%) | 90.6 |
| Open Portuguese LLM - FaQuAD NLI | 80.01 | Macro F1 (%) | 89.2 |
| Open Chinese LLM - C-Eval Semantic | 87.08 | Accuracy (%) | 83.4 |
| Open Chinese LLM - ARC Challenge | 58.36 | Accuracy (%) | 80 |
| Open Chinese LLM - HellaSwag | 64.73 | Accuracy (%) | 77.5 |
Interactive version: theaggregate.ai/model?slug=smaug-llama-3-70b-instruct · How It Works · Data refreshed daily, snapshot 2026-10-20.