Smaug-Mixtral-v0.1: benchmark results
Abacus.AI's Smaug-series fine-tune of Mixtral-8x7B using their DPO-Positive (DPOP) preference-training technique. Provider: Abacus.AI. Released 2024-02-18. Access: Open.
Unified ELO 1476 ± 1, rank #808 of 1392 rated models, from 14 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| Open LLM Leaderboard - MuSR | 12.99 | Score | 71.3 |
| Open Chinese LLM - TruthfulQA MC | 56.32 | Accuracy (%) | 69.3 |
| Open LLM Leaderboard - IFEval | 55.54 | Score | 67.2 |
| Open LLM Leaderboard - BBH | 31.92 | Score | 59.2 |
| Open LLM Leaderboard - GPQA | 6.82 | Score | 57.4 |
| Open Chinese LLM - ARC Challenge | 53.16 | Accuracy (%) | 53.6 |
| Open Chinese LLM - C-Eval Semantic | 70.53 | Accuracy (%) | 52.8 |
| Open LLM Leaderboard - MMLU-Pro | 26.13 | Score | 47.6 |
| Open Chinese LLM - CMMLU | 53.02 | Accuracy (%) | 46.3 |
| Open Chinese LLM - WinoGrande | 62.59 | Accuracy (%) | 45.8 |
| Open LLM Leaderboard - MATH Level 5 | 9.52 | Score | 45.7 |
| Open Chinese LLM - HellaSwag | 59.47 | Accuracy (%) | 44.5 |
Interactive version: theaggregate.ai/model?slug=smaug-mixtral-v0-1 · How It Works · Data refreshed daily, snapshot 2026-09-05.