Smaug-Mixtral-v0.1: benchmark results
Abacus.AI's Smaug-series fine-tune of Mixtral-8x7B using their DPO-Positive (DPOP) preference-training technique. Provider: Abacus.AI. Released 2024-02-18. Access: Open.
Aggregate Rating 1449 ± 40, rank #3803 of 10916 rated models, from 14 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| Open LLM Leaderboard - MuSR | 42.98 | Score | 70.3 |
| Open Chinese LLM - TruthfulQA MC | 56.32 | Accuracy (%) | 69.1 |
| Open LLM Leaderboard - IFEval | 55.54 | Score | 67.2 |
| Open LLM Leaderboard - GPQA | 30.12 | Score | 57.4 |
| Open LLM Leaderboard - BBH | 51.62 | Score | 57 |
| Open Chinese LLM - ARC Challenge | 53.16 | Accuracy (%) | 53.4 |
| Open Chinese LLM - C-Eval Semantic | 70.53 | Accuracy (%) | 53 |
| Open LLM Leaderboard - MMLU-Pro | 33.52 | Score | 47.6 |
| Open Chinese LLM - CMMLU | 53.02 | Accuracy (%) | 46.4 |
| Open Chinese LLM - WinoGrande | 62.59 | Accuracy (%) | 45.7 |
| Open LLM Leaderboard - MATH Level 5 | 9.52 | Score | 45.7 |
| Open Chinese LLM - HellaSwag | 59.47 | Accuracy (%) | 44.4 |
Interactive version: theaggregate.ai/model?slug=smaug-mixtral-v0-1 · How It Works · Data refreshed daily, snapshot 2026-11-08.