EMO Reference - MMLU-Pro (560-question subset) (greedy zero-shot): leaderboard
Table 1 zero-shot Base column only: the untouched model, greedy decoding, max_new_tokens 8,192 (12,288 on AIME; 16,384 for DeepSeek-R1-0528-Qwen3-8B), vLLM. Stratified 560-question MMLU-Pro subset (14 categories x 40), ten options. TextGrad and EMO columns are test-time optimisation methods and are not published. Checkpoints per the paper's Table 6 (Hugging Face ids); the paper marks the Qwen3, Qwen3.5 and DeepSeek-R1 distill checkpoints as models with an explicit long chain-of-thought mode. Metric: Accuracy (%). Higher is better. Source: https://arxiv.org/html/2610.22160v1
Metric: Accuracy (%). Source: arxiv.org. Saturation forecast: Around January 2027. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 32B | 71.4 | #442 |
| 2 | Qwen 3 14B | 71.4 | #535 |
| 3 | Qwen 3 8B | 63.6 | #671 |
| 4 | Qwen 3.5 9B | 62.9 | #373 |
| 5 | DeepSeek R1 0528 Qwen3 8B | 62 | #760 |
| 6 | Qwen 2.5 7B Instruct | 54.8 | #858 |
| 7 | Llama 3.1 8B Instruct | 45.4 | #1029 |
| 8 | Llama 3.2 3B Instruct | 34.8 | #1347 |
No result here: #4 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=emo-reference-mmlu-pro-560-question-subset-greedy-zero-shot · How It Works · Data refreshed daily, snapshot 2026-10-18.