EMO Reference - GPQA-Diamond (greedy zero-shot): leaderboard
Table 1 zero-shot Base column only: the untouched model, greedy decoding, max_new_tokens 8,192 (12,288 on AIME; 16,384 for DeepSeek-R1-0528-Qwen3-8B), vLLM. 198 four-option questions. TextGrad and EMO columns are test-time optimisation methods and are not published. Checkpoints per the paper's Table 6 (Hugging Face ids); the paper marks the Qwen3, Qwen3.5 and DeepSeek-R1 distill checkpoints as models with an explicit long chain-of-thought mode. Metric: Accuracy (%). Higher is better. Source: https://arxiv.org/html/2610.22160v1
Metric: Accuracy (%). Source: arxiv.org. Saturation forecast: Around January 2027. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 32B | 59.6 | #442 |
| 2 | Qwen 3 14B | 52.5 | #535 |
| 3 | Qwen 3 8B | 49 | #671 |
| 4 | DeepSeek R1 0528 Qwen3 8B | 46 | #760 |
| 5 | Qwen 3.5 9B | 39.4 | #373 |
| 6 | Qwen 2.5 7B Instruct | 32.8 | #858 |
| 7 | Llama 3.1 8B Instruct | 29.3 | #1029 |
| 8 | Llama 3.2 3B Instruct | 26.8 | #1347 |
No result here: #4 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=emo-reference-gpqa-diamond-greedy-zero-shot · How It Works · Data refreshed daily, snapshot 2026-10-18.