DAR-MGSM - MSA - Localized: leaderboard
DAR-MGSM Table3:250 MGSM-Rev2 problems in MSA and five Arabic dialects, translated vs culturally localized. Exact HF/API identities in AppendixD; reasoning disabled for Gemma4,Qwen3.6Plus,GPT5.4. Greedy local/Qwen decoding; GPT provider sampling default. Local Arabic/Qwen BF16,1024 output tokens; Gemma512 tokens. Single attempt, exact normalized numeric answers.
Metric: Accuracy (%). Source: arxiv.org. Saturation forecast: Around January 2027. 11 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 (Non-reasoning) | 89.2 | #82 (GPT-5.4) |
| 2 | Qwen 3.6 Plus (Non-reasoning) | 84.8 | #122 (Qwen 3.6 Plus) |
| 3 | Gemma 4 E4B (IT) (Non-reasoning) | 81.2 | |
| 4 | Gemma 4 E2B (IT) (Non-reasoning) | 71.6 | #789 (Gemma 4 E2B (IT)) |
| 5 | Qwen 2.5 7B Instruct | 69.6 | #861 |
| 6 | c4ai-command-r7B-arabic-02-2025 | 69.2 | #780 |
| 7 | Jais-2-8B-Chat | 68.8 | #877 |
| 8 | Atlas-Chat-9B | 61.6 | #1033 |
No result here: #4 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=dar-mgsm-msa-localized · How It Works · Data refreshed daily, snapshot 2026-10-22.