MSQA: leaderboard
MSQA asks short culture-specific questions written by native speakers in their own language, each with one verifiable answer grounded in local sources. This board covers all 1,064 questions across 11 language groups. Answers are free-form, judged for containing the gold answer, and scored as the harmonic mean of the correct share and the correct-given-attempted share, so both wrong answers and refusals cost points (ByteDance Seed, Nanjing University and M-A-P, 2026).
Metric: F-score (%; harmonic mean of CO, the share of fully correct answers, and CGA, the correct share of attempted answers, over all 1,064 questions; free-form answers to natively sourced culture-specific short questions asked in their own language, five runs per model, scored by a Gemini-3.1-Pro gold-answer containment judge). Source: arxiv.org. Saturation forecast: Around February 2027. 21 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 68.7 | #94 |
| 2 | GPT-5.5 | 55.6 | #49 |
| 3 | Claude Opus 4.6 | 52.8 | #102 |
| 4 | GPT-5.4 | 50.9 | #123 |
| 5 | DeepSeek V4 Pro | 50.8 | #188 |
| 6 | Seed 2.1 Pro | 50.4 | #135 |
| 7 | Claude Opus 4.7 | 49 | #79 |
| 8 | GPT-5.2 (High) | 45.4 | #176 (GPT-5.2) |
| 9 | Seed 2.0 Pro (High) | 44.3 | #211 (Seed 2.0 Pro) |
| 10 | GLM-5 | 42.1 | #260 |
| 11 | Seed 2.0 Pro (Medium) | 41.2 | #211 (Seed 2.0 Pro) |
| 12 | Seed 2.1 Turbo | 40.8 | #160 |
| 13 | Gemini 2.5 Flash | 40.6 | #453 |
| 14 | DeepSeek V3.2 | 40.4 | #406 |
| 15 | Qwen 3.5 Plus (Thinking) | 40.4 | #232 (Qwen 3.5 Plus) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=msqa · How It Works · Data refreshed daily, snapshot 2026-11-13.