MSQA - Cultural Products and Symbols: leaderboard
MSQA asks short culture-specific questions written by native speakers in their own language, each with one verifiable answer grounded in local sources. This board covers the questions about local brands, crafts, foods, genres and named symbols. Answers are free-form, judged for containing the gold answer, and scored as the harmonic mean of the correct share and the correct-given-attempted share, so both wrong answers and refusals cost points (ByteDance Seed, Nanjing University and M-A-P, 2026).
Metric: F-score (%; harmonic mean of CO, the share of fully correct answers, and CGA, the correct share of attempted answers, over the 208 questions of this cultural dimension; free-form answers to natively sourced culture-specific short questions asked in their own language, five runs per model, scored by a Gemini-3.1-Pro gold-answer containment judge). Source: arxiv.org. Saturation forecast: Around February 2027. 21 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) | 66.1 | #94 |
| 2 | GPT-5.5 | 55 | #49 |
| 3 | Claude Opus 4.6 | 50.6 | #102 |
| 4 | GPT-5.4 | 49.2 | #123 |
| 5 | GPT-5.2 (High) | 46.7 | #176 (GPT-5.2) |
| 6 | DeepSeek V4 Pro | 46.2 | #188 |
| 7 | Claude Opus 4.7 | 45.6 | #79 |
| 8 | Seed 2.1 Pro | 45.5 | #135 |
| 9 | Qwen 3.5 Plus (Thinking) | 41.4 | #232 (Qwen 3.5 Plus) |
| 10 | GLM-5 | 39.8 | #260 |
| 11 | Kimi K2.6 | 39.3 | #178 |
| 12 | Seed 2.0 Pro (High) | 38.9 | #211 (Seed 2.0 Pro) |
| 13 | Kimi K2.5 | 38.7 | #256 |
| 14 | DeepSeek V3.2 | 38.7 | #406 |
| 15 | Seed 2.0 Pro (Medium) | 38.4 | #211 (Seed 2.0 Pro) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=msqa-cultural-products-and-symbols · How It Works · Data refreshed daily, snapshot 2026-11-13.