Linguini: leaderboard
894 questions from International Linguistics Olympiad problems in 75 mostly extremely low-resource languages. Each problem carries the examples needed to solve it, so the model infers the language's rules from a small sample instead of recalling them. Exact match with no extra solved problems as examples. Higher is better. Source: arxiv.org/abs/2409.12126.
Metric: Exact Match (%; 0 in-context examples, 894 questions). Source: arxiv.org. Saturation forecast: Estimated already saturated. 13 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude 3 Opus (20240229) | 24.05 | #428 |
| 2 | GPT-4o (2024-05-13) | 14.65 | #338 |
| 3 | Claude 3 Sonnet (20240229) | 12.3 | #697 |
| 4 | GPT-4 Turbo | 8.72 | #419 |
| 5 | Llama 3 70B | 8.17 | #635 |
| 6 | GPT-4 Preview (0125) | 6.38 | #500 |
| 7 | Claude 3 Haiku (20240307) | 6.04 | #780 |
| 8 | Llama 3 70B Instruct | 4.81 | #643 |
| 9 | Llama 2 70B | 4.7 | #1046 |
| 10 | Mixtral 8x7B | 2.46 | #1096 |
| 11 | Qwen 1.5 110B Chat | 1.45 | #695 |
| 12 | Llama 2 70B Chat | 0.89 | #1278 |
| 13 | gemma-2B | 0.34 | #1555 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=linguini · How It Works · Data refreshed daily, snapshot 2026-11-07.