IOLBench - Short Answer: leaderboard
Short-answer problems from past International Linguistics Olympiads (666 text problems), scored by string match against the answer key. The model infers an unfamiliar language's rules from the problem's examples. Higher is better. Source: arxiv.org/abs/2501.04249.
Metric: Accuracy (%; string match, 666 text problems). Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5 | 61.12 | #108 |
| 2 | Claude Opus 4 | 41.9 | #172 |
| 3 | Gemini 2.5 Pro | 39.76 | #161 |
| 4 | Grok 4 | 38.11 | #191 |
| 5 | Claude Sonnet 4 | 36.84 | #209 |
| 6 | GPT-4o | 26.87 | #358 |
| 7 | GPT-4 | 24.48 | #454 |
| 8 | GPT-4o Mini | 20.62 | #601 |
| 9 | Gemini 2.5 Flash | 19.76 | #255 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=iolbench-short-answer · How It Works · Data refreshed daily, snapshot 2026-11-07.