KOR-Bench: leaderboard
Knowledge-Orthogonal Reasoning: 1,250 questions under 125 newly written rules for operations, logic, ciphers, puzzles and counterfactual worlds, which the model must apply as stated rather than from what it already knows. Chat models, zero-shot. Higher is better. Source: arxiv.org/abs/2410.06526.
Metric: Accuracy (%; zero-shot, 1,250 questions). Source: arxiv.org. Saturation forecast: Estimated already saturated. 38 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | O1 Preview (2024-09-12) | 72.88 | #220 |
| 2 | O1 Mini (2024-09-12) | 70.16 | #414 |
| 3 | Claude 3.5 Sonnet (20240620) | 58.96 | #309 |
| 4 | GPT-4o (2024-05-13) | 58 | #338 |
| 5 | Llama 3.1 405B Instruct | 55.36 | #472 |
| 6 | Qwen 2.5 32B Instruct | 54.72 | #511 |
| 7 | GPT-4 Turbo | 53.52 | #419 |
| 8 | Mistral Large 2 (Jul) | 53.12 | #498 |
| 9 | Qwen 2.5 72B Instruct | 52.16 | #470 |
| 10 | Llama 3.1 70B Instruct | 50 | #570 |
| 11 | Yi Large | 50 | #565 |
| 12 | Qwen 2.5 14B Instruct | 49.36 | #657 |
| 13 | Llama 3 70B Instruct | 49.2 | #643 |
| 14 | DeepSeek V2.5 | 47.76 | #503 |
| 15 | Qwen 2 72B Instruct | 47.04 | #584 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=kor-bench · How It Works · Data refreshed daily, snapshot 2026-11-07.