MIPLIB-NL (Pass@8): leaderboard
Metric: Pass@8 accuracy (%): an instance counts as solved when any of 8 independent samples reaches the optimum (MIPLIB-NL: natural-language optimization-modeling problems reverse-generated one-to-one from MIPLIB 2017 mixed-integer programs (223 instances, the 8 infeasible and 15 open ones excluded from scoring), with model and data specified separately; the model writes the formulation and solver code, which are executed; an instance is solved when the solver objective matches the certified optimum within 1e-6 relative error; temperature 0.6; direct prompting, no tools); higher is better. Source: arxiv.org. Saturation forecast: Around July 2027. 14 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.1 | 42.38 | #131 |
| 2 | Gemini 3 Pro (Preview) | 39.52 | #64 |
| 3 | DeepSeek V3.2 (Thinking) | 33.33 | #198 (DeepSeek V3.2) |
| 4 | Claude Sonnet 4.5 (Thinking) | 32.38 | #138 (Claude Sonnet 4.5) |
| 5 | GPT-5.1 Codex | 28.75 | #154 |
| 6 | Qwen 3 Max (Preview) (Thinking) | 27.62 | #224 (Qwen 3 Max (Preview)) |
| 7 | Claude Sonnet 4.5 | 25.71 | #138 |
| 8 | DeepSeek V3.2 (Non-reasoning) | 23.81 | #198 (DeepSeek V3.2) |
| 9 | Qwen 3 Max (Non-reasoning) | 23.33 | #201 (Qwen 3 Max) |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #8 Claude Fable 5.1.
Interactive version: theaggregate.ai/benchmark?slug=miplib-nl-pass-8 · How It Works · Data refreshed daily, snapshot 2026-10-11.