CP-SynC-XL - Python + OR-Tools: leaderboard
Solver synthesis on CP-SynC-XL (Wang et al., 2026): for each of 100 natural-language combinatorial problems from CSPLib and PyCSP3 the LLM writes a Python model for the OR-Tools CP-SAT constraint solver, under the baseline prompt, refined only on smoke and scaling checks. Share of the 4,577 instances whose output passes the reference verifier for feasibility and optimality within 256 seconds, in percent; higher is better.
Metric: Correct instances (%; share of the 4,577 CP-SynC-XL instances of 100 CSPLib and PyCSP3 combinatorial problems whose returned solution passes the reference verifier (feasibility, COP optimality) within 256 s; constraint modelling: the LLM writes a Python model for the OR-Tools CP-SAT solver, baseline prompt; final artefact after smoke and scaling-check refinement; greedy decoding). Source: arxiv.org. Saturation forecast: Around February 2027. 3 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) (Medium) | 71.6 | #91 (Gemini 3.1 Pro (Preview)) |
| 2 | GPT-5.3 Codex (Medium) | 70.2 | #114 (GPT-5.3 Codex) |
| 3 | DeepSeek V3.2 | 43.9 | #403 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=cp-sync-xl-python-plus-or-tools · How It Works · Data refreshed daily, snapshot 2026-11-12.