CP-SynC-XL - Python (Heuristic Prompt): leaderboard

Solver synthesis on CP-SynC-XL (Wang et al., 2026): for each of 100 natural-language combinatorial problems from CSPLib and PyCSP3 the LLM writes a Python solver that performs the search itself, under a prompt that also asks it to optimize search efficiency, refined only on smoke and scaling checks. Share of the 4,577 instances whose output passes the reference verifier for feasibility and optimality within 256 seconds, in percent; higher is better.

Metric: Correct instances (%; share of the 4,577 CP-SynC-XL instances of 100 CSPLib and PyCSP3 combinatorial problems whose returned solution passes the reference verifier (feasibility, COP optimality) within 256 s; native algorithmic search: the LLM writes a Python solver, heuristic prompt that also asks the LLM to optimize search efficiency; final artefact after smoke and scaling-check refinement; greedy decoding). Source: arxiv.org. Saturation forecast: Around February 2027. 3 models tracked.

Top models

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=cp-sync-xl-python-heuristic-prompt · How It Works · Data refreshed daily, snapshot 2026-11-12.