CodeSpecBench-Repo: leaderboard

CodeSpecBench (2026) asks models to write executable precondition and postcondition functions that specify the intended behaviour of a program; the repository-level set uses the 500 SWE-bench Verified issues, injecting the specifications around the issue-relevant functions. Pass rate, the share of specifications that accept every valid test and reject every invalid one, in percent; higher is better.

Metric: Pass rate (%), the share of generated executable pre/postcondition specifications that are both correct (accept every valid test) and complete (reject every invalid test), on CodeSpecBench-Repo (500 SWE-bench Verified issues; specifications injected around the issue-relevant functions, trigger tests on the fixed and buggy versions with UTBoost-augmented tests); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Sonnet 4.5Thinking20.2#261
2Gemini 2.5 Pro18.2#276
3GPT-5 Mini9.6#364
4GPT-58.6#178
5DeepSeek V3.2Non-reasoning6.8#419
6QwQ-32B3.6#907
7GPT-OSS-20B3.4#1183
8GPT-OSS-120B3.2#704
9Gemini 2.5 Flash2.8#463
10Qwen 3 32B2#972
11Qwen 3 14B1.2#1269
12Qwen 3 8BNon-reasoning0.8#1873
13Qwen 3 4B0.2#2348
14Qwen 3 1.7B0#5860
15Qwen 3 0.6B0#8350

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=codespecbench-repo · How It Works · Data refreshed daily, snapshot 2026-11-21.