CodeSpecBench-Repo - Completeness: leaderboard

CodeSpecBench (2026) asks models to write executable precondition and postcondition functions that specify the intended behaviour of a program; the repository-level set uses the 500 SWE-bench Verified issues, injecting the specifications around the issue-relevant functions. Completeness, the share of specifications that reject every invalid test, in percent; higher is better.

Metric: Completeness (%), the share of generated executable pre/postcondition specifications that reject every invalid test, on CodeSpecBench-Repo (500 SWE-bench Verified issues; specifications injected around the issue-relevant functions, trigger tests on the fixed and buggy versions with UTBoost-augmented tests); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1GPT-558.6#178
2Claude Sonnet 4.5Thinking57.2#261
3Gemini 2.5 Pro53#276
4GPT-5 Mini45.8#364
5GPT-OSS-120B36.2#704
6GPT-OSS-20B31.6#1183
7DeepSeek V3.2Non-reasoning30.2#419
8QwQ-32B25#907
9Qwen 3 14B17.8#1269
10Gemini 2.5 Flash17.6#463
11Qwen 3 32B15.6#972
12Qwen 3 8BNon-reasoning10#1873
13Qwen 3 4B1.2#2348
14Qwen 3 1.7BNon-reasoning0.2#5860
15Qwen 3 0.6B0#8350

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=codespecbench-repo-completeness · How It Works · Data refreshed daily, snapshot 2026-11-21.