CodeSpecBench-Repo - Correctness: leaderboard

CodeSpecBench (2026) asks models to write executable precondition and postcondition functions that specify the intended behaviour of a program; the repository-level set uses the 500 SWE-bench Verified issues, injecting the specifications around the issue-relevant functions. Correctness, the share of specifications that accept every valid test, in percent; higher is better.

Metric: Correctness (%), the share of generated executable pre/postcondition specifications that accept every valid test, on CodeSpecBench-Repo (500 SWE-bench Verified issues; specifications injected around the issue-relevant functions, trigger tests on the fixed and buggy versions with UTBoost-augmented tests); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Sonnet 4.5Thinking37.4#267
2Gemini 2.5 Pro30.8#280
3DeepSeek V3.2Non-reasoning20.2#425
4GPT-5 Mini19.6#373
5GPT-518.8#185
6QwQ-32B10.6#913
7GPT-OSS-20B9.4#1188
8Gemini 2.5 Flash9.2#466
9Qwen 3 32B8.8#983
10GPT-OSS-120B8.6#712
11Qwen 3 14B6#1284
12Qwen 3 8B2#1890
13Qwen 3 4B0.6#2379
14Qwen 3 1.7BNon-reasoning0.2#5913
15Qwen 3 0.6B0#8357

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=codespecbench-repo-correctness · How It Works · Data refreshed daily, snapshot 2026-11-22.