PostcondBench - Completeness: leaderboard
Comp@1 in percent: the expected share of methods whose generated postconditions pass every test and also reject every mutant of the method, for 420 documented methods from 121 Python and Java repositories (PostcondBench, 2026), one sampled postcondition set per method, averaged over code-only, comment-only and code-plus-comment inputs. Higher is better.
Metric: Comp@1 times 100 (%): expected share of methods whose generated postcondition set is test-correct and kills every mutant, one sampled postcondition set per method (k = 1, five samples drawn), averaged over the Python and Java splits (210 repository methods each) and three input settings (code only, comments only, code plus comments); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 5 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | GPT-5 | 25.5 | #178 | |
| 2 | Claude Sonnet 4.5 | Thinking | 20.7 | #261 |
| 3 | Llama 4 Maverick | 9.4 | #1110 | |
| 4 | Qwen 3 32B | 5.5 | #972 | |
| 5 | Gemma 3 27B | 4.4 | #1663 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=postcondbench-completeness · How It Works · Data refreshed daily, snapshot 2026-11-21.