PBT-Bench (Baseline Prompt): leaderboard

PBT-Bench (2026) gives an OpenHands coding agent 100 Python library problems with injected semantic bugs and asks it to write tests that expose them; this board uses the open-ended baseline prompt without property-based testing scaffolding. Bug recall in percent, mean of three runs; higher is better.

Metric: Bug recall (%) on the 100 problems under the open-ended baseline prompt (write tests that expose the injected bugs, no property-based testing scaffolding), OpenHands agent scaffold, mean of three runs, each problem weighted equally; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 8 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude Sonnet 4.676.7#173
2GLM-5.166.6#198
3DeepSeek V3.264.2#419
4Gemini 3 Flash56.2#175
5Qwen 3.6 Plus53.5#238
6Grok 4.1 Fast50.1#427
7Step 3.5 Flash37.8#526
8PBT-Bench Qwen 3.5-30B-A3B (checkpoint unspecified)31.4

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=pbt-bench-baseline-prompt · How It Works · Data refreshed daily, snapshot 2026-11-21.