ObligBench - Control - Zero-Shot Execution: leaderboard
ObligBench recorded control cohort; all 700 Java tasks, zero-shot. Success is the source's generated-test execution_passed outcome after compilation; generation failures, compile failures, test failures, timeouts and Panta placeholder outputs remain in the denominator. This measures successful execution of generated tests, not their fault-detection adequacy. BVA/CFA/oracle-strength diagnostics use different conditional eligibility and are not pooled into this rate. Exact API/FP8 labels, source-selected run counts and generation temperatures are retained. GPT-5.4 planning/feedback mix recorded temperatures 0/0.2; Panta temperature is unreported. No inferred reasoning-effort setting or cross-strategy best-score selection is applied. Metric: Generated test execution pass rate (%).
Metric: Generated test execution pass rate (%). Source: arxiv.org. Saturation forecast: Around January 2027. 2 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 | 93.43 | #82 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=obligbench-control-zero-shot-execution · How It Works · Data refreshed daily, snapshot 2026-10-27.