ObligBench - Treatment - Zero-Shot Execution: leaderboard
ObligBench recorded treatment cohort; all 895 Java tasks, zero-shot. Success is the source's generated-test execution_passed outcome after compilation; generation failures, compile failures, test failures, timeouts and Panta placeholder outputs remain in the denominator. This measures successful execution of generated tests, not their fault-detection adequacy. BVA/CFA/oracle-strength diagnostics use different conditional eligibility and are not pooled into this rate. Exact API/FP8 labels, source-selected run counts and generation temperatures are retained. GPT-5.4 planning/feedback mix recorded temperatures 0/0.2; Panta temperature is unreported. No inferred reasoning-effort setting or cross-strategy best-score selection is applied. Metric: Generated test execution pass rate (%).
Metric: Generated test execution pass rate (%). Source: arxiv.org. Saturation forecast: Around January 2027. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.6 Flash | 88.6 | #57 |
| 2 | GPT-5.4 | 68.04 | #82 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #6 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=obligbench-treatment-zero-shot-execution · How It Works · Data refreshed daily, snapshot 2026-10-27.