Source-Test Repair - Codex Challenging 27 Cases: leaderboard
Codex0.142.0-alpha.1 with GPT-5.4 high effort on27 challenging Java repair cases from6projects, selected among155 failed GPT5.4low RQ1 cases and commits modifying at least100Java sourcefiles. Separate original/modified workspaces, groundtruth testpatches and Git history removed, onlytargettest editable; one finalrepair. Baseline versus explicit symptom/cause/supportcontext remain separate runconfigurations. Native successrate describes this selected cohort, not the500case evaluation.
Metric: Successful repairs (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 2 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 (high) (TestRepair Run: harness=Codex0.142.0-alpha.1; context=symptom-cause-support) | 74.1 | |
| 2 | GPT-5.4 (high) (TestRepair Run: harness=Codex0.142.0-alpha.1; context=baseline) | 44.4 |
Interactive version: theaggregate.ai/benchmark?slug=source-test-repair-codex-challenging-27-cases · How It Works · Data refreshed daily, snapshot 2026-11-05.