Source-Test Repair - Hunk-level + Failure log Compilable: leaderboard
Source-test co-evolution repair on 500 Java cases from 15 repositories; Java17/Maven3.9.9, temperature0.6. Native fractions retain denominator500. Executable means repaired tests pass, not semantic equivalence to developer intent; compilable measures build validity. Context and pipeline configurations are separate. Hunk-level + Failure log; Compilable.
Metric: Compilable fraction. Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 (low) (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=single-stage) | 0.66 | |
| 2 | GPT-5.4 (low) (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=multi-stage) | 0.64 | |
| 3 | Qwen3-Coder-480B-Instruct (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=single-stage) | 0.62 | |
| 4 | GPT-5.4 (low) (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=baseline) | 0.61 | |
| 5 | DeepSeek-V3.2 (Non-reasoning) (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=multi-stage) | 0.58 | |
| 6 | Qwen3-Coder-480B-Instruct (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=baseline) | 0.57 | |
| 7 | Qwen3-Coder-480B-Instruct (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=multi-stage) | 0.57 | |
| 8 | DeepSeek-V3.2 (Non-reasoning) (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=single-stage) | 0.54 | |
| 9 | DeepSeek-V3.2 (Reasoning) (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=baseline%20high%20thinking) | 0.53 | |
| 10 | DeepSeek-V3.2 (Non-reasoning) (TestRepair Run: context=Hunk-level%20%2B%20Failure%20log; pipeline=baseline) | 0.52 |
Interactive version: theaggregate.ai/benchmark?slug=source-test-repair-hunk-level-plus-failure-log-compilable · How It Works · Data refreshed daily, snapshot 2026-11-05.