CI-Repair-Bench - Fault Localization: leaderboard
Fault localization on CI-Repair-Bench (2026), 567 real GitHub Actions failures from 103 Python repositories, repaired by a fixed pipeline with only the LLM varied and validated by re-running the original CI workflow: the share of failures whose top-ranked file is a ground-truth repair location, with agent-based log analysis. Percent; higher is better.
Metric: Top-1 localization recall (%): share of the 567 real GitHub Actions CI failure instances from 103 Python repositories, repaired by the paper's fixed reference pipeline (CI log analysis, fault localization, patch generation) with only the LLM varied, and validated by full CI re-execution under the original workflow whose top-ranked predicted file is a ground-truth repair-relevant file, with the default agent-based log analysis; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 4 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5 Mini (2025-08-07) | 45.33 | #345 |
| 2 | GPT-4o Mini (2024-07-18) | 42.5 | #1484 |
| 3 | CI-Repair-Bench DeepSeek-Coder (checkpoint unspecified) | 42.5 | |
| 4 | CI-Repair-Bench DeepSeek-Chat V3 (checkpoint unspecified) | 41.62 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=ci-repair-bench-fault-localization · How It Works · Data refreshed daily, snapshot 2026-11-21.