CI-Repair-Bench (BM25 Log Retrieval): leaderboard

Continuous-integration repair on CI-Repair-Bench (2026), 567 real GitHub Actions failures from 103 Python repositories, repaired by a fixed pipeline with only the LLM varied and validated by re-running the original CI workflow, with BM25 retrieval of log segments instead of agent-based log analysis. Pass@1 in percent; higher is better.

Metric: Pass@1 (%): share of the 567 real GitHub Actions CI failure instances from 103 Python repositories, repaired by the paper's fixed reference pipeline (CI log analysis, fault localization, patch generation) with only the LLM varied, and validated by full CI re-execution under the original workflow fixed by the first repair attempt when the log analysis stage uses BM25 retrieval of log segments instead of agent-based iterative analysis; higher is better. Source: arxiv.org. Saturation forecast: Around August 2028. 4 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-5 Mini (2025-08-07)10.6#345
2CI-Repair-Bench DeepSeek-Coder (checkpoint unspecified)9.9
3CI-Repair-Bench DeepSeek-Chat V3 (checkpoint unspecified)9.5
4GPT-4o Mini (2024-07-18)1.9#1484

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=ci-repair-bench-bm25-log-retrieval · How It Works · Data refreshed daily, snapshot 2026-11-21.