ComBench (Compilation Error Repair, OpenHands Agent) - Compile Success: leaderboard
Metric: Compile success (%) on the 200 reproducible real CI compilation errors from Bitcoin, OpenSSL, RocksDB and LLVM: the patched project builds in the reproduced CI job without the original or any new compilation error, agentic repair in the OpenHands framework (version 0.55.0, at most 100 iterations): the model starts from the raw compiler error log only and must localize and fix the error by inspecting files and running builds (one retry when no legal diff is produced), temperature 0.2; higher is better. Source: arxiv.org. Saturation forecast: Around January 2027. 2 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 4 | 65.5 | #194 |
| 2 | Qwen 3 32B | 37 | #424 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #8 Claude Fable 5.1.
Interactive version: theaggregate.ai/benchmark?slug=combench-compilation-error-repair-openhands-agent-compile-success · How It Works · Data refreshed daily, snapshot 2026-10-11.