LintEval - Pass@1: leaderboard
LintEval (Deng et al., EMNLP 2026 Findings; arXiv 2610.20027): refactor a C++, Python or Java competitive-programming file so that the linter alerts listed in the prompt (clang-tidy, Pylint, PMD) go away while the program keeps working. 498 LintEval-Lite programs, one turn, zero-shot, greedy decoding. Score: Pass@1, the share (%) of refactored files that compile and pass every generated test. An unchanged copy would score 100%, so read it with LintEval - Reduction@P. Higher is better. Source: https://arxiv.org/abs/2610.20027
Metric: Pass@1 (%; share of the 498 LintEval-Lite C++/Python/Java programs whose single-turn, zero-shot, greedy refactor compiles and passes every generated test). Source: arxiv.org. Saturation forecast: Around June 2027. 14 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 Mini | 68.3 | #208 |
| 2 | GPT-5.2 | 67.1 | #109 |
| 3 | Qwen 3.6 35B A3B | 65.3 | #224 |
| 4 | Qwen 3.6 27B | 64.3 | #168 |
| 5 | Qwen 2.5 Coder 32B Instruct | 61.7 | #524 |
| 6 | DeepSeek V3.2 | 57.6 | #201 |
| 7 | Qwen 2.5 Coder 7B Instruct | 53 | #939 |
| 8 | Qwen 2.5 Coder 3B Instruct | 52 | #1021 |
| 9 | Qwen 3 4B | 45.8 | #829 |
| 10 | Llama 3.2 3B Instruct | 38.8 | #1347 |
| 11 | Qwen 3.5 9B | 37.5 | #377 |
| 12 | Llama 3.2 1B Instruct | 33.7 | #1534 |
| 13 | Qwen 3 8B | 33.3 | #670 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=linteval-pass-1 · How It Works · Data refreshed daily, snapshot 2026-10-19.