Delulu - CodeBLEU: leaderboard
CodeBLEU (0-100) of the fill-in-the-middle completion of a code model against the gold completion on Delulu (2026), an execution-verified multilingual fill-in-the-middle benchmark whose 1,951 samples pair a real completion with a hallucinated one; greedy decoding. Higher is better.
Metric: CodeBLEU (0-100): n-gram, syntax-tree and dataflow agreement with the gold completion, on all 1,951 Delulu fill-in-the-middle samples (seven languages, four injected hallucination types), greedy decoding, at most 256 tokens; base FIM models are truncated to the gold completion's line count; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 11 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 2.5 Coder 32B Instruct | 53.1 | #1547 |
| 2 | Qwen 2.5 Coder 14B Instruct | 51.5 | #1832 |
| 3 | DeepSeek Coder V2 Lite Instruct | 48.9 | #3814 |
| 4 | CodeLlama-13B-hf | 47.8 | |
| 5 | Qwen 2.5 Coder 7B Instruct | 47.2 | #3914 |
| 6 | Qwen 2.5 Coder 3B Instruct | 47.2 | #3737 |
| 7 | starcoder2-15B | 46.6 | #8431 |
| 8 | CodeLlama-7B-hf | 46.3 | #9222 |
| 9 | starcoder2-7B | 44.5 | #9403 |
| 10 | Qwen2.5-Coder-1.5B-Instruct | 42.3 | #8221 |
| 11 | Qwen2.5-Coder-0.5B-Instruct | 36 | #8832 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=delulu-codebleu · How It Works · Data refreshed daily, snapshot 2026-11-21.