Delulu - CodeBLEU: leaderboard

CodeBLEU (0-100) of the fill-in-the-middle completion of a code model against the gold completion on Delulu (2026), an execution-verified multilingual fill-in-the-middle benchmark whose 1,951 samples pair a real completion with a hallucinated one; greedy decoding. Higher is better.

Metric: CodeBLEU (0-100): n-gram, syntax-tree and dataflow agreement with the gold completion, on all 1,951 Delulu fill-in-the-middle samples (seven languages, four injected hallucination types), greedy decoding, at most 256 tokens; base FIM models are truncated to the gold completion's line count; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 11 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Qwen 2.5 Coder 32B Instruct53.1#1547
2Qwen 2.5 Coder 14B Instruct51.5#1832
3DeepSeek Coder V2 Lite Instruct48.9#3814
4CodeLlama-13B-hf47.8
5Qwen 2.5 Coder 7B Instruct47.2#3914
6Qwen 2.5 Coder 3B Instruct47.2#3737
7starcoder2-15B46.6#8431
8CodeLlama-7B-hf46.3#9222
9starcoder2-7B44.5#9403
10Qwen2.5-Coder-1.5B-Instruct42.3#8221
11Qwen2.5-Coder-0.5B-Instruct36#8832

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=delulu-codebleu · How It Works · Data refreshed daily, snapshot 2026-11-21.