Delulu - Exact Match: leaderboard

Exact match (%) of the fill-in-the-middle completion of a code model with the gold completion on Delulu (2026), an execution-verified multilingual fill-in-the-middle benchmark whose 1,951 samples pair a real completion with a hallucinated one; greedy decoding. Higher is better.

Metric: Exact match (%) of the completion with the gold completion, byte for byte, on all 1,951 Delulu fill-in-the-middle samples (seven languages, four injected hallucination types), greedy decoding, at most 256 tokens; base FIM models are truncated to the gold completion's line count; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 11 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Qwen 2.5 Coder 32B Instruct54.1#1547
2CodeLlama-13B-hf47.3
3DeepSeek Coder V2 Lite Instruct46.9#3814
4CodeLlama-7B-hf46.8#9222
5Qwen 2.5 Coder 14B Instruct46.2#1832
6Qwen 2.5 Coder 3B Instruct44#3737
7Qwen 2.5 Coder 7B Instruct40.2#3914
8Qwen2.5-Coder-1.5B-Instruct37.8#8221
9Qwen2.5-Coder-0.5B-Instruct22.8#8832
10starcoder2-15B21.5#8431
11starcoder2-7B9.7#9403

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=delulu-exact-match · How It Works · Data refreshed daily, snapshot 2026-11-21.