Delulu - Edit Similarity: leaderboard

Character-level edit similarity (0-100) of the fill-in-the-middle completion of a code model to the gold completion on Delulu (2026), an execution-verified multilingual fill-in-the-middle benchmark whose 1,951 samples pair a real completion with a hallucinated one; greedy decoding. Higher is better.

Metric: Edit similarity (0-100): character-level normalized Levenshtein similarity to the gold completion, on all 1,951 Delulu fill-in-the-middle samples (seven languages, four injected hallucination types), greedy decoding, at most 256 tokens; base FIM models are truncated to the gold completion's line count; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 11 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Qwen 2.5 Coder 32B Instruct76.5#1547
2DeepSeek Coder V2 Lite Instruct71.7#3814
3Qwen 2.5 Coder 14B Instruct71.6#1832
4CodeLlama-13B-hf70.1
5CodeLlama-7B-hf69.5#9222
6Qwen 2.5 Coder 3B Instruct69.3#3737
7Qwen2.5-Coder-1.5B-Instruct63.2#8221
8Qwen 2.5 Coder 7B Instruct62.5#3914
9Qwen2.5-Coder-0.5B-Instruct62.5#8832
10starcoder2-15B60#8431
11starcoder2-7B59.6#9403

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=delulu-edit-similarity · How It Works · Data refreshed daily, snapshot 2026-11-21.