Delulu - Exact Match: leaderboard
Exact match (%) of the fill-in-the-middle completion of a code model with the gold completion on Delulu (2026), an execution-verified multilingual fill-in-the-middle benchmark whose 1,951 samples pair a real completion with a hallucinated one; greedy decoding. Higher is better.
Metric: Exact match (%) of the completion with the gold completion, byte for byte, on all 1,951 Delulu fill-in-the-middle samples (seven languages, four injected hallucination types), greedy decoding, at most 256 tokens; base FIM models are truncated to the gold completion's line count; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 11 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 2.5 Coder 32B Instruct | 54.1 | #1547 |
| 2 | CodeLlama-13B-hf | 47.3 | |
| 3 | DeepSeek Coder V2 Lite Instruct | 46.9 | #3814 |
| 4 | CodeLlama-7B-hf | 46.8 | #9222 |
| 5 | Qwen 2.5 Coder 14B Instruct | 46.2 | #1832 |
| 6 | Qwen 2.5 Coder 3B Instruct | 44 | #3737 |
| 7 | Qwen 2.5 Coder 7B Instruct | 40.2 | #3914 |
| 8 | Qwen2.5-Coder-1.5B-Instruct | 37.8 | #8221 |
| 9 | Qwen2.5-Coder-0.5B-Instruct | 22.8 | #8832 |
| 10 | starcoder2-15B | 21.5 | #8431 |
| 11 | starcoder2-7B | 9.7 | #9403 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=delulu-exact-match · How It Works · Data refreshed daily, snapshot 2026-11-21.