CanItEdit (Lazy): leaderboard
Instructional code editing (CanItEdit, Cassano et al., COLM 2024) with the lazy instruction, a terse user-style request, for each of the 105 hand-written Python problems, checked by a hidden test suite. pass@1 in percent estimated from 20 samples at temperature 0.2; higher is better.
Metric: pass@1 (%; share of the 105 hand-written Python editing problems whose edited code passes the hidden test suite, estimated from 20 samples per problem at temperature 0.2, top-p 0.95; lazy instructions, a terse user-style request). Source: arxiv.org. Saturation forecast: Estimated already saturated. 25 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-4 (0613) | 51.95 | #1214 |
| 2 | GPT-3.5 Turbo (0125) | 42.71 | #3519 |
| 3 | CodeLlama-70B-Instruct-hf | 37.52 | #6548 |
| 4 | deepseek-coder-6.7B-instruct | 31.65 | #7937 |
| 5 | starcoder2-15B | 31.48 | #8536 |
| 6 | deepseek-coder-6.7B-base | 27.76 | #9230 |
| 7 | starcoder | 27.62 | #9634 |
| 8 | Mixtral 8x7B Instruct | 24.9 | #4484 |
| 9 | CodeLlama-34B-Instruct-hf | 24.15 | #7428 |
| 10 | CodeLlama-7B-Instruct-hf | 23.49 | #9444 |
| 11 | deepseek-coder-1.3B-instruct | 17.27 | #9590 |
| 12 | CodeLlama-13B-Instruct-hf | 16.89 | #9291 |
| 13 | starcoder2-7B | 13.76 | #9557 |
| 14 | starcoder2-3B | 13.33 | #9854 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=canitedit-lazy · How It Works · Data refreshed daily, snapshot 2026-11-12.