CanItEdit (Descriptive): leaderboard
Instructional code editing (CanItEdit, Cassano et al., COLM 2024): 105 hand-written Python problems, each a before program and a descriptive instruction that spells out the required corrective, perfective or adaptive change, checked by a hidden test suite. pass@1 in percent estimated from 20 samples at temperature 0.2; higher is better.
Metric: pass@1 (%; share of the 105 hand-written Python editing problems whose edited code passes the hidden test suite, estimated from 20 samples per problem at temperature 0.2, top-p 0.95; descriptive instructions that spell out the change). Source: arxiv.org. Saturation forecast: Estimated already saturated. 25 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-4 (0613) | 63.33 | #1214 |
| 2 | GPT-3.5 Turbo (0125) | 48.14 | #3519 |
| 3 | CodeLlama-70B-Instruct-hf | 45.05 | #6548 |
| 4 | starcoder2-15B | 41.95 | #8536 |
| 5 | deepseek-coder-6.7B-instruct | 41.03 | #7937 |
| 6 | starcoder | 37.1 | #9634 |
| 7 | CodeLlama-7B-Instruct-hf | 32.83 | #9444 |
| 8 | deepseek-coder-6.7B-base | 32.62 | #9230 |
| 9 | CodeLlama-34B-Instruct-hf | 30.63 | #7428 |
| 10 | Mixtral 8x7B Instruct | 30.1 | #4484 |
| 11 | CodeLlama-13B-Instruct-hf | 26.9 | #9291 |
| 12 | deepseek-coder-1.3B-instruct | 26.22 | #9590 |
| 13 | starcoder2-7B | 25.1 | #9557 |
| 14 | starcoder2-3B | 15.95 | #9854 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=canitedit-descriptive · How It Works · Data refreshed daily, snapshot 2026-11-12.