ScratchWorld - Debug: leaderboard
Debug tasks from ScratchWorld (Zhang et al., 2026): find and repair the logical errors in a broken Scratch project so that it behaves as specified, checked by unit tests run in the Scratch VM. Composite mode, where the agent reads the program as pseudocode and edits it through semantic commands such as adding or connecting blocks instead of the GUI. The score is the percentage of the 20 tasks on which every unit test passes; higher is better.
Metric: Success Rate (%). Source: huggingface.co. Saturation forecast: Around January 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 2.5 Pro | 75 | #152 |
| 2 | Claude Sonnet 4.5 | 75 | #142 |
| 3 | DeepSeek V3.2 | 75 | #205 |
| 4 | GPT-5 | 65 | #98 |
| 5 | Qwen 3 Coder 30B A3B Instruct | 30 | #415 |
| 6 | Qwen 2.5 Coder 7B Instruct | 5 | #951 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=scratchworld-debug · How It Works · Data refreshed daily, snapshot 2026-10-20.