C3-Bench (Controllable Code Completion) - Scale Control: leaderboard
C3-Bench's Scale-Control task (Alibaba Group with USTC, UCAS and CASIA, 2026): given the Python code around a missing segment, a model must complete exactly the requested scope, such as the rest of a partial line, a set number of lines or one if, for or while block, over 909 tasks built from HumanEval and SAFIM. Completions are checked automatically by AST node type and line count, with no unit tests; the score is the percentage that match the requested scope, higher is better. Base code models are prompted in fill-in-the-middle token format and chat models in a ChatML template, and the authors' Qwen2.5-Coder-C3 models are fine-tuned on synthetic data in this task format.
Metric: Instruction-Following Rate (%; 909 SCC tasks: AST node-type and line-count check of the requested completion scope). Source: arxiv.org. Saturation forecast: Estimated already saturated. 42 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | O1 (2024-12-17) | 59.6 | #310 |
| 2 | Claude 3.5 Sonnet (20241022) | 50.8 | #635 |
| 3 | O1 Preview | 48.9 | #483 |
| 4 | O1 Mini | 44.7 | #761 |
| 5 | Claude 3.5 Haiku | 26.4 | #1563 |
| 6 | GPT-4o (2024-08-06) | 24.1 | #731 |
| 7 | GPT-4o (2024-11-20) | 24.1 | #864 |
| 8 | GPT-4o Mini (2024-07-18) | 22.5 | #1512 |
| 9 | GPT-4o (2024-05-13) | 20.8 | #734 |
| 10 | DeepSeek V3 | 20.2 | #705 |
| 11 | DeepSeek V3 (0324) | 20.2 | #704 |
| 12 | Qwen 2.5 Coder 32B Instruct | 16.9 | #1553 |
| 13 | Qwen 2.5 Coder 3B Instruct | 14.2 | #3610 |
| 14 | Qwen 2.5 Coder 14B Instruct | 13.5 | #1843 |
| 15 | Qwen 2.5 72B Instruct | 9.4 | #1093 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=c3-bench-controllable-code-completion-scale-control · How It Works · Data refreshed daily, snapshot 2026-11-12.