TRACK - Code Generation (MeLLo): leaderboard
Knowledge-conflict reasoning in TRACK (Testing Reasoning Amid Conflicting Knowledge; Feng et al., EACL 2026): 500 BigCodeBench-derived coding problems that require specific library API versions, where the needed facts contradict or extend what the model learned in pre-training, answered through MeLLo, which keeps the facts in an external memory and retrieves them by question decomposition. Holistic pass in percent (a correct final answer and reasoning that entails every required fact), higher is better.
Metric: Holistic pass (%; BigCodeBench-derived coding problems that require specific library API versions, 500 items; the final answer is correct and the reasoning entails every required fact; answered through MeLLo, which keeps the facts in an external memory and retrieves them by question decomposition). Source: arxiv.org. Saturation forecast: Estimated already saturated. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 8B | 24.8 | #1838 |
| 2 | Qwen 3 4B | 21.4 | #2297 |
| 3 | Qwen 3 1.7B | 12.5 | #5551 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=track-code-generation-mello · How It Works · Data refreshed daily, snapshot 2026-11-12.