TRACK - Code Generation (Facts in Context, Thinking): leaderboard
Knowledge-conflict reasoning in TRACK (Testing Reasoning Amid Conflicting Knowledge; Feng et al., EACL 2026): 500 BigCodeBench-derived coding problems that require specific library API versions, where the needed facts contradict or extend what the model learned in pre-training, answered with the missing facts appended to the prompt and longer thinking enabled. Holistic pass in percent (a correct final answer and reasoning that entails every required fact), higher is better.
Metric: Holistic pass (%; BigCodeBench-derived coding problems that require specific library API versions, 500 items; the final answer is correct and the reasoning entails every required fact; answered with the missing facts appended to the prompt and longer thinking enabled). Source: arxiv.org. Saturation forecast: Estimated already saturated. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | O4 Mini (2025-04-16) | 40.6 | #362 |
| 2 | Qwen 3 8B (Thinking) | 26.5 | #1838 (Qwen 3 8B) |
| 3 | Qwen 3 4B (Reasoning) | 22.8 | #2297 (Qwen 3 4B) |
| 4 | Qwen 3 1.7B (Thinking) | 15.5 | #5551 (Qwen 3 1.7B) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=track-code-generation-facts-in-context-thinking · How It Works · Data refreshed daily, snapshot 2026-11-12.