XYBench Stack Overflow - Underlying Goal Solution: leaderboard

XYBench, Ai2 release of November 13, 2026. These results use the 1,557 Stack Overflow questions, not all 8,115 questions across the four source sets. A user asks about a mistaken approach (Y) to an underlying goal (X). Five named models answer once per question in the single-turn experiment. Claude Sonnet 4.5 is the automated judge; the publisher validated its criteria with two human reviewers on 500 responses. Each metric remains distinct. The top human answer is a separate source comparator, not Median Human or Median Expert; random-pick control and the repeated emphasis table are not additional model observations. No generation settings or model checkpoints beyond the named source identities are guessed. Percentage of responses that include a solution to the underlying goal, even if the rest of the answer focuses on the mistaken approach. Higher is better.

Metric: Rate (%). Source: allenai.org. Saturation forecast: Around February 2028. 5 models tracked. Updated 14 Nov 2026.

Top models

#ModelScoreOverall rank
1GPT-571#167
2DeepSeek V4 Pro66#191
3Qwen 3.5 9B64#814
4Gemini 3.1 Pro (Preview)62#93
5Claude Sonnet 4.659#165

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=xybench-stack-overflow-underlying-goal-solution · How It Works · Data refreshed daily, snapshot 2026-11-16.