XYBench Stack Overflow - Underlying Goal Emphasis: leaderboard

XYBench, Ai2 release of November 13, 2026. These results use the 1,557 Stack Overflow questions, not all 8,115 questions across the four source sets. A user asks about a mistaken approach (Y) to an underlying goal (X). Five named models answer once per question in the single-turn experiment. Claude Sonnet 4.5 is the automated judge; the publisher validated its criteria with two human reviewers on 500 responses. Each metric remains distinct. The top human answer is a separate source comparator, not Median Human or Median Expert; random-pick control and the repeated emphasis table are not additional model observations. No generation settings or model checkpoints beyond the named source identities are guessed. Percentage of written responses focused primarily on the underlying goal, rather than the stated approach or an equal balance. Higher is better.

Metric: Rate (%). Source: allenai.org. Saturation forecast: Not forecast. 5 models tracked. Updated 14 Nov 2026.

Top models

#ModelScoreOverall rank
1GPT-541#167
2Qwen 3.5 9B41#814
3Gemini 3.1 Pro (Preview)40#93
4Claude Sonnet 4.636#165
5DeepSeek V4 Pro35#191

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=xybench-stack-overflow-underlying-goal-emphasis · How It Works · Data refreshed daily, snapshot 2026-11-16.