XYBench Stack Overflow - Underlying Goal Solution: leaderboard
XYBench, Ai2 release of November 13, 2026. These results use the 1,557 Stack Overflow questions, not all 8,115 questions across the four source sets. A user asks about a mistaken approach (Y) to an underlying goal (X). Five named models answer once per question in the single-turn experiment. Claude Sonnet 4.5 is the automated judge; the publisher validated its criteria with two human reviewers on 500 responses. Each metric remains distinct. The top human answer is a separate source comparator, not Median Human or Median Expert; random-pick control and the repeated emphasis table are not additional model observations. No generation settings or model checkpoints beyond the named source identities are guessed. Percentage of responses that include a solution to the underlying goal, even if the rest of the answer focuses on the mistaken approach. Higher is better.
Metric: Rate (%). Source: allenai.org. Saturation forecast: Around February 2028. 5 models tracked. Updated 14 Nov 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5 | 71 | #167 |
| 2 | DeepSeek V4 Pro | 66 | #191 |
| 3 | Qwen 3.5 9B | 64 | #814 |
| 4 | Gemini 3.1 Pro (Preview) | 62 | #93 |
| 5 | Claude Sonnet 4.6 | 59 | #165 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=xybench-stack-overflow-underlying-goal-solution · How It Works · Data refreshed daily, snapshot 2026-11-16.