MarketBench - Self-Assessment Calibration: leaderboard

Self-assessment on MarketBench (2026): before attempting each of 93 SWE-bench Lite tasks the agent states the probability that it will solve the task, scored against its realized result. Brier skill score relative to a constant base-rate forecast, at most 1 and negative when worse than that baseline; higher is better.

Metric: Brier skill score (at most 1; 0 matches a constant base-rate forecast, negative is worse) of the model's stated probability that it will solve each of 93 SWE-bench Lite tasks in one attempt, scored against its realized pass or fail; higher is better. Source: arxiv.org. Saturation forecast: Around May 2027. 6 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude Opus 4.50.06#165
2Claude Sonnet 4.50.018#267
3Gemini 3 Pro (Preview)-0.111#123
4GPT-5.2 Pro-0.111#139
5GPT-5.2-0.195#190
6GPT-5 Mini-0.305#373

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=marketbench-self-assessment-calibration · How It Works · Data refreshed daily, snapshot 2026-11-22.