MarketBench - Self-Assessment Calibration: leaderboard
Self-assessment on MarketBench (2026): before attempting each of 93 SWE-bench Lite tasks the agent states the probability that it will solve the task, scored against its realized result. Brier skill score relative to a constant base-rate forecast, at most 1 and negative when worse than that baseline; higher is better.
Metric: Brier skill score (at most 1; 0 matches a constant base-rate forecast, negative is worse) of the model's stated probability that it will solve each of 93 SWE-bench Lite tasks in one attempt, scored against its realized pass or fail; higher is better. Source: arxiv.org. Saturation forecast: Around May 2027. 6 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.5 | 0.06 | #165 |
| 2 | Claude Sonnet 4.5 | 0.018 | #267 |
| 3 | Gemini 3 Pro (Preview) | -0.111 | #123 |
| 4 | GPT-5.2 Pro | -0.111 | #139 |
| 5 | GPT-5.2 | -0.195 | #190 |
| 6 | GPT-5 Mini | -0.305 | #373 |
No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=marketbench-self-assessment-calibration · How It Works · Data refreshed daily, snapshot 2026-11-22.