Probability Operator Inference: leaderboard
A benchmark (Haverford College, 2026) of logical inference over probability operators: 14,320 generated yes or no questions ask whether a conclusion with a gradable epistemic modal such as must, probably or might follows from its premises, across fifteen valid and invalid templates. Overall accuracy in percent, zero-shot with uncertain answers counted wrong; higher is better.
Metric: Overall accuracy (%; micro-average over all responses on 14,320 procedurally generated English yes or no prompts over fifteen inference templates with gradable epistemic modals (must, probably, might), varying question form, negation and surface content; zero-shot, temperature 0, first answer token parsed, uncertain responses counted as incorrect; higher is better). Source: arxiv.org. Saturation forecast: Around February 2027. 29 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.7 | 75.6 | #79 |
| 2 | Claude Sonnet 4.6 | 75.1 | #161 |
| 3 | GPT-5.4 Mini | 65.6 | #389 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=probability-operator-inference · How It Works · Data refreshed daily, snapshot 2026-11-13.