Probability Operator Inference - Competence Floor: leaderboard

The competence floor on the probability-operator inference benchmark (Haverford College, 2026): the lower of the accuracy on questions whose correct answer is yes and on those whose correct answer is no, in percent. A constant answerer scores 0 and a coin flip 50; higher is better.

Metric: Competence floor (%; the lower of the accuracy on items whose correct answer is Yes and on items whose correct answer is No, each the mean of its valid and invalid template cells, so a constant responder scores 0 and a coin flip 50, on 14,320 procedurally generated English yes or no prompts over fifteen inference templates with gradable epistemic modals (must, probably, might), varying question form, negation and surface content; zero-shot, temperature 0, first answer token parsed, uncertain responses counted as incorrect; higher is better). Source: arxiv.org. Saturation forecast: Around February 2027. 29 models tracked.

Top models

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=probability-operator-inference-competence-floor · How It Works · Data refreshed daily, snapshot 2026-11-13.