Our Benchmarks — evaluations The Aggregate runs itself

Everything else on this site is collected from public leaderboards. These are the evaluations The Aggregate runs itself: question banks it built, and Guesswork, the live contest for predicting benchmark scores. Apart from Guesswork, these results are shown on this page only and do not feed the rating.

Long-tail book recall

Closed-book questions about details in 60 public-domain books, from the famous to the obscure. Run Sep 2–9, 2026, 282 items.

Each question asks for one specific detail from a Project Gutenberg book, such as a minor character's name or the place where a scene happens, and the answer key is a verbatim span of the book. The 60 books sit on five rungs by how often their Wikipedia article is read, from the canon down to books almost nobody opens, so the bank keeps separating models long after the famous titles are solved.

Answers are matched against the key; an answer that misses the exact wording goes to a second-pass judge that accepts equivalent answers. Saying “unknown” counts as wrong. Reasoning effort was set to high.

Exotic fact recall

Closed-book lookups from five public registries: proteins, ISBNs, museum objects, asteroids and network ports. Run Sep 8–9, 2026, 85 items.

Each question asks for one exact field from a public registry: a UniProt protein entry, the title behind an ISBN, the accession number of a painting at the Met, the number of a named asteroid, or the TCP port that IANA assigns to a service. Every key is exact, so no judge is involved.

Each registry contributes well-known entries and a random sample from its long tail. The score counts the 85 questions that every model in the original panel answered, 49 of them from the tail. Reasoning effort was set to high.

Asteroid numbers

Name 48 freshly sampled asteroids' catalogue numbers, from memory. Run Sep 9, 2026, 48 items.

A follow-up to the exotic recall bank, where DeepSeek and Gemini beat the Claude models on asteroids. 48 named asteroids were drawn at random from NASA/JPL's catalogue (numbers 1,000 to 20,000, first observed before 2000), and each model gave the permanent number for each name, closed book, graded by exact match. The reversal held on the fresh sample.

A reply cut off by the output limit counts as a miss; that happened at most once per model. Reasoning effort was set to high.

Day-one recall check

The 34-question recall test every new free model gets on the day it appears. Run Aug 31 – Oct 3, 2026, 34 items.

A short version of the book recall bank: 34 questions about 12 public-domain books in three tiers, famous, mid and obscure. It is the first step of the battery we run on every new free model the day it appears, so it is the one recall board where free models sit next to the flagships.

Graded by matching the key, with no second-pass judge. Saying “unknown” counts as wrong; an empty reply does not count at all. Reasoning effort was set to high where the route accepts it.

Forecasting resolved questions

Probabilities for 76 prediction-market questions that no model in the panel could look up from memory. Run Sep 1, 2026, 76 items.

Each question is a yes-or-no market on Manifold that resolved between May and August 2026. Before forecasting, every model is asked whether it already knows the outcome, and any question that any panel model answers correctly is dropped for everyone. That leaves 76 questions the models have to forecast rather than remember.

The score is the Brier score: the mean squared gap between the forecast probability and what happened, lower is better. Always answering 50% scores 0.250, and always answering the share of questions that resolved yes scores 0.249. No model beat that base rate: they were directionally right a little more often than chance and far too confident.

Inversion and execution ladder

24 generated puzzles: trace a short program, or undo a hidden digit transform, at rising difficulty. Run Aug 28 – Oct 3, 2026, 24 items.

Half the items give the model a short generated program and ask for its output; the other half give the result of a hidden digit transform and ask for the input that produced it. Six rungs of difficulty and exact-match answers, on items generated for this ladder from a fixed seed.

Run at each provider's default reasoning effort, which this ladder is very sensitive to. The scores show what a model does with its out-of-the-box settings, not its ceiling.

Puzzle portfolio ladder

30 generated puzzles in five families: logic grids, knapsacks, document facts, specifications and bug hunts. Run Aug 29 – Oct 3, 2026, 30 items.

Six items from each of five generators whose answers are fixed by construction: zebra logic grids, knapsack optimisation, fact lookup in a long document, checking an implementation against its specification, and finding a planted bug. Difficulty rises within each family, and the bank is frozen.

Run at each provider's default reasoning effort, like the ladder above.

Also run here

• MEGA-Bench Hanoi re-run. Gemini 3.8 Flash on MEGA-Bench's 14-item Tower of Hanoi planning task, scored 58.2% with MEGA-Bench's own prompts and scorer (September 20). The row sits on the public board, and the check found five of its fourteen reference answers impossible.
• Predicting a small model from its name. Gemini 3.8 Flash predicted, question by question, where Llama 3.2 1B would fail on 498 MMLU-Pro and GSM8K items, knowing only its name: AUROC 0.71, but it took the 33% model for a 66% one (September 21).
• Verbatim recital. Continuing a Project Gutenberg paragraph or an Apollo 11 transcript window word for word from a 12-word cue: every model tested scored 0 of 30 paragraphs and 0 of 24 transcript windows at 100 words (September 9).
• Classifier qualification suite. The tests a model must pass before it becomes the benchmark classifier behind Task Explorer.

Interactive version: theaggregate.ai/our-benchmarks · How It Works · Data refreshed daily, snapshot 2026-10-09.