How It Works
Company and country rankings
Companies and countries are ranked by their strongest currently ranked model’s default Aggregate Rating. The rating and standard error belong to that model. Exact ties share a rank. Country attribution follows the catalog’s developer-country metadata; unknown and multi-country entries do not receive a national rank. Deprecated and superseded merged rows are omitted, private and low-coverage labels retained, and entities without a ranked model remain unranked. Developer counts include named developers with unranked models. These are rankings of tracked AI models, not entire national AI industries. The Open weights preset uses the catalog’s access classification, not a license certification.
The daily pipeline collects public leaderboard and benchmark-report results, normalizes model names, and fuses equivalent sources into one score matrix. The default ranking fits observed scores with compute budgets unified using a hierarchical continuous-response IRT model: each model has an ability estimate, and each benchmark has difficulty, discrimination and response-noise estimates. Dependent benchmarks share weight, and source balancing reduces the influence of prolific publishers. Aggregate Rating uses an Elo-like numerical scale centered near 1500, with conditional standard errors. Its points are not calibrated chess win odds.
Reviewed ranking exclusions keep unsuitable benchmarks out of the fit while retaining their catalog results. Eligible private and low-coverage models keep their ranks with preliminary labels. A hybrid model combining ability, skill axes, and provider factors predicts missing scores. Saturation forecasts estimate when benchmarks stop providing useful differences among leading models. For bounded metrics, future dates follow fitted response curves toward about 90% of the scoring range; metrics without a usable ceiling retain a forward threshold scan. Dates are conditional model projections, not validated deadlines. Missing-score prediction accuracy is tracked against later scraped scores; this does not validate saturation dates.
The Aggregate is built and run by Mikhail Doroshenko, an AI researcher and co-author of Humanity’s Last Exam. It is an independent project with no affiliation with any AI lab. Every ranked score comes from a public leaderboard or a published report and links back to where it came from; the evaluations the site runs itself, such as Guesswork, are shown as its own work.
The changelog on What’s New is computed by the pipeline with no language model involved. The story above it is written by Claude from that changelog during the morning audit. When a published number turns out to be wrong, the fix ships with the next daily publish and the boards it touched appear in What’s New as changes; corrections that changed what a visitor saw are listed on the interactive page.
Interactive version: theaggregate.ai/how-it-works · How It Works · Data refreshed daily, snapshot 2026-10-15.