AInsteinBench: leaderboard

A model drives the CodeAct agent inside a Docker container holding a real scientific code base (PySCF, Qiskit, Einstein Toolkit, AMReX, OpenMM, RDKit) and must fix a merged pull request's scientific bug or rebuild an ablated feature until the repository's tests pass; the score is the share of all 244 problems resolved. Unlike SWE-bench, the repositories are research codes whose tests check physical and numerical correctness. ByteDance Seed, which also makes Doubao-Seed-1.8, wrote the benchmark and ran every model.

Metric: % Resolved. Source: arxiv.org. Saturation forecast: Around May 2027. 7 models tracked. Updated 17 Nov 2026.

Top models

#ModelSettingScoreOverall rank
1Claude Opus 4.544#161
2Gemini 3 Pro (Preview)42.8#120
3Seed 1.836.7#250
4GPT-5High35.4#177
5Claude Sonnet 4.533.7#258
6GPT-5.132.9#235
7GLM-4.625.9#496

No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.

Interactive version: theaggregate.ai/benchmark?slug=ainsteinbench · How It Works · Data refreshed daily, snapshot 2026-11-17.