AInsteinBench: leaderboard
A model drives the CodeAct agent inside a Docker container holding a real scientific code base (PySCF, Qiskit, Einstein Toolkit, AMReX, OpenMM, RDKit) and must fix a merged pull request's scientific bug or rebuild an ablated feature until the repository's tests pass; the score is the share of all 244 problems resolved. Unlike SWE-bench, the repositories are research codes whose tests check physical and numerical correctness. ByteDance Seed, which also makes Doubao-Seed-1.8, wrote the benchmark and ran every model.
Metric: % Resolved. Source: arxiv.org. Saturation forecast: Around May 2027. 7 models tracked. Updated 17 Nov 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | Claude Opus 4.5 | 44 | #161 | |
| 2 | Gemini 3 Pro (Preview) | 42.8 | #120 | |
| 3 | Seed 1.8 | 36.7 | #250 | |
| 4 | GPT-5 | High | 35.4 | #177 |
| 5 | Claude Sonnet 4.5 | 33.7 | #258 | |
| 6 | GPT-5.1 | 32.9 | #235 | |
| 7 | GLM-4.6 | 25.9 | #496 |
No result here: #1 Claude Fable 5.2, #2 GPT-6.2 Astra, #3 Gemini 4 Carbon.
Interactive version: theaggregate.ai/benchmark?slug=ainsteinbench · How It Works · Data refreshed daily, snapshot 2026-11-17.