Singularity-Bench - Pretraining: leaderboard

Singularity-Bench - Pretraining — Singularity-Bench v1 Table5; six engineering tasks with native model/harness pairs. Pretraining,midtraining,RL and engine-building each12hours on4GB300; coding-harness6hours on1GB300; serverless-cluster4hours on16GB300. Held-out downstream task performance, not builder output speed or cost. Stars mean beating the human baseline; exact stars and native cost cells remain in row provenance. Section6.3binds the Opus5.5andSol inference-engine cells to max effort; unreported settings are not inferred for other cells. No lowest-cost or best-harness pooling. Crosses denote failed/rejected submissions; their individual raw metrics remain absent rather than invented numeric values. AppendixA.1 assigns those failures zero contribution to the published Index, which is retained natively. Human baseline is an engineering reference, not a fitted population median. Separate harnesses retain their native labels and documented source controls. Metric: Held-out validation perplexity.

Metric: Held-out validation perplexity. Source: arxiv.org. Saturation forecast: Around January 2028. 7 models tracked. Updated 15 Nov 2026.

Top models

#ModelScoreOverall rank
1Claude Opus 5.512.77#5
2Claude Opus 512.89#20
3GPT-6.1 Sol13.33#10
4Kimi K313.6#56
5GLM-5.314.81#68
6DeepSeek V4.1 Flash16.15#62
7MiniMax-M316.76#272

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #6 GPT-6 Astra.

Interactive version: theaggregate.ai/benchmark?slug=singularity-bench-pretraining · How It Works · Data refreshed daily, snapshot 2026-11-16.