AutoformBench: leaderboard
Meta FAIR benchmark of 341 graduate- and research-level Lean 4 autoformalization tasks across 24 mathematical sources, evaluated in a shared network-disabled Harbor agent loop with Lean-kernel verification and a $200 per-task cost ceiling.
Metric: Solved (%) of 341 Lean 4 formalization tasks. Source: github.com. Saturation forecast: Around March 2027. 9 models tracked. Updated 20 Nov 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | GPT-6 Astra | Max | 82.4 | #10 |
| 2 | Claude Opus 5 | Max | 61.88 | #26 |
| 3 | GPT-5.6 Sol | xHigh | 58.94 | #36 |
| 4 | Claude Fable 5 | Max | 53.08 | #32 |
| 5 | DeepSeek V4.1 Flash | Max | 52.79 | #69 |
| 6 | Gemini 3.7 Flash | High | 52.49 | #42 |
| 7 | Kimi K3 | Max | 48.39 | #61 |
| 8 | Claude Sonnet 5 | High | 44.57 | #118 |
| 9 | Claude Opus 4.8 | High | 43.99 | #64 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=autoformbench · How It Works · Data refreshed daily, snapshot 2026-11-20.