Vals AI Autoform-Hard: leaderboard
Vals AI's hardest slice of Meta FAIR's AutoformBench: 56 Lean 4 formalization tasks from graduate and research textbooks (algebraic geometry, number theory, Lie theory and more) that none of the ten frontier models in the original study solved. Each agent gets one attempt in the same harness (a shell, the Lean toolchain and prebuilt Mathlib) within 3 hours and $200, and a task counts only if its proofs pass clean-room grading and an independent kernel replay; an errored run counts as unsolved, so a 0% row is a real score.
Metric: Accuracy (%). Source: www.vals.ai. Saturation forecast: Around February 2027. 8 models tracked. Updated 19 Nov 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | GPT-6.2 Astra | Max | 87.5 | #4 |
| 2 | GPT-6.2 Sol | Max | 85.71 | #12 |
| 3 | Claude Fable 5.2 | High | 85.71 | #1 |
| 4 | Gemini 4 Carbon | High | 71.43 | #5 |
| 5 | Gemini 4 Hydrogen | xHigh | 64.29 | #16 |
| 6 | Gemini 4 Argon | High | 48.21 | #18 |
| 7 | Beam 1.0 | 0 | #213 | |
| 8 | MiniMax-M3.1-Flash | Max | 0 | #121 |
No result here: #3 Gemini 4 Neon, #9 Claude Opus 5.5, #10 GPT-6 Astra.
Interactive version: theaggregate.ai/benchmark?slug=vals-ai-autoform-hard · How It Works · Data refreshed daily, snapshot 2026-11-20.