LiveFMBench: leaderboard
Formal specification is essential for rigorous program verification, yet writing correct specifications remains costly and difficult to automate. Although large language models (LLMs) and agents have shown promising progress, their true capabilities and failure modes remain unclear. We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generatio
Metric: Pass@1 Before (Pre2025) (self-reported). Source: benchmarklist.com. Saturation forecast: Around January 2027. 11 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 235B A22B 2507 (Thinking) | 37.11 | #260 (Qwen 3 235B A22B 2507) |
| 2 | Qwen 3 32B | 32.37 | #443 |
| 3 | Kimi K2 0905 | 30.44 | #288 |
| 4 | Qwen 3 Coder 480B A35B Instruct | 30.37 | #315 |
| 5 | DeepSeek V3 (0324) | 24.29 | #340 |
| 6 | Qwen 3 235B A22B 2507 Instruct | 15.83 | #300 |
| 7 | Llama 3.3 70B Instruct | 11.64 | #534 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=livefmbench · How It Works · Data refreshed daily, snapshot 2026-10-20.