FinHardBench: leaderboard
FinHardBench (COLM 2026) gives an LLM the natural-language specification of one of 33 financial FPGA modules, from fixed-point arithmetic and trading indicators to order books and option pricing, and checks the single-shot Verilog it returns against a self-checking testbench. Score is the share of 99 generations per model, three per task, that pass simulation, in percent; higher is better.
Metric: Simulation pass rate (%; share of 99 single-shot generations per model, 33 financial FPGA module tasks x 3 trials, whose Verilog passes the self-checking testbench of its task in Icarus Verilog; the prompt holds only the natural-language specification; via OpenRouter at temperature 0.7). Source: arxiv.org. Saturation forecast: Around February 2027. 6 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 4.6 | 61 | #161 |
| 2 | GPT-5.4 | 61 | #123 |
| 3 | Gemini 3.1 Flash Lite (Preview) | 46 | #418 |
| 4 | DeepSeek V3.2 | 25 | #406 |
| 5 | Mistral Large 3 | 23 | #817 |
| 6 | MiniMax-M2.7 | 19 | #518 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=finhardbench · How It Works · Data refreshed daily, snapshot 2026-11-13.