AssertLLM2 - Bug Kill Ratio: leaderboard

Share (%) of mutated buggy versions of each design that the proven generated assertions catch on AssertLLM2 (2026), where LLMs write SystemVerilog assertions from the specifications of 83 real-world hardware designs, checked by formal property verification; mean of three generations per design. Higher is better.

Metric: Bug kill ratio (%): share of mutated buggy RTL variants (20 single-bug variants per design, or 5 bugs in a merged bug-hunting variant) on which a proven assertion fails, SystemVerilog assertions generated from the structured specification of each of 83 AssertLLM2 designs and checked in Cadence JasperGold, mean of three independent generations per design (the paper's default average setting); higher is better. Source: arxiv.org. Saturation forecast: Around July 2028. 6 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude Sonnet 4.511.69#261
2GPT-5.210.19#185
3Gemini 2.5 Pro9.76#276
4DeepSeek V3.29.2#419
5Qwen 3 Coder Plus6.82#720
6Llama 4 Maverick5.08#1110

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=assertllm2-bug-kill-ratio · How It Works · Data refreshed daily, snapshot 2026-11-21.