Intent2Tx - State Equivalence: leaderboard

Intent2Tx (2026) evaluates LLMs on translating natural-language Web3 intents into executable Ethereum transactions, with instances derived from real mainnet traces across eleven protocol categories. This board is the percentage of single-step transactions whose execution on a forked mainnet changes account state like the reference, without demonstrations; higher is better.

Metric: State equivalence (from 0-1, times 100): share of generated single-step transactions whose execution on a forked Ethereum mainnet (fork one block before the original transaction) changes account-level state like the reference transaction (within 1%), over instances whose reference executes, direct inference without demonstrations; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 2.5 Flash61.6#463
2Gemini 2.5 Pro60.5#276
3GLM-4.659.2#498
4Kimi K2 090556#570
5GPT-5.254.1#185
6Qwen3 Coder Next52#725
7Claude Haiku 4.551.3#583
8codestral-250848.1#1511
9DeepSeek V3.248#419
10Claude Opus 4.547.2#160
11Qwen 3 235B A22B 2507 Instruct40.9#616
12GPT-5.2 Codex39.4#182
13GLM-4 32B (0414)37.7#1505
14Llama 3.3 70B Instruct31#1385
15Llama 3.1 8B Instruct20.8#4600

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=intent2tx-state-equivalence · How It Works · Data refreshed daily, snapshot 2026-11-21.