Intent2Tx (Retrieval) - State Equivalence: leaderboard

Intent2Tx (2026) evaluates LLMs on translating natural-language Web3 intents into executable Ethereum transactions, with instances derived from real mainnet traces across eleven protocol categories. This board is the percentage of single-step transactions whose execution on a forked mainnet changes account state like the reference, with three retrieved demonstrations; higher is better.

Metric: State equivalence (from 0-1, times 100): share of generated single-step transactions whose execution on a forked Ethereum mainnet (fork one block before the original transaction) changes account-level state like the reference transaction (within 1%), over instances whose reference executes, with the three most similar instances retrieved as demonstrations; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 16 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Claude Opus 4.570.6#165
2Gemini 2.5 Pro70.2#280
3GPT-5.269.7#190
4Qwen3 Coder Next69.7#741
5GPT-5.2 Codex69.7#186
6Gemini 2.5 Flash69.2#466
7GLM-4.667.2#496
8Kimi K2 090565.5#578
9Qwen 3 235B A22B 2507 Instruct65.3#625
10Claude Haiku 4.564.7#587
11GLM-4 32B (0414)64.3#1525
12DeepSeek V3.263.5#425
13codestral-250862.6#1522
14Llama 3.3 70B Instruct57.3#1394
15Ministral 3 14B52.3#1885

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=intent2tx-retrieval-state-equivalence · How It Works · Data refreshed daily, snapshot 2026-11-22.