Intent2Tx: leaderboard

Intent2Tx (2026) evaluates LLMs on translating natural-language Web3 intents into executable Ethereum transactions, with instances derived from real mainnet traces across eleven protocol categories. This board is the final weighted score on a 0 to 100 scale, combining JSON validity, the correct contract and function, parameter accuracy and exact end-to-end match with weights 0.1, 0.3, 0.2 and 0.4, on single-step intents without demonstrations; higher is better.

Metric: Final weighted score (from 0-1, times 100): per-instance 0.1 x JSON format validity + 0.3 x logic score (correct contract address, name and function) + 0.2 x parameter score + 0.4 x pass@1 (exact end-to-end match), averaged over instances, generating an Ethereum transaction (contract, address, function, parameters, ETH value) as JSON from a natural-language intent and an address book, on Intent2Tx instances derived from 300 days of real Ethereum mainnet traces, single-step intents (one transaction), direct inference without demonstrations; higher is better. Source: arxiv.org. Saturation forecast: Around 2031. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 2.5 Pro38#276
2GPT-5.238#185
3Claude Opus 4.537#160
4DeepSeek V3.235#419
5GLM-4.634#498
6Kimi K2 090534#570
7Claude Haiku 4.533#583
8Gemini 2.5 Flash32#463
9GLM-4 32B (0414)31#1505
10Qwen3 Coder Next31#725
11codestral-250831#1511
12Qwen 3 235B A22B 2507 Instruct30#616
13GPT-5.2 Codex29#182
14Llama 3.3 70B Instruct28#1385
15Llama 3.1 8B Instruct25#4600

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Example tasks

I want to swap 2122.6 TSUKA for at least 385.6 RENA and send it to 0xd90b34fc8c9544f831fce314eccbf83ded40d37e.

Source

Swap wstETH to WETH on Uniswap, then transfer the WETH to wallet 0x8BE0b09F2d5DE9F6CCd1aDb4aa6Ec920Fd1d3e18.

Source

Interactive version: theaggregate.ai/benchmark?slug=intent2tx · How It Works · Data refreshed daily, snapshot 2026-11-21.