Intent2Tx (Retrieval): leaderboard

Intent2Tx (2026) evaluates LLMs on translating natural-language Web3 intents into executable Ethereum transactions, with instances derived from real mainnet traces across eleven protocol categories. This board is the final weighted score on a 0 to 100 scale, combining JSON validity, the correct contract and function, parameter accuracy and exact end-to-end match with weights 0.1, 0.3, 0.2 and 0.4, on single-step intents with three retrieved similar instances as demonstrations; higher is better.

Metric: Final weighted score (from 0-1, times 100): per-instance 0.1 x JSON format validity + 0.3 x logic score (correct contract address, name and function) + 0.2 x parameter score + 0.4 x pass@1 (exact end-to-end match), averaged over instances, generating an Ethereum transaction (contract, address, function, parameters, ETH value) as JSON from a natural-language intent and an address book, on Intent2Tx instances derived from 300 days of real Ethereum mainnet traces, single-step intents, with the three most similar instances retrieved as demonstrations; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 16 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 2.5 Flash70#463
2Claude Opus 4.570#160
3Gemini 2.5 Pro69#276
4GLM-4.669#498
5GPT-5.267#185
6Claude Haiku 4.567#583
7Kimi K2 090567#570
8GPT-5.2 Codex66#182
9DeepSeek V3.265#419
10GLM-4 32B (0414)65#1505
11Qwen 3 235B A22B 2507 Instruct64#616
12Qwen3 Coder Next63#725
13codestral-250861#1511
14Llama 3.3 70B Instruct58#1385
15Ministral 3 14B52#1870

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Example tasks

I want to swap 2122.6 TSUKA for at least 385.6 RENA and send it to 0xd90b34fc8c9544f831fce314eccbf83ded40d37e.

Source

Swap wstETH to WETH on Uniswap, then transfer the WETH to wallet 0x8BE0b09F2d5DE9F6CCd1aDb4aa6Ec920Fd1d3e18.

Source

Interactive version: theaggregate.ai/benchmark?slug=intent2tx-retrieval · How It Works · Data refreshed daily, snapshot 2026-11-21.