MCPGen - Tool Creation (Individual Pass): leaderboard
Tool creation in MCPGen (Yang et al., Shanghai Jiao Tong University, 2026): the model implements each of 911 withheld tools of 100 executable Model Context Protocol projects from its schema, workflow context and project interfaces. Percent of generated tools that pass their own tool-level tests when substituted one at a time, single-turn at temperature 0; higher is better.
Metric: Individual pass rate (%; share of the 911 withheld workflow tools whose generated implementation passes its tool-level tests when substituted alone; single-turn generation, temperature 0, 100 executable MCP projects from 16 domains). Source: arxiv.org. Saturation forecast: Not forecast. 11 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 4.5 | 63.8 | #244 |
| 2 | Claude Haiku 4.5 (20251001) | 62.3 | #529 |
| 3 | Qwen 3 Coder 30B A3B Instruct | 58.5 | #973 |
| 4 | GPT-5.1 | 58.3 | #225 |
| 5 | O3 | 56.7 | #224 |
| 6 | Gemini 3 Pro (Preview) | 53.5 | #111 |
| 7 | GPT-4o Mini | 53.1 | #1627 |
| 8 | GPT-5.2 | 52.2 | #172 |
| 9 | Gemini 3 Flash (Preview) | 41.3 | #141 |
| 10 | Qwen 3 Next 80B A3B Instruct | 40.1 | #763 |
| 11 | MiniMax-M2.1 | 39.3 | #569 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=mcpgen-tool-creation-individual-pass · How It Works · Data refreshed daily, snapshot 2026-11-12.