MCP-GRANITE - Tool Selection F1: leaderboard

MCP-GRANITE (Paschalides, Symeonides, Pallis and Dikaiakos, 2026) runs locally served models as tool-using agents on 81 multi-step scenarios in nine edge-computing domains, each exposed through a mock MCP server at four tool-interface granularities, three times each. Tool-selection F1 in percent between the tools the agent called and the scenario's gold tools, averaged over runs; higher is better.

Metric: Tool-selection F1 (%; per-run F1 of the tools the agent called against the scenario's gold tool set at that granularity, averaged over the 972 runs per model: 81 multi-step scenarios in nine edge domains, each at four tool-interface granularities from fine-grained primitives to one dispatch tool, three repetitions; a Google ADK agent over MCP with the model served locally through Ollama, temperature 0, at most 10 turns; printed as a fraction with its standard deviation and scaled to percent). Source: arxiv.org. Saturation forecast: Around February 2027. 9 models tracked.

Top models

No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.

Interactive version: theaggregate.ai/benchmark?slug=mcp-granite-tool-selection-f1 · How It Works · Data refreshed daily, snapshot 2026-11-13.