MechVQA - Item Localization: leaderboard

MechVQA (2026) tests multimodal LLMs on understanding mechanical engineering drawings, with 20,778 questions on 3,281 drawings scored as accuracy by three LLM judges against reference answers. This board covers the Item Localization subtask of the recognition capability, as accuracy in percent on the test split; higher is better.

Metric: Accuracy (%) on the Item Localization subtask (Recognition capability), MechVQA test split (drawing-level 8:1:1 split of 20,778 questions on 3,281 mechanical drawings), answers judged against the reference by three LLM judges (GPT-OSS-120B, DeepSeek-V3.2, Kimi-k2); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3 Pro (Preview)64.03#123
2GLM-4.6V63.31#721
3GPT-562.59#185
4Qwen 3 VL 32B Instruct56.83#607
5Claude Sonnet 4.556.12#267
6Qwen 3 VL Plus52.88#514
7GPT-4o40.29#754
8MiMo-VL-7B-SFT38.85#1097
9Qwen 3 VL 30B A3B Instruct37.41#903
10Qwen 3 VL 4B Instruct33.09#1348
11MiniCPM-V-4.530.22#1451
12Gemma 3 27B (IT)24.46#1341
13GPT-4o Mini23.02#1680
14InternVL3.5-8B19.42#1618
15Llama 3.2 11B Instruct11.51#5213

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=mechvqa-item-localization · How It Works · Data refreshed daily, snapshot 2026-11-22.