MechVQA - Consistency Judgment: leaderboard

MechVQA (2026) tests multimodal LLMs on understanding mechanical engineering drawings, with 20,778 questions on 3,281 drawings scored as accuracy by three LLM judges against reference answers. This board covers the Consistency Judgment subtask of the judging capability, as accuracy in percent on the test split; higher is better.

Metric: Accuracy (%) on the Consistency Judgment subtask (Judging capability), MechVQA test split (drawing-level 8:1:1 split of 20,778 questions on 3,281 mechanical drawings), answers judged against the reference by three LLM judges (GPT-OSS-120B, DeepSeek-V3.2, Kimi-k2); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3 Pro (Preview)82.67#123
2GPT-570.67#185
3Qwen 3 VL 32B Instruct69.33#607
4GLM-4.6V69.33#721
5Qwen 3 VL Plus68#514
6GPT-4o66.67#754
7Claude Sonnet 4.564#267
8Qwen 3 VL 30B A3B Instruct57.33#903
9GPT-4o Mini49.33#1680
10Qwen 3 VL 4B Instruct49.33#1348
11MiniCPM-V-4.546.67#1451
12MiMo-VL-7B-SFT46.67#1097
13Gemma 3 27B (IT)40#1341
14InternVL3.5-8B34.67#1618
15Llama 3.2 11B Instruct28#5213

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=mechvqa-consistency-judgment · How It Works · Data refreshed daily, snapshot 2026-11-22.