MMRareBench - Diagnosis: leaderboard

MMRareBench (2026) diagnosis track: from a de-identified rare-disease case narrative with diagnosis mentions masked and several medical images, the model names the primary diagnosis. Rubric score (0-100) from a strict Qwen3-VL-235B judge; higher is better.

Metric: Diagnosis track (T1) score: open-ended primary diagnosis from a masked rare-disease case narrative and its medical images, graded 0-100 by a Qwen3-VL-235B judge with a strict track-specific rubric that awards credit only when a criterion is fully met (exact match weighted three times disease category and specificity, capped at 20 without an exact match); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 23 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3 Flash (Preview)76.6#152
2GPT-575.9#178
3Gemini 2.5 Pro74#276
4Claude Sonnet 4.568.5#261
5GPT-4o62.5#751
6Gemini 2.5 Flash62.2#463
7Qwen 3 VL 235B A22B Instruct58.1#532
8Qwen 2.5 VL 72B Instruct52.6#841
9Hulu-Med-14B52#1607
10Lingshu-32B49.8#1113
11Qwen 3 VL 30B A3B Instruct46.9#895
12Qwen 2.5 VL 32B Instruct46.9#1062
13MedGemma-27B-IT43.7#1350
14HuluMed-7B42.2#2049
15Hulu-Med-4B39.8#2392

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=mmrarebench-diagnosis · How It Works · Data refreshed daily, snapshot 2026-11-21.