MMRareBench - Diagnosis F1: leaderboard

MMRareBench (2026) diagnosis track scored deterministically: token-level F1 (%) between the predicted primary diagnosis and the gold diagnosis or its aliases, from masked rare-disease case narratives with multiple medical images; higher is better.

Metric: Diagnosis track (T1) token-level F1 (%, SQuAD-style normalization, maximum over the gold diagnosis aliases) of the predicted primary diagnosis from a masked rare-disease case narrative and its medical images; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 23 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3 Flash (Preview)84.7#156
2Gemini 2.5 Pro80.1#280
3GPT-576.1#185
4Claude Sonnet 4.570.8#267
5GPT-4o70#754
6Gemini 2.5 Flash70#466
7Qwen 3 VL 235B A22B Instruct67.3#541
8Qwen 2.5 VL 72B Instruct61#849
9Hulu-Med-14B58.2#1621
10Qwen 3 VL 30B A3B Instruct56.3#903
11Lingshu-32B53.3#1125
12Claude Haiku 4.551.8#587
13MedGemma-27B-IT51.3#1359
14Qwen 2.5 VL 32B Instruct49.6#1065
15HuluMed-7B49#2061

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=mmrarebench-diagnosis-f1 · How It Works · Data refreshed daily, snapshot 2026-11-22.