EHRBench - Diagnosis: leaderboard

EHRBench (2026) tests LLMs on clinical decision questions generated from real patient records and verified against biomedical knowledge bases, posed as multiple-choice questions with 4 to 6 options. This board is accuracy in percent on the diagnosis-decision questions (complete a withheld diagnosis of an encounter); higher is better.

Metric: Accuracy (%) on the diagnosis-decision questions (complete a withheld diagnosis of an encounter), EHR-grounded multiple-choice clinical decision questions (4, 5 and 6 options) built from MIMIC-III, MIMIC-IV and PROMOTE encounter trajectories with knowledge-base verification, JSON-constrained answers, deterministic decoding; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 31 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-5.2 Instant72.02#304
2GPT-4.169.87#505
3Llama 3.3 70B Instruct68.35#1385
4GPT-5 Chat68.26#449
5Qwen 3 32B67.97#972
6GLM-4 32B (0414)67.09#1505
7Qwen 2.5 32B66.48#1344
8GPT-4.1 Mini66.41#789
9Mistral Small 366.2#1796
10GPT-5 Mini65.4#364
11m1-32B-1k63.07#1240
12EHRBench llama3-70B (checkpoint unspecified)62.21
13Qwen 3 4B59.67#2348
14doctor-r1-8B58.74#1624
15glm-4-9B (0414)58.36#2301

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=ehrbench-diagnosis · How It Works · Data refreshed daily, snapshot 2026-11-21.