EHRBench - Treatment: leaderboard

EHRBench (2026) tests LLMs on clinical decision questions generated from real patient records and verified against biomedical knowledge bases, posed as multiple-choice questions with 4 to 6 options. This board is accuracy in percent on the treatment-decision questions; higher is better.

Metric: Accuracy (%) on the treatment-decision questions, EHR-grounded multiple-choice clinical decision questions (4, 5 and 6 options) built from MIMIC-III, MIMIC-IV and PROMOTE encounter trajectories with knowledge-base verification, JSON-constrained answers, deterministic decoding; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 31 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1GPT-5 Chat80.45#449
2GPT-5.2 Instant80.13#304
3GPT-4.180.1#505
4Llama 3.3 70B Instruct79.05#1385
5GPT-4.1 Mini77.9#789
6GLM-4 32B (0414)77.9#1505
7EHRBench llama3-70B (checkpoint unspecified)77.63
8Qwen 3 32B77.34#972
9Qwen 2.5 32B76.87#1344
10GPT-5 Mini76.17#364
11Mistral Small 375.02#1796
12m1-32B-1k74.84#1240
13Qwen 3 8B74.49#1873
14doctor-r1-8B74.49#1624
15GPT-4.1 Nano74.03#2056

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=ehrbench-treatment · How It Works · Data refreshed daily, snapshot 2026-11-21.