EHRNote-ChatQA - All Turns Correct: leaderboard
EHRNote-ChatQA (2026) poses multi-turn 5-way multiple-choice questions over the full sequence of MIMIC-IV discharge summaries of one patient, pairing each content question with an evidence-grounding question about which notes and sections support the answer. This board is the share of the 967 patient conversations in which every turn, content and evidence, is answered correctly, in percent; higher is better.
Metric: Sample-level strict paired accuracy (%): share of the 967 patient samples in which every content and evidence-grounding turn is answered correctly, on 967 multi-turn samples, each over the full set of MIMIC-IV discharge summaries of one patient (1 to 5 notes), 8,036 content and 8,036 paired evidence-grounding 5-way multiple-choice questions (chance 20), chat history kept across turns, greedy decoding where settable; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 22 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 | 40.02 | #123 |
| 2 | Gemini 3 Flash (Preview) | 35.76 | #144 |
| 3 | Qwen 3 Next 80B A3B Instruct | 19.86 | #766 |
| 4 | Qwen 3 30B A3B 2507 Instruct | 10.03 | #1089 |
| 5 | Llama 4 Scout Instruct | 9.51 | #1543 |
| 6 | GPT-5.4 Mini | 8.69 | #389 |
| 7 | DeepSeek R1 Distill Llama 70B | 8.38 | #1637 |
| 8 | DeepSeek R1 Distill Qwen 32B | 4.55 | #1788 |
| 9 | Qwen 3 4B 2507 Instruct | 3 | #2163 |
| 10 | Ministral-3-3B-Instruct-2512 | 1.76 | #2579 |
| 11 | DeepSeek R1 Distill Qwen 14B | 1.45 | #2582 |
| 12 | Ministral-3-8B-Instruct-2512 | 1.14 | #1858 |
| 13 | Phi-4 Mini Instruct | 0.41 | #6419 |
| 14 | Phi-3.5-mini-instruct | 0.31 | #5733 |
| 15 | DeepSeek R1 Distill Llama 8B | 0.1 | #6754 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=ehrnote-chatqa-all-turns-correct · How It Works · Data refreshed daily, snapshot 2026-11-13.