FHIR-AgentBench (Single-Turn + Retriever): leaderboard
Answering 2,931 clinician-sourced questions (from EHRSQL-2024) about single patients whose de-identified MIMIC-IV records are served through an HL7 FHIR API; FHIR-AgentBench (Lee et al., 2025). The LLM runs as a single-turn agent with a FHIR resource retriever. Answer correctness in percent, judged by o4-mini against the SQL-derived answer; higher is better.
Metric: Answer correctness (%; o4-mini judge of the free-text answer against the SQL-derived answer, 2,931 questions; one turn with a FHIR resource retriever). Source: arxiv.org. Saturation forecast: Estimated already saturated. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 32B | 23 | #960 |
| 2 | O4 Mini | 22 | #354 |
| 3 | Gemini 2.5 Flash | 21 | #452 |
| 4 | Llama 3.3 70B | 17 | #1511 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=fhir-agentbench-single-turn-plus-retriever · How It Works · Data refreshed daily, snapshot 2026-11-12.