MamaBench: leaderboard
Metric: Bias Trap Rate (%; share of pairs with a correct base diagnosis whose counterfactual is still wrong; 217 expert-authored maternal and paediatric patient narratives, each paired with a minimally perturbed counterfactual whose diagnosis differs (371 pathologies); diagnoses judged by Claude Opus 4.5 at temperature 0; standalone model at temperature 0, no retrieval; lower is better). Source: arxiv.org. Saturation forecast: Around December 2026. 5 models tracked.
Top models
| # | Model | Score |
|---|---|---|
| 1 | GPT-5.4 | 25.6 |
| 2 | Claude Sonnet 4.6 | 25.8 |
| 3 | Nemotron 3 Super | 35 |
| 4 | Kimi K2.5 | 40 |
| 5 | GLM-5 | 50.4 |
Interactive version: theaggregate.ai/benchmark?slug=mamabench · How It Works · Data refreshed daily, snapshot 2026-09-29.