AI Consilium — Role reasoning structured: leaderboard
190 held-out NEJM cases; single candidate model performs role assignment, all role-specific opinions and all review/revision/finalization calls; up to 5 rounds. Physician-calibrated GPT-OSS-120B judge, ordinal 0–3 criterion ratings pooled over numeric applicable cells. Component-specific criterion/denominator sets stay separate; not-evaluable cells omitted, infrastructure failures not converted to zero. The calibration cohort's mixed 3+1 worker pipeline is not this held-out experiment. Short Qwen3/Qwen3-Next/Gemma labels do not establish checkpoint variant; source-qualified identities must remain distinct until run manifests resolve them. Metric: Mean clinical quality (0–3). Higher is better. Source: https://arxiv.org/html/2610.22298v1
Metric: Mean clinical quality (0–3). Source: arxiv.org. Saturation forecast: Estimated already saturated. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.6 Sol | 2.89 | #18 |
| 2 | GLM-5.2 | 2.46 | #71 |
| 3 | DeepSeek V4 Pro | 2.44 | #101 |
| 4 | GPT-OSS-120B | 2.42 | #336 |
| 5 | Qwen 3.5 27B | 2.26 | #185 |
| 6 | Gemma 4 31B | 2.25 | #197 |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #7 Gemini 4 Argon.
Interactive version: theaggregate.ai/benchmark?slug=ai-consilium-role-reasoning-structured · How It Works · Data refreshed daily, snapshot 2026-10-17.