ABRA - Hard: leaderboard
Hard radiology tasks (localizing new lesions across longitudinal studies, end-to-end BI-RADS reporting from breast MRI) for an agent operating an OHIF imaging viewer and an Orthanc DICOM server through 21 tools (ABRA, 2026), scored as 0.2 planning plus 0.3 execution plus 0.5 outcome on a 0-100 scale. Higher is better.
Metric: Hard-tier score (longitudinal new-lesion localisation, end-to-end BI-RADS reporting from breast MRI), composite 0.2 Planning + 0.3 Execution + 0.5 Outcome, each normalised to 0-1, times 100; the agent operates an OHIF viewer and an Orthanc DICOM server through 21 tools, temperature 0, provider-default reasoning effort, single run; tasks that hit the turn cap without submitting score Outcome 0; higher is better. Source: arxiv.org. Saturation forecast: Not forecast. 10 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Kimi K2.5 | 46 | #266 |
| 2 | Mistral Large 3 | 45 | #832 |
| 3 | ABRA Qwen 3.5 (checkpoint unspecified) | 45 | |
| 4 | Gemini 3.1 Pro (Preview) | 44 | #99 |
| 5 | Claude Sonnet 4.6 | 44 | #173 |
| 6 | GPT-5.4 Nano (2026-03-17) | 42 | #473 |
| 7 | Ministral 3 14B | 41 | #1870 |
| 8 | GPT-5.4 (2026-03-05) | 40 | #92 |
| 9 | ABRA Gemma 4 (checkpoint unspecified) | 37 | |
| 10 | Gemini 3 Flash (Preview) | 36 | #152 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=abra-hard · How It Works · Data refreshed daily, snapshot 2026-11-21.