ABRA - Hard: leaderboard

Hard radiology tasks (localizing new lesions across longitudinal studies, end-to-end BI-RADS reporting from breast MRI) for an agent operating an OHIF imaging viewer and an Orthanc DICOM server through 21 tools (ABRA, 2026), scored as 0.2 planning plus 0.3 execution plus 0.5 outcome on a 0-100 scale. Higher is better.

Metric: Hard-tier score (longitudinal new-lesion localisation, end-to-end BI-RADS reporting from breast MRI), composite 0.2 Planning + 0.3 Execution + 0.5 Outcome, each normalised to 0-1, times 100; the agent operates an OHIF viewer and an Orthanc DICOM server through 21 tools, temperature 0, provider-default reasoning effort, single run; tasks that hit the turn cap without submitting score Outcome 0; higher is better. Source: arxiv.org. Saturation forecast: Not forecast. 10 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Kimi K2.546#266
2Mistral Large 345#832
3ABRA Qwen 3.5 (checkpoint unspecified)45
4Gemini 3.1 Pro (Preview)44#99
5Claude Sonnet 4.644#173
6GPT-5.4 Nano (2026-03-17)42#473
7Ministral 3 14B41#1870
8GPT-5.4 (2026-03-05)40#92
9ABRA Gemma 4 (checkpoint unspecified)37
10Gemini 3 Flash (Preview)36#152

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=abra-hard · How It Works · Data refreshed daily, snapshot 2026-11-21.