CXR-ContraBench: leaderboard
CXR-ContraBench (2026) probes negated-option attraction in medical vision-language models with chest X-ray multiple-choice questions whose options are a present finding, its negated form and another finding. This board is the CheXpert direct presence probe of 235 questions under plain multiple-choice prompting; accuracy in percent with chance at 33.3, higher is better.
Metric: Accuracy (%) on the CheXpert direct presence probe of CXR-ContraBench: 235 three-option questions from 82 CheXpert validation studies asking which finding is present, with the present finding, its negated form (No X) and another finding as options; base multiple-choice prompting without chain of thought or the authors' verifier, exact match; chance is 33.3; higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 7 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | LLaVA-Med-1.5-7B | 74.04 | |
| 2 | GPT-4o | 62.55 | #751 |
| 3 | LLaVA-v1.5-7B | 56.6 | #7630 |
| 4 | MedGemma-27B-IT | 51.91 | #1350 |
| 5 | CheXagent-2-3B | 37.87 | |
| 6 | MedGemma-4B-IT | 31.49 | #3282 |
| 7 | Qwen 2.5 VL 7B Instruct | 30.21 | #1902 |
No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=cxr-contrabench · How It Works · Data refreshed daily, snapshot 2026-11-21.