EXCISE - MedDistractQA Average: leaderboard

EXCISE v1 medical distractor robustness: 1273 held-out MedQA test questions; MedQA-TD and MedDistractQA each have Nonliteral/Bystander variants; BiasMedQA has 1273 five-option questions per bias. First-answer-token logits over answer options, bfloat16/eager attention, batch 1/no KV unless stated. Qwen 3-8B thinking off for first-token rows; separately labeled reasoning baselines use generated answers. EXCISE attributes by attention blocking and gates using hidden-state discordance; layer/threshold/blocking chosen on development, never test labels. CoT greedy max 1024 tokens; Qwen/Llama use JSON, Meerkat model-card prompt/final-answer extraction. KV cache used for Qwen/Llama CoT MMLU-PM/MedDistractQA and all Meerkat CoT. Exact inference methods remain distinct evaluated systems. The same held-out task is assembled across disjoint model/method populations; table-specific readout, masking, projection, and perturbation controls are retained in the named system/task. Forced-choice head-search masks use 382 development pairs and 887 selection questions; random matched masks average 5 runs. Gold-cue controls target the correct option. Projection batches use 20 random 50-question partitions or a fixed unlabeled development projection. SimCSE gate threshold is development-tuned; MPNet/SapBERT reuse the original threshold. Native position averages, two-type averages and seven-bias averages are separate; no score or denominator is reconstructed. EXCISE - MedDistractQA Average. Metric: Accuracy (%).

Metric: Accuracy (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 31 models tracked.

Top models

Interactive version: theaggregate.ai/benchmark?slug=excise-meddistractqa-average · How It Works · Data refreshed daily, snapshot 2026-10-28.