EXCISE - MedQA-TD Bystander End: leaderboard
EXCISE v1 medical distractor robustness: 1273 held-out MedQA test questions; MedQA-TD and MedDistractQA each have Nonliteral/Bystander variants; BiasMedQA has 1273 five-option questions per bias. First-answer-token logits over answer options, bfloat16/eager attention, batch 1/no KV unless stated. Qwen 3-8B thinking off for first-token rows; separately labeled reasoning baselines use generated answers. EXCISE attributes by attention blocking and gates using hidden-state discordance; layer/threshold/blocking chosen on development, never test labels. CoT greedy max 1024 tokens; Qwen/Llama use JSON, Meerkat model-card prompt/final-answer extraction. KV cache used for Qwen/Llama CoT MMLU-PM/MedDistractQA and all Meerkat CoT. Exact inference methods remain distinct evaluated systems. The same held-out task is assembled across disjoint model/method populations; table-specific readout, masking, projection, and perturbation controls are retained in the named system/task. Forced-choice head-search masks use 382 development pairs and 887 selection questions; random matched masks average 5 runs. Gold-cue controls target the correct option. Projection batches use 20 random 50-question partitions or a fixed unlabeled development projection. SimCSE gate threshold is development-tuned; MPNet/SapBERT reuse the original threshold. Native position averages, two-type averages and seven-bias averages are separate; no score or denominator is reconstructed. EXCISE - MedQA-TD Bystander End. Metric: Accuracy (%).
Metric: Accuracy (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 84 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Phi-4 (Harness: EXCISE S2A plus Excise) | 65.5 | |
| 2 | Phi-4 (Harness: EXCISE Excise) | 65.2 | |
| 3 | Phi-4 (Harness: EXCISE Batches of 50 questions) | 65.18 | |
| 4 | Phi-4 (Harness: EXCISE forced-choice Head mask, tuned on development data) | 65.04 | |
| 5 | Phi-4 (Harness: EXCISE Fixed: dev. MedQA + MedQA-TD) | 63.55 | |
| 6 | Llama-3-Meerkat-8B (Harness: EXCISE CoT + Excise) | 63.32 | |
| 7 | Llama-3.1-8B-Instruct (Harness: EXCISE CoT + Excise) | 60.88 | |
| 8 | Qwen3-8B (no thinking) (Harness: EXCISE CoT + Excise) | 59.94 | |
| 9 | Llama-3.1-8B-Instruct (Harness: EXCISE forced-choice Head mask, tuned on development data) | 58.84 | |
| 10 | Llama-3-Meerkat-8B (Harness: EXCISE Excise) | 57.66 |
Interactive version: theaggregate.ai/benchmark?slug=excise-medqa-td-bystander-end · How It Works · Data refreshed daily, snapshot 2026-10-28.