EXCISE - MMLU Professional Medicine: leaderboard
EXCISE v1 medical distractor robustness: 1273 held-out MedQA test questions; MedQA-TD and MedDistractQA each have Nonliteral/Bystander variants; BiasMedQA has 1273 five-option questions per bias. First-answer-token logits over answer options, bfloat16/eager attention, batch 1/no KV unless stated. Qwen 3-8B thinking off for first-token rows; separately labeled reasoning baselines use generated answers. EXCISE attributes by attention blocking and gates using hidden-state discordance; layer/threshold/blocking chosen on development, never test labels. CoT greedy max 1024 tokens; Qwen/Llama use JSON, Meerkat model-card prompt/final-answer extraction. KV cache used for Qwen/Llama CoT MMLU-PM/MedDistractQA and all Meerkat CoT. Exact inference methods remain distinct evaluated systems. The same held-out task is assembled across disjoint model/method populations; table-specific readout, masking, projection, and perturbation controls are retained in the named system/task. Forced-choice head-search masks use 382 development pairs and 887 selection questions; random matched masks average 5 runs. Gold-cue controls target the correct option. Projection batches use 20 random 50-question partitions or a fixed unlabeled development projection. SimCSE gate threshold is development-tuned; MPNet/SapBERT reuse the original threshold. Native position averages, two-type averages and seven-bias averages are separate; no score or denominator is reconstructed. EXCISE - MMLU Professional Medicine. Metric: Accuracy (%).
Metric: Accuracy (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 34 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen3-8B (no thinking) (Harness: EXCISE CoT) | 86.03 | |
| 2 | Qwen3-8B (no thinking) (Harness: EXCISE CoT + Excise) | 85.66 | |
| 3 | Qwen3-8B (no thinking) (Harness: EXCISE CoT + I3C) | 84.93 | |
| 4 | Qwen3-8B (no thinking) (Harness: EXCISE S2A) | 83.46 | |
| 5 | Llama-3-Meerkat-8B (Harness: EXCISE CoT) | 83.46 | |
| 6 | Qwen3-8B (no thinking) (Harness: EXCISE Excise) | 83.09 | |
| 7 | Qwen3-8B (no thinking) (Harness: EXCISE first-token) | 82.72 | |
| 8 | Qwen3-8B (no thinking) (Harness: EXCISE Ignore instruction) | 82.72 | |
| 9 | Llama-3-Meerkat-8B (Harness: EXCISE CoT + I3C) | 82.35 | |
| 10 | Qwen3-8B (no thinking) (Harness: EXCISE top-1 Avg. Attention) | 81.99 |
Interactive version: theaggregate.ai/benchmark?slug=excise-mmlu-professional-medicine · How It Works · Data refreshed daily, snapshot 2026-10-28.