EXCISE Off-case 1 Sentences - Nonliteral: leaderboard
EXCISE v1 medical distractor robustness: 1273 held-out MedQA test questions; MedQA-TD and MedDistractQA each have Nonliteral/Bystander variants; BiasMedQA has 1273 five-option questions per bias. First-answer-token logits over answer options, bfloat16/eager attention, batch 1/no KV unless stated. Qwen 3-8B thinking off for first-token rows; separately labeled reasoning baselines use generated answers. EXCISE attributes by attention blocking and gates using hidden-state discordance; layer/threshold/blocking chosen on development, never test labels. CoT greedy max 1024 tokens; Qwen/Llama use JSON, Meerkat model-card prompt/final-answer extraction. KV cache used for Qwen/Llama CoT MMLU-PM/MedDistractQA and all Meerkat CoT. Exact inference methods remain distinct evaluated systems. The same held-out task is assembled across disjoint model/method populations; table-specific readout, masking, projection, and perturbation controls are retained in the named system/task. Forced-choice head-search masks use 382 development pairs and 887 selection questions; random matched masks average 5 runs. Gold-cue controls target the correct option. Projection batches use 20 random 50-question partitions or a fixed unlabeled development projection. SimCSE gate threshold is development-tuned; MPNet/SapBERT reuse the original threshold. Native position averages, two-type averages and seven-bias averages are separate; no score or denominator is reconstructed. EXCISE Off-case 1 Sentences - Nonliteral. Metric: Accuracy (%).
Metric: Accuracy (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Llama-3-Meerkat-8B (Harness: EXCISE known distractor) | 61.6 | |
| 2 | Llama-3.1-8B-Instruct (Harness: EXCISE known distractor) | 60.2 | |
| 3 | Qwen3-8B (no thinking) (Harness: EXCISE known distractor) | 60.2 | |
| 4 | Llama-3-Meerkat-8B (Harness: EXCISE Excise) | 54.4 | |
| 5 | Llama-3.1-8B-Instruct (Harness: EXCISE Excise) | 53.3 | |
| 6 | Qwen3-8B (no thinking) (Harness: EXCISE Excise) | 52 | |
| 7 | Llama-3.1-8B-Instruct (Harness: EXCISE w/o IA) | 45.2 | |
| 8 | Llama-3-Meerkat-8B (Harness: EXCISE w/o IA) | 43.9 | |
| 9 | Qwen3-8B (no thinking) (Harness: EXCISE w/o IA) | 43.6 | |
| 10 | Llama-3-Meerkat-8B (Harness: EXCISE first-token) | 42.8 |
Interactive version: theaggregate.ai/benchmark?slug=excise-off-case-1-sentences-nonliteral · How It Works · Data refreshed daily, snapshot 2026-10-28.