MADE - Jaccard: leaderboard

Label-set agreement on MADE (2026), a living multi-label benchmark of FDA medical device adverse event reports with 1,154 hierarchical labels: the per-report Jaccard index between predicted and gold label sets, averaged over the test reports, with models prompted with ten kNN-retrieved training reports. Percent; higher is better.

Metric: Sample-level Jaccard index (%, the 0-1 score times 100) between the predicted and gold label sets with ten-shot prompting with kNN-retrieved training reports, the task instructions and the label list, on the 10,288-report stratified (truncated) MADE test set of FDA medical device adverse event reports from July 2024 to June 2025 with 1,154 hierarchical IMDRF product and patient problem labels, greedy decoding (GPT-5 at its default medium reasoning effort); higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1GPT-557#178
2GPT-4.157#505
3DeepSeek R1 052850#428
4Qwen 3 235B A22B Instruct49#691
5Qwen 3 235B A22BThinking48#628
6Qwen 3 30B A3BThinking47#1156
7Llama-3.3-nemotron-super-49B-v1.546#920
8GPT-OSS-120B45#704
9GLM 4.5 Air44#886
10Llama 3.1 70B Instruct43#1571
11Qwen 3 4BReasoning43#2348
12Qwen 3 30B A3B Instruct43#1053
13Qwen 3 4B Instruct41#1696
14Llama 3.1 8B Instruct22#4600
15Kimi K27#443

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=made-jaccard · How It Works · Data refreshed daily, snapshot 2026-11-21.