Mat-Pref - IID: leaderboard
Mat-Pref (Leung, Kim and Park, 2026) asks which element substitution at a crystallographic site of an inorganic host best raises or lowers its formation energy or band gap, with three or four candidates graded against Materials Project DFT values. This board covers 89 logical questions on held-out hosts of the eight training structure families, each scored by majority vote over its distractor permutations, in percent (higher is better); rows include models the authors fine-tuned on the training split.
Metric: Logical-question accuracy (%; majority vote over each logical question's distractor permutations, three- or four-choice ionic substitution questions graded against Materials Project DFT values; 89 logical questions on held-out hosts of the eight training structure families). Source: arxiv.org. Saturation forecast: Estimated already saturated. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Qwen 3 235B A22B 2507 Instruct | 43.8 | #619 |
| 2 | Qwen 2.5 72B Instruct | 37.1 | #1093 |
| 3 | DeepSeek V3 | 36 | #705 |
| 4 | Llama 3.3 70B Instruct | 33.7 | #1353 |
| 5 | Llama 3.1 8B Instruct | 18 | #4277 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=mat-pref-iid · How It Works · Data refreshed daily, snapshot 2026-11-12.