LitXBench: leaderboard

LitXBench (2026) tests extracting complete experiments from scientific papers: on LitXAlloy, 19 alloy papers with 1,426 annotated measurements, the model writes each material with its processing chain, microstructure and measurements as Python objects from the OCR text of the paper. Overall weighted F1 from 0 to 1 for language models called through an API, after Hungarian matching of extracted to annotated materials; higher is better.

Metric: Overall F1 (0-1), the weighted combination of measurement (0.5), process condition (0.2), material set (0.15) and configuration (0.15) F1 for extracting every experiment (materials, process chains, microstructure configurations and measurements) as Python objects from the OCR text of LitXAlloy's 19 alloy papers (1,426 measurements; figures excluded), extracted materials matched to the annotated ones by the Hungarian algorithm, mean of three runs; models called through the API with Pydantic AI; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 6 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3.1 Pro (Preview)0.77#99
2Gemini 3 Flash (Preview)0.74#152
3GPT-5.2High0.73#185
4Claude Opus 4.60.72#113
5GPT-5 Mini0.68#364
6Claude Haiku 4.50.65#583

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=litxbench · How It Works · Data refreshed daily, snapshot 2026-11-21.