LitXBench (Coding Agents) - Materials: leaderboard

LitXBench (2026) tests extracting complete experiments from scientific papers: on LitXAlloy, 19 alloy papers with 1,426 annotated measurements, the model writes each material with its processing chain, microstructure and measurements as Python objects from the OCR text of the paper. F1 from 0 to 1 on the set of extracted materials, for coding agents (Claude Code, Codex, Gemini CLI); higher is better.

Metric: Material F1 (0-1), whether the right set of materials is extracted for extracting every experiment (materials, process chains, microstructure configurations and measurements) as Python objects from the OCR text of LitXAlloy's 19 alloy papers (1,426 measurements; figures excluded), extracted materials matched to the annotated ones by the Hungarian algorithm, mean of three runs; coding agents (Claude Code, Codex, Gemini CLI); higher is better. Source: arxiv.org. Saturation forecast: Estimated already saturated. 3 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Gemini 3.1 Pro (Preview)0.98#102
2GPT-5.2 CodexHigh0.95#186
3Claude Opus 4.60.94#114

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=litxbench-coding-agents-materials · How It Works · Data refreshed daily, snapshot 2026-11-22.