ARFBench - Macro-F1: leaderboard

Time series question answering on software incident data in ARFBench (2026, Datadog): multiple-choice questions over multivariate production telemetry from real incidents, with the series shown as chart images. Multiclass macro-F1 over all 750 questions, with answers mapped to per-category semantic classes, in percent; higher is better.

Metric: Multiclass macro-F1 (%) over the 750 ARFBench questions, with answers mapped to per-category semantic answer classes, for the time series rendered as chart images (Matplotlib plots, three images for paired questions), with the series description and channel names as text; few-shot multiple-choice prompting with shuffled answer choices, temperature 0.05 (default for models without the setting), reasoning models at medium effort with 2,000 output tokens, blank or invalid answers counted wrong; higher is better. Source: arxiv.org. Saturation forecast: Around April 2028. 11 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1GPT-551.9#185
2GPT-5.4Medium51.4#138
3Gemini 3 Pro49.6#146
4Toto-1.0-QA-Experimental 32B (TSFM-VLM)48.9
5Claude Opus 4.646.7#114
6Qwen 3 VL 32B (post-trained)46.6
7Qwen 3 VL 32B45.1#624
8GPT-4.144#506
9GPT-4o42.4#754
10Claude Sonnet 4.537.9#267
11Qwen 3 VL 8B34.7#927

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=arfbench-macro-f1 · How It Works · Data refreshed daily, snapshot 2026-11-22.