ARFBench - Magnitude: leaderboard

Time series question answering on software incident data in ARFBench (2026, Datadog): multiple-choice questions over multivariate production telemetry from real incidents, with the series shown as chart images. Accuracy on the Magnitude questions (how far the anomaly deviates from expected behavior), in percent; higher is better.

Metric: Accuracy (%) on the 76 Magnitude questions (how far the anomaly deviates from the expected counterfactual) of ARFBench, for the time series rendered as chart images (Matplotlib plots, three images for paired questions), with the series description and channel names as text; few-shot multiple-choice prompting with shuffled answer choices, temperature 0.05 (default for models without the setting), reasoning models at medium effort with 2,000 output tokens, blank or invalid answers counted wrong; higher is better. Source: arxiv.org. Saturation forecast: Not forecast. 11 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1GPT-4.168.4#505
2GPT-565.8#178
3Toto-1.0-QA-Experimental 32B (TSFM-VLM)64.5
4Qwen 3 VL 32B (post-trained)63.2
5GPT-4o61.8#751
6Qwen 3 VL 32B59.2#622
7Claude Opus 4.657.9#113
8GPT-5.4Medium57.9#135
9Qwen 3 VL 8B57.9#917
10Gemini 3 Pro56.7#145
11Claude Sonnet 4.553.9#261

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=arfbench-magnitude · How It Works · Data refreshed daily, snapshot 2026-11-21.