ARFBench - Categorization: leaderboard

Time series question answering on software incident data in ARFBench (2026, Datadog): multiple-choice questions over multivariate production telemetry from real incidents, with the series shown as chart images. Accuracy on the Categorization questions (which type of anomaly occurs, such as a level shift or transient spike), in percent; higher is better.

Metric: Accuracy (%) on the 104 Categorization questions (level shift, transient spike, seasonality, variance or trend change) of ARFBench, for the time series rendered as chart images (Matplotlib plots, three images for paired questions), with the series description and channel names as text; few-shot multiple-choice prompting with shuffled answer choices, temperature 0.05 (default for models without the setting), reasoning models at medium effort with 2,000 output tokens, blank or invalid answers counted wrong; higher is better. Source: arxiv.org. Saturation forecast: Around December 2027. 11 models tracked. Updated 7 Oct 2026.

Top models

#ModelSettingScoreOverall rank
1Toto-1.0-QA-Experimental 32B (TSFM-VLM)66.3
2Claude Opus 4.663.5#113
3GPT-559.6#178
4GPT-4.156.7#505
5GPT-5.4Medium56.7#135
6Gemini 3 Pro54.8#145
7Claude Sonnet 4.553.8#261
8Qwen 3 VL 32B (post-trained)52.9
9GPT-4o51.9#751
10Qwen 3 VL 32B50#622
11Qwen 3 VL 8B45.2#917

No result here: #1 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=arfbench-categorization · How It Works · Data refreshed daily, snapshot 2026-11-21.