ARFBench - Macro-F1: leaderboard
Time series question answering on software incident data in ARFBench (2026, Datadog): multiple-choice questions over multivariate production telemetry from real incidents, with the series shown as chart images. Multiclass macro-F1 over all 750 questions, with answers mapped to per-category semantic classes, in percent; higher is better.
Metric: Multiclass macro-F1 (%) over the 750 ARFBench questions, with answers mapped to per-category semantic answer classes, for the time series rendered as chart images (Matplotlib plots, three images for paired questions), with the series description and channel names as text; few-shot multiple-choice prompting with shuffled answer choices, temperature 0.05 (default for models without the setting), reasoning models at medium effort with 2,000 output tokens, blank or invalid answers counted wrong; higher is better. Source: arxiv.org. Saturation forecast: Around April 2028. 11 models tracked. Updated 7 Oct 2026.
Top models
| # | Model | Setting | Score | Overall rank |
|---|---|---|---|---|
| 1 | GPT-5 | 51.9 | #185 | |
| 2 | GPT-5.4 | Medium | 51.4 | #138 |
| 3 | Gemini 3 Pro | 49.6 | #146 | |
| 4 | Toto-1.0-QA-Experimental 32B (TSFM-VLM) | 48.9 | ||
| 5 | Claude Opus 4.6 | 46.7 | #114 | |
| 6 | Qwen 3 VL 32B (post-trained) | 46.6 | ||
| 7 | Qwen 3 VL 32B | 45.1 | #624 | |
| 8 | GPT-4.1 | 44 | #506 | |
| 9 | GPT-4o | 42.4 | #754 | |
| 10 | Claude Sonnet 4.5 | 37.9 | #267 | |
| 11 | Qwen 3 VL 8B | 34.7 | #927 |
No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.
Interactive version: theaggregate.ai/benchmark?slug=arfbench-macro-f1 · How It Works · Data refreshed daily, snapshot 2026-11-22.