FWBench: leaderboard
Forecast Workflow Bench (FWBench; Nagashima, 2026) tests language-model agents that buy time-series forecasts under a credit budget and submit hourly capacity plans for 1,251 electricity-demand and cycle-hire cases, scored against observed demand. The decision score averages, per case, the normalized excess loss over the hindsight-optimal plan and the share of the forecast budget spent, with invalid plans penalized; lower is better, and 0 is the hindsight optimum bought without forecasts.
Metric: Decision score S (lower is better; 0 is the hindsight-optimal plan bought with no forecast credits; per case S is half the sum of the excess decision loss over the perfect-demand plan, scaled by the hour-of-day policy's mean excess loss, and the forecast credits spent over the budget of 113,726; invalid plans take a loss penalty above every feasible loss; averaged by series and lead and weighted over electricity seasons and cycle hire; 1,251 cases: 1,091 EIA-930 electricity and 160 TfL cycle-hire, 12 tool calls per case, forecasts from Chronos-Bolt Tiny, Chronos-2 and TimesFM 2.5). Source: arxiv.org. Saturation forecast: Around April 2028. 10 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-6 Astra (High) | 0.4 | #6 (GPT-6 Astra) |
| 2 | GPT-5.4 Nano (High) | 2.64 | #633 (GPT-5.4 Nano) |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=fwbench · How It Works · Data refreshed daily, snapshot 2026-11-12.