BAGEN - Warehouse - F1@All: leaderboard
BAGEN (Budget-Aware Agent, arXiv 2606.00198) on a warehouse-operations simulation built from real enterprise data, with three coupled budgets: time (weeks), warehouse occupancy (item-weeks) and cumulative cost (USD). The agent manages inventory over 24 weeks (12 turns); at every prefix of its own logged rollouts a model must say whether a target cash level can still be reached within all three budgets. As the task is continuous cash maximization, it is scored on challenge-conditioned probes balanced 50/50 between reachable and unreachable targets. Unlike task-success or cost leaderboards, "it asks the agent itself to predict its remaining" budget mid-task. Score: all-turn feasibility macro-F1 (%) over reachable and unreachable probes. Each model is scored on its own rollouts, so item sets differ per model; settings from the paper's Table 5 (Claude effort low, GPT-5.2 reasoning none, Gemini and Qwen thinking low). Higher is better. Source: https://arxiv.org/abs/2606.00198
Metric: All-turn feasibility macro-F1 (%). Source: arxiv.org. Saturation forecast: Around April 2027. 5 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) (Low) | 67 | #64 (Gemini 3.1 Pro (Preview)) |
| 2 | Claude Sonnet 4.6 (Low) | 64.9 | #94 (Claude Sonnet 4.6) |
| 3 | GPT-5.2 (Non-reasoning) | 63.4 | #115 (GPT-5.2) |
| 4 | Claude Opus 4.7 (Low) | 63.2 | #51 (Claude Opus 4.7) |
| 5 | Qwen 3 235B A22B 2507 Instruct | 60.8 | #315 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=bagen-warehouse-f1-all · How It Works · Data refreshed daily, snapshot 2026-11-05.