BAGEN - Sokoban - F1@All: leaderboard
BAGEN (Budget-Aware Agent, arXiv 2606.00198) on Sokoban puzzle rollouts, remaining cost in tokens: a model replays every prefix of its own logged agent rollouts and must say whether the task can still finish within budget. Unlike task-success or cost leaderboards, "it asks the agent itself to predict its remaining" cost mid-task. Score: all-turn feasibility macro-F1 (%) over feasible and impossible prefixes. Each model is scored on its own rollouts, so item sets differ per model; settings from the paper's Table 5 (Claude effort low, GPT-5.2 reasoning none, Gemini and Qwen thinking low). Higher is better. Source: https://arxiv.org/abs/2606.00198
Metric: All-turn feasibility macro-F1 (%). Source: arxiv.org. Saturation forecast: Around February 2027. 5 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.1 Pro (Preview) (Low) | 61.9 | #64 (Gemini 3.1 Pro (Preview)) |
| 2 | Claude Sonnet 4.6 (Low) | 53.6 | #94 (Claude Sonnet 4.6) |
| 3 | Claude Opus 4.7 (Low) | 45.6 | #51 (Claude Opus 4.7) |
| 4 | GPT-5.2 (Non-reasoning) | 40.6 | #115 (GPT-5.2) |
| 5 | Qwen 3 235B A22B 2507 Instruct | 12.6 | #315 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=bagen-sokoban-f1-all · How It Works · Data refreshed daily, snapshot 2026-11-05.