Δτ-Bench - Cost SPE: leaderboard
Δτ-Bench (ZJUNLP, arXiv 2610.34238) keeps each user goal fixed while the tool ecosystem moves through six versions in which "interfaces change, functions split, workflows merge, and alternative tools become available". Unlike tau-bench, whose tools stay fixed, the agent must recover from disrupted workflows and find newly cheaper paths. 397 test tasks; reasoning effort high for models that expose it. Codex-GPT-5.5 (GPT-5.5 inside the Codex harness) is not included. Score: Cost SPE (%), success-weighted path efficiency by execution cost: the reference path's cost over the agent's, capped at 1, averaged over all tasks with failures scoring 0. Higher is better. Source: https://arxiv.org/abs/2610.34238
Metric: Cost success-weighted path efficiency (%). Source: arxiv.org. Saturation forecast: Around 2032. 8 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemini 3.6 Flash | 25.88 | #57 |
| 2 | GPT-5.5 | 23.42 | #33 |
| 3 | Claude Sonnet 5 | 20.6 | #65 |
| 4 | GLM-5.2 | 19.84 | #80 |
| 5 | Kimi K3 | 17.83 | #35 |
| 6 | GPT-OSS-120B | 17.16 | #346 |
| 7 | Qwen 3.8 Max | 16.84 | #34 |
| 8 | DeepSeek V4 Pro | 14.31 | #112 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=bench-cost-spe · How It Works · Data refreshed daily, snapshot 2026-11-05.