DuMateBench: leaderboard
DuMateBench (2026) rebuilds 200 multi-tool workflow tasks from privacy-screened sessions of a production agent platform and runs agent harnesses with different base models in containers with missing dependencies, network and tool faults and distractor files. Final score in percent: 0.3 times the deterministic checklist pass rate plus 0.7 times a Gemini judge rubric score; higher is better.
Metric: Final score (%; 0.3 x partial pass rate + 0.7 x Gemini-3.1-Pro-Preview judge score of the produced artifacts against task-specific rubrics; 200 tasks reconstructed from anonymized, privacy-screened DuMate production sessions, each in an isolated Docker container with insufficient, unstable and noisy environment conditions; each agent keeps its native control loop and runtime settings). Source: arxiv.org. Saturation forecast: Around February 2027. 20 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | DeepSeek V4 Pro | 82.23 | #188 |
| 2 | Claude Opus 4.8 | 81.81 | #56 |
| 3 | GPT-5.5 | 80.44 | #49 |
| 4 | GLM-5.2 | 77.61 | #138 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=dumatebench · How It Works · Data refreshed daily, snapshot 2026-11-13.