ISO-Bench (vLLM) - Hard Success: leaderboard
Metric: Hard success rate (%) on the 39 vLLM inference-optimization tasks: the patch beats or matches the human fix's measured speedup, whatever code it changes; each agent gets the repository at the commit before an expert's merged performance pull request, the benchmark showing the bottleneck and a 120-minute budget, and must write an optimization patch that keeps outputs identical; the patch's latency (TTFT) or throughput change is measured against the human fix (within 5 percent counts as similar); higher is better. Source: arxiv.org. Saturation forecast: Around August 2027. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Sonnet 4.5 | 56.4 | #138 |
| 2 | GPT-5 (High) | 33.3 | #91 (GPT-5) |
No result here: #3 Claude Opus 5.5, #5 GPT-6 Astra, #8 Claude Fable 5.1.
Interactive version: theaggregate.ai/benchmark?slug=iso-bench-vllm-hard-success · How It Works · Data refreshed daily, snapshot 2026-10-11.