AISI Cyber CTFs (inference-scaling export) @ 500K tokens: leaderboard
UK AI Security Institute capture-the-flag suite from How Inference Compute Shapes Frontier LLM Evaluation (arXiv 2606.17930; AISI export of 2026-09-16 in the Every Eval Ever datastore): 109 CTF tasks (106-108 evaluated per model), 5 independent trajectories each, ReAct-style Inspect agent with context compaction. Score: cumulative flag-capture rate within the first 500K total tokens of a trajectory, mean over tasks (%). The export counts more tasks than the 71 the paper's text gives; the older chart-read AISI Cyber CTF board is a separate measurement. Higher is better. Source: https://arxiv.org/abs/2606.17930
Metric: Cumulative flag-capture rate within 500K total tokens (%; mean over tasks of the share of 5 trajectories that captured the flag). Source: arxiv.org. Saturation forecast: Around August 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.3 Codex | 40.75 | #67 |
| 2 | Claude Opus 4.6 | 40 | #62 |
| 3 | Claude Opus 4.5 (20251101) | 36.64 | #73 |
| 4 | GPT-5.4 (2026-03-05) | 36.3 | #58 |
| 5 | GPT-5.2 Codex | 33.33 | #93 |
| 6 | GPT-5 | 31.45 | #98 |
| 7 | Claude Sonnet 4.5 | 27.99 | #142 |
| 8 | O3 (2025-04-16) | 26.89 | #120 |
| 9 | GPT-5.1 Codex | 26.6 | #158 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=aisi-cyber-ctfs-inference-scaling-export-500k-tokens · How It Works · Data refreshed daily, snapshot 2026-10-19.