AISI Cyber CTFs (inference-scaling export) @ 50M tokens: leaderboard
UK AI Security Institute capture-the-flag suite from How Inference Compute Shapes Frontier LLM Evaluation (arXiv 2606.17930; AISI export of 2026-09-16 in the Every Eval Ever datastore): 109 CTF tasks (106-108 evaluated per model), 5 independent trajectories each, ReAct-style Inspect agent with context compaction and a 50M-token cap (100M for Claude Opus 4.5). Score: cumulative flag-capture rate within 50M total tokens, mean over tasks (%). The export counts more tasks than the 71 the paper's text gives; the older chart-read AISI Cyber CTF board is a separate measurement. Higher is better. Source: https://arxiv.org/abs/2606.17930
Metric: Cumulative flag-capture rate within 50M total tokens (%; mean over tasks of the share of 5 trajectories that captured the flag). Source: arxiv.org. Saturation forecast: Around January 2027. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-5.4 (2026-03-05) | 84.81 | #58 |
| 2 | GPT-5.3 Codex | 83.36 | #67 |
| 3 | Claude Opus 4.6 | 81.12 | #62 |
| 4 | GPT-5.2 Codex | 73.83 | #93 |
| 5 | Claude Opus 4.5 (20251101) | 72.52 | #73 |
| 6 | GPT-5.1 Codex | 65.28 | #158 |
| 7 | GPT-5 | 55.97 | #98 |
| 8 | Claude Sonnet 4.5 | 55.19 | #142 |
| 9 | O3 (2025-04-16) | 51.89 | #120 |
No result here: #3 Claude Opus 5.5, #6 GPT-6 Astra, #7 GPT-6.1 Sol.
Interactive version: theaggregate.ai/benchmark?slug=aisi-cyber-ctfs-inference-scaling-export-50m-tokens · How It Works · Data refreshed daily, snapshot 2026-10-19.