CUDAHercules - Single-Kernel Mean Speed (Tool-Augmented): leaderboard
CUDAHercules (Li et al., 2026) asks models to write CUDA against expert reference implementations, with correctness checked by the validator of each task over several problem sizes or randomized inputs. This board is mean expert-relative speed on the 20 architecture-general single-kernel tasks, under tool-augmented search, up to 20 rounds with build logs, hardware specifications and GPU profiling tools: the expert runtime divided by the generated runtime, averaged over all tasks with failed tasks counting zero, so 1 matches the expert code; higher is better.
Metric: Mean expert-relative speed (the expert reference runtime divided by the generated code's runtime, averaged over the 20 architecture-general Class 1 single-kernel tasks, a task whose code fails the validator counting 0, so 1 matches the expert; tool-augmented: up to 20 rounds with build logs, hardware specifications and GPU profiling tools (CUDAForge)). Source: arxiv.org. Saturation forecast: Around July 2027. 4 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Claude Opus 4.6 | 0.4 | #102 |
| 2 | GPT-5.4 | 0.32 | #123 |
| 3 | Qwen 3.5 122B A10B | 0.17 | #331 |
No result here: #1 Claude Fable 5.2, #2 Gemini 4 Carbon, #5 Claude Opus 5.5.
Interactive version: theaggregate.ai/benchmark?slug=cudahercules-single-kernel-mean-speed-tool-augmented · How It Works · Data refreshed daily, snapshot 2026-11-13.