Inkling 2 Small Launch - PMPP-Hard - AUC (task-hours): leaderboard
69 GPU kernel-optimization tasks; four independent attempts; 10-hour budget. Each model uses its maximum available reasoning setting, PI harness and shared compaction policy. Task/grading fixes applied; two independent LLM audits per solution and third adjudication on disagreement. AUC spans 0–10 hours (max 690 task-hours); mean and pass@4 are counts out of 69. No maximum-effort name is converted into an unsupported provider-specific setting.
Metric: AUC (task-hours). Source: thinkingmachines.ai. Saturation forecast: Rough model projection: around 2026. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | GPT-6 Astra (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 616.39 | |
| 2 | Kimi K3.1 Pro (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 583.86 | |
| 3 | Claude Opus 5.5 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 499.65 | |
| 4 | DeepSeek V4.1 Flash (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 497.36 | |
| 5 | Inkling 2 Small (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 494.27 | |
| 6 | Mistral Large 4 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 478.39 | |
| 7 | Kimi K3 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 453.58 | |
| 8 | GLM 5.3 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 430.96 | |
| 9 | GLM 5.2 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 389.12 |
Interactive version: theaggregate.ai/benchmark?slug=inkling-2-small-launch-pmpp-hard-auc-task-hours · How It Works · Data refreshed daily, snapshot 2026-11-06.