Inkling 2 Small Launch - PMPP-Hard - Pass@4: leaderboard
69 GPU kernel-optimization tasks; four independent attempts; 10-hour budget. Each model uses its maximum available reasoning setting, PI harness and shared compaction policy. Task/grading fixes applied; two independent LLM audits per solution and third adjudication on disagreement. AUC spans 0–10 hours (max 690 task-hours); mean and pass@4 are counts out of 69. No maximum-effort name is converted into an unsupported provider-specific setting.
Metric: Pass@4. Source: thinkingmachines.ai. Saturation forecast: Rough model projection: around 2026. 9 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Kimi K3 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 66 | |
| 2 | Claude Opus 5.5 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 64 | |
| 3 | Inkling 2 Small (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 64 | |
| 4 | GPT-6 Astra (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 63 | |
| 5 | Kimi K3.1 Pro (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 63 | |
| 6 | DeepSeek V4.1 Flash (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 62 | |
| 7 | Mistral Large 4 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 61 | |
| 8 | GLM 5.3 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 61 | |
| 9 | GLM 5.2 (Benchmark Sweep Run: source=Inkling2SmallPMPP; PI harness; max available reasoning; 4 attempts; 10h) | 61 |
Interactive version: theaggregate.ai/benchmark?slug=inkling-2-small-launch-pmpp-hard-pass-4 · How It Works · Data refreshed daily, snapshot 2026-11-06.