starcoder2-15B: benchmark results
BigCode's StarCoder2-15B, a 15B code model trained on 600+ programming languages from The Stack v2 using Fill-in-the-Middle over 4+ trillion tokens, not an instruction-following model. Provider: BigCode. Released 2024-02-28. Access: Open.
Aggregate Rating 1293 ± 16, rank #1495 of 1656 rated models, from 44 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| EffiBench - NET | 2.59 | Normalized Execution Time | 100 |
| EffiBench - NMU | 1.71 | Normalized Memory Usage | 100 |
| ShaderMatch | 18.78 | Clone Match Rate (%) | 100 |
| CanItEdit (Descriptive) | 41.95 | pass@1 (%; share of the 105 hand-written Python editing prob | 70.8 |
| CRUXEval | 47.1 | Output Prediction pass@1 (%) | 66.7 |
| CanItEdit (Lazy) | 31.48 | pass@1 (%; share of the 105 hand-written Python editing prob | 66.7 |
| DomainCodeBench | 88.96 | Composite Score (%) | 66.7 |
| Open LLM Leaderboard v1 - GSM8K | 52.24 | Accuracy (%) (5-shot) | 64.1 |
| GSM8K | 57.7 | Accuracy (%) | 61.3 |
| Big Code Memorization - HumanEval-ET pass@1 | 40.24 | HumanEval-ET pass@1 (%) | 52.9 |
| EvoEval Tool Use | 48 | Pass@1 (%) | 51 |
| BigCode Models Leaderboard | 44.1 | HumanEval Python Pass@1 (%) | 49.2 |
Interactive version: theaggregate.ai/model?slug=starcoder2-15b · How It Works · Data refreshed daily, snapshot 2026-10-20.