starcoder: benchmark results
BigCode's original StarCoder, a 15.5B code model released under the OpenRAIL-M license; the linked model card provides no further training or architecture details here. Provider: BigCode. Released 2023-05-04. Access: Open.
Aggregate Rating 1211 ± 11, rank #1577 of 1656 rated models, from 116 benchmark results.
Strongest benchmark results
| Benchmark | Score | Metric | Percentile |
|---|---|---|---|
| ToolBench - WebShop Long | 2.6 | Task Score | 97.7 |
| EffiBench - NET | 2.84 | Normalized Execution Time | 81.7 |
| ToolBench - Google Sheets | 48 | Task Score | 74.4 |
| ToolBench - Open Weather | 91 | Task Score | 74.4 |
| ToolBench - Home Search | 82 | Task Score | 68.6 |
| MMLU-by-task - Econometrics | 30.7 | Accuracy (%) | 68.2 |
| ToolBench - Tabletop | 21.9 | Task Score | 67.4 |
| CanItEdit (Descriptive) | 37.1 | pass@1 (%; share of the 105 hand-written Python editing prob | 62.5 |
| MMLU-by-task - College Mathematics | 33 | Accuracy (%) | 61.7 |
| ToolBench - VirtualHome | 19.4 | Task Score | 60.5 |
| ToolBench Leaderboard | 49.75 | Average Task Score | 60.5 |
| EffiBench - NMU | 1.91 | Normalized Memory Usage | 58.5 |
Interactive version: theaggregate.ai/model?slug=starcoder · How It Works · Data refreshed daily, snapshot 2026-10-20.