FLOWSTEP Abstracted Calibration Zero-Shot - BLEU-4: leaderboard
FLOWSTEP v1 evaluates next-step prediction for GitHub Actions. It uses a repository-level 8:1:1 split and 13630 full test steps. Abs and Raw are distinct abstraction conditions. Generation uses temperature 0 and one candidate; calibrated script-model selection uses 1000 random test Script instances. The full pipeline combines RoBERTa type/action heads with Sonnet 4.5 one-shot prompting or a trained T5-small; the weights are trained on distinct task sets. Judge validation uses 350 incorrect scripts, 2 human annotators, and 3 ordinal functional classes. Signed agreement and correlation measure judgment quality, not generation accuracy. The unchanged T5/GH-WCOM one-shot reference rows are included only once, in the zero-shot population. Metric: BLEU-4 (%).
Metric: BLEU-4 (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 7 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | FLOWSTEP Abstracted Claude-Sonnet-4.5 Zero-Shot | 31.03 | |
| 2 | FLOWSTEP Abstracted Claude-Haiku-4.5 Zero-Shot | 29.4 | |
| 3 | FLOWSTEP Abstracted DeepSeek-V3.2-Exp Zero-Shot | 29.23 | |
| 4 | FLOWSTEP Abstracted GPT-4.1 Zero-Shot | 23.7 | |
| 5 | FLOWSTEP Abstracted GPT-3.5-turbo Zero-Shot | 13 | |
| 6 | FLOWSTEP Abstracted GH-WCOM trained baseline | 11.22 | |
| 7 | FLOWSTEP Abstracted T5-small trained baseline | 11.1 |
Interactive version: theaggregate.ai/benchmark?slug=flowstep-abstracted-calibration-zero-shot-bleu-4 · How It Works · Data refreshed daily, snapshot 2026-10-28.