FLOWSTEP Raw Action Ablation - Top5: leaderboard
FLOWSTEP v1 evaluates next-step prediction for GitHub Actions. It uses a repository-level 8:1:1 split and 13630 full test steps. Abs and Raw are distinct abstraction conditions. Generation uses temperature 0 and one candidate; calibrated script-model selection uses 1000 random test Script instances. The full pipeline combines RoBERTa type/action heads with Sonnet 4.5 one-shot prompting or a trained T5-small; the weights are trained on distinct task sets. Judge validation uses 350 incorrect scripts, 2 human annotators, and 3 ordinal functional classes. Signed agreement and correlation measure judgment quality, not generation accuracy. Native Table 5 row labels inconsistently repeat the without-contrastive wording. Exact row indices and labels are retained without inventing a corrected training recipe. These arms cannot be aliased to a backbone. Metric: Top-k accuracy (%).
Metric: Top-k accuracy (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 3 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | FLOWSTEP Raw RoBERTa source arm1 [Finetune w/ o Contrastive Learning + Classification head] | 94.17 | |
| 2 | FLOWSTEP Raw RoBERTa source arm2 [Finetune w/ o Contrastive Learning] | 91.26 | |
| 3 | FLOWSTEP Raw RoBERTa source arm3 [Finetune w/o Contrastive Learning] | 52.49 |
Interactive version: theaggregate.ai/benchmark?slug=flowstep-raw-action-ablation-top5 · How It Works · Data refreshed daily, snapshot 2026-10-28.