SLATE API Selection - Task Success: leaderboard
Long-horizon e-commerce API selection from a thousand-tool library with simulator-verified dependencies. Simulator-verified success rate (%). SLATE v1, hierarchical e-commerce plans, wide distractor tool library. Both native decision model and matched backbone score identical candidate selections through native probability readouts. A step executes only if its prerequisites were selected correctly; task success requires every reference terminal outcome, step accuracy scores allsubsteps. Mean over three option orders; Table4 exact HuggingFace checkpoints/revisions/readout types are pinned. No LLM-judge or mere process-exit proxy.
Metric: Simulator-verified success rate (%). Source: arxiv.org. Saturation forecast: Rough model projection: around 2026. 12 models tracked.
Top models
| # | Model | Score | Overall rank |
|---|---|---|---|
| 1 | Gemma 4 12B (IT) (Benchmark Sweep Run: source=SLATE; backbone-revision=43c9d014; native-probability-readout; three-option-orders) | 5 | |
| 2 | OpenJev-Flash-9B (Benchmark Sweep Run: source=SLATE; decision-revision=b77b0fff; readout=label probabilities (merged) [28]; three-option-orders) | 4.7 | |
| 3 | AgentDecision-4B (Benchmark Sweep Run: source=SLATE; decision-revision=f0e13bab; readout=label probabilities (LoRA) [29]; three-option-orders) | 4.3 | |
| 4 | Qwen 3 4B (Benchmark Sweep Run: source=SLATE; backbone-revision=1cfa9a72; native-probability-readout; three-option-orders) | 3.3 | |
| 5 | Qwen 3.5 9B (Benchmark Sweep Run: source=SLATE; backbone-revision=c2022362; native-probability-readout; three-option-orders) | 2.7 | |
| 6 | gemma-4-E4B-it (Benchmark Sweep Run: source=SLATE; backbone-revision=ee0ef602; native-probability-readout; three-option-orders) | 1.7 | |
| 7 | Jevify-Tulu-3-8B (Benchmark Sweep Run: source=SLATE; decision-revision=3918311a; readout=label probabilities (LoRA) [30]; three-option-orders) | 1 | |
| 8 | Strands-Decider-12B (Benchmark Sweep Run: source=SLATE; decision-revision=1202634a; readout=pointer head (LoRA) [27]; three-option-orders) | 1 | |
| 9 | Tulu-3-8B-DPO (Benchmark Sweep Run: source=SLATE; backbone-revision=a7beb67e; native-probability-readout; three-option-orders) | 0.3 | |
| 10 | Jevify-SmolLM3-3B (Benchmark Sweep Run: source=SLATE; decision-revision=553e4965; readout=label probabilities (LoRA) [30]; three-option-orders) | 0.3 |
Interactive version: theaggregate.ai/benchmark?slug=slate-api-selection-task-success · How It Works · Data refreshed daily, snapshot 2026-11-06.