Chat2Workflow - Pass Rate: leaderboard

Chat2Workflow (2026) tests whether language models build executable Dify workflows from multi-turn natural-language requests drawn from real business workflows, run against test cases. Pass rate in percent, the share of generated workflows that pass format, node and structure checks; higher is better.

Metric: Pass rate (%) on Chat2Workflow (79 instructions in 27 multi-turn tasks, pooled over six domains): share of generated workflows that pass the format, node-selection and structure checks (valid JSON converted to a Dify YAML workflow with every mandatory node), judged by GPT-5.1; higher is better. Source: arxiv.org. Saturation forecast: Around February 2027. 15 models tracked. Updated 7 Oct 2026.

Top models

#ModelScoreOverall rank
1Gemini 3 Pro (Preview)80.17#123
2Claude Sonnet 4.571.31#267
3GLM-4.769.2#383
4GPT-5.267.51#190
5GLM-4.662.45#496
6Kimi K2 (Thinking)57.81
7DeepSeek V3.151.48#511
8GPT-5.147.26#248
9DeepSeek V3.243.46#425
10Qwen 3 Coder 480B A35B Instruct42.19#655
11Qwen 3 235B A22B40.93#633
12Qwen 3 32B36.71#983
13Kimi K236.29#448
14Qwen 3 14B27.43#1284
15Qwen 3 8B13.92#1890

No result here: #2 Claude Fable 5.2, #3 Gemini 4 Neon, #4 GPT-6.2 Astra.

Interactive version: theaggregate.ai/benchmark?slug=chat2workflow-pass-rate · How It Works · Data refreshed daily, snapshot 2026-11-22.