Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language

Buqiang Xu; Guozhou Zheng; Ningyu Zhang; Shuofei Qiao; Yijun Wang; Yi Zhong; Zifei Shan

Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language

T0 review · 3 major / 2 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Large language models still cannot reliably turn natural language into correct, stable, and deployable visual workflows, even with an agentic baseline.

desk verdict Useful industrial NL-to-workflow benchmark framing with a modest agentic gain, but abstract-only so the deployability claim is still uncheckable. read the letter →

arxiv 2604.19667 v2 pith:WOKFU2YA submitted 2026-04-21 cs.CL cs.AIcs.CVcs.LGcs.MA

Yi Zhong , Buqiang Xu , Yijun Wang , Zifei Shan , Shuofei Qiao , Guozhou Zheng , Ningyu Zhang This is my paper · ORCID

classification cs.CLcs.AIcs.CVcs.LGcs.MA

keywords executablevisualworkflowsnaturallanguagetoworkflowChat2Workflowagenticbaselineautomationlargemodelsmulti-roundinteractionindustrialdeployment

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Chat2Workflow, a benchmark built from real-world business workflows so that any generated workflow can be transformed and deployed directly onto industrial platforms such as Dify and Coze. It shows that state-of-the-art language models often capture high-level intent yet fail to produce correct, stable, and executable visual workflows when requirements are complex or evolve across multi-round interaction. An agentic baseline proposed by the authors raises resolve rate by as much as 6.05 percent, but a substantial practical gap remains. The work therefore positions the benchmark as a foundation for measuring and advancing industrial-grade automation of workflow construction that today is still done almost entirely by hand.

What carries the argument

Chat2Workflow—the benchmark of real-world business workflow instances, each constructed so a generated workflow can be transformed and directly deployed to platforms such as Dify and Coze—together with the agentic baseline that iteratively refines the workflow under multi-round dialogue.

What would settle it

Take a held-out set of live production workflows from Dify or Coze, generate candidates with the same models and agent, deploy them without manual repair, and check whether end-to-end execution success still matches the reported resolve rates.

Watch

Extended reading notes

Core claim

State-of-the-art language models can often capture high-level intent from natural language descriptions of business processes, yet they struggle to generate correct, stable, and executable visual workflows—especially under complex and evolving requirements. The authors formalize this gap with Chat2Workflow, a benchmark whose instances are designed for direct deployment on platforms such as Dify and Coze, and show that an agentic baseline improves resolve rate by up to 6.05 percent while leaving a large real-world gap.

Load-bearing premise

That the curated real-world business instances and the transform-and-deploy pipeline to platforms such as Dify and Coze truly measure industrial executability, so resolve rate reflects deployable correctness rather than benchmark-specific formatting success.

Editorial extensions

If this is right

Resolve-rate numbers on Chat2Workflow become a concrete yardstick for industrial workflow automation progress.
Future models or agents can be scored on whether they close the remaining gap before any auto-generated workflow is deployed.
The transform-and-deploy pipeline lets generated outputs be tested on live platforms rather than only in simulation.
Multi-round requirement evolution is treated as a first-class evaluation axis rather than a side concern.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Closing the remaining gap may require tighter coupling of formal workflow verification with language-model generation rather than pure agentic prompting.
Success on this benchmark would cut the manual engineering cost of building production visual workflows.
Similar deployability-first benchmarks could be built for other visual or low-code programming domains.
The modest 6.05 percent gain suggests iterative agent scaffolding alone is insufficient and that new inductive biases may be needed.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing to know: this is a new evaluation artifact for turning natural language into executable visual workflows that claim direct deployability to Dify/Coze, plus a small agentic baseline lift (up to 6.05% resolve rate) and an honest admission that a large gap remains. That is the contribution.

What is actually new is the task framing and the industrial grounding. Manual workflow engineering is expensive and real; a benchmark built from real business workflows, with multi-round evolving requirements and a transform-and-deploy path to actual platforms, is a sensible measurement instrument for LLM agents and low-code automation. The abstract does not overclaim a paradigm shift. It reports that SOTA models often get high-level intent but fail on correctness, stability, and executability, and that their agentic baseline helps only modestly. That posture is useful. Shipping code is a plus.

The soft spots are exactly what you would expect from abstract-only. We cannot see dataset size, splits, metric definitions, how multi-round evolution is simulated, what “resolve” scores (syntax vs control flow vs runtime), or the fidelity of the transform pipeline to Dify/Coze schemas. The stress-test concern is fair as a risk: if resolve mostly rewards benchmark-specific formatting rather than end-to-end platform executability, both the gap and the 6.05% gain lose industrial force. That is not proven failure; it is missing evidence. Circularity looks ordinary for a benchmark paper, not load-bearing self-definition. Novelty and significance sit in the mid range—solid applied evaluation, not first-principles work.

Who it is for: people building LLM agents for enterprise automation, low-code platforms, and anyone who needs a harder, more deployable workflow generation test than pure text-to-code. A serious referee should see the full paper, data, and harness. I would not desk-reject on the abstract; I would send it out and demand the metric and pipeline details. Bring it to reading group only if someone is actively working on agentic workflow tools; otherwise wait for the full artifact. I would not cite it yet from the abstract alone, but I would track the repo.

Referee Report

3 major / 2 minor

Summary. The manuscript introduces Chat2Workflow, a benchmark for multi-round natural-language generation of executable visual workflows, constructed from real-world business workflows and designed so that outputs can be transformed and deployed to industrial platforms such as Dify and Coze. It reports that state-of-the-art language models often capture high-level intent but struggle to produce correct, stable, and executable workflows under complex and evolving requirements, and proposes an agentic baseline that improves resolve rate by up to 6.05%, while arguing that a substantial real-world gap remains. Code is released at a public repository.

Significance. If the benchmark construction, deployability pipeline, and evaluation protocol hold under full scrutiny, Chat2Workflow would be a practically relevant contribution for industrial automation of visual workflow authoring—an area where manual multi-round engineering is costly and error-prone. Explicit framing around evolving requirements, platform deployability (Dify/Coze), and a public code release are strengths. The reported modest agentic gain alongside a large residual gap would usefully position the resource as a foundation for subsequent work rather than a solved task.

major comments (3)

[Abstract] The abstract’s central industrial claim—that generated workflows can be transformed and directly deployed to platforms such as Dify and Coze—depends on an uninspectable transform-and-deploy pipeline. Without a full-paper specification of transformation rules, platform schema constraints, and how syntax vs. control-flow vs. runtime failures are scored, it is not possible to verify that “resolve rate” measures true deployable correctness rather than benchmark-specific formatting success. This premise is load-bearing for both the reported SOTA gap and the 6.05% gain.
[Abstract] Resolve rate is presented as the primary success measure for multi-round NL-to-workflow generation under evolving requirements, but the abstract does not define how multi-round requirement evolution is simulated, how partial credit is assigned, or what constitutes a resolved instance. Without these definitions (and associated dataset size, splits, and error bars), the 6.05% gain and the “large real-world gap” claim cannot be assessed for statistical or practical significance.
[Abstract] The abstract asserts construction from a “large collection of real-world business workflows” and SOTA struggle under “complex and evolving requirements,” yet supplies no instance counts, complexity stratification, baseline inventory, or ablation of the agentic method. These elements are load-bearing for the claim that the residual gap is industrial rather than artifactual; they must appear with reproducible detail in the full manuscript.

minor comments (2)

[Abstract] The abstract is generally clear, but “resolve rate” should be briefly glossed on first use (e.g., end-to-end deployable success under the stated transform pipeline) so readers can interpret the 6.05% figure without the full paper.
[Abstract] Naming the concrete SOTA models and the agentic baseline architecture in the abstract would help readers gauge the strength of the comparison before consulting the full text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical benchmark-plus-baseline paper with no derivation that reduces predictions to inputs by construction.

full rationale

Chat2Workflow is a benchmark introduction and empirical evaluation paper, not a first-principles derivation. The abstract claims construction of a real-world workflow benchmark, an agentic baseline, and measured resolve-rate gains (up to 6.05%) against SOTA models, with residual industrial gap. There are no equations, fitted parameters renamed as predictions, uniqueness theorems, ansatzes smuggled via self-citation, or self-definitional reductions of the form X derives Y where X is defined via Y. Ordinary benchmark design (authors curate instances and define a resolve metric tied to transform-and-deploy to Dify/Coze) is not circularity under the enumerated patterns; it is standard evaluation practice and does not force the reported model failures or baseline gains by construction. With only the abstract available, no load-bearing self-citation chain or definitional loop is quotable. Score 0 is the honest finding.

Assumptions & free parameters 0 free parameters · 3 assumptions · 2 invented entities

Abstract-only audit. The central claims rest on domain assumptions about what counts as an executable industrial workflow and on the authors' evaluation construct (resolve rate under a transform-to-Dify/Coze pipeline). No free numeric parameters are disclosed in the abstract. The main invented entity is the benchmark itself; independent evidence would be public data plus third-party re-runs on the named platforms.

assumptions (3)

domain assumption Generated workflows that pass the paper's transform-and-deploy checks on platforms such as Dify and Coze are treated as industrially executable.
Abstract states each instance is designed so generated workflows can be transformed and directly deployed; this equates platform acceptance with the target property of executability.
ad hoc to paper Resolve rate is a valid primary measure of success for multi-round NL-to-workflow generation under evolving requirements.
The abstract reports up to 6.05% resolve-rate gains without defining the metric in the abstract; the claim strength depends on that metric choice.
domain assumption The curated collection of real-world business workflows is representative of industrial workflow construction difficulty.
Benchmark validity for 'industrial-grade automation' depends on coverage and difficulty of the source workflows, which the abstract asserts but does not detail.

invented entities (2)

Chat2Workflow benchmark
purpose: Provide instances and evaluation for generating executable visual workflows from natural language with deployability to industrial platforms.
The benchmark is the paper's primary artifact. Independent evidence would require public instances, evaluation harness, and third-party deployment checks; abstract claims code availability but full independent handles are not verifiable here.
Agentic baseline for Chat2Workflow
purpose: Improve multi-round generation of executable workflows relative to direct LLM generation.
Presented as a robust baseline yielding up to 6.05% resolve-rate gains; without methods details it is an author-defined system rather than an independently established entity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language." pith.science (2026). https://pith.science/paper/WOKFU2YA

@misc{pith2026260419667,
  author       = {Pith},
  title        = {Pith review of: Chat2Workflow: A Benchmark for Generating Executable Visual Workflows with Natural Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WOKFU2YA}},
  note         = {Machine review of arXiv:2604.19667}
}

read the original abstract

At present, executable visual workflows have emerged as a mainstream paradigm in real-world industrial deployments, offering strong reliability and controllability. However, in current practice, such workflows are almost entirely constructed through manual engineering: developers must carefully design workflows, write prompts for each step, and repeatedly revise the logic as requirements evolve -- making development costly, time-consuming, and error-prone. To study whether large language models can automate this multi-round interaction process, we introduce Chat2Workflow, a benchmark for generating executable visual workflows directly from natural language, and propose a robust agentic baseline to improve performance. The benchmark is built from a large collection of real-world business workflows, with each instance designed so that the generated workflow can be transformed and directly deployed to practical workflow platforms such as Dify and Coze. Experimental results show that while state-of-the-art language models can often capture high-level intent, they struggle to generate correct, stable, and executable workflows, especially given complex and evolving requirements. Although our agentic baseline yields up to 6.05% resolve rate gains, the remaining real-world gap positions Chat2Workflow as a foundation for advancing industrial-grade automation. Code is available at https://github.com/zjunlp/Chat2Workflow.