REVIEW 5 cited by
WorkflowLLM: Enhancing Workflow Orchestration Capability of Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent advancements in large language models (LLMs) have driven a revolutionary paradigm shift in process automation from Robotic Process Automation to Agentic Process Automation by automating the workflow orchestration procedure based on LLMs. However, existing LLMs (even the advanced OpenAI GPT-4o) are confined to achieving satisfactory capability in workflow orchestration. To address this limitation, we present WorkflowLLM, a data-centric framework elaborately designed to enhance the capability of LLMs in workflow orchestration. It first constructs a large-scale fine-tuning dataset WorkflowBench with 106,763 samples, covering 1,503 APIs from 83 applications across 28 categories. Specifically, the construction process can be divided into three phases: (1) Data Collection: we collect real-world workflow data from Apple Shortcuts and RoutineHub, transcribing them into Python-style code. We further equip them with generated hierarchical thought via ChatGPT. (2) Query Expansion: we prompt ChatGPT to generate more task queries to enrich the diversity and complexity of workflows. (3) Workflow Generation: we leverage an annotator model trained on collected data to generate workflows for synthesized queries. Finally, we merge the synthetic samples that pass quality confirmation with the collected samples to obtain the WorkflowBench. Based on WorkflowBench, we fine-tune Llama-3.1-8B to obtain WorkflowLlama. Our experiments show that WorkflowLlama demonstrates a strong capacity to orchestrate complex workflows, while also achieving notable generalization performance on previously unseen APIs. Additionally, WorkflowBench exhibits robust zero-shot generalization capabilities on an out-of-distribution task planning dataset, T-Eval. Our data and code are available at https://github.com/OpenBMB/WorkflowLLM.
Forward citations
Cited by 5 Pith papers
-
SoK: Intent-Oriented Systematization of Multi-Turn LLM Jailbreaks
Multi-turn LLM jailbreaks succeed based on how harmful intent is organized across turns, not on interaction length, and detection should shift to session and cross-session scope.
-
Procedural Knowledge Is Not Low-Rank: Why LoRA Fails to Internalize Multi-Step Procedures
On multi-step procedural tasks, LoRA fine-tuning underperforms full fine-tuning at every rank tested because procedural knowledge requires high-rank weight updates.
-
ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution
ASPERA generates a benchmark of 250 executable assistant tasks and finds that LLMs, even with full API documentation, solve only 10 to 80 percent of them.
-
Workflow as Knowledge: Semantic Persistence for LLM-Mediated Workflows
A conceptual model proposes representing LLM workflow definitions, instances, and inference records as persistent typed objects in a shared knowledge substrate, distinguishing deterministic derive from LLM-mediated infer.
-
Context-Aware Scientific Knowledge Extraction on Linked Open Data using Large Language Models
WISE combines LLM filtering, word-overlap scoring, and adaptive stopping in a recursive tree search to extract and synthesize knowledge from linked web sources, reporting higher recall and detail than general LLM base...
Discussion (0). Sign in to comment.