Pith. sign in

REVIEW 4 major objections 6 minor 62 references

Routine: A Structural Planning Framework for LLM Agent System in Enterprise

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A structured planning script lifts an enterprise HR agent's multi-step tool-calling accuracy from 41.1% to 96.3%.

desk verdict Routine is a useful engineering recipe, but the headline gains mostly reflect handing the model the tool names, not better planning. read the letter →

arxiv 2507.14447 v2 pith:CKXMOHIZ submitted 2025-07-19 cs.AI cs.CL

Guancheng Zeng , Xueyi Chen , Jiawang Hu , Shaohua Qi , Yaxuan Mao , Zhantao Wang , Yifan Nie , Shuang Li
show 8 more authors
Qiuyang Feng Pengxu Qiu Yujia Wang Wenqiang Han Linyan Huang Gang Li Jingjing Mo Haowen Hu
This is my paper · ORCID
classification cs.AIcs.CL
keywords RoutineLLMagentsmulti-steptoolcallingenterpriseagentsystemsinstructionfollowingplanningframeworkknowledgedistillationselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the reason enterprise LLM agents fail at multi-step tool calling is not raw model capability but plan representation: when a plan is written as a Routine, a structured script with numbered steps, each carrying a name, description, and an assigned tool, the execution model no longer has to infer the workflow. In an HR agent scenario with 25 available tools, giving GPT-4o such a Routine raised its overall per-step accuracy from 41.1% to 96.3%, and gave Qwen3-14B a lift from 32.6% to 83.3%. A reader should care because this converts an expensive, unreliable planning problem into a cheaper instruction-following problem, and the paper shows the same Routine can be used to distill scenario-specific training data that lets a small model reach 95.5%, close to GPT-4o's performance.

What carries the argument

The load-bearing object is the Routine itself: a numbered natural-language script in which every step names its purpose, describes what to do, and designates exactly one tool, with branches represented as 'Branch X-n Step i' entries so the execution model can track where it is in the workflow. The complete system wraps the Routine in a system prompt together with a variable-memory dictionary and a standardized tool interface, and a separate procedure-memory mechanism retrieves the one relevant Routine for a query. The routine converts planning into constrained execution: tool selection becomes reading a designated tool name, parameter passing is mediated by short keys that the memory module expands before the tool call, and the final output is generated by a dedicated summarization tool rather than free text from the execution model.

What would settle it

Re-run the same HR scenario end-to-end without splitting traces: let the agent decide branch conditions and the next step from its own prior tool results, and score each of the 200 queries as a full success only if every tool call and parameter in the whole trajectory is correct. If overall per-step accuracy stays near 96% while full-task success drops substantially, the gain is largely an artifact of oracle step context rather than a real end-to-end capability.

Watch

Extended reading notes

Core claim

The central claim is that a Routine—a structured, well-formatted plan composed of step numbers, step names, step descriptions, and explicit tool assignments, plus optional input/output parameter descriptions and branch notation—acts as a stable intermediate representation between a planning model and an execution model. Supplying this Routine in the system prompt turns each tool call from a reasoning and search problem into a direct 'use this tool at this step' instruction. The paper demonstrates the claim in a deployed HR agent scenario: with Routine guidance, GPT-4o's overall accuracy (structure, tool selection, and parameters all correct) rose from 41.1% to 96.3%, and Qwen3-14B rose from 32.6% to 83.3%. Fine-tuning Qwen3-14B on a generic Routine-following dataset improved its routine-guided accuracy to 88.2%, while fine-tuning on 537 scenario-specific tool-calling samples distilled via Routine brought it to 95.5%, approaching GPT-4o's 96.3%.

Load-bearing premise

The accuracy claim depends on the test being a fair measure of real task success: each of the 1,148 samples is a per-step slice distilled from GPT-4o traces under the same Routine format being tested, each slice is given the correct preceding tool-call history, and free-text parameters are checked for existence and type rather than exact content.

Editorial extensions

If this is right

  • Providing a Routine with explicit tool names is the dominant driver of the accuracy gain; removing tool names drops accuracy by roughly 5 to 15 points for most tested models, while adding I/O parameter descriptions helps weaker models and modestly helps some strong ones.
  • A general Routine-following fine-tuning dataset improves execution accuracy when a Routine is present, but it weakens a model's autonomous planning ability in the no-Routine condition, so the two capabilities trade off.
  • Scenario-specific distillation through Routines lets a 14B model internalize procedural knowledge and achieve near-GPT-4o accuracy, and adding an explicit Routine on top of the specialized model pushes it even higher.
  • Branching Routines are slightly harder to follow than linear ones; the accuracy gap is small for strong models but pronounced for weaker ones, so branch logic should be introduced only on top of a solid tool-calling foundation.
  • When several candidate Routines are present, accuracy can drop because the model may try to merge steps; the paper therefore concludes that the memory module should aim to return a single most relevant Routine.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The per-step evaluation gives each sample the correct preceding tool-call history, so the reported 96.3% likely overstates end-to-end task completion; a natural follow-up experiment would compare per-step accuracy with full-trajectory success on the same 200 queries.
  • If Routine's effect is general, the same annotation-and-evaluation recipe should transfer to other enterprise domains (finance, IT operations, procurement) with a similar 25-tool tool server; the paper's HR-only evidence leaves that transfer open.
  • Distillation from a routine-annotated teacher suggests a practical deployment path: one domain expert annotates a handful of routines, and a small local model is fine-tuned on distilled traces, reducing dependence on large proprietary models at inference time.
  • The multi-routine interference results imply retrieval precision matters more than recall for procedure memory; ranking routines with a confidence threshold and returning only the top match may improve stability.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Routine, a structured planning framework that provides execution models with step-by-step natural-language plans containing step names, descriptions, and explicit tool assignments, with optional I/O parameter descriptions and branch notation. The framework is evaluated in an enterprise HR agent scenario with 25 MCP tools and seven hand-annotated Routines, using a test set of 1,148 per-step samples decomposed from 200 traces distilled by GPT-4o. Table 2 reports large gains in overall tool-calling accuracy for GPT-4o (41.1% to 96.3%) and Qwen3-14B (32.6% to 83.3%) when Routine is provided, and fine-tuning Qwen3-14B on a common Routine-following dataset plus a scenario-specific distilled dataset raises accuracy to 95.5%. The paper also presents ablations on Routine components, generation methods, and Routine quantity.

Significance. The paper addresses a practical problem—stability of multi-step tool calling in enterprise agents—and proposes a simple, interpretable solution that appears to work across multiple models. The experimental coverage is broad: eight base models, fine-tuning with two data sources, and systematic ablations. The authors are transparent that the explicit tool name is a core component of the framework and that the current approach relies on expert-provided drafts. If the evaluation concerns are resolved, Routine could be a useful practical contribution to applied LLM-agent research. However, the current evaluation methodology (per-step scoring with oracle context, self-distilled test data, and the absence of a control that separates plan presence from plan content) limits the strength of the central claim and requires revision.

major comments (4)
  1. [§5.1.2] The evaluation is per-step and teacher-forced: the 1,148 test samples are decomposed from 200 traces distilled by GPT-4o under the same Routine structure, and each sample is scored with the correct preceding history. Consequently, the overall accuracy measures the model's ability to imitate the teacher's next step given oracle context, not its ability to complete a multi-step task without error propagation. Additionally, the gold labels are the teacher's tool calls without independent verification (e.g., human annotation or ground truth), so the reported accuracy is an agreement rate with GPT-4o, not a measure of task correctness. The authors should either report end-to-end accuracy on complete traces or clearly frame the results as per-step imitation scores.
  2. [§5.1.3, §5.3.1] The no-Routine baseline provides no plan, while the Routine condition provides a full plan that includes the correct tool name for every step. The headline gains (41.1→96.3 for GPT-4o) therefore conflate the effect of having a plan with the effect of the plan containing the tool assignments. The ablation in §5.3.1 (Table 3) shows that removing the tool name reduces accuracy but still leaves a structured plan, and the remaining gain over the no-plan baseline indicates that the step descriptions themselves are valuable. To isolate the contribution of the structured format per se, the authors should add a control condition with an unstructured natural-language plan containing the same step descriptions but no explicit tool names, or a structured plan without tool names. Without this control, the claim that the Routine formalism itself drives the improvement is not fully supported.
  3. [§4.2.3, §5.1.2] Both the 537 scenario-specific training queries and the 200 test queries are generated from the same 5–6 query templates per sub-scenario (10 sub-scenarios total). The paper does not state whether the test queries are disjoint from the training queries in terms of template and filled-in entity values, nor does it report the similarity between the two sets. If the test set shares template structure with the training data, the fine-tuned model's 95.5% accuracy may reflect template memorization rather than generalization to new phrasings. The authors should clarify the exact relationship (e.g., new entity values, new template variations) and, ideally, evaluate on a held-out set with novel templates.
  4. [Table 2, §5.2.1] The claim that 'tool selection errors were identified as the main cause of failure, accounting for over 85% of all errors' is not directly supported by the table, because the reported tool selection accuracy is computed only on the subset of samples that passed the structural check. Similarly, the statement that the improvement in tool selection accuracy is the main driver of the overall improvement cannot be verified from the conditional metrics alone; different conditions have different subset sizes at each stage. The authors should provide an unconditional error breakdown (e.g., the number or fraction of samples that fail at each stage) to substantiate these conclusions.
minor comments (6)
  1. [Abstract] The phrase 'resulting in an accuracy increase to 88.2% on scenario-specific evaluations' is ambiguous; it should specify that this is the Common fine-tuned Qwen3-14B under the 'Routine w/ Branch' condition in Table 2.
  2. [Table 2 caption] 'Routing-following dataset' should be 'Routine-following dataset'.
  3. [§3.1.1] The branch notation in the main text (e.g., 'Branch X-n Step i') differs from the JSON notation in Appendix A.1 (e.g., 'x-n_i'); the authors should state explicitly that these are equivalent representations.
  4. [§4.2.3] The paper states that 537 user queries were obtained after data cleaning, but it is unclear whether the 200 evaluation queries are a subset of these 537 or a separate set; please clarify the relationship.
  5. [§5.3.3] The multi-Routine interference experiment does not specify how the interference Routines are selected (e.g., random, semantically similar, or from different sub-scenarios); this should be stated for reproducibility.
  6. [Appendix B.1] 'se text parsing tools' appears to be a typo for 'use text parsing tools'.

Circularity Check

1 steps flagged · score 4.0 of 10

Headline gains (41.1%→96.3%) are scored against labels distilled from the very GPT-4o-under-Routine configuration being tested, so the absolute 'accuracy' numbers are partly self-consistency; the directional claim retains independent support from the no-Routine baseline and the tool-name ablation.

  1. self definitional [§4.2.3, distillation of the 537 and 200 sample sets; §5.1.2, test data decomposition; headline results in Table 2 and the Abstract.]
    "we used GPT-4o equipped with Routines to distill a set of 537 single-turn, multi-step user queries for the HR agent system scenario after data cleaning ... In addition, we distilled a set of 200 single-turn, multi-step scenario-based user queries, totaling 1,148 tool calls. This dataset was used to evaluate model performance across different variants."

    The gold-standard tool calls are produced by GPT-4o guided by Routine prompts, i.e., by the exact intervention under test. Overall Accuracy (§5.1.1) counts a case correct only if structure, tool, and parameters all match that reference, so Routine-condition accuracy measures agreement with the system's own output distribution: GPT-4o's 96.3% is largely self-consistency (same model, same Routine format, deterministic decoding), and the fine-tuned Qwen's 95.5% is imitation fidelity to the teacher that wrote both the 537 training and the 1,148 test labels. The reduction is partial, not a tautology: the no-Routine baseline is judged against identical labels and scores 41.1%, so the directional claim has independent content.

full rationale

This paper contains no equation-level derivation chain; its claims are empirical measurements (Tables 2–5). The central causal claim—Routine guidance improves multi-step tool-calling accuracy relative to autonomous planning—is supported by contrasts that could have come out otherwise: the same test samples yield 41.1% for GPT-4o without Routine versus 96.3% with it, and the §5.3.1 ablation shows removing the explicit tool name drops accuracy by 5–15 points, an effect the paper openly interprets as converting tool selection into a specified-tool execution task. That interpretation caveat is an attribution concern, not circularity, and the paper's transparency about it counts in its favor. The single structural defect approaching circularity is the provenance of the evaluation reference. §4.2.3 and §5.1.2 state that the 537 training traces and the 200-query/1,148-call test set were both distilled with GPT-4o 'equipped with Routines'; Overall Accuracy is defined as exact agreement with those labels. The Routine-condition scores therefore measure, in part, the tested systems' consistency with the intervention's own output distribution (GPT-4o's 96.3% is the same model under the same prompt discipline; the student's 95.5% is imitation fidelity to the teacher that authored its labels). Because the no-Routine baseline is graded on identical labels at 32–44%, the comparison does not collapse, and the directional result is not forced. Self-citations are present but not load-bearing: [45] is the authors' prior single-step pipeline, explicitly extended here, and [15] is the OpenAI Cookbook that originally introduced Routine; neither functions as an external proof or a uniqueness constraint, and §2.2–2.3 attribute both openly. External scaffolding exists (BFCL AST evaluator, BUTTON dataset). Per the reviewing rule, the missing disclosure in §6—the self-distilled test reference—is flagged: the limitation section discusses expert-draft dependence but never mentions that the test labels were generated by the tested configuration. Net: partial circularity in the reference standard with independent directional content, so score 4 rather than 6.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No new physical or formal entities are introduced; Routine is a prompt format built from existing components. The central assumptions are about the validity of the self-distilled evaluation set, the per-step evaluation protocol, and the correctness of the manually drafted Routines and tool schemas.

assumptions (4)
  • domain assumption The 200-query evaluation set distilled by GPT-4o with Routines represents correct ground-truth tool calls for the HR scenario.
    Used in §4.2.3 and §5.1.2 to create the test set; no human verification is reported.
  • domain assumption Per-step accuracy computed with true preceding tool-call history estimates end-to-end execution accuracy.
    Evaluation samples are decomposed from full traces and scored independently (§5.1.2), which can overstate whole-task success.
  • domain assumption The seven manually annotated Routines correctly and completely cover the HR sub-scenarios.
    The Routines are the independent variable of the study and were annotated by the authors (§4.1); an incorrect Routine would invalidate the comparison.
  • domain assumption Tool descriptions in the MCP server are accurate and sufficient for the models to select tools.
    The execution model relies on tool names and parameter schemas (§3.3, §4.2.1); no analysis of tool description quality is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Routine: A Structural Planning Framework for LLM Agent System in Enterprise." pith.science (2026). https://pith.science/paper/CKXMOHIZ

@misc{pith2026250714447,
  author       = {Pith},
  title        = {Pith review of: Routine: A Structural Planning Framework for LLM Agent System in Enterprise},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CKXMOHIZ}},
  note         = {Machine review of arXiv:2507.14447}
}
read the original abstract

The deployment of agent systems in an enterprise environment is often hindered by several challenges: common models lack domain-specific process knowledge, leading to disorganized plans, missing key tools, and poor execution stability. To address this, this paper introduces Routine, a multi-step agent planning framework designed with a clear structure, explicit instructions, and seamless parameter passing to guide the agent's execution module in performing multi-step tool-calling tasks with high stability. In evaluations conducted within a real-world enterprise scenario, Routine significantly increases the execution accuracy in model tool calls, increasing the performance of GPT-4o from 41.1% to 96.3%, and Qwen3-14B from 32.6% to 83.3%. We further constructed a Routine-following training dataset and fine-tuned Qwen3-14B, resulting in an accuracy increase to 88.2% on scenario-specific evaluations, indicating improved adherence to execution plans. In addition, we employed Routine-based distillation to create a scenario-specific, multi-step tool-calling dataset. Fine-tuning on this distilled dataset raised the model's accuracy to 95.5%, approaching GPT-4o's performance. These results highlight Routine's effectiveness in distilling domain-specific tool-usage patterns and enhancing model adaptability to new scenarios. Our experimental results demonstrate that Routine provides a practical and accessible approach to building stable agent workflows, accelerating the deployment and adoption of agent systems in enterprise environments, and advancing the technical vision of AI for Process.

Figures

Figures reproduced from arXiv: 2507.14447 by the authors.

Figure 1
Figure 1. The mechanism of Routine guiding an LLM agent through multi-step tool calls Furthermore, we synthesized a total of 4,209 common training data with Rou￾tine generation based on an open multi￾tool dataset3 . After training, Qwen3-14B achieved an accuracy of 88.2%, further im￾proving its Routine instruction-following capability. Additionally, we performed data distillation based on Routine in real en￾terprise scenarios… view at source ↗
Figure 2
Figure 2. Framework of our Routine-based agent system. To begin, the system uses expert￾annotated planning prompts to generate Routines. During runtime, it processes user input and system parameters, completing tasks through the interaction between the modules, and ultimately provides a final summary based on the observation from a dedicated summarization tool. 2.3 Tool Calling Data Synthesis and Post-Training Training data q… view at source ↗
Figure 3
Figure 3. AI Optimization and Management of Routines 3.1.2 AI-Powered Routine Generation and Optimization While general-purpose models can effectively deconstruct problems into natural language Routines in open-ended scenarios, they struggle in enterprise contexts due to a lack of domain-specific knowledge. This makes it difficult for them to directly and reliably generate contextualized Routines without assistance from domai… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Its core function is to optimize parame [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The agent executes the model training process. The whole process includes the common Routine following capability training and the scenario-specific tool calling capability training, and can be divided into three modules: data synthesis, model fine-tuning and model eva…
Figure 6
Figure 6. Figure 6: The common Routine-following dataset filtering pipeline. 1. Routine Text Verification: Basic verification of generated Routine texts, discarding entries with empty responses or abnormal outputs. 2. Removal of Natural Language Summaries: As final agent-level summaries a…
Figure 7
Figure 7. Figure 7: AST evaluation workflow for our enterprise scenario. The process is mainly based on the BFCL framework with categorizing errors into three primary types: structural, tool selection, and parameter errors. Note: The content of free text parameters is not assessed via exa…
Figure 8
Figure 8. Figure 8: Different ways of Agent Planning The AI optimization step brought sig￾nificant and universal performance im￾provements. The performance of GPT￾4o, bridged most of the gap between ba￾sic usability and high reliability. The Qwen3 series models have also achieved stable i…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 44 canonical work pages

  1. [1]

    An Introduction to MultiAgent Systems

    Michael Wooldridge. An Introduction to MultiAgent Systems . John Wiley & Sons, 2nd edition, 2009

    work page 2009

  2. [2]

    A survey on large language model based autonomous agents

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), March 2024

    2024

  3. [3]

    Data interpreter: An llm agent for data science, 2024

    Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Yaying Fei, Yuheng Cheng, Zhibin Gou, Zongze Xu, and Chenglin Wu. Data interprete...

    2024

  4. [4]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. arXiv preprint arXiv:2305.15334, 2023

    arXiv 2023

  5. [5]

    Extracting knowledge graphs from user stories using langchain, 2025

    Thayná Camargo da Silva. Extracting knowledge graphs from user stories using langchain, 2025

    work page 2025

  6. [6]

    From mind to machine: The rise of manus ai as a fully autonomous digital agent, 2025

    Minjie Shen and Qikai Yang. From mind to machine: The rise of manus ai as a fully autonomous digital agent, 2025

    work page 2025

  7. [7]

    Orchestrating agents and data for enterprise: A blueprint architecture for compound ai, 2025

    Eser Kandogan, Nikita Bhutani, Dan Zhang, Rafael Li Chen, Sairam Gurajada, and Estevam Hruschka. Orchestrating agents and data for enterprise: A blueprint architecture for compound ai, 2025

    work page 2025

  8. [8]

    Flowbench: Revisiting and benchmarking workflow-guided planning for llm-based agents, 2024

    Ruixuan Xiao, Wentao Ma, Ke Wang, Yuchuan Wu, Junbo Zhao, Haobo Wang, Fei Huang, and Yongbin Li. Flowbench: Revisiting and benchmarking workflow-guided planning for llm-based agents, 2024

    work page 2024

Show all 62 references
  1. [9]

    Reducing tool hallucination via reliability alignment, 2025

    Hongshen Xu, Zichen Zhu, Lei Pan, Zihan Wang, Su Zhu, Da Ma, Ruisheng Cao, Lu Chen, and Kai Yu. Reducing tool hallucination via reliability alignment, 2025

    2025

  2. [10]

    Flowmind: Automatic workflow generation with llms, 2024

    Zhen Zeng, William Watson, Nicole Cho, Saba Rahimi, Shayleen Reynolds, Tucker Balch, and Manuela Veloso. Flowmind: Automatic workflow generation with llms, 2024

    2024

  3. [11]

    Beyond snippet assistance: A workflow-centric framework for end-to-end ai-driven code generation

    Vladimir Sonkin and C˘at˘alin Tudose. Beyond snippet assistance: A workflow-centric framework for end-to-end ai-driven code generation. Computers, 14(3):94, 2025. 18

    2025

  4. [12]

    From words to workflows: Automating business processes, 2024

    Laura Minkova, Jessica López Espejel, Taki Eddine Toufik Djaidja, Walid Dahhane, and El Hassane Ettifouri. From words to workflows: Automating business processes, 2024

    2024

  5. [13]

    Shah, and Christopher Re

    Michael Wornow, Avanika Narayan, Krista Opsahl-Ong, Quinn McIntyre, Nigam H. Shah, and Christopher Re. Automating the enterprise with foundation models, 2024

    2024

  6. [14]

    Ida: Breaking barriers in no-code ui automation through large language models and human-centric design, 2024

    Segev Shlomov, Avi Yaeli, Sami Marreed, Sivan Schwartz, Netanel Eder, Offer Akrabi, and Sergey Zeltyn. Ida: Breaking barriers in no-code ui automation through large language models and human-centric design, 2024

    2024

  7. [15]

    Using reasoning for routine generation

    OpenAI Cookbook. Using reasoning for routine generation. https://cookbook.openai. com/examples/o1/using_reasoning_for_routine_generation, 2024. Accessed on July 16, 2025

    2024

  8. [16]

    Tool-planner: Task planning with clusters across multiple tools, 2025

    Yanming Liu, Xinyue Peng, Jiannan Cao, Shi Bo, Yuwei Zhang, Xuhong Zhang, Sheng Cheng, Xun Wang, Jianwei Yin, and Tianyu Du. Tool-planner: Task planning with clusters across multiple tools, 2025

    2025

  9. [17]

    Towards large language model-based personal agents in the enterprise: Current trends and open problems

    Vinod Muthusamy, Yara Rizk, Kiran Kate, Praveen Venkateswaran, Vatche Isahagian, Ashu Gulati, and Parijat Dube. Towards large language model-based personal agents in the enterprise: Current trends and open problems. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findin...

    2023

  10. [18]

    Auto-gpt for online decision making: Benchmarks and additional opinions, 2023

    Hui Yang, Sifu Yue, and Yunzhong He. Auto-gpt for online decision making: Benchmarks and additional opinions, 2023

    2023

  11. [19]

    Multi-agent collaboration: Harnessing the power of intelligent llm agents, 2023

    Yashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of intelligent llm agents, 2023

    2023

  12. [20]

    Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023

    2023

  13. [21]

    Chang and Longling Geng

    Edward Y . Chang and Longling Geng. Sagallm: Context management, validation, and transac- tion guarantees for multi-agent llm planning, 2025

    2025

  14. [22]

    Agentif: Benchmarking instruction following of large language models in agentic scenarios, 2025

    Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. Agentif: Benchmarking instruction following of large language models in agentic scenarios, 2025

    2025

  15. [23]

    The cost of dynamic reasoning: Demystifying ai agents and test-time scaling from an ai infrastructure perspective, 2025

    Jiin Kim, Byeongjun Shin, Jinha Chung, and Minsoo Rhu. The cost of dynamic reasoning: Demystifying ai agents and test-time scaling from an ai infrastructure perspective, 2025

    2025

  16. [24]

    Runaway is ashamed, but helpful: On the early-exit behavior of large language model-based agents in embodied environments, 2025

    Qingyu Lu, Liang Ding, Siyi Cao, Xuebo Liu, Kanjian Zhang, Jinxia Zhang, and Dacheng Tao. Runaway is ashamed, but helpful: On the early-exit behavior of large language model-based agents in embodied environments, 2025

    2025

  17. [25]

    Manus: Markerless grasp capture using articulated 3d gaussians, 2024

    Chandradeep Pokhariya, Ishaan N Shah, Angela Xing, Zekun Li, Kefan Chen, Avinash Sharma, and Srinath Sridhar. Manus: Markerless grasp capture using articulated 3d gaussians, 2024

    2024

  18. [26]

    Agentorchestra: A hierarchical multi-agent framework for general-purpose task solving, 2025

    Wentao Zhang, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. Agentorchestra: A hierarchical multi-agent framework for general-purpose task solving, 2025

    2025

  19. [27]

    ReAct: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Nir, and Anirudh Nir. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023

    2023

  20. [28]

    Plan-and-act: Improving planning of agents for long-horizon tasks, 2025

    Lutfi Eren Erdogan, Nicholas Lee, Sehoon Kim, Suhong Moon, Hiroki Furuta, Gopala Anu- manchipalli, Kurt Keutzer, and Amir Gholami. Plan-and-act: Improving planning of agents for long-horizon tasks, 2025

    2025

  21. [29]

    Le, Ed H

    Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V . Le, Ed H. Chi, and Denny Zhou. Natural plan: Benchmarking llms on natural language planning, 2024

    2024

  22. [30]

    Nl2plan: Robust llm-driven planning from minimal text descriptions, 2024

    Elliot Gestrin, Marco Kuhlmann, and Jendrik Seipp. Nl2plan: Robust llm-driven planning from minimal text descriptions, 2024

    2024

  23. [31]

    L. Diao, C. Yang, and Z. Zhang. Guidebench: Benchmarking domain-oriented guideline following for llm agents. arXiv, 2025. 19

    2025

  24. [32]

    Chain-of-thought prompting elicits reasoning in large language models, 2023

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023

    2023

  25. [33]

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Ni...

    2022

  26. [34]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instruc- tions, 2023

    2023

  27. [35]

    J. Zhou, T. Lu, and S. et al. Mishra. Instruction-following evaluation for large language models. arXiv, 2023

    2023

  28. [36]

    B. Wen, P. Ke, and X. et al. Gu. Complexbench: Benchmarking complex instruction-following with multiple constraints composition. arXiv, 2024

    2024

  29. [37]

    Zhang, H

    X. Zhang, H. Yu, and C. et al. Fu. Iopo: Empowering llms with complex instruction following via input-output preference optimization. arXiv, 2024

    2024

  30. [38]

    Synthetic data rl: Task definition is all you need

    Yiduo Guo, Zhen Guo, Chuanwei Huang, Zi-Ang Wang, Zekai Zhang, Haofei Yu, Huishuai Zhang, and Yikang Shen. Synthetic data rl: Task definition is all you need. arXiv preprint arXiv:2505.17063, 2025

    2025 arXiv

  31. [39]

    Griffiths

    Yujia Qin, Shengding Hu, Zixuan Liu, Shuo Wang, Yilun Zhao, Shunyu Yao, Yilun Chen, Weizhi Chen, and Thomas L. Griffiths. ToolBench: An open platform for tool-usage evaluation. arXiv preprint arXiv:2307.16789, 2023

    2023 arXiv

  32. [40]

    Toolace: Winning the points of llm function calling

    Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. Toolace: Winning the points of llm function calling. arXiv preprint arXiv:2409.00920, 2024

    2024 arXiv

  33. [41]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023

    2023 arXiv

  34. [42]

    Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets

    Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh RN, et al. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems , 37:54463...

    2024

  35. [43]

    Kimi-k2: An agentic framework for tool-augmented language models, 2024

    Moonshot AI. Kimi-k2: An agentic framework for tool-augmented language models, 2024. Accessed: 2025-07-16

    2024

  36. [44]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  37. [45]

    Adaptable and precise: Enterprise- scenario llm function-calling capability training pipeline, 2024

    Guancheng Zeng, Wentao Ding, Beining Xu, Chi Zhang, Wenqiang Han, Gang Li, Jingjing Mo, Pengxu Qiu, Xinran Tao, Wang Tao, and Haowen Hu. Adaptable and precise: Enterprise- scenario llm function-calling capability training pipeline, 2024

    2024

  38. [46]

    Tool learning with large language models: a survey

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-rong Wen. Tool learning with large language models: a survey. Frontiers of Computer Science, 19(8), January 2025

    2025

  39. [47]

    Redefining enterprise data management with ai-powered automation, 2024

    Priyanka Neelakrishnan. Redefining enterprise data management with ai-powered automation, 2024

    2024

  40. [48]

    Ai-powered decision intelligence in enterprise systems engineering, 2025

    Guguloth and Kumar. Ai-powered decision intelligence in enterprise systems engineering, 2025

    2025

  41. [49]

    Ai-driven innovations in enterprise systems, 2025

    Zeebaree and Mustafa. Ai-driven innovations in enterprise systems, 2025

    2025

  42. [50]

    Integrating artificial intelligence in enterprise architecture, 2025

    Gopalaswamy. Integrating artificial intelligence in enterprise architecture, 2025. 20

    2025

  43. [51]

    Mcp-radar: A multi-dimensional benchmark for evaluating tool use capabilities in large language models

    Xuanqi Gao, Siyi Xie, Juan Zhai, Shqing Ma, and Chao Shen. Mcp-radar: A multi-dimensional benchmark for evaluating tool use capabilities in large language models. arXiv preprint arXiv:2505.16700, 2025

    2025

  44. [52]

    A survey on progress in llm alignment from the perspective of reward design, 2025

    Miaomiao Ji, Yanqiu Wu, Zhibin Wu, Shoujin Wang, Jian Yang, Mark Dras, and Usman Naseem. A survey on progress in llm alignment from the perspective of reward design, 2025

    2025

  45. [53]

    Longfunceval: Measuring the effective- ness of long context models for function calling, 2025

    Kiran Kate, Tejaswini Pedapati, Kinjal Basu, Yara Rizk, Vijil Chenthamarakshan, Subhajit Chaudhury, Mayank Agarwal, and Ibrahim Abdelaziz. Longfunceval: Measuring the effective- ness of long context models for function calling, 2025

    2025

  46. [54]

    Evaluating large language models in vulnerability detection under variable context windows, 2025

    Jie Lin and David Mohaisen. Evaluating large language models in vulnerability detection under variable context windows, 2025

    2025

  47. [55]

    Knowledge injection via prompt distillation, 2024

    Kalle Kujanpää, Harri Valpola, and Alexander Ilin. Knowledge injection via prompt distillation, 2024

    2024

  48. [56]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Lu Wang

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Lu Wang. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022

    2022

  49. [57]

    Zero: Memory optimiza- tions toward training trillion parameter models, 2020

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models, 2020

    2020

  50. [58]

    Flashattention-2: Faster attention with better parallelism and work partitioning, 2023

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023

    2023

  51. [59]

    Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E

    Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Advances in Neural Information Processing Syst...

    2024

  52. [60]

    Y . Ye. Task memory engine: Spatial memory for robust multi-step llm agents, 2025

    2025

  53. [61]

    Evaluating llm-based agents for multi-turn conversations: A survey, 2025

    Shengyue Guan, Haoyi Xiong, Jindong Wang, Jiang Bian, Bin Zhu, and Jian guang Lou. Evaluating llm-based agents for multi-turn conversations: A survey, 2025

    2025

  54. [62]

    step": "1

    Maggie Huan, Yuetai Li, Tuney Zheng, Xiaoyu Xu, Seungone Kim, Minxin Du, Radha Pooven- dran, Graham Neubig, and Xiang Yue. Does math reasoning improve general llm capabilities? understanding transferability of llm reasoning. arXiv preprint arXiv:2507.00432, 2025. 21 A Appendix...

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.