Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges

T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces DialogTool, a multi-turn dialogue benchmark covering the full lifecycle of stateful tool use, and reports that state-of-the-art LLMs, including GPT-4o, still fail to use tools reliably over long dialogues.

desk verdict A useful but overstated 'stateful' benchmark: the long-horizon results are solid, but the environment appears to be deterministic lookups, and the novelty needs a head-to-head with ToolSandbox and tau-bench. read the letter →

arxiv 2505.13328 v1 pith:INL6MKT7 submitted 2025-05-19 cs.CL

Hongru Wang , Wenyu Huang , Yufei Wang , Yuanhao Xi , Jianqiao Lu , Huan Zhang , Nan Hu , Zeming Liu
show 2 more authors
Jeff Z. Pan Kam-Fai Wong
This is my paper · ORCID
classification cs.CL
keywords toolusemulti-turndialoguestatefulinteractionLLMagentsbenchmarkAPIcalltask-orientedlong-horizonevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing benchmarks usually test LLM tool use in single turns: pick the right API and fill its arguments once. This paper argues that real assistants face stateful, multi-turn tool use, where the same tool must be created, selected, executed, and referenced across many exchanges. To measure that, it builds DialogTool, a dialogue dataset re-encoded from task-oriented dialogues into Apps, APIs, and arguments, plus VirtualMobile, an environment that actually executes API calls and tracks state. Evaluating 13 LLMs across six tasks, the paper finds that no model clears 80% on tool creation or utilization, and that GPT-4o's tool-execution accuracy falls from 75.2% on dialogues under 10 turns to 35.3% on dialogues over 40 turns. The paper concludes that current state-of-the-art models are not yet reliable tool users over long horizons.

What carries the argument

The load-bearing artifact is the DialogTool/VirtualMobile pair. DialogTool re-labels existing task-oriented dialogue annotations: domains become Apps, intents become APIs, slots become arguments, and database lookups become API calls, while system actions are preserved as a fine-grained action set (request, confirm, inform, notify_success/failure, and so on). VirtualMobile implements every API in Python, stores the result of each call as database state, and checks whether a generated function returns the same output as the ground truth. This lets the benchmark score the whole lifecycle—creating a tool, deciding when to use it, selecting it, filling its arguments, and generating a role-styled response—under a changing environment state.

What would settle it

Compare GPT-4o's DialogTool execution accuracy when the full environment state is re-injected before every turn; if accuracy stops falling as turn count grows, the reported long-horizon failure is a state-tracking artifact of context, not a robust limit of tool-use ability, while if it still falls, the benchmark measures a genuine skill deficit.

Watch

Extended reading notes

Core claim

The central discovery is that state-of-the-art LLMs still cannot perform well to use tools over long horizons. On DialogTool, GPT-4o—the strongest model tested—achieves 66.7% tool creation, 77.8% tool selection, and 68.7% tool execution, and these numbers drop sharply as dialogues lengthen: with more than 40 turns, awareness falls to 52.3%, selection to 54.4%, and execution to 35.3%. No tested model exceeds 80% on any creation or utilization task, and most errors in execution come from missing arguments rather than wrong API names. The paper attributes the difficulty to the stateful nature of multi-turn interactions: the agent must remember previous calls, decide when not to call a tool, and convert tool results into role-consistent responses.

Load-bearing premise

The automatic re-labelling of task-oriented dialogue annotations into Apps, APIs, and arguments assumes that this transformation preserves the semantics of real stateful tool use; if it drops real-world dependencies like side effects, confirmation semantics, or failure recovery, the benchmark measures a synthetic re-encoding rather than genuine tool-use ability.

Editorial extensions

If this is right

  • Models that score well on single-turn API selection can still be unreliable in real assistants, because long-horizon execution degrades sharply with turn count.
  • Exposing tool spaces hierarchically—choose the App first, then the API—improves selection accuracy relative to flat selection over all APIs.
  • Complex APIs with many required arguments, such as train-ticket booking, are disproportionately hard for models to create from descriptions, suggesting code-generation ability is bottlenecked by argument count.
  • Tool execution errors are dominated by missing arguments rather than wrong API choices, so improving argument grounding and formatting may yield the largest gains.
  • Because role-consistent response quality tracks tool-utilization accuracy, response generation cannot be improved in isolation from upstream tool calls.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The benchmark's re-labeling of task-oriented dialogues may not capture real-world side effects and failure recovery; if genuine API calls have dependencies like partial bookings or idempotency requirements, the reported gaps could be either larger or smaller than in real deployment.
  • The turn-count degradation suggests a state-tracking bottleneck: a model given an explicit, re-injected state summary at each turn might keep accuracy flat, which would isolate memory loss from tool-use skill.
  • The same dataset could be extended to multi-agent setups where each App is an agent, testing whether modular orchestration reduces the long-horizon drop.
  • Tool creation's exact-output pass criterion is strict; allowing functional equivalence might change relative rankings, especially for models that produce semantically identical code with different structure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces DialogTool, a multi-turn dialogue dataset re-annotated from SGD and MultiWOZ by mapping services to Apps, intents to APIs, slots to arguments, and database lookups to API calls, together with VirtualMobile, a Python environment that simulates API calls. Six tasks are defined across three stages: tool creation, tool utilization (awareness, selection, execution), and role-consistent response generation and role play. The authors evaluate 13 LLMs and report that even GPT-4o achieves at most 77.8% on tool selection and 68.7% on tool execution, with performance degrading in longer dialogues (Table 5). They conclude that current state-of-the-art LLMs cannot reliably perform stateful tool use over long horizons.

Significance. The paper addresses a real gap: most tool-use benchmarks are single-turn or focus on a subtask, and the proposed pipeline for converting task-oriented dialogues into tool-use annotations is economical and transparent, with prompts and error analyses included. The hierarchical App-to-API selection experiment and the per-API breakdowns are useful contributions. However, the benchmark's core novelty—stateful tool use—is not actually exercised by the environment as described, because API implementations are deterministic lookup functions with no side effects; moreover, the artifacts are not released, and the closest prior work (Toolsandbox, τ-bench) is cited but not compared. If the statefulness issue is addressed and artifacts are released, DialogTool could be a valuable resource for the community.

major comments (4)
  1. [Section 3.3 (footnote 2); Section 4.2] The environment's API implementations are explicitly deterministic: footnote 2 states 'Given same input in the dialogue, it can produce same output', and Section 3.3 describes the database as storing unique results sampled from the original dialogues. Under this design, APIs flagged as is_transactional (Section 3.2), e.g., ReserveHotel or ReserveCar, have no side effects: a second identical call returns the same result and does not consume availability, create a record, or alter later search results. Consequently, the 'stateful tool interactions' evaluated by DialogTool reduce to multi-turn context tracking and slot filling; dependencies between calls (search, then reserve, then search again) are not modeled. The assertion in Section 4.2 that 'the environmental state will be updated to e_t' after execution is not supported by any described state-transition mechanism. Because the paper's central conclusion (abstract; Table 5) is that LLMs fail at 'stateful' tool use over long horizons, this gap is load-bearing. The authors should either implement genuine state changes for transactional APIs and verify that subsequent calls reflect them, or explicitly reframe the benchmark as multi-turn context-dependent tool use without world-state side effects and temper the claims accordingly.
  2. [Section 3 (Dataset and Environment); Section 1] No dataset, environment code, or evaluation scripts are released or linked anywhere in the manuscript, despite the contribution statement saying the environment 'ensur[es] the reproducibility'. For a benchmark paper, the artifacts are the main deliverable; without them, readers cannot run DialogTool or VirtualMobile, and the reported numbers cannot be independently checked. An availability statement with a URL or an anonymized release is required before the benchmark claims can be substantiated.
  3. [Section 4.3; Tables 3-4] The paper states that 13 models are evaluated, but Section 4.3 lists only 12: ChatGLM3-6B, Qwen1.5-7B/14B/72B-Chat, Mistral-7B-Instruct, Llama-2-7B/13B/70B, Llama-3-8B/70B, GPT-3.5, and GPT-4o. Vicuna-13B appears in Tables 3 and 4 (and is needed to reach 13) but its version, source, and prompting configuration are never described. This omission makes the Vicuna results unreproducible and leaves the model roster ambiguous.
  4. [Related Work; Table 1] The references include Toolsandbox (Lu et al., 2024), which is explicitly described as 'a stateful, conversational, interactive evaluation benchmark for LLM tool use,' and τ-bench (Yao et al., 2024), yet neither is cited or compared in the body, and Table 1 omits them. The abstract's and Section 1's claim that DialogTool is 'the first benchmark' for stateful tool use is therefore not positioned against the closest existing work. The authors should either add a comparison and justify their novelty (e.g., tool creation, role play, and the specific six-task decomposition) or soften the 'first' claim.
minor comments (4)
  1. [Section 3.4] The text says the dataset contains 'approximately 16k dialogues and 33k turns', but Table 2 reports 329,964 turns; the '50% during training and 36% during evaluation' multi-App percentages also do not match Table 2 (10,739/16,142 ≈ 66.5% and 360/900 = 40%). Please correct these inconsistencies.
  2. [Various] Typos and style issues throughout: 'alternatively' should be 'interchangeably' (footnote 1), 'infeaible' (Section 1), 'Culumn' in Table 1, 'outforms' (Section 4.4), 'reproductivity' (Section 4.3), and inconsistent 'QWen'/'Qwen' spellings.
  3. [Appendix D.3; Section 4.3] The human evaluation uses 50 sampled responses per model with three annotators, but no inter-annotator agreement or score variance is reported; the GPT-4o-based role consistency score is assigned by GPT-4o, which is itself one of the evaluated models. The claim that 'we do not observe significant differences' when using Llama3.1-70B as judge (footnote 12) would benefit from concrete numbers.
  4. [Figures 3 and 4] Figure 3 and Figure 4 are referenced before they appear in the text, and the captions in the appendix are not self-contained; consider moving the figures into the main text or adding a pointer to the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: benchmark construction and evaluations are externally grounded and empirical.

full rationale

The paper's derivation chain is not circular. DialogTool is built by re-labeling external task-oriented dialogue datasets (SGD, MultiWOZ): 'we regard the service/domain, slots and intents in TDD as different Apps, Arguments, and APIs, and transform every database lookup operation in the dialogue into an API function call' (Section 1). The benchmark labels therefore come from pre-existing annotations, not from the models under test. Tool creation is scored by executing generated Python against stored database rows sampled from those source dialogues ('we sample every database lookup operation from the original dialogue datasets and store all unique returned results as the database'), which is an external consistency check rather than a fitted prediction. Utilization tasks (awareness, selection, execution) are evaluated by exact-match accuracy against those same annotations; the reported long-horizon degradation (Table 5) is an empirical outcome, not an artifact of the scoring rule. Role-consistency scores use GPT-4o, but the paper reports a human-alignment check and alternative-model agreement ('We also conduct a human evaluation to validate the alignment... We try other models such as Llama3.1-70B-Instruct and we do not observe significant differences'). The few self-citations (e.g., Wang et al. 2024b for the tool-call paradigm and model selection, Huang et al. 2024 for metrics) are conventional and non-load-bearing: none supplies a premise whose truth is assumed in order to derive the paper's conclusion. The skeptic's concern about VirtualMobile's deterministic 'same input, same output' APIs (footnote 2) is a construct-validity criticism of how faithfully the environment models stateful side effects; it does not show that any result is equivalent to its inputs by construction. No fitted parameter is later relabeled as a prediction, and no uniqueness theorem or ansatz is imported from prior work. Therefore no circular step meets the quotation-plus-reduction bar.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claims rest on the dataset transformation and environment fidelity, not on fitted parameters. No free parameters or invented entities are introduced.

assumptions (3)
  • domain assumption Reformulating task-oriented dialogue annotations (domains, intents, slots, database lookups) as App/API/argument interactions preserves the stateful semantics of real tool use.
    Section 3.2 'Setting Shift' transforms SGD/MultiWOZ annotations via a Python script into tool-learning labels; the benchmark's validity depends on this equivalence.
  • domain assumption The manually implemented APIs in VirtualMobile are correct and deterministic, and the stored database covers the returns needed for all dialogues.
    Section 3.3 states each App/API is manually implemented and correctness is ensured; no automated verification is described.
  • domain assumption GPT-4o's role-consistency scores are a valid proxy for human judgment of role consistency.
    Section 4.3 uses GPT-4o to score role consistency and reports a human-evaluation alignment check, but with only 50 samples per model and no inter-annotator agreement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges." pith.science (2026). https://pith.science/paper/INL6MKT7

@misc{pith2026250513328,
  author       = {Pith},
  title        = {Pith review of: Rethinking Stateful Tool Use in Multi-Turn Dialogues: Benchmarks and Challenges},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/INL6MKT7}},
  note         = {Machine review of arXiv:2505.13328}
}
read the original abstract

Existing benchmarks that assess Language Models (LMs) as Language Agents (LAs) for tool use primarily focus on stateless, single-turn interactions or partial evaluations, such as tool selection in a single turn, overlooking the inherent stateful nature of interactions in multi-turn applications. To fulfill this gap, we propose \texttt{DialogTool}, a multi-turn dialogue dataset with stateful tool interactions considering the whole life cycle of tool use, across six key tasks in three stages: 1) \textit{tool creation}; 2) \textit{tool utilization}: tool awareness, tool selection, tool execution; and 3) \textit{role-consistent response}: response generation and role play. Furthermore, we build \texttt{VirtualMobile} -- an embodied virtual mobile evaluation environment to simulate API calls and assess the robustness of the created APIs\footnote{We will use tools and APIs alternatively, there are no significant differences between them in this paper.}. Taking advantage of these artifacts, we conduct comprehensive evaluation on 13 distinct open- and closed-source LLMs and provide detailed analysis at each stage, revealing that the existing state-of-the-art LLMs still cannot perform well to use tools over long horizons.

Figures

Figures reproduced from arXiv: 2505.13328 by the authors.

Figure 1
Figure 1. A typical example to show the entire life cycle [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The pipeline of DialogTool collection by 1) Setting shift: transfer the setting of existing dialogue datasets; 2) Role Play: then rewriting the utterances from system side into role-consistent utterances. (Wang et al., 2023d), resulting in more longer in￾teraction time , such as character.ai4 . Most existing work focuses on character roles rather than assistant roles. For example, CharacterEval (Tu et al., 2024) eva… view at source ↗
Figure 4
Figure 4. The performance of two prompting strategies. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (10 more)
Figure 5
Figure 5. Figure 5: This is a sample JSON configuration of Rents App which contains 3 distinct APIs. We also provide name, format and possible values for categorical argu￾ments. In this app, the base required arguments are empty. C Prompt Details You are a helpful assistant and you are go…
Figure 6
Figure 6. Figure 6: The prompt used to prompt LLM to create tool in python code. Given a dialogue between user and dialogue system, and a role config for dialogue sys￾tem, please assign a consistency score ac￾cording to all utterances by the dialogue system. The consistency of a role refe…
Figure 7
Figure 7. Figure 7: The prompt used to prompt LLM to assign the [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: The prompt used to prompt LLM to generate [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 11
Figure 11. Figure 11: The prompt used to prompt LLM to decide API and corresponding arguments after decided App under the hierarchical setting. Given the API description and the ex￾isting dialogue history, please gener￾ate one API request that should be in￾voked to complete the user’s curr…
Figure 9
Figure 9. Figure 9: The prompt used to prompt LLM to generate [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: The prompt used to prompt LLM to decide App first under the hierarchical setting. Given the API description and the existing dialogue history, please generate one API request that should be invoked to complete the user’s current query, and output it in the format of […
Figure 12
Figure 12. Figure 12: The prompt used to prompt LLM to decide App, API and all related arguments at the same time under the flat setting. D Analysis D.1 Tool Selection To explore the specfic performance of different LLMs on each API, we provide the accuracy of each API for each LLM as show…
Figure 13
Figure 13. Figure 13: The tool creation performance of different LLMs on each API. We use same colour to indicate the API [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: The performance of different LLMs on each API. We use same colour to indicate the API comes from [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WeClawArena: An Auditable Sandbox and Benchmark for Cross-User Agents Collaboration and Security in Human-Centered Agent Networks

    cs.AI 2026-08 conditional novelty 7.0 of 10

    A benchmark and Docker sandbox that tests whether owner-scoped AI agents can collaborate on real tasks without being manipulated into privacy leaks, poisoned evidence, or invalid approvals.

  2. Experience-Evolving Multi-Turn Tool-Use Agent with Hybrid Episodic-Procedural Memory

    cs.LG 2025-12 conditional novelty 6.0 of 10

    An LLM agent that builds a tool-transition graph with state summaries from past experience improves tool selection and RL exploration by large margins on multi-turn benchmarks.

Reference graph

Works this paper leans on

58 extracted references · 9 canonical work pages · cited by 2 Pith papers

  1. [1]

    AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

    2024

  2. [2]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  3. [3]

    Simon Baron-Cohen. 1991. Precursors to a theory of mind: Understanding attention in others. Natural theories of mind: Evolution, development and simulation of everyday mindreading, 1:233--251

    work page 1991

  4. [4]

    Pawe Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, I \ n igo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Ga s i \'c . 2018. https://doi.org/10.18653/v1/D18-1547 M ulti WOZ - a large-scale multi-domain W izard-of- O z dataset for task-oriented dialogue modelling . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proce...

    doi:10.18653/v1/d18-1547 2018

  5. [5]

    Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241

    arXiv 2018

  6. [6]

    Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320--335

    2022

  7. [7]

    Yingqiang Ge, Yujie Ren, Wenyue Hua, Shuyuan Xu, Juntao Tan, and Yongfeng Zhang. 2023. https://arxiv.org/abs/2312.03815 Llm as os, agents as apps: Envisioning aios, agents and the aios-agent ecosystem . Preprint, arXiv:2312.03815

    arXiv 2023

  8. [8]

    Gerd Gigerenzer. 1991. From tools to theories: A heuristic of discovery in cognitive psychology. Psychological review, 98(2):254

    1991

Show all 58 references
  1. [9]

    Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018. Slot-gated modeling for joint slot filling and intent prediction. In Proceedings of The 16th Annual Conference of the North American Chapter of the Association for Compu...

    2018

  2. [10]

    Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, and Qun Liu. 2024. https://arxiv.org/abs/2401.17167 Planning, creation, usage: Benchmarking llms for comprehensive tool utiliza...

    2024 arXiv

  3. [11]

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. https://arxiv.org/abs/2201.07207 Language models as zero-shot planners: Extracting actionable knowledge for embodied agents . Preprint, arXiv:2201.07207

    2022 arXiv

  4. [12]

    Vojt e ch Hude c ek and Ondrej Dusek. 2023. https://doi.org/10.18653/v1/2023.sigdial-1.21 Are large language models all you need for task-oriented dialogue? In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 216--228, Pragu...

    2023 doi

  5. [13]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  6. [14]

    Mihir Kale and Abhinav Rastogi. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.527 Template guided text generation for task-oriented dialogue . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6505--6520, Online. Assoc...

    2020 doi

  7. [15]

    Wai-Chung Kwan, Huimin Wang, Hongru Wang, Zezhong Wang, Bin Liang, Xian Wu, Yefeng Zheng, and Kam-Fai Wong. 2024. https://aclanthology.org/2024.lrec-main.837 J o TR : A joint transformer and reinforcement learning framework for dialogue policy learning . In Proceedings of the ...

    2024

  8. [16]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. https://arxiv.org/abs/2005.11401 Retrieval-augmented generation for knowledge-int...

    2021 arXiv

  9. [17]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.187 API -bank: A comprehensive benchmark for tool-augmented LLM s . In Proceedings of the 2023 Conference on Empiric...

    2023 doi

  10. [18]

    Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. https://arxiv.org/abs/2209.07753 Code as policies: Language model programs for embodied control . Preprint, arXiv:2209.07753

    2023 arXiv

  11. [19]

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2023. https...

    2023 arXiv

  12. [20]

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2024. https://arxiv.org/abs/2408.04682 Toolsandbox: A stateful, conversational, interactive evaluation benchmark for llm tool ...

    2024 arXiv

  13. [21]

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. 2023. https://arxiv.org/abs/2304.09842 Chameleon: Plug-and-play compositional reasoning with large language models . Preprint, arXiv:2304.09842

    2023 arXiv

  14. [22]

    Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. https://arxiv.org/abs/2401.13178 Agentboard: An analytical evaluation board of multi-turn llm agents . Preprint, arXiv:2401.13178

    2024 arXiv

  15. [23]

    Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. https://arxiv.org/abs/2311.12983 Gaia: a benchmark for general ai assistants . Preprint, arXiv:2311.12983

    2023 arXiv

  16. [24]

    Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022...

    2022 arXiv

  17. [25]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. https://arxiv.org/abs/2305.15334 Gorilla: Large language model connected with massive apis . Preprint, arXiv:2305.15334

    2023 arXiv

  18. [26]

    Baolin Peng, Chenguang Zhu, Chunyuan Li, Xiujun Li, Jinchao Li, Michael Zeng, and Jianfeng Gao. 2020. Few-shot natural language generation for task-oriented dialog. arXiv preprint arXiv:2002.12328

    2020 arXiv

  19. [27]

    Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    2018

  20. [28]

    Libo Qin, Wenbo Pan, Qiguang Chen, Lizi Liao, Zhou Yu, Yue Zhang, Wanxiang Che, and Min Li. 2023 a . https://doi.org/10.18653/v1/2023.emnlp-main.363 End-to-end task-oriented dialogue: A survey of tasks, methods, and future directions . In Proceedings of the 2023 Conference on ...

    2023 doi

  21. [29]

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, ...

    2023 arXiv

  22. [30]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023 c . https://arxiv.org/abs/2307.16789 Toolll...

    2023 arXiv

  23. [31]

    Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8689--8696

    2020

  24. [32]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. https://arxiv.org/abs/2303.17580 Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face . Preprint, arXiv:2303.17580

    2023 arXiv

  25. [33]

    Raphael Shu, Elman Mansimov, Tamer Alkhouli, Nikolaos Pappas, Salvatore Romeo, Arshit Gupta, Saab Mansour, Yi Zhang, and Dan Roth. 2022. https://arxiv.org/abs/2212.09946 Dialog2api: Task-oriented dialogue with api description and example programs . Preprint, arXiv:2212.09946

    2022 arXiv

  26. [34]

    Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. 2023. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427

    2023 arXiv

  27. [35]

    Ryuichi Takanobu, Hanlin Zhu, and Minlie Huang. 2019. https://doi.org/10.18653/v1/D19-1010 Guided dialog policy learning: Reward estimation for multi-domain task-oriented dialog . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the...

    2019 doi

  28. [36]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  29. [37]

    Quan Tu, Shilong Fan, Zihang Tian, and Rui Yan. 2024. https://arxiv.org/abs/2401.01275 Charactereval: A chinese benchmark for role-playing conversational agent evaluation . Preprint, arXiv:2401.01275

    2024 arXiv

  30. [38]

    Dingmin Wang, Chenghua Lin, Qi Liu, and Kam-Fai Wong. 2021. https://doi.org/10.18653/v1/2021.naacl-main.27 Fast and scalable dialogue state tracking with explicit modular decomposition . In Proceedings of the 2021 Conference of the North American Chapter of the Association for...

    2021 doi

  31. [39]

    Hongru Wang, Minda Hu, Yang Deng, Rui Wang, Fei Mi, Weichao Wang, Yasheng Wang, Wai-Chung Kwan, Irwin King, and Kam-Fai Wong. 2023 a . https://doi.org/10.18653/v1/2023.findings-emnlp.641 Large language models as source planner for personalized knowledge-grounded dialogues . In...

    2023 doi

  32. [40]

    Pan, and Kam-Fai Wong

    Hongru Wang, Yujia Qin, Yankai Lin, Jeff Z. Pan, and Kam-Fai Wong. 2024 a . https://doi.org/10.1145/3626772.3661381 Empowering large language models: Tool learning for real-world interaction . In Proceedings of the 47th International ACM SIGIR Conference on Research and Develo...

    2024

  33. [41]

    Hongru Wang, Huimin Wang, Zezhong Wang, and Kam-Fai Wong. 2022 a . Integrating pretrained language model for dialogue policy evaluation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6692--6696. IEEE

    2022

  34. [42]

    Hongru Wang, Lingzhi Wang, Yiming Du, Liang Chen, Jingyan Zhou, Yufei Wang, and Kam-Fai Wong. 2023 b . https://arxiv.org/abs/2311.16789 A survey of the evolution of language model-based dialogue systems . Preprint, arXiv:2311.16789

    2023 arXiv

  35. [43]

    Hongru Wang, Rui Wang, Fei Mi, Yang Deng, Zezhong Wang, Bin Liang, Ruifeng Xu, and Kam-Fai Wong. 2023 c . https://doi.org/10.18653/v1/2023.findings-emnlp.806 Cue- C o T : Chain-of-thought prompting for responding to in-depth dialogue questions with LLM s . In Findings of the A...

    2023 doi

  36. [44]

    Pan, and Kam-Fai Wong

    Hongru Wang, Rui Wang, Boyang Xue, Heming Xia, Jingtao Cao, Zeming Liu, Jeff Z. Pan, and Kam-Fai Wong. 2024 b . https://doi.org/10.18653/v1/2024.emnlp-main.856 A pp B ench: Planning of multiple API s from various APP s for complex user instruction . In Proceedings of the 2024 ...

    2024 doi

  37. [45]

    Hongru Wang, Boyang Xue, Baohang Zhou, Tianhua Zhang, Cunxiang Wang, Huimin Wang, Guanhua Chen, and Kam-Fai Wong. 2025. https://aclanthology.org/2025.naacl-long.331/ Self- DC : When to reason and when to act? self divide-and-conquer for compositional unknown questions . In Pro...

    2025

  38. [46]

    Ruoyao Wang, Peter Jansen, Marc-Alexandre C \^o t \'e , and Prithviraj Ammanabrolu. 2022 b . https://doi.org/10.18653/v1/2022.emnlp-main.775 S cience W orld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language...

    2022 doi

  39. [47]

    Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2024 c . https://arxiv.org/abs/2309.10691 Mint: Evaluating llms in multi-turn interaction with tools and language feedback . Preprint, arXiv:2309.10691

    2024 arXiv

  40. [48]

    Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, Zhaoxiang Zhang, Wanli Ouyang, Ke Xu, Wenhu Chen, Jie Fu, and Junran Peng. 2023 d . https://arxiv.org/abs/2310.00746 Rolellm: Benchmarking, e...

    2023 arXiv

  41. [49]

    Heng-Da Xu, Xian-Ling Mao, Puhai Yang, Fanshu Sun, and Heyan Huang. 2024. https://doi.org/10.18653/v1/2024.acl-long.152 Rethinking task-oriented dialogue systems: From complex modularity to zero-shot autonomous agent . In Proceedings of the 62nd Annual Meeting of the Associati...

    2024 doi

  42. [50]

    Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yu Niu, Hua Wu, Haifeng Wang, and Shihang Wang. 2022. https://doi.org/10.18653/v1/2022.findings-acl.207 Long time no see! open-domain conversation with long-term persona memory . In Findings of the Association for Computational Linguis...

    2022 doi

  43. [51]

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. https://arxiv.org/abs/2406.12045 -bench: A benchmark for tool-agent-user interaction in real-world domains . Preprint, arXiv:2406.12045

    2024 arXiv

  44. [52]

    Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. https://doi.org/10.18653/v1/2020.nlp4convai-1.13 M ulti WOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines . In Proceedings of th...

    2020 doi

  45. [53]

    Xiaoying Zhang, Baolin Peng, Kun Li, Jingyan Zhou, and Helen Meng. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.891 SGP - TOD : Building task bots effortlessly via schema-guided LLM prompting . In Findings of the Association for Computational Linguistics: EMNLP 2023, ...

    2023 doi

  46. [54]

    Jinfeng Zhou, Zhuang Chen, Dazhen Wan, Bosi Wen, Yi Song, Jifan Yu, Yongkang Huang, Libiao Peng, Jiaming Yang, Xiyao Xiao, Sahand Sabour, Xiaohan Zhang, Wenjing Hou, Yijia Zhang, Yuxiao Dong, Jie Tang, and Minlie Huang. 2023. https://arxiv.org/abs/2311.16832 Characterglm: Cust...

    2023 arXiv

  47. [55]

    Qi Zhu, Kaili Huang, Zheng Zhang, Xiaoyan Zhu, and Minlie Huang. 2020. https://doi.org/10.1162/tacl_a_00314 C ross WOZ : A large-scale C hinese cross-domain task-oriented dialogue dataset . Transactions of the Association for Computational Linguistics, 8:281--295

    2020 doi

  48. [56]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. https://arxiv.org/abs/2306.13304 Toolqa: A dataset for llm question answering with external tools . Preprint, arXiv:2306.13304

    2023 arXiv

  49. [57]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  50. [58]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.