{"id":"334193f9-2428-4395-a381-69e0f474ffea","arxiv_id":"2505.13328","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DialogTool, a multi-turn dialogue benchmark with stateful tool use and a virtual mobile environment, shows that current LLMs struggle over long horizons, with GPT-4o reaching at most 77.8% in tool selection and 68.7% in tool execution.","lead":"This paper introduces DialogTool, a multi-turn dialogue benchmark for evaluating language models as assistants that use tools, along with a simulated mobile environment called VirtualMobile. It tests 13 AI models and finds that even the best ones make many mistakes on long conversations that require creating, choosing, and executing tool calls.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VirtualMobile's deterministic 'same input, same output' APIs appear to preclude state-changing transactional calls, so the benchmark's central claim of evaluating stateful tool use may rest on an environment with no side effects.","rationale":"Agree with the reader that construct validity of the re-labelling is the key risk, but I locate it more sharply: the environment's determinism guarantee (Section 3.3, footnote 2) appears to rule out the very state changes that define stateful tool use. The reader's weakest_assumption focuses on semantic loss during the SGD/MultiWOZ-to-API transformation; the more concrete failure mode is that even if the labels are faithful, the simulated APIs do not execute with persistent side effects. Without side effects, a model is never required to update a mental or external world model across calls; it only needs to read the dialogue text and fill arguments. The drop with turn count in Table 5 could then be an artifact of long-context degradation on classification and span extraction tasks rather than a deficit in stateful tool-use reasoning. This is load-bearing for the abstract's general claim. That said, the empirical results still document poor accuracy on these subtasks, the dataset is large, and the environment is released, so the concern warrants a reproducibility condition rather than rejection. The reader's CONDITIONAL verdict stands; I add one specific release-time check, exposing and testing the environment's state transitions, to the conditions. This is an honest, checkable test of the central claim.","tokens_in":21571,"tokens_out":10674,"duration_ms":108989,"concrete_test":"Run VirtualMobile's transactional API ReserveHotel twice with identical arguments and verify whether the second call returns a different result (e.g., failure, changed availability, or a distinct confirmation token). Then run SearchHotel with the same city and date and check whether the just-booked hotel is excluded or flagged as unavailable. If both queries return byte-identical outputs, the environment has no side effects. As a complement, inspect the repository's API implementations for any mutation of the database or state attributes inside transactional functions (e.g., writing to self.db) and for any downstream query that reads such mutations. Passing this test, by observing state changes, would resolve the concern; failing it would confirm the benchmark is stateless.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states that each API is manually implemented so that 'Given same input in the dialogue, it can produce same output' (footnote 2). For a genuinely stateful transactional API (e.g., ReserveHotel, flagged is_transactional in Section 3.2), a second call with identical arguments should change the world: availability decreases, a confirmation record is created, or a later SearchHotel excludes the booked option. A deterministic lookup-table environment cannot represent such side effects, because the database is built by 'sampling every database lookup operation from the original dialogue datasets and storing all unique returned results' (Section 3.3) rather than modelling a persistent world state. Consequently, the six tasks, especially tool awareness/selection/execution, may reduce to slot filling and action classification over dialogue context, not to stateful tool use with dependencies across calls. The abstract's conclusion that LLMs 'cannot perform well to use tools over long horizons' then rests on a benchmark whose core statefulness property is not actually exercised. This does not invalidate the dataset as a re-annotation of task-oriented dialogues, but it weakens the paper's claim to be the first benchmark for the whole lifecycle of stateful tool use.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DialogTool, a multi-turn dialogue dataset re-annotated from SGD and MultiWOZ by mapping services to Apps, intents to APIs, slots to arguments, and database lookups to API calls, together with VirtualMobile, a Python environment that simulates API calls. Six tasks are defined across three stages: tool creation, tool utilization (awareness, selection, execution), and role-consistent response generation and role play. The authors evaluate 13 LLMs and report that even GPT-4o achieves at most 77.8% on tool selection and 68.7% on tool execution, with performance degrading in longer dialogues (Table 5). They conclude that current state-of-the-art LLMs cannot reliably perform stateful tool use over long horizons.","tokens_in":21799,"tokens_out":8913,"duration_ms":76228,"significance":"The paper addresses a real gap: most tool-use benchmarks are single-turn or focus on a subtask, and the proposed pipeline for converting task-oriented dialogues into tool-use annotations is economical and transparent, with prompts and error analyses included. The hierarchical App-to-API selection experiment and the per-API breakdowns are useful contributions. However, the benchmark's core novelty—stateful tool use—is not actually exercised by the environment as described, because API implementations are deterministic lookup functions with no side effects; moreover, the artifacts are not released, and the closest prior work (Toolsandbox, τ-bench) is cited but not compared. If the statefulness issue is addressed and artifacts are released, DialogTool could be a valuable resource for the community.","major_comments":[{"comment":"The environment's API implementations are explicitly deterministic: footnote 2 states 'Given same input in the dialogue, it can produce same output', and Section 3.3 describes the database as storing unique results sampled from the original dialogues. Under this design, APIs flagged as is_transactional (Section 3.2), e.g., ReserveHotel or ReserveCar, have no side effects: a second identical call returns the same result and does not consume availability, create a record, or alter later search results. Consequently, the 'stateful tool interactions' evaluated by DialogTool reduce to multi-turn context tracking and slot filling; dependencies between calls (search, then reserve, then search again) are not modeled. The assertion in Section 4.2 that 'the environmental state will be updated to e_t' after execution is not supported by any described state-transition mechanism. Because the paper's central conclusion (abstract; Table 5) is that LLMs fail at 'stateful' tool use over long horizons, this gap is load-bearing. The authors should either implement genuine state changes for transactional APIs and verify that subsequent calls reflect them, or explicitly reframe the benchmark as multi-turn context-dependent tool use without world-state side effects and temper the claims accordingly.","section":"Section 3.3 (footnote 2); Section 4.2"},{"comment":"No dataset, environment code, or evaluation scripts are released or linked anywhere in the manuscript, despite the contribution statement saying the environment 'ensur[es] the reproducibility'. For a benchmark paper, the artifacts are the main deliverable; without them, readers cannot run DialogTool or VirtualMobile, and the reported numbers cannot be independently checked. An availability statement with a URL or an anonymized release is required before the benchmark claims can be substantiated.","section":"Section 3 (Dataset and Environment); Section 1"},{"comment":"The paper states that 13 models are evaluated, but Section 4.3 lists only 12: ChatGLM3-6B, Qwen1.5-7B/14B/72B-Chat, Mistral-7B-Instruct, Llama-2-7B/13B/70B, Llama-3-8B/70B, GPT-3.5, and GPT-4o. Vicuna-13B appears in Tables 3 and 4 (and is needed to reach 13) but its version, source, and prompting configuration are never described. This omission makes the Vicuna results unreproducible and leaves the model roster ambiguous.","section":"Section 4.3; Tables 3-4"},{"comment":"The references include Toolsandbox (Lu et al., 2024), which is explicitly described as 'a stateful, conversational, interactive evaluation benchmark for LLM tool use,' and τ-bench (Yao et al., 2024), yet neither is cited or compared in the body, and Table 1 omits them. The abstract's and Section 1's claim that DialogTool is 'the first benchmark' for stateful tool use is therefore not positioned against the closest existing work. The authors should either add a comparison and justify their novelty (e.g., tool creation, role play, and the specific six-task decomposition) or soften the 'first' claim.","section":"Related Work; Table 1"}],"minor_comments":[{"comment":"The text says the dataset contains 'approximately 16k dialogues and 33k turns', but Table 2 reports 329,964 turns; the '50% during training and 36% during evaluation' multi-App percentages also do not match Table 2 (10,739/16,142 ≈ 66.5% and 360/900 = 40%). Please correct these inconsistencies.","section":"Section 3.4"},{"comment":"Typos and style issues throughout: 'alternatively' should be 'interchangeably' (footnote 1), 'infeaible' (Section 1), 'Culumn' in Table 1, 'outforms' (Section 4.4), 'reproductivity' (Section 4.3), and inconsistent 'QWen'/'Qwen' spellings.","section":"Various"},{"comment":"The human evaluation uses 50 sampled responses per model with three annotators, but no inter-annotator agreement or score variance is reported; the GPT-4o-based role consistency score is assigned by GPT-4o, which is itself one of the evaluated models. The claim that 'we do not observe significant differences' when using Llama3.1-70B as judge (footnote 12) would benefit from concrete numbers.","section":"Appendix D.3; Section 4.3"},{"comment":"Figure 3 and Figure 4 are referenced before they appear in the text, and the captions in the appendix are not self-contained; consider moving the figures into the main text or adding a pointer to the appendix.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper has potential, but the central statefulness claim needs to be fixed and the artifacts released. The novelty claim should also be checked against Toolsandbox and τ-bench, both of which are in the reference list but not discussed. If the authors can demonstrate genuine state transitions in VirtualMobile and release the code and data, I would be supportive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: DialogTool is a genuinely useful resource for the tool-use community, but the paper's central novelty—stateful tool use—is the weakest part. The long-horizon results are real and worth having; the statefulness claim, as described, is much weaker.\n\nWhat the paper does well: it re-labels SGD/MultiWOZ into a large multi-turn tool-use benchmark (16k dialogues, six tasks, three stages). The setting shift is clearly explained and the prompts are in the appendix. Tool creation is evaluated by executing generated code against stored database lookups—that is concrete. The error analyses (missing vs. mismatched arguments, API errors, the long-horizon drop from 75.2% to 35.3% execution for GPT-4o) are informative. The finding that even the best closed model sits below 80% on selection and 70% on execution is useful pressure.\n\nWhere it gets soft: the environment looks deterministic by design. Footnote 2 says 'given same input... same output,' and the sample API code in the appendix just filters a static database. Nothing in the example writes or mutates state. The paper flags APIs as transactional (ReserveHotel, ReserveCar) and says the environment tracks state, but it doesn't show how a second identical call changes availability or creates a record. If the environment really is pure lookup, then 'stateful' reduces to multi-turn context tracking—which is still worth testing, but is not the 'whole lifecycle of stateful tool use' the title promises. The authors should either build in actual side effects or re-label the claim.\n\nThe novelty claim also needs qualification. ToolSandbox and tau-bench exist, are cited in the references, and are not discussed or compared in the related work. Table 1 doesn't include them. That weakens 'first benchmark' more than the text admits. Reproducibility is another gap: no release link for code or data, no confidence intervals or significance tests, and the human evaluation is 50 responses per model with three students and no agreement numbers.\n\nNone of this is fatal. The core empirical finding—performance degrades sharply with turn count—is consistent across models and is a real contribution. The paper deserves a serious referee, and if the authors release the code and either show the environment does handle side effects or drop the world-state claim, it could become a useful community benchmark. My advice: send it to review, with the statefulness issue as the central question.\n\nBest","headline":"A useful but overstated 'stateful' benchmark: the long-horizon results are solid, but the environment appears to be deterministic lookups, and the novelty needs a head-to-head with ToolSandbox and tau-bench.","tokens_in":22346,"tokens_out":4403,"would_cite":false,"duration_ms":41084,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces DialogTool, a multi-turn dialogue benchmark covering the full lifecycle of stateful tool use, and reports that state-of-the-art LLMs, including GPT-4o, still fail to use tools reliably over long dialogues.","keywords":["tool use","multi-turn dialogue","stateful interaction","LLM agents","benchmark","API call","task-oriented dialogue","long-horizon evaluation"],"falsifier":"Compare GPT-4o's DialogTool execution accuracy when the full environment state is re-injected before every turn; if accuracy stops falling as turn count grows, the reported long-horizon failure is a state-tracking artifact of context, not a robust limit of tool-use ability, while if it still falls, the benchmark measures a genuine skill deficit.","tokens_in":21398,"feed_emoji":"📱","tokens_out":7639,"duration_ms":67834,"temperature":0.7,"pith_summary":"Existing benchmarks usually test LLM tool use in single turns: pick the right API and fill its arguments once. This paper argues that real assistants face stateful, multi-turn tool use, where the same tool must be created, selected, executed, and referenced across many exchanges. To measure that, it builds DialogTool, a dialogue dataset re-encoded from task-oriented dialogues into Apps, APIs, and arguments, plus VirtualMobile, an environment that actually executes API calls and tracks state. Evaluating 13 LLMs across six tasks, the paper finds that no model clears 80% on tool creation or utilization, and that GPT-4o's tool-execution accuracy falls from 75.2% on dialogues under 10 turns to 35.3% on dialogues over 40 turns. The paper concludes that current state-of-the-art models are not yet reliable tool users over long horizons.","feed_headline":"Long dialogues halve GPT-4o's API-call success rate","feed_subtitle":"New DialogTool benchmark shows state-aware tool use decays fast as conversations stretch past 20 turns.","key_machinery":"The load-bearing artifact is the DialogTool/VirtualMobile pair. DialogTool re-labels existing task-oriented dialogue annotations: domains become Apps, intents become APIs, slots become arguments, and database lookups become API calls, while system actions are preserved as a fine-grained action set (request, confirm, inform, notify_success/failure, and so on). VirtualMobile implements every API in Python, stores the result of each call as database state, and checks whether a generated function returns the same output as the ground truth. This lets the benchmark score the whole lifecycle—creating a tool, deciding when to use it, selecting it, filling its arguments, and generating a role-styled response—under a changing environment state.","core_discovery":"The central discovery is that state-of-the-art LLMs still cannot perform well to use tools over long horizons. On DialogTool, GPT-4o—the strongest model tested—achieves 66.7% tool creation, 77.8% tool selection, and 68.7% tool execution, and these numbers drop sharply as dialogues lengthen: with more than 40 turns, awareness falls to 52.3%, selection to 54.4%, and execution to 35.3%. No tested model exceeds 80% on any creation or utilization task, and most errors in execution come from missing arguments rather than wrong API names. The paper attributes the difficulty to the stateful nature of multi-turn interactions: the agent must remember previous calls, decide when not to call a tool, and convert tool results into role-consistent responses.","pith_inferences":["The benchmark's re-labeling of task-oriented dialogues may not capture real-world side effects and failure recovery; if genuine API calls have dependencies like partial bookings or idempotency requirements, the reported gaps could be either larger or smaller than in real deployment.","The turn-count degradation suggests a state-tracking bottleneck: a model given an explicit, re-injected state summary at each turn might keep accuracy flat, which would isolate memory loss from tool-use skill.","The same dataset could be extended to multi-agent setups where each App is an agent, testing whether modular orchestration reduces the long-horizon drop.","Tool creation's exact-output pass criterion is strict; allowing functional equivalence might change relative rankings, especially for models that produce semantically identical code with different structure."],"forward_implications":["Models that score well on single-turn API selection can still be unreliable in real assistants, because long-horizon execution degrades sharply with turn count.","Exposing tool spaces hierarchically—choose the App first, then the API—improves selection accuracy relative to flat selection over all APIs.","Complex APIs with many required arguments, such as train-ticket booking, are disproportionately hard for models to create from descriptions, suggesting code-generation ability is bottlenecked by argument count.","Tool execution errors are dominated by missing arguments rather than wrong API choices, so improving argument grounding and formatting may yield the largest gains.","Because role-consistent response quality tracks tool-utilization accuracy, response generation cannot be improved in isolation from upstream tool calls."],"supporting_citations":[{"why":"Supplies the Schema-Guided Dialogue annotations (services, intents, slots, transactional flags) that DialogTool re-encodes into Apps, APIs, and arguments.","marker":"Rastogi et al. (2020)"},{"why":"Supplies MultiWOZ dialogues used as a second seed source for diverse multi-domain conversations.","marker":"Budzianowski et al. (2018)"},{"why":"Establishes the tool-selection and tool-execution evaluation paradigm that DialogTool extends to stateful multi-turn settings.","marker":"Patil et al. (2023)"},{"why":"API-Bank provides a multi-turn dialogue tool benchmark with tool-augmented LLMs and response evaluation, which DialogTool contrasts with.","marker":"Li et al. (2023)"},{"why":"ToolBench supplies a large-scale real-world API set, and DialogTool compares against its single-turn tool selection focus.","marker":"Qin et al. (2023c)"},{"why":"UltraTool evaluates tool creation, selection, and execution, and is the direct prior work on lifecycle evaluation that DialogTool extends with statefulness.","marker":"Huang et al. (2024)"},{"why":"Provides evidence that advanced LLMs struggle over long horizons even in simple domains, used in the limitation discussion to motivate the benchmark.","marker":"Yao et al. (2024)"}],"fun_headline_variants":["Long dialogues slash GPT-4o's tool execution to 35%","DialogTool benchmark: stateful tool use crumbles after 40 turns","GPT-4o tool execution falls to 35% in stateful multi-turn dialogues","Stateful tool use defeats LLMs: GPT-4o only 35% after 40 turns","LLMs forget tools in long dialogues: execution drops to 35%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatic re-labelling of task-oriented dialogue annotations into Apps, APIs, and arguments assumes that this transformation preserves the semantics of real stateful tool use; if it drops real-world dependencies like side effects, confirmation semantics, or failure recovery, the benchmark measures a synthetic re-encoding rather than genuine tool-use ability.","fun_headline_variants_meta":{"raw":{"variants":["Long dialogues slash GPT-4o's tool execution to 35%","DialogTool benchmark: stateful tool use crumbles after 40 turns","GPT-4o tool execution falls to 35% in stateful multi-turn dialogues","Stateful tool use defeats LLMs: GPT-4o only 35% after 40 turns","LLMs forget tools in long dialogues: execution drops to 35%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001009,"raw_usage":{"total_tokens":4256,"prompt_tokens":929,"completion_tokens":3327,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":3222}},"tokens_in":545,"tokens_out":3327,"duration_ms":23752,"temperature":1.0,"reasoning_tokens":3222,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:14:57.072776+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare GPT-4o's DialogTool execution accuracy when the full environment state is re-injected before every turn; if accuracy stops falling as turn count grows, the reported long-horizon failure is a state-tracking artifact of context, not a robust limit of tool-use ability, while if it still falls, the benchmark measures a genuine skill deficit.","supporting_citations":[],"review_version":1}