{"id":"98b26b09-5b53-45f3-b92f-9cd5e7d8de9a","arxiv_id":"2505.08402","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A prompt-based framework improves LLM tool-use accuracy on ToolQA by generating tool parameters with tool-specific handler structures instead of one generic structure.","lead":"TUMS is a framework that helps large language models use external tools more accurately by splitting tool calls into smaller parameter steps. The authors report large accuracy gains on the ToolQA benchmark, but the gains are uneven and depend on hand-crafted prompts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The multi-structure handler claim is confounded by manual, ToolQA-tuned structure assignment; a transfer or automatic-selection test is needed to settle it.","rationale":"The reader's weakest assumption — that manual, subjective structure assignment tuned with knowledge of ToolQA is the load-bearing confound — matches my reading of the paper. The central claim requires that multi-structure parameter-level processing generalizes beyond the specific tools and datasets used in the experiments, but the method section provides no automatic or principled rule for choosing among the three handler structures. The ablation shows a large drop when collapsing to one structure, which is real evidence that the structured prompts matter, but it does not isolate the structure taxonomy from per-tool prompt engineering, because the same authors who designed the prompts also chose which structure each tool should use after seeing the benchmark. The preference-based hint in Appendix 6.2 further demonstrates that the authors resorted to benchmark-specific guidance to improve hard-question performance, reinforcing the concern that the approach is not a general mechanism. I therefore agree with the CONDITIONAL verdict: the idea is plausible and the ablation is suggestive, but the generality claim needs transfer evidence or an automatic structure-selection component before it can be accepted as a framework-level contribution. No additional fatal flaw emerged from my review; the main risk is empirical, not logical, and a transfer test would directly settle it.","tokens_in":12061,"tokens_out":3775,"duration_ms":40394,"concrete_test":"Apply the published TUMS prompts unchanged to a held-out tool suite (e.g., ToolBench or API-Bank) with a pre-registered mapping from tool documentation to Direct/Parallel/Serial structures, and compare against one-structure TUMS and ReAct under the same Qwen1.5-72B backbone. If the multi-structure gain over one-structure disappears or reverses on unseen tools, the Section 3.3 manual mapping, not parameter-level processing, is the active ingredient. A cheaper internal check: design the structure mapping using only seven of the eight ToolQA datasets and measure the held-out dataset's easy and hard scores against the reported tables; a large drop on the held-out dataset would confirm benchmark-specific tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central attribution — that parameter-level multi-structure handlers cause the reported gains — rests on Section 3.3's heuristic assignment of each tool to Direct, Parallel, or Serial generation based on the authors' 'subjective complexity,' together with hand-written few-shot exemplars for each handler. Because all ToolQA tools and datasets are known at design time, this mapping can encode benchmark-specific solution paths, and Appendix 6.2 makes that tuning explicit with preference-based hints recommending SQLInterpreter for certain hard questions. The ablation in Figure 3 (TUMS vs. TUMS-OS) holds exemplars approximately fixed, so it demonstrates that per-tool structured prompting helps on ToolQA, but it does not show that the structure taxonomy transfers to unseen tools: the framework has no mechanism for selecting a handler structure for a new tool, and the claim that the processor 'is capable of expansion' is not an automatic selection procedure. The reported hard-question average is also fragile, driven heavily by Yelp (+157.1%) while Flight (-45.5%) and Airbnb (-30.0%) regress. If the manual mapping and hints are the active ingredient rather than parameter-level processing per se, the central claim reduces to a benchmark-specific prompt-engineering result. The paper provides no evidence on other benchmarks and no automatic structure selection, so this concern is load-bearing for the claimed generality.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TUMS, a four-module framework for LLM tool use consisting of an intent recognizer, a task decomposer, a subtask processor with multiple parameter-generation handlers (Direct, Parallel, and Serial), and an executor. The core claim is that transforming tool-level processing into parameter-level processing via multi-structure handlers improves tool-use accuracy. Experiments on ToolQA with Qwen1.5-72B report an average 19.6% improvement over the previous best baseline on easy questions and 50.6% on hard questions, along with ablations of the handler structures and of the intent recognizer, and an efficiency analysis.","tokens_in":12332,"tokens_out":5102,"duration_ms":51137,"significance":"If the attribution claimed in the paper were established, the idea of assigning structure-specific parameter-generation procedures according to tool complexity would be a useful and easily adoptable contribution to tool-augmented LLMs. The paper deserves credit for a clear modular design, a published prompt suite, an ablation of the multi-structure mechanism, and a cost analysis. However, the current evidence is in-sample: the handler assignments and the preference-based hints are hand-designed for ToolQA, and the hard-question average is dominated by a single dataset, so the general significance of the framework is not yet demonstrated.","major_comments":[{"comment":"The assignment of each tool to the Direct, Parallel, or Serial handler structure is made by the authors' 'subjective complexity', and Appendix 6.2 introduces preference-based hints that are manually written for specific ToolQA questions. Because the paper provides no automatic structure-selection mechanism and no transfer experiment to unseen tools or datasets, the experiments do not establish that parameter-level multi-structure handlers, rather than benchmark-specific prompt engineering, are the cause of the reported improvements.","section":"Section 3.3 and Appendix 6.2"},{"comment":"The abstract's claim of 'almost an overall enhancement' is not supported on the hard benchmark: in Table 2, TUMS degrades relative to ReAct on Flight-hard (11.0 to 6.0) and Airbnb-hard (10.0 to 7.0), while the 50.6% average is heavily driven by Yelp-hard (14.0 to 36.0). The authors themselves note 'unstable performance in hard questions of table datasets'; the discussion should analyze per-dataset variability instead of relying on the average, and should report variance or significance testing.","section":"Tables 1 and 2, Section 4.2"},{"comment":"The Qwen and CoT baselines are not given access to any tools, so their near-zero scores on tool-dependent datasets are not informative for a tool-use comparison. The meaningful tool-using baselines are ReAct and Chameleon, and against them TUMS is not uniformly better (for example, it is worse than ReAct on Flight-hard). The main comparison should be reframed accordingly, and paired or resampling-based significance tests should be reported.","section":"Section 4.1 and following paragraph"},{"comment":"The ablation TUMS vs. TUMS-OS is described as holding exemplars fixed while removing the multi-structure handlers, but the paper does not specify how a Direct-only prompt is constructed for tools that originally used Parallel or Serial exemplars, and no variance or significance information is given. The ablation therefore does not conclusively isolate the multi-structure handlers from the accompanying tool-specific exemplars, which is the central attribution of the paper.","section":"Figure 3 and Section 4.2"},{"comment":"The TUMS-PRE variant introduces preference-based hints that are manually written for specific hard questions after observing the framework's bottlenecks, as illustrated by the Airbnb question in Appendix 6.2. The reported 40.13% correct rate for TUMS-PRE is therefore a benchmark-specific result and should not be used as evidence for the general effectiveness of the framework; it should be clearly labeled as an in-sample variant or removed from the contribution claims.","section":"Appendix 6.2"}],"minor_comments":[{"comment":"The arrows in the TUMS rows are not defined; the reader cannot tell whether the percentages compare TUMS to ReAct specifically or to the best baseline in each column, and the dash on Coffee-easy should be explained.","section":"Tables 1 and 2"},{"comment":"The x-axis labels 'On Easy Questions' and 'On Hard Questions' are confusing because both the total cost and the average cost appear on the same plot; the figure should be clarified or split into separate panels.","section":"Figure 4"},{"comment":"References 29 and 30 are identical citations of Shi et al., 'Learning to use tools via cooperative and interactive agents'; the duplicate should be removed and the in-text citations reconciled.","section":"References"},{"comment":"The paper states that the code has been open-sourced in GitHub, but no repository URL or identifier is provided; please add one for reproducibility.","section":"Appendix 6.1"},{"comment":"There are several typos and style inconsistencies, including 'Prompt F or' in Figure 5, 'such markable performance improvement' in Section 4.2, and the first-person 'Acknowledgments' section; these should be corrected in a final revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper's central mechanism is plausible and the modular framework is easy to build on, but the current evidence does not separate the multi-structure handler design from ToolQA-specific prompt engineering. I recommend requiring either a transfer experiment to an unseen toolset or an automatic structure-selection procedure, plus a re-analysis of the hard benchmark without the preference-based tricks, before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core of this paper is simple to state: instead of forcing every tool call through one parameter-generation prompt, TUMS routes each tool to one of three handler structures (direct, parallel, serial) based on the tool's parameter count and complexity. The easy-question results on ToolQA are consistent and meaningful, and the ablation against a one-structure variant (TUMS-OS) supports the claim that per-tool structured prompting helps, at least on this benchmark. The intent recognizer's efficiency analysis is a nice touch. Credit where due: this is an honest, readable empirical paper with a plausible mechanism.\n\nThe soft spots are real and, in one place, load-bearing. The assignment of each tool to a handler structure is manual, based on the authors' 'subjective complexity,' and several prompts and exemplars were clearly written with ToolQA's tools and questions in mind. Appendix 6.2 makes the benchmark-specific tuning explicit: TUMS-PRE recommends SQLInterpreter on certain hard questions because it is a 'shorter solving path.' That is engineering for the test set, not a general method. The stress-test note is right: the paper offers no mechanism for selecting a handler structure for a new tool, so the claimed generality of the multi-structure taxonomy is unsupported. The hard-question average of +50.6% is also fragile, driven mostly by Yelp (+157%) while Flight (-45.5%) and Airbnb (-30%) regress. The 'almost an overall enhancement' phrase in the introduction overstates what the tables show.\n\nOther issues are conventional but still matter: no variance or significance testing, no code link (the text says 'open-sourced' without a URL), and the Qwen/CoT baselines are denied access to tools, making their near-zero scores a strawman. None of these kills the paper, but together they weaken the central attribution. The framework is a modest extension of existing decomposition and prompting ideas, not a new capability.\n\nWho is this for? Researchers working on tool-augmented LLMs who want a practical prompt-engineering recipe and a data point that per-tool prompt structures can pay off on a known benchmark. It deserves a serious referee, but the authors should be pushed to add a transfer test across unseen tools or an automatic structure-selection heuristic, report error bars, and publish the code before this can be trusted as more than a benchmark-specific result.","headline":"Solid easy-question gains on ToolQA, but the paper's central attribution to multi-structure handlers is undercut by hand-tuned, benchmark-specific design choices.","tokens_in":12830,"tokens_out":1605,"would_cite":false,"duration_ms":18181,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Generating tool parameters with per-tool prompt structures raises LLM tool-use accuracy by 19.6% on easy and 50.6% on hard ToolQA benchmark questions.","keywords":["large language models","tool use","tool-augmented LLMs","parameter generation","multi-structure handlers","task decomposition","question answering","ToolQA"],"falsifier":"Take a held-out toolset, assign each tool's handler structure automatically from its schema or without looking at the test questions, run TUMS against ReAct, and check whether the advantage survives. A second check: give the one-structure variant the same few-shot examples inside a single combined prompt, and if it then matches TUMS, the gain comes from exemplar count or prompt length rather than from the structural split.","tokens_in":11889,"feed_emoji":"🧰","tokens_out":12081,"duration_ms":102689,"temperature":0.7,"pith_summary":"The paper claims that failed tool calls in large language models come mostly from bad parameters, not bad tool choice, because models generate parameters for every tool with one uniform, coarse-grained prompt. It proposes TUMS, which routes a question through an intent recognizer, a decomposer that plans one tool call per subtask, and a subtask processor whose multi-structure handlers generate parameters three ways: directly for simple tools, in parallel for tools with many parameters, and serially for intricate calls such as SQL queries. On the ToolQA benchmark of tool-using question answering, TUMS averages 19.6% higher accuracy on easy questions and 50.6% higher on hard questions than the ReAct baseline, which interleaves reasoning with tool calls. The paper takes its ablation as evidence that the handler split, not the added planning modules, drives the gain: keeping the same framework but using only the direct handler drops the correct rate to about 19.9%, below ReAct.","feed_headline":"LLM tool calls improve up to 50.6% with per-tool parameter handlers","feed_subtitle":"Splitting parameter generation into direct, parallel, and serial handlers beats ReAct on ToolQA easy and hard questions.","key_machinery":"The central mechanism is the multi-structure handler set inside the subtask processor: three few-shot prompt structures, direct, parallel, and serial, each matched to tools by a heuristic rule based on parameter count and usage difficulty. The parallel structure handles tools whose parameters are many but independent, and the serial structure handles tools whose parameters must be discovered sequentially, with each stage's output feeding the next. The framework's own ablation demonstrates that the mechanism, not the surrounding pipeline, carries the accuracy gain: replacing all three structures with only the direct one lowers the correct rate by roughly 19.9 percentage points to below the ReAct baseline. The intent recognizer contributes mainly efficiency, reducing the average number of model queries needed per correct answer.","core_discovery":"The discovery, on the paper's own terms, is that parameter generation should be moved from the tool level to the parameter level. Because tools differ in how many parameters they need and how hard they are to invoke, a single generation structure produces missing, irrelevant, or wrong parameters, so the authors assign each tool to one of three handler structures based on its difficulty. Direct Generation handles simple few-parameter calls; Parallel Generation splits multi-condition filters into categories such as time, space, and object, generates each condition separately, and integrates them; Serial Generation builds complex calls like SQL queries step by step, from query framework to column-to-value mapping to final synthesis. Wrapped in an intent recognizer that narrows the toolset and a decomposer that iterates subtask-tool pairs against execution results, the design yields the reported ToolQA gains, and a preference-based variant that favors shorter solving paths reaches a 40.13% overall correct rate against ReAct's 29.93%.","pith_inferences":["If the framework is to generalize, the structure choice itself needs automation: a model that selects direct, parallel, or serial generation from a tool's own schema, tested on tools not seen during design, would establish transferability.","A matched-exemplar ablation, in which the one-structure variant receives the same few-shot examples inside a single combined prompt, would separate the effect of structure choice from the effect of extra exemplars, which the reported comparison leaves entangled.","The parameter-level idea is not confined to question answering: any agent that builds arguments of varying size and coupling, such as REST API calls with optional fields or code interpreters, could use the same direct-parallel-serial split.","TUMS-PRE's preference for short tool paths suggests that a cheap planning prior can substitute for expensive search over tool sequences, a hypothesis worth testing on agents that currently rely on tree or graph search for tool planning."],"forward_implications":["Other tool-augmented agents could adopt the same parameter-level split without retraining, because each tool's generation structure is simply a few-shot prompt and the gain should transfer to any sufficiently capable instruction-following model.","The one-structure ablation implies that intent recognition and task decomposition alone will not beat ReAct on ToolQA; the handler split is what pushes accuracy above the baselines.","Because the intent recognizer lowers the average number of model queries per correct answer, a pipeline with more planning stages can still be cheaper per success than a simpler one.","The processor is explicitly expandable to more structures, so the three-handler design is a starting point rather than a ceiling for covering additional tool types.","Preference-based hints that encode \"the shorter solving path is the better path\" push the overall correct rate from 29.93% for ReAct to 40.13% for TUMS-PRE, pointing to planning priors as a further lever."],"supporting_citations":[{"why":"Supplies the ToolQA benchmark, its 13 tools and eight datasets, and the GPT-3.5-Turbo baseline results that TUMS must beat.","marker":"[46]"},{"why":"ReAct is the primary baseline; its 29.93% overall correct rate is the comparison point for TUMS-PRE, and the one-structure ablation falls below it.","marker":"[41]"},{"why":"Chameleon is the second tool-using baseline, providing the controller-style multi-tool planning comparison.","marker":"[20]"},{"why":"Qwen1.5-72B-Chat is the backbone LLM on which all methods, including TUMS, are evaluated.","marker":"[34]"},{"why":"Chain-of-Thought is the reasoning baseline and the prompting antecedent for few-shot structured generation.","marker":"[37]"}],"fun_headline_variants":["TUMS lifts LLM tool accuracy 50.6% by handling params per-tool","One-size-fits-all LLM params? TUMS says no, lifts accuracy","Parameter-level handling: TUMS beats ReAct on ToolQA","Split LLM param generation: TUMS wins on easy and hard tools"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hand-made assignment of each tool to a direct, parallel, or serial handler, which the paper itself labels as based on 'subjective complexity' and which was made with knowledge of the benchmark's tools, is what produces the reported gains; if those structure choices do not transfer to other tool sets, the improvement could vanish.","fun_headline_variants_meta":{"raw":{"variants":["TUMS lifts LLM tool accuracy 50.6% by handling params per-tool","One-size-fits-all LLM params? TUMS says no, lifts accuracy","Parameter-level handling: TUMS beats ReAct on ToolQA","Split LLM param generation: TUMS wins on easy and hard tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000976,"raw_usage":{"total_tokens":4180,"prompt_tokens":1012,"completion_tokens":3168,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":3085}},"tokens_in":628,"tokens_out":3168,"duration_ms":22753,"temperature":1.0,"reasoning_tokens":3085,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:55:10.168254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out toolset, assign each tool's handler structure automatically from its schema or without looking at the test questions, run TUMS against ReAct, and check whether the advantage survives. A second check: give the one-structure variant the same few-shot examples inside a single combined prompt, and if it then matches TUMS, the gain comes from exemplar count or prompt length rather than from the structural split.","supporting_citations":[{"cited_title":"Advances in Neural Information Processing Systems 36 (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the ToolQA benchmark, its 13 tools and eight datasets, and the GPT-3.5-Turbo baseline results that TUMS must beat."},{"cited_title":"Advances in Neural Information Processing Systems36 (2024)","cited_arxiv_id":null,"evidence_quote":"Chameleon is the second tool-using baseline, providing the controller-style multi-tool planning comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Qwen1.5-72B-Chat is the backbone LLM on which all methods, including TUMS, are evaluated."}],"review_version":1}