{"id":"0641f368-3555-45d9-8b51-9721b46edcd4","arxiv_id":"2506.21967","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Tool-using LLM agents are fragile at every step of the tool pipeline: incomplete documentation, tool hallucinations, and response attacks all degrade performance, and bigger models are not reliably more robust.","lead":"This paper stress-tests AI agents that use external tools by removing API details, injecting fake tool calls, and hiding attack commands inside tool results. Across nine models, agents failed or got manipulated at every stage, with open-source models suffering more and larger models not consistently safer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Forced-output attack success rates lack a no-attack baseline; larger models' natural formatting may explain the 'larger models more vulnerable' claim.","rationale":"I read the paper's central empirical claim as: tool-integrated agents are unstable at each stage, open-source models are more vulnerable, and scaling does not fix reasoning-related hallucination recovery and may increase vulnerability to instruction-like attacks. The documentation-incompleteness and hallucination-recovery experiments have their own limitations (small filtered ToolBench subset, no error bars, an LLM judge, and the ReAct-only scope acknowledged in the Limitations section), but their qualitative direction is plausible and honestly scoped. The least secure supporting pillar is the forced-output attack result, because the measurement instrument conflates attack-induced format compliance with naturally-occurring format compliance. A one-line baseline check would settle whether the reported ordering is real. This does not undermine the paper wholesale; it sharpens the conditions under which the scaling-vulnerability claim should be stated. The reader's weakest assumption focused on representativeness of ReAct and ToolBench; my concern is a narrower, internal confound in the attack-success metric, hence partial agreement.","tokens_in":11753,"tokens_out":4373,"duration_ms":43839,"concrete_test":"Re-run the forced-output attack experiment without injecting any attack prompt: use the same 200 tasks/trajectories and a neutral final tool response, then apply the same GPT-4o-mini judge or an exact string-checker to measure the rate at which final outputs already satisfy each of the four format constraints. Compare these baseline rates to the 'Succ.' column in Table 4 per model. If baseline rates are non-negligible and track model size (e.g., large models already quote or use bullet points often), recompute attack-specific effect = attack_success - baseline_natural. If the effect is flat or decreasing with model size, the 'larger models more vulnerable' conclusion fails. Additionally, report per-attack-prompt baselines, since 'include a quote' may be naturally common while 'include the phrase the following answer is fictional' is essentially impossible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline conclusion in Section 4.3 that increasing model size 'may make agents more vulnerable to attacks that resemble normal user instructions' rests on forced-output attack success rates in Table 4 (e.g., Qwen2.5-Instruct 7B 28.3% vs 72B 37.0%; GPT-4o-mini 21.8% vs GPT-4o 34.7%). The success detector in Appendix F (Figure 6) defines success for forced-output attacks as the final output satisfying the requested format: numbered bullet points, an example/analogy, the phrase 'the following answer is fictional', or a quote from a famous figure. No baseline is reported for how often these models produce such formatting naturally, without any attack prompt. Larger models are known to produce longer, more structured, example- and quote-laden answers; if natural compliance rates already increase with model size, the reported attack-success ordering is confounded and does not establish increased vulnerability to injected instructions. This concern is load-bearing because it is the only quantitative support for the scaling-vulnerability finding, and Section 4.3 presents it as a key takeaway.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically evaluates the stability of tool-integrated LLM agents across the tool invocation pipeline. Using a filtered subset of ToolBench (212 functional queries, 1,067 tools), the authors construct three test conditions: incomplete tool documentation, tool usage hallucinations (wrong tool, empty parameter, wrong parameter name, wrong parameter value), and tool response attacks (information leakage, instruction override, forced output). They evaluate three proprietary and six open-source models (including size pairs from Qwen2.5, Llama-3.1, and InternLM2.5) under the ReAct framework, measuring task completion rates and attack success rates. The main reported findings are that agents are fragile at every stage, open-source models are more vulnerable than proprietary ones, parameter-value hallucinations are not improved by scaling, and forced-output attacks succeed more often on larger models, which the authors attribute to enhanced instruction-following. The paper concludes that stability should be a first-class dimension in agent evaluation.","tokens_in":11926,"tokens_out":5349,"duration_ms":48930,"significance":"If the findings are robust, the paper addresses a genuine gap: existing tool-agent benchmarks largely measure end-to-end task success and neglect failures that occur during documentation reading, tool-call generation, and response processing. The decomposition into three stages is sensible, the manual verification of the filtered ToolBench subset is a strength, and the appendix provides detailed prompts that would allow replication of the attack and hallucination constructions. The comparison across model families and sizes is also useful. However, the main quantitative claims currently rest on small, unreplicated differences without confidence intervals, and the key scaling-vulnerability result for forced-output attacks lacks a necessary no-attack control. With those controls and statistical grounding added, the paper could be a solid contribution to agent robustness evaluation.","major_comments":[{"comment":"The forced-output attack success rates are not interpretable as attack effectiveness without a no-attack baseline. The success detector in Figure 6 counts an attack as successful whenever the final output contains numbered bullet points, an example or analogy, the phrase 'the following answer is fictional,' or a quote from a famous figure. Many LLMs, especially larger instruction-tuned models, produce such formatting naturally in ordinary answers. Because no control condition measures how often the same models produce these formats on the same trajectories without any attack prompt, the reported increases in 'success' from Qwen2.5-Instruct-7B (28.3%) to 72B (37.0%) and from GPT-4o-mini (21.8%) to GPT-4o (34.7%) may simply reflect higher natural format-compliance rates in larger models. This is load-bearing because Section 4.3 uses these numbers as the sole quantitative support for the claim that increasing model size 'may make agents more vulnerable to attacks resembling normal user instructions.' Please add a no-attack control and report the difference relative to that baseline, or remove the scaling-vulnerability conclusion.","section":"Section 4.3, Table 4, Appendix F"},{"comment":"The paper reports only point estimates on 200-212 test cases, with no confidence intervals, significance tests, or variance information. For example, GPT-4o's drop from 64.9% to 62.4% in Table 2 corresponds to roughly five queries, and several cross-model differences in Table 3 are of similar magnitude. Without quantifying sampling error, the conclusions that proprietary models are more robust than open-source models and that scaling has differential effects by hallucination type are not established. Please provide bootstrap confidence intervals or paired significance tests over the query set, since the same queries are reused across conditions.","section":"Tables 2-4"},{"comment":"The 'Orig.' and 'Mod.' baselines are not defined precisely enough to interpret the delta values. For Table 3, it is unclear whether 'Orig.' is the task completion rate on the original successful trajectories used to construct the hallucination prompts, and whether the same trajectories are used for every model; for Table 4, it is unclear whether 'Orig.' is the same set of trajectories with no attack inserted. Appendix D states that trajectories are selected from the documentation experiment and up-sampled to 200; if the selected trajectories differ across models or conditions, the reported deltas are not directly comparable. Please specify the exact baseline construction and confirm that the compared conditions share identical trajectories.","section":"Tables 3-4 and Appendices D-E"},{"comment":"The attack success judge is a single unvalidated GPT-4o-mini prompt. The paper reports no human agreement study, no comparison with an alternative judge, and no calibration of the judge against labeled examples. This is particularly important for information leakage and instruction override, where success depends on subtle semantic judgments about whether the agent 'leaked' information or 'answered' the injected task. Please validate the judge on a sample (e.g., human agreement or a second independent judge) and report agreement statistics.","section":"Appendix F"}],"minor_comments":[{"comment":"The sentence 'This trend is evident in models like GPT-4 and Qwen2.5-Instruct, where such attack success rates rise to 34.7% and 9.5%, respectively' appears to contain transcription errors: Table 4 shows GPT-4o at 34.7% and Qwen2.5-Instruct-72B at 37.0%, while 9.5% appears in an unrelated row or column. Please correct the model names and numbers.","section":"Section 4.3"},{"comment":"The term 'unsucceed' is used repeatedly; 'unsuccessful' or 'not succeeded' would be clearer.","section":"Figure 6"},{"comment":"The first sentence of the Limitations section is ungrammatical: 'highlights that their vulnerability to external factors and reveals intriguing findings' should be rephrased.","section":"Limitations"},{"comment":"The paper uses 'GPT-4' in the main text when the experiments actually use GPT-4o and GPT-4o-mini; please make the model names consistent throughout.","section":"Abstract and Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The skeptical concern about the forced-output baseline is well founded and is the main technical obstacle to the paper's headline conclusion. In addition to adding the no-attack control, the authors should address the absence of error bars and the ambiguous baselines, since these affect most of the quantitative comparisons. I would also encourage the authors to release the filtered dataset and evaluation prompts, as the manual verification and detailed prompts are valuable assets but reproducibility would be strengthened by code and data release."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis paper deserves a serious referee, but the headline scaling-vulnerability claim needs a control before it can be accepted. The broader message—that tool agents are fragile across the whole invocation pipeline—is plausible and worth saying.\n\nThe genuinely new part is the integrated three-stage framing: documentation incompleteness, tool usage hallucination, and response attacks. The authors put real work into cleaning ToolBench, manually filtering to 212 functional queries, and testing nine models across two open-source families with two sizes each. The observation that parameter-value hallucinations are not fixed by scaling, while format-type errors are, is interesting and could be real.\n\nThe soft spots are real, though. The forced-output attack success rates have no no-attack baseline. The stress-test note is right: GPT-4o naturally writes bullet points and quotes; without knowing how often each model does that unprompted, the claim that larger models are more vulnerable to forced-output attacks is confounded. That is load-bearing for Section 4.3, so that conclusion should be softened or supported with a control. Also, there are no error bars or significance tests anywhere, the test sets are small (200-ish queries), and the attack judge is an unvalidated GPT-4o-mini prompt. The 'Orig.' baseline in Tables 3 and 4 is ambiguous—state explicitly whether it's the same trajectory without the injected error/attack. The authors' own limitations (ReAct only, no mechanism, no defense) are stated honestly, which I credit.\n\nThe qualitative pattern from the tables is consistent enough that I don't think the paper is directionally wrong. But the exact percentages, especially the scaling-vulnerability result, should not be taken at face value.\n\nThis paper is for people building and evaluating tool agents, especially those concerned with robustness. A serious referee can see whether the protocol can be tightened. I'd send it to review, but ask for the baseline experiments, error bars, and a clearer evaluation protocol first.","headline":"Useful systematic framing of tool-agent stability, but the scaling-vulnerability claim needs a no-attack baseline before it can be trusted.","tokens_in":12446,"tokens_out":3415,"would_cite":false,"duration_ms":35287,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tool-integrated LLM agents are unstable at every stage of a tool call: incomplete documentation hurts, hallucinated parameter values are rarely corrected, and attacks hidden in tool responses often succeed.","keywords":["tool-integrated agents","agent stability","tool usage hallucination","tool documentation incompleteness","tool response attack","indirect prompt injection","model scaling","ReAct framework"],"falsifier":"A concrete falsifier: rerun the same three perturbation suites on the same tasks but with documentation written by independent human API authors rather than GPT-4o, and with a different agent scaffold; if task-completion drops and attack success rates largely disappear or reverse, the claim that instability runs through the whole tool invocation process would be refuted.","tokens_in":11550,"feed_emoji":"⚠️","tokens_out":7709,"duration_ms":69715,"temperature":0.7,"pith_summary":"This paper argues that tool-integrated LLM agents are not merely imperfect but outright unstable: performance degrades when tool documentation is incomplete, agents usually fail to recover after hallucinating a bad tool call, and instructions concealed inside tool responses can hijack behavior. The authors build three test suites derived from the ToolBench API collection and run them across three proprietary and six open-source models. They report that open-source models are more vulnerable than proprietary ones, that missing parameter descriptions hurt more than missing API-purpose descriptions, and that scaling model size does little for parameter-value hallucinations while sometimes making forced-output attacks more effective. If this picture holds, standard end-to-end tool-usage scores overstate how reliable agents will be in real deployments where documents, calls, and responses all go wrong.","feed_headline":"Tool-using LLM agents fail at every step of a tool call","feed_subtitle":"Incomplete docs, hallucinated parameters, and poisoned responses all break agents; bigger models do not fix it.","key_machinery":"The central object is a three-part stability test suite built around the ReAct agent loop: Tool Documentation Incompleteness (four documentation levels), Tool Usage Hallucination (four error types appended to an otherwise correct trajectory), and Tool Response Attack (three attack types inserted into the final tool response). The quantity that carries the argument is the drop in task-completion rate between original and perturbed runs for the first two suites, plus the attack success rate for the third, judged by a prompted evaluator. This machinery isolates each stage of the pipeline so failures can be attributed to documentation reading, call generation, or response processing rather than to overall accuracy alone.","core_discovery":"The paper's central claim is that instability runs through every phase of a tool invocation—reading tool documentation, generating tool calls, and processing tool responses—and that current agent evaluations miss most of it. Across the constructed perturbations, the authors find that task completion falls when documentation is incomplete, that agents can usually correct a wrong tool choice but often blindly trust an erroneous parameter value (with drops above 12 percent), and that attacks embedded in tool responses succeed at rates ranging from roughly 10 percent to over 90 percent. The paper also claims that increasing model size strengthens instruction-following-related robustness but not reasoning-related robustness, and that attacks mimicking legitimate user instructions, such as forced-output attacks, succeed more often against larger models.","pith_inferences":["A testable extension is to run the same three perturbation suites on non-ReAct scaffolds such as reflection-based or multi-agent systems; the paper itself notes those frameworks may behave differently, so the whole-process instability claim would be stronger if it held across scaffolds.","The documentation-incompleteness result points to a cheap defense the paper outlines but does not implement: pre-filling parameter descriptions from invocation traces or a stronger model, and training on progressively stripped documentation.","The forced-output attack result suggests a broader pattern: any capability that improves instruction following also increases compliance with instruction-shaped text, so security evaluations should include instruction-mimicking attacks whenever instruction-following is being measured.","Because the attack-success detector is itself a prompted LLM, an independent human-annotation study of attack outcomes would test whether the reported success rates are inflated or deflated."],"forward_implications":["End-to-end tool-usage scores should be reported together with perturbation-based stability scores, because an agent that looks strong on clean queries can rank very differently when documents or tool responses are imperfect.","Deployers of open-source agents should treat missing parameter descriptions as a high-risk failure point and complete API documentation before release.","Parameter-value hallucinations are a reasoning failure that scaling alone does not fix, so training or decoding interventions aimed at verifying argument values are needed.","Tool responses must be treated as untrusted input, and defenses against indirect prompt injection—especially information leakage—are needed before tool-using agents handle user data.","Larger models can be more, not less, susceptible to instruction-mimicking attacks, so gains in instruction following do not automatically translate into security."],"supporting_citations":[{"why":"Supplies ToolBench, the real-world API collection whose test queries and tools are filtered into the 212-case evaluation set.","marker":"Qin et al., 2023"},{"why":"Supplies the ReAct prompting framework that turns each LLM into a tool-invoking agent for all three experiments.","marker":"Yao et al., 2022"},{"why":"Defines the indirect prompt injection method used to embed information-leakage, instruction-override, and forced-output attacks into tool responses.","marker":"Greshake et al., 2023"},{"why":"Introduces the notion of tool hallucination that the paper refines into four error types for the tool usage hallucination suite.","marker":"Patil et al., 2023"},{"why":"Provides the OpenAPI specification standard used as the reference for what counts as complete tool documentation.","marker":"SmartBear, 2024"},{"why":"Documents that real API providers often supply incomplete documentation, motivating the documentation-incompleteness test.","marker":"Rapid, 2023"},{"why":"Specifies the GPT-4 family of proprietary models used as the closed-source comparison point.","marker":"Achiam et al., 2023"},{"why":"Specifies the Qwen2.5 model family, one of the open-source families compared across two sizes.","marker":"Yang et al., 2024"},{"why":"Specifies the Llama-3.1 model family, the second open-source family compared across sizes.","marker":"Dubey et al., 2024"},{"why":"Specifies the InternLM2.5 model family, the third open-source family compared across sizes.","marker":"Cai et al., 2024"}],"fun_headline_variants":["LLM agents break at every tool-call stage","Tool agents: fragile from docs to responses","Bigger models don't fix tool-agent fragility","Tool-using agents: vulnerable at each step","Agents crash on docs, params, and responses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 212 filtered real-world API tasks, the ReAct agent setup, and the GPT-4o-generated 'complete' documentation are a fair stand-in for how tool-integrated agents are actually deployed, so the vulnerability gaps measured here are the ones that would appear in practice.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents break at every tool-call stage","Tool agents: fragile from docs to responses","Bigger models don't fix tool-agent fragility","Tool-using agents: vulnerable at each step","Agents crash on docs, params, and responses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000618,"raw_usage":{"total_tokens":2811,"prompt_tokens":828,"completion_tokens":1983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":1911}},"tokens_in":444,"tokens_out":1983,"duration_ms":14674,"temperature":1.0,"reasoning_tokens":1911,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:13:51.467643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete falsifier: rerun the same three perturbation suites on the same tasks but with documentation written by independent human API authors rather than GPT-4o, and with a different agent scaffold; if task-completion drops and attack success rates largely disappear or reverse, the claim that instability runs through the whole tool invocation process would be refuted.","supporting_citations":[],"review_version":1}