Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes a five-category taxonomy of parameter-filling failures in LLM tool agents and shows, via 15 perturbation methods, that hallucinated parameter names stem from the LLM itself while the other four failure patterns are…

desk verdict Useful taxonomy and perturbation suite for parameter-filling failures, but the causal claim about inherent LLM limitations outruns the experimental design. read the letter →

arxiv 2507.15296 v1 pith:KMML47GG submitted 2025-07-21 cs.SE cs.AI

Qian Xiong , Yuekai Huang , Ziyou Jiang , Zhiyuan Chang , Yujia Zheng , Tianhao Li , Mingyang Li This is my paper
classification cs.SEcs.AI
keywords toolagentsparameterfillingLLMhallucinationfailuretaxonomyinputperturbationLLaMatoolchainreliabilityempiricalstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper establishes a five-category taxonomy of parameter-filling failures in LLM tool agents—missing information, redundant information, hallucinated parameter names, task deviation, and specification mismatch—and then asks where each category comes from. To answer that, the authors apply 15 perturbation methods to the three input sources an agent reads (tool documents, user queries, tool returns) and measure how often four LLMs shift from correct behavior into each failure category. The central result is that hallucinated parameter names stay rare and nearly unchanged under every perturbation, indicating they come from the LLM's own parameter-name inference, while degraded input sources are the main drivers of the other four failure patterns. If correct, this gives tool-agent builders a concrete division of labor: fix input quality to handle the common failures, and treat hallucinated parameter names as a model-level problem to be addressed by better schema grounding or validation rather than by cleaning the inputs.

What carries the argument

The central machinery is the parameter-filling step inside the agent's invocation chain, turned into a test subject by treating initial correct trajectories as test oracles and applying perturbations to the three input sources the agent reads. The failure taxonomy itself is the central object, built through grounded-theory open coding and constant comparative analysis of behavior trajectories until annotator agreement surpassed a Cohen's Kappa of 0.9. The 15 perturbation methods are the experimental instrument: six alter tool documents (removing descriptions or examples, swapping or corrupting descriptions and types), four alter user queries (removing or complicating parameters, adding noise), and five alter tool returns (fuzzing keys, adding ID prefixes, changing key naming conventions, corrupting JSON format). Failures are scored by comparing each perturbed trajectory against the original trajectory, using the failure taxonomy plus ROUGE-L with a 0.8 threshold to judge task deviation and specification mismatch.

What would settle it

Select a random sample of the 600 behavior trajectories for one LLM and have independent annotators label every parameter value against the original user query and the tool schema, without seeing the paper's oracle labels. If a meaningful fraction of cases the paper counts as 'passed' contain parameter values that are wrong but happened to let the tool succeed, the oracle is contaminated and the taxonomy plus the 'hallucinated names are inherent' conclusion would not be reproduced.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that parameter failures in API-based tool agents form a stable five-category taxonomy and that the categories have different input-source sensitivities. Hallucinated parameter names are the outlier: their failure rate stays low and roughly flat across all 15 perturbations, which the paper takes as evidence that the LLM's internal parameter-name generation is the cause. By contrast, wrong-type descriptions in tool documents are the strongest trigger of specification mismatch and task deviation; removing the first or last parameter from a user query pushes task-deviation rates above 50% on nearly all tested LLMs; and corrupt JSON in tool returns is the top trigger of missing-information failures. The results also show that over half of failed trajectories exhibit multiple failure patterns, so failures transfer along the chain.

Load-bearing premise

The study assumes the initial successful behavior trajectories produced by ToolLLaMa are correct enough to serve as the standard of comparison; if those trajectories contain hidden mistakes, the taxonomy and every perturbation-induced failure rate are measured against a contaminated reference.

Editorial extensions

If this is right

  • If the taxonomy holds, tool-agent builders can treat the five failure categories as a checklist for tracing an invocation failure back to its source.
  • Since wrong-type descriptions in tool documents are the strongest trigger of specification mismatch and task deviation, adding a parameter data-type verification step before invocation should reduce these failures.
  • Since removing a parameter from the user query pushes task deviation above 50% on nearly all tested LLMs, query-completeness checking or user-facing templates become a practical safeguard.
  • Because corrupt JSON in tool returns is the top trigger of missing-information failures, standardizing return formats and avoiding truncated returns should improve downstream parameter parsing.
  • Hallucinated parameter names will not disappear by cleaning inputs; they need model-side remedies such as stronger tool-schema grounding or post-invocation validation against the schema.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors only gesture at is that the failure transfer effect (more than half of failed trajectories show multiple patterns) means per-perturbation failure rates understate end-to-end damage: one small input defect can cascade through the chain before the final answer.
  • The 'hallucinated names are inherent to the LLM' conclusion is inferred from stability across perturbations; a direct test would compare the same perturbations on models with and without tool-specific fine-tuning, which would separate schema-grounding ability from general language-model behavior.
  • The perturbation harness itself is transferable: the same 15 methods could be run on non-API tools (command-line or library calls) or in multi-turn conversations to see whether the taxonomy and the input-source attributions survive those settings.
  • A natural extension is to use the taxonomy as a diagnostic layer in production: classify each failed call into one of the five categories and route it to the appropriate remedy, which the paper does not implement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies parameter-filling failures in LLM tool-agent systems. The authors construct a taxonomy of five failure categories (Missing Information, Redundant Information, Hallucination Name, Task Deviation, Specification Mismatch) by applying grounded theory to behavior trajectories obtained from ToolLLaMa. They then design 15 perturbation methods targeting three input sources (tool documents, user queries, tool returns), run the perturbed inputs through four LLMs, and compare the resulting trajectories as test cases against 'initial correct behavior trajectories' treated as oracles. Based on the failure rates in Table 1, they conclude that hallucinated parameter names stem primarily from inherent LLM limitations, while input-source problems mainly cause other failure patterns, and they offer design suggestions for tool agents.

Significance. If the central attribution claim were established, the taxonomy and the 15 perturbation methods would be a useful empirical contribution to tool-agent evaluation and debugging, and the released code and dataset would support further research. The taxonomy itself is plausible and the perturbation suite is thoughtfully designed. However, the paper's headline causal conclusion—that 'parameter name hallucination failure primarily stems from inherent LLM limitations'—is not uniquely supported by the current experimental design, because there is no unperturbed baseline row and because the oracle trajectories are not independently validated. These issues are load-bearing rather than cosmetic.

major comments (4)
  1. [§5.2, Eq. (1), Table 1] The central causal claim is under-supported by the absence of an unperturbed control condition. Eq. (1) defines the failure rate against Ntotal, but Table 1 reports only perturbation-condition rates; there is no row for the failure rate of the initial, unperturbed trajectories. Low and flat Hallucination Name rates across all 15 perturbations (for example, 0.00–0.33% for ToolLLaMA-v2 and 0.00–0.17% for GPT-4o-mini) cannot by themselves show that the failure is 'inherent' to the LLM: the same pattern would arise if the baseline hallucination rate were already nonzero, or if the oracle systematically classified hallucinated parameter names as passing. The authors should add an unperturbed baseline row and report confidence intervals or significance tests for the differences.
  2. [§4 (oracle definition), §5.1 (filtering)] The manuscript states 'we treat the initial correct behavior trajectories as test oracles' but never specifies how correctness was established. If an 'initial correct' trajectory contains an undetected failure—such as a hallucinated parameter name, of which Appendix A.4 itself shows examples—then Eq. (1) counts that failure as a pass, depressing the measured Hallucination Name rate and biasing the attribution toward 'inherent LLM limitations.' The authors should describe the oracle-validation procedure (e.g., independent annotation, tool-execution success checks, or a held-out correctness criterion) and report agreement on the oracle labels, not only on the taxonomy coding in Section 3.
  3. [§5.2 (Rouge-L threshold), §5.1 (preprocessing)] The use of a fixed Rouge-L threshold of 0.8 for detecting Task Deviation and Specification Mismatch is arbitrary, and no sensitivity analysis is provided. The observed conclusions (e.g., that user-query removal leads to severe task deviation) could shift with a different threshold. Similarly, the preprocessing step that 'excluded data with originally unsolvable queries' may remove naturally occurring failures and thereby distort the failure rates; the authors should report how many queries were excluded, on what basis, and whether the main conclusions are robust to these choices.
  4. [§3 (taxonomy construction) and §5 (evaluation)] The taxonomy is mined from ToolLLaMa's own behavior trajectories in Section 3 and then applied to score ToolLLaMA-v2 and other models in Section 5; its fit to the ToolLLaMA family is partly guaranteed by construction. The perturbation analysis is an independent probe and mitigates this concern, but the claim of a comprehensive, generalizable taxonomy would be stronger if the categories were validated on an independently constructed failure corpus or if the taxonomy were annotated on trajectories from more than one agent framework. As written, the paper should temper the 'comprehensive' wording and acknowledge this circularity explicitly.
minor comments (5)
  1. [§1 (title and Figure 1)] The 'Butterfly Effect' metaphor is used in the title and abstract but is never defined or connected to a measurable phenomenon; consider adding an explicit explanation of how small parameter errors propagate through the toolchain.
  2. [§5.1 (data description)] The paper says 600 behavior trajectories are obtained for each investigated LLM, but it is not stated whether these 600 queries are the same across all four LLMs or are generated/model-specific; this is important for comparability across rows in Table 1.
  3. [§3 (Grounded Theory)] The description of the grounded-theory process reports a Cohen's Kappa score 'above 0.9' but does not give the actual score or the number of annotation rounds; please report the final agreement value and the number of trajectories annotated.
  4. [§4.2 (User Query Perturbation)] There is a typo in the bullet item: 'RP F' should be 'RPF', and the same inconsistency appears in Table 1 as 'W D' instead of 'WD'.
  5. [Table 1 (statistical presentation)] The table reports rates to two decimal places without any measure of uncertainty; since these are proportions from finite samples, adding confidence intervals or an explicit statement of the number of trials per cell would greatly improve interpretability.

Circularity Check

1 steps flagged · score 6.0 of 10

The oracle is defined as the same model's own unperturbed trajectories, so the measured failure rates and the 'inherent LLM limitation' conclusion reduce to self-consistency rather than ground truth.

  1. self definitional [Section 4 (Methodology); Section 5.1 (Data Preprocessing); Section 5.2 (Setting), Eq. (1)]
    "Based on this concept, we treat the initial correct behavior trajectories as test oracles and perturb them to generate the test samples. ... For each 600 LLM-specific initial data, we apply 15 perturbation methods ... We use oracles and the derived failure taxonomy. The parameter filling behavior of the LLM is compared against the oracle. If no failure patterns are detected, the test case is considered passed, otherwise, it is marked as failed."

    The oracle is each model's own unperturbed trajectories ('LLM-specific initial data'), so Npass in Eq. (1) measures self-consistency under perturbation, not correctness against tool requirements. The failure taxonomy was itself mined from ToolLLaMa trajectories in Section 3 ('After obtaining the behavior trajectories, they will be utilized for mining failure patterns'), and those trajectories contained the failures being classified. Thus for ToolLLaMA-v2 (and arguably all models) the 'correct' reference is not independently verified; it is the same behavioral stream from which the categories were induced.

full rationale

The central circular step is the construction of the test oracle. Section 4 states that 'the initial correct behavior trajectories' are used as oracles, and Section 5.1 clarifies that these are the 600 'LLM-specific initial data' obtained for each evaluated model. That makes the reference standard the same system whose parameter-filling failures the paper is trying to explain. In parallel, Section 3 mined the failure taxonomy from ToolLLaMa's behavioral trajectories, which by definition contained failures; otherwise there would have been no patterns to discover. Consequently, the computation of Failure Rate (FR = 1 - Npass/Ntotal) in Eq. (1) is not anchored to an external ground truth such as tool API specifications, but to the model's own unperturbed behavior. Any perturbed trajectory that differs from this baseline in a way the taxonomy labels as a failure is counted as a failure, while failures already present in the oracle are implicitly treated as correct. This directly undermines the abstract's headline attribution that 'parameter name hallucination failure primarily stems from inherent LLM limitations': the low, flat Hallucination Name rates in Table 1 are measured against a contaminated or self-referential reference, so the claim that input sources do not affect this failure mode is not uniquely supported. The perturbation methods themselves are concrete and externally meaningful, and the taxonomy categories have semantic definitions, so the paper is not entirely fabricated; however, the load-bearing causal conclusion about failure sources reduces, at least partially, to the way the oracle and taxonomy were defined from the very trajectories under analysis. No self-citation chain is involved; the circularity is internal to the experimental design.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The paper contributes an empirical taxonomy rather than a mathematical derivation, so its ledger is dominated by evaluation choices and representativeness assumptions. The Rouge-L threshold and oracle construction are the main hand-set quantities; the claimed comprehensiveness rests on the transferability of a taxonomy built from one agent family.

free parameters (1)
  • Rouge-L threshold = 0.8
    Chosen in Section 5.2 to decide when Task Deviation and Specification Mismatch count as failures; no sensitivity analysis is provided, so the threshold influences all reported rates.
assumptions (5)
  • domain assumption Initial correct behavior trajectories from ToolLLaMa are valid test oracles for all four evaluated LLMs.
    Section 4 treats initial trajectories as oracles; if they contain hidden errors, all failure-rate measurements inherit that contamination.
  • domain assumption The five-category taxonomy built from ToolLLaMa behavior transfers to other LLMs and tool sets.
    Section 3 mines patterns from ToolLLaMa and Section 5.2 applies them to GPT-3.5, GPT-4o-mini, Qwen2.5-Plus, and ToolLLaMA-v2 without validating taxonomy completeness on those models.
  • domain assumption Each perturbation method changes only the intended input source.
    Section 4 assumes RD, RPF, FK, etc. isolate effects of tool document, user query, or tool return; in practice a corrupted JSON format may also change semantic content.
  • ad hoc to paper Rouge-L similarity to the oracle at threshold 0.8 is a faithful detector of Task Deviation and Specification Mismatch.
    Section 5.2 sets the threshold without calibration; reported failure rates depend on this choice.
  • domain assumption The benchmark queries and four evaluated LLMs are representative of real tool-agent usage.
    Data come from ToolEyes/ToolBench and results are limited to English, single-turn, API-style invocations, as the Limitations section acknowledges.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems." pith.science (2026). https://pith.science/paper/KMML47GG

@misc{pith2026250715296,
  author       = {Pith},
  title        = {Pith review of: Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KMML47GG}},
  note         = {Machine review of arXiv:2507.15296}
}
read the original abstract

The emergence of the tool agent paradigm has broadened the capability boundaries of the Large Language Model (LLM), enabling it to complete more complex tasks. However, the effectiveness of this paradigm is limited due to the issue of parameter failure during its execution. To explore this phenomenon and propose corresponding suggestions, we first construct a parameter failure taxonomy in this paper. We derive five failure categories from the invocation chain of a mainstream tool agent. Then, we explore the correlation between three different input sources and failure categories by applying 15 input perturbation methods to the input. Experimental results show that parameter name hallucination failure primarily stems from inherent LLM limitations, while issues with input sources mainly cause other failure patterns. To improve the reliability and effectiveness of tool-agent interactions, we propose corresponding improvement suggestions, including standardizing tool return formats, improving error feedback mechanisms, and ensuring parameter consistency.

Figures

Figures reproduced from arXiv: 2507.15296 by the authors.

Figure 1
Figure 1. The process by which a tool agent resolves a user query, as well as the parameter issues that may occur [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The process of systematically mining the failure patterns using the Grounded Theory Approach [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The process of data preprocessing in our experiment. Enhanced behavioral trajectories were obtained [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Heatmap of the correlation between the transi [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use

    cs.AI 2026-07 unverdicted novelty 7.0 of 10

    Static SFT and RL training for tool-use agents leads to performance drops under open-world distributional shifts across perception, interaction, reasoning and internalization; perturbation-augmented fine-tuning is pro...

  2. When Agents Look the Same: Quantifying Distillation-Induced Similarity in Tool-Use Behaviors

    cs.CL 2026-04 unverdicted novelty 7.0 of 10

    New RPS and AGS metrics show within-family distilled LLM agents have 5.9 pp higher tool-use graph similarity than cross-family pairs, with some models exceeding their teachers.

  3. Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures

    cs.AI 2026-07 conditional novelty 6.0 of 10

    The paper introduces a 41-mode taxonomy that assigns each agent failure to an interaction edge and a fault side, and shows LLM judges can reproduce the labels with Cohen's κ=0.76.

  4. Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments

    cs.AI 2026-05 unverdicted novelty 5.0 of 10

    NoisyAgent trains LLM agents with controlled user and tool noise to improve robustness in stochastic environments while also boosting clean-benchmark performance.

Reference graph

Works this paper leans on

31 extracted references · 8 canonical work pages · cited by 4 Pith papers

  1. [1]

    Pan, Shuyi Yang, Lakshya A

    Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. 2025. https://api.semanticscholar.org/CorpusID:277103715 Why do multi-agent llm systems fail? ArXiv, abs/2503.13657

    arXiv 2025

  2. [2]

    Glaser and Anselm L

    Barney G. Glaser and Anselm L. Strauss. 1967. The Discovery of Grounded Theory: Strategies for Qualitative Research. Aldine Publishing Company

    work page 1967

  3. [3]

    Shijue Huang, Wanjun Zhong, Jianqiao Lu, Qi Zhu, Jiahui Gao, Weiwen Liu, Yutai Hou, Xingshan Zeng, Yasheng Wang, Lifeng Shang, Xin Jiang, Ruifeng Xu, and Qun Liu. 2024. https://api.semanticscholar.org/CorpusID:267320882 Planning, creation, usage: Benchmarking llms for comprehensive tool utilization in real-world complex scenarios . ArXiv, abs/2401.17167

    arXiv 2024

  4. [4]

    Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2023. https://api.semanticscholar.org/CorpusID:263672025 Metatool benchmark for large language models: Deciding whether to use tools and which to use . ArXiv, abs/2310.03128

    arXiv 2023

  5. [5]

    S. H. Khandkar. 2009. Open coding. (23)

    work page 2009

  6. [6]

    Richard Landis and Gary G

    J. Richard Landis and Gary G. Koch. 1977. http://www.jstor.org/stable/2529310 The measurement of observer agreement for categorical data . Biometrics, 33(1):159--174

    arXiv 1977

  7. [7]

    Minghao Li, Feifan Song, Yu Bowen, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. https://api.semanticscholar.org/CorpusID:258179056 Api-bank: A comprehensive benchmark for tool-augmented llms . In Conference on Empirical Methods in Natural Language Processing

    work page 2023

  8. [8]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

Show all 31 references
  1. [9]

    Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Felix Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, Zirui Wang, and Ruoming Pang. 2024. https://api.semanticscholar.org/CorpusID:271843446 Toolsandbox: A stateful, conversational, interactive evaluation bench...

    2024 arXiv

  2. [10]

    Kangyun Ning, Yisong Su, Xueqiang Lv, Yuanzhe Zhang, Jian Liu, Kang Liu, and Jinan Xu. 2024. https://api.semanticscholar.org/CorpusID:271270637 Wtu-eval: A whether-or-not tool usage evaluation benchmark for large language models . ArXiv, abs/2407.12823

    2024 arXiv

  3. [11]

    OpenAI. 2022. OpenAI: Introducing ChatGPT . https://openai.com/blog/chatgpt

    2022

  4. [12]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o/ Hello gpt-4o

    2024

  5. [13]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. https://api.semanticscholar.org/CorpusID:258865184 Gorilla: Large language model connected with massive apis . ArXiv, abs/2305.15334

    2023 arXiv

  6. [14]

    Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun

    Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. https://api.semanticscholar.org/CorpusID:260334759 T...

    2023 arXiv

  7. [15]

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Jirong Wen. 2024. https://api.semanticscholar.org/CorpusID:270067624 Tool learning with large language models: A survey . ArXiv, abs/2405.17935

    2024 arXiv

  8. [16]

    Maddison, and Tatsunori Hashimoto

    Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2023. https://api.semanticscholar.org/CorpusID:262944419 Identifying the risks of lm agents with an lm-emulated sandbox . ArXiv, abs/2309.15817

    2023 arXiv

  9. [17]

    Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yue Ting Zhuang. 2023. https://api.semanticscholar.org/CorpusID:265506220 Taskbench: Benchmarking large language models for task automation . ArXiv, abs/2311.18760

    2023 arXiv

  10. [18]

    Simranjit Singh, Michael Fore, and Dimitrios Stamoulis. 2024. https://api.semanticscholar.org/CorpusID:269502562 Evaluating tool-augmented agents in remote sensing platforms . ArXiv, abs/2405.00709

    2024 arXiv

  11. [19]

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. https://api.semanticscholar.org/CorpusID:259108190 Toolalpaca: Generalized tool learning for language models with 3000 simulated cases . ArXiv, abs/2306.05301

    2023 arXiv

  12. [20]

    Jize Wang, Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. 2024 a . https://api.semanticscholar.org/CorpusID:271097480 Gta: A benchmark for general tool agents . ArXiv, abs/2407.08713

    2024 arXiv

  13. [21]

    Pei Wang, Yanan Wu, Zekun Moore Wang, Jiaheng Liu, Xiaoshuai Song, Zhongyuan Peng, Ken Deng, Chenchen Zhang, Jiakai Wang, Junran Peng, Ge Zhang, Hangyu Guo, Zhaoxiang Zhang, Wenbo Su, and Boyuan Zheng. 2024 b . https://api.semanticscholar.org/CorpusID:273350927 Mtu-bench: A mu...

    2024 arXiv

  14. [22]

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Qin Liu, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, and 11 others. 2023. https://api.semanticscholar....

    2023 arXiv

  15. [23]

    Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang, Jingren Zhou, Junyang Lin, and 25 others. 2024. https://api...

    2024 arXiv

  16. [24]

    Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Munan Ning, and Li Yuan. 2023. https://api.semanticscholar.org/CorpusID:263608740 Llm lies: Hallucinations are not bugs, but features as adversarial examples . ArXiv, abs/2310.01469

    2023 arXiv

  17. [25]

    Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024 a . https://api.semanticscholar.org/CorpusID:266693212 Tooleyes: Fine-grained evaluation for tool learning capabilities of large lang...

    2024 arXiv

  18. [26]

    Junjie Ye, Sixian Li, Guanyu Li, Caishuang Huang, Songyang Gao, Yilong Wu, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024 b . https://api.semanticscholar.org/CorpusID:267740273 Toolsword: Unveiling safety issues of large language models in tool learning across three stages . ArXi...

    2024 arXiv

  19. [27]

    Michal Young. 2001. https://api.semanticscholar.org/CorpusID:10111645 Test oracles

    2001

  20. [28]

    Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2024 a . https://api.semanticscholar.org/CorpusID:273098793 Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents . ArXi...

    2024 arXiv

  21. [29]

    Yuanhe Zhang, Zhenhong Zhou, Wei Zhang, Xinyue Wang, Xiaojun Jia, Yang Liu, and Sen Su. 2024 b . https://api.semanticscholar.org/CorpusID:274822339 Crabs: Consuming resource via auto-generation for llm-dos attack under black-box settings

    2024

  22. [30]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  23. [31]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.