Pith. sign in

REVIEW 4 major objections 5 minor 60 references

DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adaptively editing tool documentation, field by field, lifts LLM agent task success by 95.89% over the original docs, the paper argues.

desk verdict A genuinely useful empirical taxonomy and a sensible adaptive-documentation framework, but the headline gains rest on a test set that is double-dipped for model and budget selection, so the magnitudes are not yet solid. read the letter →

arxiv 2608.10037 v1 pith:XPYQHHCH submitted 2026-08-10 cs.LG cs.AI

You Lu , Kun Zhang , Bihuan Chen , Xin Peng This is my paper · ORCID
classification cs.LGcs.AI
keywords tooldocumentationLLMagentsinformationfieldsexecutiontracediagnosisadaptiveoptimizationtasksuccessratefield-leveleditingfailedfeedback
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that tool documentation is not a fixed input but a tunable resource for LLM agents. It surveys 14 tool-use datasets, identifies 17 common information fields, and finds that adding or removing a single field changes task success by 6.34 percentage points on average, with effect directions flipping across task domains, LLM backbones, and agent architectures. To exploit this, the paper proposes DocsChisel, which analyzes failed execution traces, plans add/remove/refine operations on documentation fields, and iteratively revises each tool's documentation using domain-level accumulated memory. The central claim is that this adaptive process improves task success rate by 95.89% over original documentation and by 75.15% over two existing optimizers, while adding modest token overhead and about 12.65 minutes of optimization time per tool. If true, documentation quality should be treated as a first-class optimization target in agent deployment, not a static artifact.

What carries the argument

The load-bearing mechanism is a closed optimization loop operating at the information-field level. First, domain-aware execution profiling partitions each tool's queries into optimization and validation sets and collects failed execution traces under the original documentation. Then an LLM diagnoses each failed trace against the current documentation and a growing domain memory, mapping failures to responsible fields; a planner merges these diagnoses into add, remove, and refine operations anchored to the original documentation; and a generator applies the operations. Each candidate documentation is re-executed on validation and optimization sets for regression-aware evaluation, and accepted candidates are stored for final selection. Domain-level memories of failure patterns, effective operations, and editing rules are updated after each iteration, which the ablation study credits with a 76.47% relative task-success gain by the fifth iteration over the no-memory variant.

What would settle it

Re-run the main comparison under a strict protocol: freeze the optimization model and iteration budget on a development split, exclude every multi-tool query from any test set whose tool also had that query in an optimization set, and then measure DocsChisel's task-success gain; if the 95.89% improvement over original documentation collapses toward the variance of repeated runs, the reported effect is largely selection-driven.

Watch

Extended reading notes

Core claim

The central discovery is heterogeneity: tool documentation across ecosystems supplies very different information fields, and the usefulness of each field depends on the agent setting. Among 17 identified fields, only tool name and functionality description appear in all 14 datasets, while usage guidance and invocation constraint appear in only two. Controlled experiments show that the same field can help in one task domain and hurt in another, help one LLM backbone and hurt another, and reverse its effect when switching from a single-agent to a multi-agent execution flow. The paper therefore concludes that no fixed documentation convention generalizes across agent settings, and that an effective optimizer must adapt field composition to the target agent. DocsChisel is the proposed instantiation of that idea, and the paper reports that it consistently improves both tool invocation correctness and task success rate across all evaluated domains, backbones, and agent architectures.

Load-bearing premise

The central claim assumes the held-out test set remained truly independent of the optimizer's design choices: the same test set was used in the sensitivity analysis to select the optimization model and iteration budget, and multi-tool queries appear in several tools' query sets, so any leakage there would inflate the reported gains.

Editorial extensions

If this is right

  • Tool documentation should become an explicit optimization artifact in agent deployment pipelines, alongside prompts, retrieval indices, and model selection.
  • Documentation built for one agent setting should not be assumed portable; teams would likely need per-domain, per-backbone, and per-architecture field choices rather than a single template.
  • Failed execution traces are a recoverable signal: organizations already logging agent failures could feed them into a similar diagnose-plan-generate loop without requiring new labeled data.
  • Because the method trades roughly 12.65 minutes of offline optimization per tool for higher success, it is practical for moderate-sized tool sets, with the main cost being compute rather than human curation.
  • The reported consistency across repeated runs suggests the optimization reduces variance in agent outcomes, not just the average success rate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • [Editorial inference] The 17-field taxonomy could be reused as an audit schema: measuring which fields a documentation set lacks would predict which agent settings are most likely to fail, before any optimization run.
  • [Editorial inference] The domain-memory mechanism resembles transfer learning across tools; if it generalizes, one could pretrain memories on a cheaper backbone and apply them to an expensive target agent, cutting optimization cost substantially.
  • [Editorial inference] A testable extension is to let the optimizer learn when to merge similar tools or split overloaded documentation, operations outside the current add/remove/refine vocabulary.
  • [Editorial inference] The 6.34 percentage-point average swing from a single field is comparable in size to typical prompt-engineering effects, suggesting documentation and prompt optimization should be co-optimized rather than tuned in isolation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an empirical study of tool documentation for LLM agents, identifying 17 information fields and showing that their effects on task success vary across task domains, LLM backbones, and agent paradigms. Based on this motivation, the authors propose DocsChisel, a framework that iteratively optimizes tool documentation at the information-field level using failed execution traces, field-level operations, and domain-level memory. The evaluation compares DocsChisel against original documentation, EasyTool, and DRAFT across 9 domains, 3 backbones, and 2 agent paradigms, reporting a 95.89% average improvement in task success rate over original documentation and 75.15% over baselines, with 3x5 repeated runs and Mann-Whitney U tests with Holm correction.

Significance. If the reported results hold, the paper makes a useful contribution to tool documentation engineering for LLM agents: it provides a systematic field-level taxonomy, demonstrates domain/backbone/paradigm dependence, and offers a practical adaptive optimization framework with reproducible artifacts and a statistically rigorous main evaluation protocol. The strengths include the 3x5 repeated evaluation, the use of non-parametric significance testing with multiple-testing correction, and the design of memory-guided field-level operations rather than unconstrained rewriting. However, the central empirical claim of consistent gains is currently undermined by the RQ5 practice of selecting the optimization model and iteration budget on the same test set used for the main RQ3 results, and by an unresolved ambiguity in how multi-tool queries are assigned to tool-specific optimization and test splits.

major comments (4)
  1. [Section IV-D and Section IV-A] The held-out test set is not independent of the optimization choices. In RQ5, the authors vary the optimization model and iteration budget and report TC and TS 'on the test set' (Section IV-D), and they use those test-set results to select Claude Haiku 4.5 as the optimization model (Table IV) and to set the default iteration budget to five (Fig. 9). The main RQ3 results are then reported with that same configuration on the same test set, directly contradicting the claim in Section IV-A that 'the test set is held out throughout the optimization process.' Because the baselines are not given the same test-based selection opportunity, the comparison is systematically favorable to DocsChisel. The authors should either move the model/iteration selection to the validation set and report RQ3 on a genuinely untouched test set, or justify that the RQ5 decisions were made before any test access and provide evidence that the selected configuration is not an artifact of test-set overfitting.
  2. [Section III-B and Section IV-A] The query splitting procedure is under-specified with respect to multi-tool queries. Section III-B states that 'a query involving multiple tools belongs to multiple tool-specific query sets,' and Section IV-A describes a global 5:1:4 split over user queries, but it is not stated whether a query assigned to the global test set can still appear in the tool-specific optimization or validation sets of another tool. If a test query can appear in the optimization set of a different tool, then the 'held-out' test queries have influenced documentation optimization indirectly, inflating the reported gains. The authors should clarify whether the split is global over queries or per tool, and if multi-tool queries are assigned to multiple tool-specific sets, they should verify and report that the global optimization, validation, and test sets are disjoint.
  3. [Section II-C and Fig. 1] The RQ2 impact analysis reports point estimates without confidence intervals or per-domain sample sizes, which weakens the load-bearing claim that information-field effects are heterogeneous across domains, backbones, and paradigms. For example, the claim that removing POF improves TS by 7.5 percentage points for Claude Haiku 4.5 but decreases it for GPT-4o and GLM-5 may be within noise if the data-analysis domain has few queries per tool. Since this heterogeneity is the stated motivation for the entire framework, the authors should report the number of queries per domain/bucket and provide uncertainty estimates or significance tests for the field-effect comparisons.
  4. [Section IV-B and Fig. 8] The average improvement percentages reported in RQ3 aggregate relative gains across heterogeneous settings, which can be misleading when baseline TS is low. For instance, a relative TS improvement of 194.22% across nine domains is driven largely by settings where the original TS is near zero. The authors should report absolute percentage-point changes alongside relative percentages, and consider reporting per-setting absolute differences in addition to the averaged relative improvements.
minor comments (5)
  1. [Section IV-A] The paper states that a stratified split over task domains and tools was performed but does not include the detailed per-domain and per-tool statistics that would allow the reader to assess balance; these details should be provided in the appendix or on the project website.
  2. [Section IV-B] In the boxplot description, the authors claim that DocsChisel reduces the interquartile range of TC by 41.56% and 52.93% compared with EasyTool and DRAFT, but it is unclear whether these IQR reductions are computed over the 15 runs per setting or pooled across settings; this should be clarified.
  3. [Section IV-D] The sentence 'GLM-5 also outperforms GPT-4o, with relative gains of 55.26% in TC and 67.67% in TS' refers to GPT-4o's lower absolute values, but the wording is easy to misread; please rephrase to make the reference point explicit.
  4. [Section IV-E] The ablation statement 'DOCSCHISEL-NoMem reduces TC and TS by 13.64% and 76.47% compared with DOCSCHISEL' is directionally confusing because NoMem is the ablated variant; it would be clearer to say 'DOCSCHISEL outperforms DOCSCHISEL-NoMem by...'.
  5. [Section V] The threats-to-validity section mentions inter-rater agreement for field identification but does not specify the number of tools or fields annotated per rater for the kappa computation; adding this detail would strengthen the reproducibility of the taxonomy.

Circularity Check

1 steps flagged · score 4.0 of 10

Main gains are reported on the same test set used to select the optimization model and iteration budget, so the headline improvements are not a clean held-out estimate.

  1. fitted input called prediction [Section IV-D (RQ5 Sensitivity Analysis), Table IV and Fig. 9; Section IV-A RQ Setup]
    "For RQ5, we examine how different optimization models and iteration budgets affect DOCSCHISEL in the data analysis domain. ... We report the average TC and TS on the test set using the optimized documentation generated under each setting. ... Claude Haiku 4.5 achieves the best performance ... We therefore set the default iteration budget to five. ... For RQ3 and RQ4, we apply DOCSCHISEL with Claude Haiku 4.5 as the optimization model ... We set the maximum number of iterations to 5 by default."

    The paper selects two global configuration choices of DocsChisel, namely the optimization LLM (Claude Haiku 4.5) and the iteration budget (5), by comparing TC and TS on the test set in RQ5. It then reports the main effectiveness numbers, including the 95.89% TS improvement over original documentation and 75.15% over baselines, using that same configuration evaluated on the same test set. Thus the held-out test set is used to tune the very method whose prediction is measured on it, while the baselines are not given an equivalent test-based selection opportunity. The reported gains are therefore a selected maximum rather than an unbiased held-out estimate.

full rationale

The derivation chain of DocsChisel is not definitionally circular: per-tool documentation is generated from the optimization set and selected on a validation set (Sections III-D and III-E), and the final comparison is made against original documentation and baselines on a nominally held-out test set. No self-citation chain or imported uniqueness theorem is load-bearing. However, the main evaluation is partially circular because RQ5 uses the test set to choose the optimization model and the iteration budget, and RQ3 then reports the headline gains with exactly that configuration on the same test set. This is a fitted-input-called-prediction pattern: the reported improvement is a selected maximum over test-set-informed choices, and baselines were not given a comparable test-based selection step, making the comparison systematically favorable to DocsChisel. An additional ambiguity in Section III-B is that tool-specific query sets overlap by definition ('a query involving multiple tools belongs to multiple tool-specific query sets'), and Section IV-A does not state that a query assigned to the global test set is excluded from other tools' optimization sets; if the split is only per tool, the claimed holdout guarantee is weakened further. These issues inflate the headline numbers but do not make the framework's output equivalent to its input by construction, so the score is moderate rather than extreme.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The main free parameters are configuration choices made using the evaluation data: the optimization LLM, iteration budget, and split ratio. The axioms are domain assumptions about the completeness of the field taxonomy, the reliability of LLM-based diagnosis, and the adequacy of validation sets for selection. No new physical entities are introduced; the information-field taxonomy and memory structures are internal constructs of the system, not entities with independent falsifiable handles outside the paper.

free parameters (4)
  • max_optimization_iterations = 5
    Set as the default based on RQ5 iteration sensitivity analysis on the data analysis domain's test set; affects all reported main results.
  • optimization_LLM = Claude Haiku 4.5
    Selected after RQ5 model sensitivity comparison among GPT-4o, GLM-5, and Claude Haiku 4.5 using the test set; the main RQ3 results use this model.
  • query_split_ratio = 5:1:4 (optimization:validation:test)
    Hand-chosen split ratio used for all experiments; no sensitivity analysis or justification is provided.
  • candidate_selection_threshold = TS_new >= TS_original
    Design choice that determines which candidate documents enter the candidate set; not tuned but directly affects final documentation selection.
assumptions (5)
  • domain assumption The 17-field taxonomy is a complete and faithful representation of tool documentation content.
    The taxonomy is manually derived from 14 datasets by three authors with consensus and a kappa of 0.862. It is used throughout to classify fields and plan operations; if it omits fields that matter, the framework's operation space is incomplete.
  • domain assumption LLM-based diagnosis reliably attributes execution failures to documentation fields.
    DocsChisel's core loop depends on the optimizer LLM correctly mapping failed traces to field-level issues (Sec. III-C, Fig. 4). No verification of diagnosis accuracy is provided beyond final task success.
  • domain assumption Validation-set task success rate is a reliable proxy for test-set performance when selecting among candidate documents.
    With only 207 validation queries total and per-tool validation sets of a few queries, selection noise is high (Sec. IV-A). The paper does not quantify this noise.
  • domain assumption The benchmarks and query sets used are representative of real-world tool-use tasks.
    Claims of generalization rest on nine domains from two benchmarks, with a limited set of LLMs and paradigms, as acknowledged in Sec. V.
  • standard math Standard statistical procedures (Mann-Whitney U test and Holm correction) are valid for the reported comparisons.
    Used in Sec. V to establish statistical significance of the main improvements; these are standard tools and are applied to repeated runs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents." pith.science (2026). https://pith.science/paper/XPYQHHCH

@misc{pith2026260810037,
  author       = {Pith},
  title        = {Pith review of: DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XPYQHHCH}},
  note         = {Machine review of arXiv:2608.10037}
}
read the original abstract

Large language models (LLMs) increasingly rely on external tools to accomplish complex real-world tasks, making tool documentation a critical grounding resource for LLM agents. Existing studies mainly focus on improving the tool-use capabilities of LLM agents, while largely treating tool documentation as a fixed input. Although several recent works attempt to optimize tool documentation through rewriting or compression, little is known about how the information contained in tool documentation affects agent performance across different settings. To bridge this gap, we conduct a large-scale empirical study on tool documentation for LLM agents. Our study reveals substantial heterogeneity in the information fields provided by existing tool documentation. Moreover, the effectiveness of different information fields is highly dependent on the task domain, LLM backbone, and agent paradigm, indicating that no fixed tool documentation can consistently generalize across diverse agent settings. Motivated by these findings, we propose DocsChisel, an adaptive tool documentation optimization framework for LLM agents. DocsChisel analyzes failed execution traces of a target LLM agent to identify documentation-related issues, and iteratively optimizes tool documentation by adding, removing, and refining information fields for each tool. We evaluate DocsChisel against two state-of-the-art baselines, i.e., EasyTool and DRAFT. Experimental results show that DocsChisel improves the task success rate of LLM agents by 95.89% over the original tool documentation and by 75.15%, on average, over existing baselines, while incurring limited optimization time and token overhead

Figures

Figures reproduced from arXiv: 2608.10037 by the authors.

Figure 1
Figure 1. Impact of Information Fields on Task Success Rate across Task Domains, LLM Backbones, and Agent Paradigms [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. shows the approach overview. First, DOCSCHISEL performs domain-aware execution profiling with the original tool documentation (Sec. III-B). In this stage, when the target Tool Set with Original Documentation Queries with Execution Ground Truth Target LLM Agent …… … … Tools 4 Final Documentation Selection Optimized Tool Documentation Task Success Candidate Documentation Set Final Documentation 2 Memory-Guided Documen… view at source ↗
Figure 3
Figure 3. Memory-Guided Documentation Optimization [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Prompt Template for Failed Trace Diagnosis [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Prompt Template for Field Operation Planning [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 7
Figure 7. Figure 7: Prompt Template for Updating the Memory regression errors [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]
Figure 8
Figure 8. Figure 8: Effectiveness Comparison across Task Domains, LLM Backbones, and Agent Paradigms [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: Results of Iteration Sensitivity 0.00 0.18 0.35 0.53 0.71 TS DOCSCHISEL DOCSCHISEL-NoMem 0 1 2 3 4 5 Iteration 0.71 0.78 0.85 0.93 TC 1.00 DOCSCHISEL DOCSCHISEL-NoMem [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Results of Ablation Study by 97.37% and 164.71%. GLM-5 also outperforms GPT-4o, with relative gains of 55.26% in TC and 67.67% in TS. These results indicate that the quality of optimized documentation is sensitive to the optimization model. Documentation optimiza￾tion…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 38 canonical work pages

  1. [1]

    (2026) Docschisel

    Anonymous. (2026) Docschisel. Accessed on May 11, 2026. [Online]. Available: https://anonymous.4open.science/r/F42D-DocsChisel

    work page 2026

  2. [2]

    (2025) Claude code

    Anthropic. (2025) Claude code. Accessed on May 13, 2026. [Online]. Available: https://www.anthropic.com/claude-code

    work page 2025

  3. [3]

    (2025) Writing effective tools for agents — with agents

    ——. (2025) Writing effective tools for agents — with agents. Accessed on May 13, 2026. [Online]. Available: https://www.anthropic.com/engi neering/writing-tools-for-agents

    2025

  4. [4]

    (2024) Coze

    ByteDance. (2024) Coze. Accessed on May 13, 2026. [Online]. Available: https://www.coze.com

    work page 2024

  5. [5]

    Automatic evaluation of api usability using complexity metrics and visualizations,

    C. R. B. de Souza and D. L. M. Bentolila, “Automatic evaluation of api usability using complexity metrics and visualizations,” inProceedings of the 31st International Conference on Software Engineering, 2009, pp. 299–302

    work page 2009

  6. [6]

    Improving api documentation usability with knowledge pushing,

    U. Dekel and J. D. Herbsleb, “Improving api documentation usability with knowledge pushing,” inProceedings of the IEEE 31st International Conference on Software Engineering, 2009, pp. 320–330

    work page 2009

  7. [7]

    (2024) Dify

    Dify.AI. (2024) Dify. Accessed on May 13, 2026. [Online]. Available: https://dify.ai

    work page 2024

  8. [8]

    Anytool: Self-reflective, hierarchical agents for large-scale api calls,

    Y . Du, F. Wei, and H. Zhang, “Anytool: Self-reflective, hierarchical agents for large-scale api calls,” inProceedings of the 41st International Conference on Machine Learning, 2024, pp. 11 812–11 829

    work page 2024

Show all 60 references
  1. [9]

    Play2prompt: Zero-shot tool instruction optimization for llm agents via tool play,

    W. Fang, Y . Zhang, K. Qian, J. Glass, and Y . Zhu, “Play2prompt: Zero-shot tool instruction optimization for llm agents via tool play,” inProceedings of the Findings of the Association for Computational Linguistics, 2025, pp. 26 274–26 290

    2025

  2. [10]

    Jtpro: A joint tool-prompt reflective optimization framework for language agents,

    S. Ghoshal, A. Mittal, J. Singh, M. Ballesteros, W. Sun, F. Tu, S. Singh, Y . Benajiba, F. Shah, S. Bharadwajet al., “Jtpro: A joint tool-prompt reflective optimization framework for language agents,”arXiv preprint arXiv:2604.19821, 2026

    2026 arXiv

  3. [11]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools,

    T. Glm, A. Zeng, B. Xu, B. Wang, C. Zhang, D. Yin, D. Zhang, D. Rojas, G. Feng, H. Zhaoet al., “Chatglm: A family of large language models from glm-130b to glm-4 all tools,”arXiv preprint arXiv:2406.12793, 2024

    2024 arXiv

  4. [12]

    (2023) Google gemini

    Google. (2023) Google gemini. Accessed on May 13, 2026. [Online]. Available: https://gemini.google.com/

    2023

  5. [13]

    Gravitas

    S. Gravitas. (2023) Autogpt: Build, deploy, and run ai agents. Accessed on May 13, 2026. [Online]. Available: https://github.com/significant-gra vitas/autogpt

    2023

  6. [14]

    Learning to rewrite tool descrip- tions for reliable llm-agent tool use,

    R. Guo, K. Dong, X. Gao, and K. Das, “Learning to rewrite tool descrip- tions for reliable llm-agent tool use,”arXiv preprint arXiv:2602.20426, 2026

    2026 arXiv

  7. [15]

    Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,

    J. He, C. Treude, and D. Lo, “Llm-based multi-agent systems for software engineering: Literature review, vision, and the road ahead,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–30, 2025

    2025

  8. [16]

    A simple sequentially rejective multiple test procedure,

    S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979

    1979

  9. [17]

    Metagpt: Meta programming for a multi- agent collaborative framework,

    S. Hong, M. Zhuge, J. Chen, X. Zheng, Y . Cheng, J. Wang, C. Zhang, S. Yau, Z. Lin, L. Zhouet al., “Metagpt: Meta programming for a multi- agent collaborative framework,” inProceedings of the International Conference on Learning Representations, vol. 2024, 2024, pp. 23 247– 23 275

    2024

  10. [18]

    Large language models for software engi- neering: A systematic literature review,

    X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024

    2024

  11. [19]

    Tool documentation enables zero-shot tool- usage with large language models,

    C.-Y . Hsieh, S.-A. Chen, C.-L. Li, Y . Fujii, A. Ratner, C.-Y . Lee, R. Krishna, and T. Pfister, “Tool documentation enables zero-shot tool- usage with large language models,”arXiv preprint arXiv:2308.00675, 2023

    2023 arXiv

  12. [20]

    Chatgpt for shaping the future of dentistry: the potential of multi-modal large language model,

    H. Huang, O. Zheng, D. Wang, J. Yin, Z. Wang, S. Ding, H. Yin, C. Xu, R. Yang, Q. Zhenget al., “Chatgpt for shaping the future of dentistry: the potential of multi-modal large language model,”International Journal of Oral Science, vol. 15, no. 1, p. 29, 2023

    2023

  13. [21]

    Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments,

    K.-H. Huang, A. Prabhakar, S. Dhawan, Y . Mao, H. Wang, S. Savarese, C. Xiong, P. Laban, and C.-S. Wu, “Crmarena: Understanding the capacity of llm agents to perform professional crm tasks in realistic environments,” inProceedings of the Conference of the Nations of the Americ...

    2025

  14. [22]

    Metatool benchmark for large language models: Deciding whether to use tools and which to use,

    Y . Huang, J. Shi, Y . Li, C. Fan, S. Wu, Q. Zhang, Y . Liu, P. Zhou, Y . Wan, N. Gonget al., “Metatool benchmark for large language models: Deciding whether to use tools and which to use,” inProceedings of the International Conference on Learning Representations, 2024, pp. 42...

    2024

  15. [23]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” inProceedings of the International Conference on Learning Representations, vol. 2024, 2024, pp. 54 107–54 157

    2024

  16. [24]

    Automatic detection of five api documentation smells: Practitioners’ perspectives,

    J. Y . Khan, M. Tawkat Islam Khondaker, G. Uddin, and A. Iqbal, “Automatic detection of five api documentation smells: Practitioners’ perspectives,” inProceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering, 2021, pp. 318–329

    2021

  17. [25]

    (2024) Langchain: Build agents faster, your way

    LangChain. (2024) Langchain: Build agents faster, your way. Accessed on May 13, 2026. [Online]. Available: https://langchain.com

    2024

  18. [26]

    Verification-guided context optimization for tool calling via hierarchical llms-as-editors,

    H. Li, S. You, F. Di Palo, Y . Qian, and A. Jain, “Verification-guided context optimization for tool calling via hierarchical llms-as-editors,” arXiv preprint arXiv:2512.13860, 2025

    2025

  19. [27]

    Api-bank: A comprehensive benchmark for tool-augmented llms,

    M. Li, Y . Zhao, B. Yu, F. Song, H. Li, H. Yu, Z. Li, F. Huang, and Y . Li, “Api-bank: A comprehensive benchmark for tool-augmented llms,” inProceedings of the Conference on Empirical Methods in Natural Language Processing, 2023, pp. 3102–3116

    2023

  20. [28]

    Soen-101: Code generation by emulating software process models using large language model agents,

    F. Lin, D. J. Kim, and T.-H. Chen, “Soen-101: Code generation by emulating software process models using large language model agents,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), 2025, pp. 1527–1539

    2025

  21. [29]

    Toolscope: Enhancing llm agent tool use through tool merging and context-aware filtering,

    M. M. Liu, D. Garcia, F. Parllaku, V . Upadhyay, S. F. A. Shah, and D. Roth, “Toolscope: Enhancing llm agent tool use through tool merging and context-aware filtering,”arXiv preprint arXiv:2510.20036, 2025

    2025 arXiv

  22. [30]

    Patterns of knowledge in api reference documentation,

    W. Maalej and M. P. Robillard, “Patterns of knowledge in api reference documentation,”IEEE Transactions on software Engineering, vol. 39, no. 9, pp. 1264–1282, 2013

    2013

  23. [31]

    Mann-whitney u test,

    P. E. McKnight and J. Najab, “Mann-whitney u test,”The Corsini encyclopedia of psychology, pp. 1–1, 2010

    2010

  24. [32]

    Collaboration challenges in building ml-enabled systems: Communication, documentation, engi- neering, and process,

    N. Nahar, S. Zhou, G. Lewis, and C. K ¨astner, “Collaboration challenges in building ml-enabled systems: Communication, documentation, engi- neering, and process,” inProceedings of the 44th international conference on software engineering, 2022, pp. 413–425

    2022

  25. [33]

    A systematic mapping study on api documentation generation approaches,

    K. Nybom, A. Ashraf, and I. Porres, “A systematic mapping study on api documentation generation approaches,” inProceedings of the 44th Euromicro Conference on Software Engineering and Advanced Applications, 2018, pp. 462–469

    2018

  26. [34]

    (2023) Introducing chatgpt

    OpenAI. (2023) Introducing chatgpt. Accessed on May 13, 2026. [Online]. Available: https://openai.com/blog/chatgpt

    2023

  27. [35]

    (2024) Gpt-4o

    OpenAI. (2024) Gpt-4o. Accessed on May 13, 2026. [Online]. Available: https://openai.com/index/hello-gpt-4o/

    2024

  28. [36]

    (2025) Introducing codex

    ——. (2025) Introducing codex. Accessed on May 13, 2026. [Online]. Available: https://openai.com/index/introducing-codex/

    2025

  29. [37]

    (2024) Openapihub

    OpenAPIHub. (2024) Openapihub. Accessed on May 13, 2026. [Online]. Available: https://www.openapihub.com

    2024

  30. [38]

    Gorilla: Large language model connected with massive apis,

    S. G. Patil, T. Zhang, X. Wang, and J. E. Gonzalez, “Gorilla: Large language model connected with massive apis,”Advances in Neural Information Processing Systems, vol. 37, pp. 126 544–126 565, 2024

    2024

  31. [39]

    An empirical study of api usability,

    M. Piccioni, C. A. Furia, and B. Meyer, “An empirical study of api usability,” inProceedings of the ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, 2013, pp. 5–14

    2013

  32. [40]

    Toolllm: Facilitating large language models to master 16000+ real-world apis,

    Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qianet al., “Toolllm: Facilitating large language models to master 16000+ real-world apis,” inProceedings of the International Conference on Learning Representations, vol. 2024, 2024, pp. 9695–9717

    2024

  33. [41]

    Towards completeness-oriented tool retrieval for large language models,

    C. Qu, S. Dai, X. Wei, H. Cai, S. Wang, D. Yin, J. Xu, and J.-R. Wen, “Towards completeness-oriented tool retrieval for large language models,” inProceedings of the 33rd ACM International Conference on Information and Knowledge Management, 2024, pp. 1930–1940

    2024

  34. [42]

    From exploration to mastery: Enabling llms to master tools via self-driven interactions,

    ——, “From exploration to mastery: Enabling llms to master tools via self-driven interactions,” inProceedings of the International Conference on Learning Representations, 2025, pp. 55 675–55 697

    2025

  35. [43]

    (2024) Rapidapi

    RapidAPI. (2024) Rapidapi. Accessed on May 13, 2026. [Online]. Available: https://rapidapi.com

    2024

  36. [44]

    Toolformer: Language models can teach themselves to use tools,

    T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,”Advances in Neural Informa- tion Processing Systems, vol. 36, pp. 68 539–68 551, 2023

    2023

  37. [45]

    Shortcutsbench: A large-scale real-world benchmark for api-based agents,

    H. Shen, Y . Li, D. Meng, D. Cai, S. Qi, L. Zhang, M. Xu, and Y . Ma, “Shortcutsbench: A large-scale real-world benchmark for api-based agents,” inProceedings of the International Conference on Learning Representations, 2025, pp. 77 772–77 797

    2025

  38. [46]

    An empirical study on evolution of api documentation,

    L. Shi, H. Zhong, T. Xie, and M. Li, “An empirical study on evolution of api documentation,” inProceedings of the International Conference on Fundamental Approaches To Software Engineering, 2011, pp. 416–431

    2011

  39. [47]

    Tool learning in the wild: Empowering language models as automatic tool agents,

    Z. Shi, S. Gao, L. Yan, Y . Feng, X. Chen, Z. Chen, D. Yin, S. Verberne, and Z. Ren, “Tool learning in the wild: Empowering language models as automatic tool agents,” inProceedings of the ACM on Web Conference, 2025, pp. 2222–2237

    2025

  40. [48]

    Workbench: a benchmark dataset for agents in a realistic workplace setting,

    O. Styles, S. Miller, P. Cerda-Mardini, T. Guha, V . Sanchez, and B. Vidgen, “Workbench: a benchmark dataset for agents in a realistic workplace setting,”arXiv preprint arXiv:2405.00823, 2024

    2024 arXiv

  41. [49]

    Live api documenta- tion,

    S. Subramanian, L. Inozemtseva, and R. Holmes, “Live api documenta- tion,” inProceedings of the 36th international conference on software engineering, 2014, pp. 643–652

    2014

  42. [50]

    Toolalpaca: Generalized tool learning for language models with 3000 simulated cases,

    Q. Tang, Z. Deng, H. Lin, X. Han, Q. Liang, B. Cao, and L. Sun, “Toolalpaca: Generalized tool learning for language models with 3000 simulated cases,”ArXiv, vol. abs/2306.05301, 2023

    2023 arXiv

  43. [51]

    Autogen: Enabling next-gen llm applications via multi-agent conversations,

    Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liuet al., “Autogen: Enabling next-gen llm applications via multi-agent conversations,” inProceedings of the 1st Conference on Language Modeling, 2024

    2024

  44. [52]

    Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Demystifying llm-based software engineering agents,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 801–824, 2025

    2025

  45. [53]

    On the tool manipulation capability of open-source large language models,

    Q. Xu, F. Hong, B. Li, C. Hu, Z. Chen, and J. Zhang, “On the tool manipulation capability of open-source large language models,”ArXiv, vol. abs/2305.16504, 2023

    2023 arXiv

  46. [54]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  47. [55]

    τ-bench: A benchmark for tool-agent-user interaction in real-world domains,

    S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “ τ-bench: A benchmark for tool-agent-user interaction in real-world domains,”arXiv preprint arXiv:2406.12045, 2024

    2024 arXiv

  48. [56]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,”arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  49. [57]

    Benchmarking llm tool-use in the wild,

    P. Yu, W. Liu, Y . Yang, J. Li, Z. Zhang, X. Feng, and F. Zhang, “Benchmarking llm tool-use in the wild,”arXiv preprint arXiv:2604.06185, 2026

    2026 arXiv

  50. [58]

    Easytool: Enhancing llm-based agents with concise tool instruction,

    S. Yuan, K. Song, J. Chen, X. Tan, Y . Shen, K. Ren, D. Li, and D. Yang, “Easytool: Enhancing llm-based agents with concise tool instruction,” inProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human La...

    2025

  51. [59]

    Glm-5: from vibe coding to agentic engineering,

    A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xieet al., “Glm-5: from vibe coding to agentic engineering,” arXiv preprint arXiv:2602.15763, 2026

    2026 arXiv

  52. [60]

    Losemb: Logic- guided semantic bridging for inductive tool retrieval,

    L. Zhuang, Q. Zhang, H. Zhou, Y . Zhang, and X. Huang, “Losemb: Logic- guided semantic bridging for inductive tool retrieval,” inProceedings of the ACM Web Conference, 2026, pp. 3835–3846

    2026

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.