REVIEW 4 major objections 5 minor 37 references
ToolFactory: Automating Tool Generation by Leveraging LLM to Understand REST API Documentations
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that messy, natural-language REST API docs can be turned into validated AI-callable tools automatically, with a prompt-tuned Llama 3 extractor and an LLM judge as the gate.
desk verdict Useful benchmark and a credible extractor, but the autonomy claim is overstated and the validation oracle is too weak for the word 'verified'. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the prompt-tuned extraction model APILlama, which freezes Llama 3 8B and trains only 20 soft-prompt tokens (81,920 parameters) that encode a 572-token JSON schema instruction, so inference overhead is small and one training run takes about two hours on a single A40 GPU. Around it, ToolFactory combines a JSON-to-tool builder (parameter-pattern matching, percent-encoding correction, required-parameter checks), a validation gate (HTTP 200 plus a GPT-4o evaluator that rejects empty or error content), and a parameter-inference module that retrieves candidate values by cosine similarity of parameter keys and descriptions against a database built from previously verified tools.
What would settle it
Take the 26 tools that APILlama generated and that passed validation, call each one with the parameter values APILlama extracted, and independently verify that the response contains the specific data described for that endpoint (for example, the requested card information, weather data, or database record). If a large fraction of these calls return error pages, empty payloads, or data for the wrong entity while still passing the GPT-4o check, the paper's recovery claim would be disproven. A second check would re-run the leave-one-API-out parameter-inference experiment with ground-truth parameter values checked by human experts rather than by the GPT-4o judge.
Extended reading notes
Core claim
ToolFactory's central discovery is that the task of turning an API documentation page into a usable tool can be reduced to extracting a fixed, small JSON object (title, endpoint name, method, URL, required and optional parameters) from free text, and that a frozen Llama 3 8B model with a 20-token trainable soft prompt can perform this extraction as well as a proprietary structured-output mode. The paper shows that the extracted JSON is sufficient to generate executable tools that pass validation, and it reports that most failures in tool generation are traceable to incorrect parameter values rather than to schema or URL errors. It then claims that a knowledge base of verified tools, queried by semantic similarity of parameter names and descriptions, can infer missing parameter values more effectively than a general-purpose LLM prompted to guess (33 vs 17 tools in a leave-one-API-out test). Together these results are presented as evidence that the pipeline generalizes across documentation styles and across application domains.
Load-bearing premise
A tool is counted as valid when the API returns HTTP 200 and a GPT-4o evaluator decides the response is not empty and not an error message, and the same model family that generated the ground-truth annotations is used in that evaluator; if this judgment accepts responses that do not actually perform the documented function, the recovery rates overstate real tool correctness.
Editorial extensions
If this is right
- Developers can generate AI-callable tools from REST API documentation in natural language without writing adapters, provided the documentation contains enough information to extract endpoints and parameters.
- The 20-token soft prompt makes inference cheaper than in-context schema prompting, and the tiny trainable footprint means an 8B model can be fine-tuned quickly on a small dataset.
- Because most failures are incorrect parameter values, the knowledge-base inference method is the key to raising tool recovery rates; the paper reports it doubled the number of tools whose missing parameters could be inferred relative to an LLM guessing directly.
- The glyco case study demonstrates that the same pipeline can produce 92 validated tools spanning multiple databases, including searching, translation, visualization, correction, and cross-database ID conversion.
Reading between the lines
- The paper's validity metric is the main thing to probe: if the GPT-4o judge accepts responses that return an error page or unrelated data, the 26/50 and 92 counts overstate the number of tools that actually perform their documented function.
- The parameter-inference method presupposes that a pool of verified tools in the same domain already exists, so a cold-start application (first tool in a new domain) would need another source of example values.
- The selection of authentication-free APIs is for evaluation convenience; the paper suggests keys could be injected as configuration, so the pipeline's transfer to authenticated APIs is plausible but untested.
- A natural next experiment is to run ToolFactory on a set of well-structured OpenAPI documents and compare extraction quality against a schema-parser baseline, which would reveal how much of the benefit comes from the LLM extraction versus the validation and inference machinery.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces ToolFactory, an open-source pipeline that extracts structured API information from natural-language REST API documentation, converts it into AI-usable tools, validates the generated tools, and uses a knowledge base of validated tools to infer missing parameter values. The pipeline is built around APILlama, a soft-prompt-tuned frozen Llama-3-8B model trained on a newly collected benchmark of 167 API documents and 744 endpoints. The authors report that APILlama achieves near-perfect schema validity, extraction quality comparable to GPT-3.5 structured mode, recovery of 26 out of 50 ground-truth passable tools, and a glycomaterials case study with 92 verified tools.
Significance. If the claims were fully supported, the paper would make a useful contribution to LLM-based tool construction: the benchmark targets diverse, semi-structured and unstructured API documentation rather than only OpenAPI or RapidAPI schemas, the pipeline is open-source, and the soft-prompting approach is parameter-efficient and practical. The strongest contribution is the benchmark itself and the demonstration that a small fine-tuned open model can approach proprietary-model extraction quality. However, the headline claim of fully autonomous conversion of arbitrary REST API documentation is not supported by the authors' own description of the pipeline, and the evaluation oracle is the same model family that generated the ground-truth annotations. These issues materially weaken the central claims but are addressable with a softened claim and additional validation.
major comments (4)
- [Section 1 and Section 3] The paper's headline claim that ToolFactory 'autonomously generates tools from any REST API documentation written in natural language, eliminating the need for human intervention' is directly contradicted by Section 3, which states 'In cases where the base URL was not specified, human annotation was necessary.' Missing base URLs are exactly the type of incomplete documentation the pipeline is meant to handle, so the stated autonomy guarantee is not supported; the manuscript should either remove the 'any documentation'/'eliminating human intervention' claim or describe and evaluate an automatic base-URL completion step.
- [Sections 3 and 4.3] The benchmark's ground-truth annotations were produced by GPT-4o in structured mode (Section 3), and tool validation is performed by a GPT-4o-based evaluator that accepts a response if it is non-empty and not an error message (Section 4.3, Appendix I.1). Because the same model family both defines and judges correctness, the central numbers (26/50 recovered tools, 92 verified glyco tools, and the parameter-inference comparison) could reflect shared LLM biases rather than actual tool correctness. Please add a human-validated sample of both the annotations and the tool-validation judgments, and report inter-annotator agreement.
- [Section 5.1 and Table 1] The extraction comparison is based on a single random 80/20 split with 34 documents and 117 endpoints in the test set, and no confidence intervals or multiple-seed results are reported. Given the small test set, the observed differences among APILlama and GPT-3.5 variants (e.g., # Matched 66 vs. 84, Parameter Recall 0.92 vs. 0.82) may not be stable; please report repeated splits or bootstrap intervals.
- [Section 5.2 and Table 2] Only 50 of the 117 endpoints in the test set pass validation even when tools are generated from ground-truth JSON, meaning a large fraction of the benchmark APIs cannot currently be validated end-to-end. This is an important scope limitation for the claim that the pipeline converts 'any REST API documentation' into working tools, and it should be discussed explicitly in the sections making the autonomy and generality claims rather than appearing only in a results table.
minor comments (5)
- [Appendix E] Appendix E contains a self-referential sentence 'For all error types other than Passed Validation, we group them into four main categories ... (see Appendix E)', which appears to be a leftover editorial instruction; please revise.
- [Throughout] The model name is inconsistent: 'APILLAMA' is used in most of the text, while 'APILlama' appears in the contribution list; please unify the naming.
- [Appendix B.2] The example input contains mojibake such as 'Pokémon', which should be rendered as 'Pokémon' for clarity.
- [Figure B1] The caption of Figure B1 contains empty links for the three example documentation pages; please include the actual URLs or remove the placeholder text.
- [Appendix D] There is a typo in 'excutable' in the introductory sentence of Appendix D; it should be 'executable'.
Circularity Check
No circular derivation: the pipeline's predictions are validated against live API responses and held-out annotations, not against fitted inputs.
full rationale
The paper's claimed derivation chain is: GPT-4o in structured mode plus human verification produces the API Extraction Benchmark annotations; APILlama is trained on an 80% split and evaluated on a disjoint 20% split using those annotations as ground truth; extracted JSON is converted into tools by deterministic scripts; tool validity is checked by live HTTP status plus a GPT-4o content evaluator; and parameter-value inference is tested in a leave-one-API-out design where success requires the live API call to pass validation. None of these steps defines the predicted quantity in terms of a fitted parameter or reduces a headline result to an input by construction. The training/evaluation split is a standard supervised setup, and the tool-validation numbers (e.g., 26/50 passable tools, 92 glyco tools) depend on real API responses, not on the model's own outputs. The most defensible concern is a shared-judge dependency: GPT-4o was used to synthesize annotations and also serves as the response-content judge, and APILlama is a distillation of that model family; this can overstate correctness if the evaluator is permissive, but it is a measurement risk, not a definitional circularity. Likewise, the Section 1 claim of eliminating the need for human intervention is contradicted by Section 3's admission that human annotation was necessary for missing base URLs, but that is an overclaim about autonomy rather than a circular derivation. No self-citation chain or imported uniqueness theorem is load-bearing. Hence score 0.
Assumptions & free parameters
free parameters (5)
- Soft prompt length =
20 virtual tokens
- Similarity threshold for parameter candidates =
0.5
- Retrieval counts for parameter inference =
5 description neighbors + 5 key neighbors = 10
- Combination sampling cap =
20
- Training hyperparameters =
learning rate 0.001, one epoch, 4-bit quantization
assumptions (5)
- domain assumption A GPT-4o-based judge plus HTTP 200 is a sufficient definition of a working tool
- domain assumption Authentication-free APIs from APIList.com represent REST APIs broadly enough for the paper's general claims
- domain assumption GPT-4o structured-mode annotations, after unspecified human checking, are reliable ground truth
- domain assumption Semantic similarity between parameter names and descriptions transfers across APIs within a domain
- ad hoc to paper Soft-prompt-tuning a frozen Llama-3-8B with 20 tokens and one epoch is sufficient to learn the extraction task
Cite this review
Pith. "Pith review of ToolFactory: Automating Tool Generation by Leveraging LLM to Understand REST API Documentations." pith.science (2026). https://pith.science/paper/MOSLKKMS
@misc{pith2026250116945,
author = {Pith},
title = {Pith review of: ToolFactory: Automating Tool Generation by Leveraging LLM to Understand REST API Documentations},
year = {2026},
howpublished = {\url{https://pith.science/paper/MOSLKKMS}},
note = {Machine review of arXiv:2501.16945}
}
read the original abstract
LLM-based tool agents offer natural language interfaces, enabling users to seamlessly interact with computing services. While REST APIs are valuable resources for building such agents, they must first be transformed into AI-compatible tools. Automatically generating AI-compatible tools from REST API documents can greatly streamline tool agent development and minimize user learning curves. However, API documentation often suffers from a lack of standardization, inconsistent schemas, and incomplete information. To address these issues, we developed \textbf{ToolFactory}, an open-source pipeline for automating tool generation from unstructured API documents. To enhance the reliability of the developed tools, we implemented an evaluation method to diagnose errors. Furthermore, we built a knowledge base of verified tools, which we leveraged to infer missing information from poorly documented APIs. We developed the API Extraction Benchmark, comprising 167 API documents and 744 endpoints in various formats, and designed a JSON schema to annotate them. This annotated dataset was utilized to train and validate ToolFactory. The experimental results highlight the effectiveness of ToolFactory. We also demonstrated ToolFactory by creating a domain-specific AI agent for glycomaterials research. ToolFactory exhibits significant potential for facilitating the seamless integration of scientific REST APIs into AI workflows.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
A survey on large lan- guage model based autonomous agents,
L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, W . X. Zhao, Z. Wei, and J. Wen, “A survey on large lan- guage model based autonomous agents,” Fron- tiers of Computer Science, vol. 18, Mar. 2024
work page 2024
-
[2]
J. Baek, S. K. Jauhar, S. Cucerzan, and S. J. Hwang, “Researchagent: Iterative research idea generation over scientific literature with large language models,” 2024
work page 2024
-
[3]
Conversational health agents: A per- sonalized llm-powered agent framework,
M. Abbasian, I. Azimi, A. M. Rahmani, and R. Jain, “Conversational health agents: A per- sonalized llm-powered agent framework,” 2024
work page 2024
-
[4]
Y. Li, Y. Yu, H. Li, Z. Chen, and K. Khashanah, “Tradinggpt: Multi-agent system with layered memory and distinct characters for enhanced fi- nancial trading performance,” 2023
work page 2023
-
[5]
Chemcrow: Aug- menting large-language models with chemistry tools,
A. M. Bran, S. Cox, O. Schilter, C. Baldassari, A. D. White, and P . Schwaller, “Chemcrow: Aug- menting large-language models with chemistry tools,” 2023
work page 2023
-
[6]
Toolllm: Facilitating large language models to master 16000+ real-world apis,
Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, X. Tang, B. Qian, S. Zhao, L. Hong, R. Tian, R. Xie, J. Zhou, M. Gerstein, D. Li, Z. Liu, and M. Sun, “Toolllm: Facilitating large language models to master 16000+ real-world apis,” 2023
work page 2023
- [7]
-
[8]
The power of scale for parameter-efficient prompt tuning,
B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” 2021
work page 2021
Show all 37 references
-
[9]
Langchain
C. Harrison, “Langchain.” https://github.com/ langchain-ai/langchain, 2024. Accessed: 2024- 10-02
2024
-
[10]
Unified structure generation for universal information extraction,
Y. Lu, Q. Liu, D. Dai, X. Xiao, H. Lin, X. Han, L. Sun, and H. Wu, “Unified structure generation for universal information extraction,” 2022
2022
-
[11]
Structured information extraction from scien- tific text with large language models,
J. Dagdelen, A. Dunn, S. Lee, N. Walker, A. S. Rosen, G. Ceder, K. A. Persson, and A. Jain, “Structured information extraction from scien- tific text with large language models,” Nature Communications, vol. 15, no. 1, p. 1418, 2024
2024
-
[12]
Llama 2: Open foundation and fine-tuned chat models,
e. a. Hugo Touvron, “Llama 2: Open foundation and fine-tuned chat models,” 2023
2023
-
[13]
Language models are few-shot learners,
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P . Dhariwal, A. Neelakantan, P . Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B...
2020
-
[14]
Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities,
Y. Zhu, X. Wang, J. Chen, S. Qiao, Y. Ou, Y. Yao, S. Deng, H. Chen, and N. Zhang, “Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities,” 2024
2024
-
[15]
Openai structured outputs
OpenAI, “Openai structured outputs.” https://platform.openai.com/docs/guides/ structured-outputs/introduction, 2024. Ac- cessed: 2024-10-02
2024
-
[16]
Rapidapi hub
RapidAPI, “Rapidapi hub.” https://rapidapi.com/ hub, 2024. Accessed: 2024-12-31
2024
-
[17]
Restgpt: Connecting large language models with real-world restful apis,
Y. Song, W . Xiong, D. Zhu, W . Wu, H. Qian, M. Song, H. Huang, C. Li, K. Wang, R. Yao, Y. Tian, and S. Li, “Restgpt: Connecting large language models with real-world restful apis,” 2023
2023
-
[18]
Automate logins with browser bots
axiom.ai, “Automate logins with browser bots.” https://axiom.ai/automate/login, 2024. Ac- cessed: 2024-10-02
2024
-
[19]
st-codesearch-distilroberta-base,
N. Reimers, “st-codesearch-distilroberta-base,” 2021
2021
-
[20]
CodeSearchNet challenge: Evaluating the state of semantic code search,
H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “CodeSearchNet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436, 2019
1909 arXiv
-
[21]
Opena- gents: An open platform for language agents in the wild,
T. Xie, F . Zhou, Z. Cheng, P . Shi, L. Weng, Y. Liu, T. J. Hua, J. Zhao, Q. Liu, C. Liu, L. Z. Liu, Y. Xu, H. Su, D. Shin, C. Xiong, and T. Yu, “Opena- gents: An open platform for language agents in the wild,” 2023
2023
-
[22]
Introducing gpts
OpenAI, “Introducing gpts.” https://openai.com/ index/introducing-gpts/, 2024. Accessed: 2024- 10-02
2024
-
[23]
Pydantic,
S. Colvin, E. Jolibois, H. Ramezani, A. G. Badaracco, T. Dorsey, D. Montague, S. Matveenko, M. Trylesinski, S. Runkle, D. Hewitt, A. Hall, and V . Plot, “Pydantic,” 01
-
[24]
Function calling
OpenAI, “Function calling.” https://openai.com/ index/function-calling-and-other-api-updates/,
-
[25]
Glycodata
F . Wang, S. Seth, S. R. Ramacharla, and S. A. Deshmukh, “Glycodata.” glycodata.org/, 2025. Accessed: 2025-1-15
2025
-
[26]
Accessed: 2025-01-10
2025
-
[27]
Glytoucan: an accessible glycan struc- ture repository,
M. Tiemeyer, K. Aoki, J. Paulson, R. D. Cum- mings, W . S. York, N. G. Karlsson, F . Lisacek, N. H. Packer, M. P . Campbell, N. P . Aoki, et al. , “Glytoucan: an accessible glycan struc- ture repository,” Glycobiology, vol. 27, no. 10, pp. 915–919, 2017
2017
-
[28]
Glygen: computational and infor- matics resources for glycoscience,
W . S. York, R. Mazumder, R. Ranzinger, N. Ed- wards, R. Kahsay, K. F . Aoki-Kinoshita, M. P . Campbell, R. D. Cummings, T. Feizi, M. Mar- tin, et al. , “Glygen: computational and infor- matics resources for glycoscience,” Glycobiol- ogy, vol. 30, no. 2, pp. 72–73, 2020
2020
-
[29]
The glycosmos portal: a unified and comprehensive web re- source for the glycosciences,
I. Yamada, M. Shiota, D. Shinmachi, T. Ono, S. Tsuchiya, M. Hosoda, A. Fujita, N. P . Aoki, Y. Watanabe, N. Fujita, et al. , “The glycosmos portal: a unified and comprehensive web re- source for the glycosciences,” Nature Methods, vol. 17, no. 7, pp. 649–650, 2020
2020
-
[30]
Kegg as a glycome infor- matics resource,
K. Hashimoto, S. Goto, S. Kawano, K. F . Aoki- Kinoshita, N. Ueda, M. Hamajima, T. Kawasaki, and M. Kanehisa, “Kegg as a glycome infor- matics resource,” Glycobiology, vol. 16, no. 5, pp. 63R–70R, 2006
2006
-
[31]
The human o-glcnacome database and meta-analysis,
E. Wulff-Fuentes, R. R. Berendt, L. Mass- man, L. Danner, F . Malard, J. Vora, R. Kah- say, and S. Olivier-Van Stichelen, “The human o-glcnacome database and meta-analysis,” Sci- entific data, vol. 8, no. 1, p. 25, 2021
2021
-
[32]
Glyconnect: glycopro- teomics goes visual, interactive, and analytical,
D. Alocci, J. Mariethoz, A. Gastaldello, E. Gasteiger, N. G. Karlsson, D. Kolarich, N. H. Packer, and F . Lisacek, “Glyconnect: glycopro- teomics goes visual, interactive, and analytical,” Journal of proteome research , vol. 18, no. 2, pp. 664–677, 2018
2018
-
[33]
The Proteins API: accessing key integrated protein and genome information,
A. Nightingale, R. Antunes, E. Alpi, B. Bursteinas, L. Gonzales, W . Liu, J. Luo, G. Qi, E. Turner, and M. Martin, “The Proteins API: accessing key integrated protein and genome information,” Nucleic Acids Research , vol. 45, pp. W539–W544, 04 2017
2017
-
[34]
Glycam web: Website builders,
Woods Group, “Glycam web: Website builders,”
-
[35]
Unilectin, a one-stop-shop to explore and study carbohydrate-binding proteins,
A. Imberty, F . Bonnardel, and F . Lisacek, “Unilectin, a one-stop-shop to explore and study carbohydrate-binding proteins,” Current Proto- cols, vol. 1, no. 11, p. e305, 2021. A API-extraction Schema We defined the JSON schema to extract API information. In pratice, this is a...
2021
-
[36]
Pubchem substance and compound databases,
S. Kim, P . A. Thiessen, E. E. Bolton, J. Chen, G. Fu, A. Gindulyte, L. Han, J. He, S. He, B. A. Shoemaker, et al. , “Pubchem substance and compound databases,” Nucleic acids research , vol. 44, no. D1, pp. D1202–D1213, 2016
2016
-
[2025]
If you use this software, please cite it as described
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.