{"id":"7e143b39-e42e-4c0e-803e-236fa487fd84","arxiv_id":"2501.16945","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ToolFactory converts natural-language REST API documentation into AI-agent tools via a soft-prompt-tuned Llama extractor, a tool validation loop, and a knowledge base for missing parameter values.","lead":"ToolFactory reads REST API documentation written in natural language and automatically produces AI-agent tools from it, reporting success on a new benchmark of 167 API documents. The significance is that research teams could plug existing scientific web services into LLM agents without hand-writing adapters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3 admits human annotation of missing base URLs, contradicting Section 1's claim that ToolFactory eliminates human intervention; the autonomy claim is false as stated.","rationale":"Good-faith reading: the concrete engineering contribution — an open pipeline, a new benchmark, and APILlama — is plausible and the experiments are relevant. However, the central claim as worded in Section 1 is not just overstated; it is contradicted by Section 3's admission that human annotation was necessary for missing base URLs. Since the paper's own target is poorly documented APIs, missing base URLs are a canonical case, not a rare edge case. The reader's weakest_assumption focused on the GPT-4o validation oracle and label circularity. Those are legitimate threats to the absolute numbers (26/50 recovered tools, 92 verified tools) and should be addressed with human annotation and independent validation, but they do not by themselves refute the automation claim. The autonomy contradiction does. I therefore partially agree with the reader: same conditional verdict, but the more load-bearing concern is internal to the methods section. The appropriate remedy is to soften the claim to 'semi-automated pipeline with human fallback for missing base URLs', release the benchmark and code, and quantify how often human annotation is needed. With that, the paper remains a solid engineering contribution; without it, the headline claim is false. Hence UNCHANGED conditional verdict.","tokens_in":15413,"tokens_out":6470,"duration_ms":56246,"concrete_test":"Release the API Extraction Benchmark and count how many of the 167 API documents lack an explicit base URL in their raw HTML, the condition Section 3 says required human annotation. If that count is nonzero, ToolFactory without any human base-URL input cannot process those documents, which falsifies the 'any documentation / no human intervention' claim. A stronger version: run the full ToolFactory pipeline on a withheld set of such documents with zero human annotation of base URLs and report the tool pass rate; if it is zero or near-zero for those documents, the Section 1 claim fails in exactly the regime the paper says it targets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's strongest claim (Section 1) is that ToolFactory 'autonomously generates tools from any REST API documentation written in natural language, eliminating the need for human intervention.' That claim is internally contradicted by the authors' own description of the benchmark in Section 3: 'In cases where the base URL was not specified, human annotation was necessary.' The pipeline therefore cannot process arbitrary documentation without a human supplying missing base URLs before extraction and tool generation can proceed. This is not a peripheral caveat: the 'any documentation' and 'eliminating human intervention' promises are the headline contribution, and the adversary is the poorly documented API, exactly the setting the paper targets. The benchmark is also curated (auth-free APIs, static HTML only, LLM-filtered pages), and Table 2 shows only 50 of 117 endpoints pass validation even with ground-truth extraction, so end-to-end autonomous generation is not demonstrated. The evaluation-oracle concern raised by the reader is real, but the autonomy contradiction is more decisive: even a perfect oracle would not fix the admitted human-in-the-loop step.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ToolFactory, an open-source pipeline that extracts structured API information from natural-language REST API documentation, converts it into AI-usable tools, validates the generated tools, and uses a knowledge base of validated tools to infer missing parameter values. The pipeline is built around APILlama, a soft-prompt-tuned frozen Llama-3-8B model trained on a newly collected benchmark of 167 API documents and 744 endpoints. The authors report that APILlama achieves near-perfect schema validity, extraction quality comparable to GPT-3.5 structured mode, recovery of 26 out of 50 ground-truth passable tools, and a glycomaterials case study with 92 verified tools.","tokens_in":15601,"tokens_out":3295,"duration_ms":28809,"significance":"If the claims were fully supported, the paper would make a useful contribution to LLM-based tool construction: the benchmark targets diverse, semi-structured and unstructured API documentation rather than only OpenAPI or RapidAPI schemas, the pipeline is open-source, and the soft-prompting approach is parameter-efficient and practical. The strongest contribution is the benchmark itself and the demonstration that a small fine-tuned open model can approach proprietary-model extraction quality. However, the headline claim of fully autonomous conversion of arbitrary REST API documentation is not supported by the authors' own description of the pipeline, and the evaluation oracle is the same model family that generated the ground-truth annotations. These issues materially weaken the central claims but are addressable with a softened claim and additional validation.","major_comments":[{"comment":"The paper's headline claim that ToolFactory 'autonomously generates tools from any REST API documentation written in natural language, eliminating the need for human intervention' is directly contradicted by Section 3, which states 'In cases where the base URL was not specified, human annotation was necessary.' Missing base URLs are exactly the type of incomplete documentation the pipeline is meant to handle, so the stated autonomy guarantee is not supported; the manuscript should either remove the 'any documentation'/'eliminating human intervention' claim or describe and evaluate an automatic base-URL completion step.","section":"Section 1 and Section 3"},{"comment":"The benchmark's ground-truth annotations were produced by GPT-4o in structured mode (Section 3), and tool validation is performed by a GPT-4o-based evaluator that accepts a response if it is non-empty and not an error message (Section 4.3, Appendix I.1). Because the same model family both defines and judges correctness, the central numbers (26/50 recovered tools, 92 verified glyco tools, and the parameter-inference comparison) could reflect shared LLM biases rather than actual tool correctness. Please add a human-validated sample of both the annotations and the tool-validation judgments, and report inter-annotator agreement.","section":"Sections 3 and 4.3"},{"comment":"The extraction comparison is based on a single random 80/20 split with 34 documents and 117 endpoints in the test set, and no confidence intervals or multiple-seed results are reported. Given the small test set, the observed differences among APILlama and GPT-3.5 variants (e.g., # Matched 66 vs. 84, Parameter Recall 0.92 vs. 0.82) may not be stable; please report repeated splits or bootstrap intervals.","section":"Section 5.1 and Table 1"},{"comment":"Only 50 of the 117 endpoints in the test set pass validation even when tools are generated from ground-truth JSON, meaning a large fraction of the benchmark APIs cannot currently be validated end-to-end. This is an important scope limitation for the claim that the pipeline converts 'any REST API documentation' into working tools, and it should be discussed explicitly in the sections making the autonomy and generality claims rather than appearing only in a results table.","section":"Section 5.2 and Table 2"}],"minor_comments":[{"comment":"Appendix E contains a self-referential sentence 'For all error types other than Passed Validation, we group them into four main categories ... (see Appendix E)', which appears to be a leftover editorial instruction; please revise.","section":"Appendix E"},{"comment":"The model name is inconsistent: 'APILLAMA' is used in most of the text, while 'APILlama' appears in the contribution list; please unify the naming.","section":"Throughout"},{"comment":"The example input contains mojibake such as 'Pokémon', which should be rendered as 'Pokémon' for clarity.","section":"Appendix B.2"},{"comment":"The caption of Figure B1 contains empty links for the three example documentation pages; please include the actual URLs or remove the placeholder text.","section":"Figure B1"},{"comment":"There is a typo in 'excutable' in the introductory sentence of Appendix D; it should be 'executable'.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful benchmark and a sensible open-source pipeline, but the headline autonomy claim is internally contradicted by the admitted human annotation step, and the evaluation relies on a single LLM family for both label generation and validation. I recommend major revision rather than rejection because the core extraction pipeline can be repaired by softening the claim, adding human validation of a sample, and reporting uncertainty around the small test-set numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful engineering paper with a headline that oversells. The benchmark and the open APILlama model are real contributions; the autonomy claim is not supported by the paper's own methods.\n\nThe new things: an annotated benchmark of 167 API documents (744 endpoints) spanning organized to unorganized docs; APILlama, a soft-prompt-tuned Llama-3-8B that gets 97% valid JSON and extraction quality close to GPT-3.5 structured mode; a retrieval-based parameter value inference that beats GPT-4o on their own test; and a glycomaterials case study yielding 92 validated tools. The soft-prompt approach is practical: 20 virtual tokens compress a 572-token schema, training takes 2 hours on one A40.\n\nWhat's good: the evaluation is task-relevant—they check structure, semantics, and functional call success, not just F1 on a label set. The error taxonomy (C1–C4) is useful. The parameter inference setting is a fair simulation of missing documentation.\n\nWhere it's soft: (1) The 'eliminating the need for human intervention' line is contradicted in Section 3, where missing base URLs required human annotation. The benchmark is also filtered to auth-free, static HTML pages. So 'any REST API documentation' is a big overreach. This is claim-wordsmithing rather than a broken pipeline, but it deserves an explicit fix. (2) The validation oracle is weak: a tool is 'verified' if it returns HTTP 200 and GPT-4o says the body isn't an error. Since GPT-4o also generated the ground-truth annotations, there's a shared-judge circularity. No human checks that the tool actually performs the documented function. That makes numbers like 26/50 recovered tools and 92 verified glyco tools upper bounds, not confirmed facts. (3) The benchmark is a single split with no error bars or inter-annotator agreement. (4) The code link is a placeholder—no artifacts to reproduce.\n\nI wouldn't call these fatal. The central extraction result is plausible and directly supported by their experiments. But the paper needs a revision that releases data/code, adds a human-evaluated subset (or at least a second LLM judge), reports variance, and adjusts the autonomy and 'verified' language.\n\nWho it's for: anyone working on LLM tool agents, especially for scientific APIs. A serious referee should look at it. I'd lean conditional accept with major revisions, not a desk reject.","headline":"Useful benchmark and a credible extractor, but the autonomy claim is overstated and the validation oracle is too weak for the word 'verified'.","tokens_in":16154,"tokens_out":3643,"would_cite":false,"duration_ms":30313,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that messy, natural-language REST API docs can be turned into validated AI-callable tools automatically, with a prompt-tuned Llama 3 extractor and an LLM judge as the gate.","keywords":["REST API documentation","tool agents","LLM","prompt tuning","structured information extraction","parameter inference","glycomaterials","automatic tool generation"],"falsifier":"Take the 26 tools that APILlama generated and that passed validation, call each one with the parameter values APILlama extracted, and independently verify that the response contains the specific data described for that endpoint (for example, the requested card information, weather data, or database record). If a large fraction of these calls return error pages, empty payloads, or data for the wrong entity while still passing the GPT-4o check, the paper's recovery claim would be disproven. A second check would re-run the leave-one-API-out parameter-inference experiment with ground-truth parameter values checked by human experts rather than by the GPT-4o judge.","tokens_in":15184,"feed_emoji":"🔧","tokens_out":7291,"duration_ms":59477,"temperature":0.7,"pith_summary":"The paper claims that natural-language REST API documentation, even when unstructured or incomplete, can be converted automatically into AI-callable tools without human-written adapters. To establish this, it builds an open-source pipeline (ToolFactory) that uses a prompt-tuned Llama 3 model to extract a minimal JSON schema from each documentation page, compiles that JSON into Python functions or OpenAPI definitions, and validates each tool by calling its endpoint and checking the response with an LLM-based evaluator. On a new benchmark of 167 API documents (744 endpoints), the extraction model recovered 26 of 50 ground-truth tools that pass validation, and the same pipeline produced 92 verified tools for a glycomaterials research agent. A sympathetic reader would care because the result suggests scientific and niche APIs with messy documentation could be integrated into LLM agents at low engineering cost.","feed_headline":"Messy REST docs become AI tools, minus hand adapters","feed_subtitle":"A prompt-tuned model extracts callable functions from messy API pages; 92 verified tools now power a glyco agent.","key_machinery":"The central mechanism is the prompt-tuned extraction model APILlama, which freezes Llama 3 8B and trains only 20 soft-prompt tokens (81,920 parameters) that encode a 572-token JSON schema instruction, so inference overhead is small and one training run takes about two hours on a single A40 GPU. Around it, ToolFactory combines a JSON-to-tool builder (parameter-pattern matching, percent-encoding correction, required-parameter checks), a validation gate (HTTP 200 plus a GPT-4o evaluator that rejects empty or error content), and a parameter-inference module that retrieves candidate values by cosine similarity of parameter keys and descriptions against a database built from previously verified tools.","core_discovery":"ToolFactory's central discovery is that the task of turning an API documentation page into a usable tool can be reduced to extracting a fixed, small JSON object (title, endpoint name, method, URL, required and optional parameters) from free text, and that a frozen Llama 3 8B model with a 20-token trainable soft prompt can perform this extraction as well as a proprietary structured-output mode. The paper shows that the extracted JSON is sufficient to generate executable tools that pass validation, and it reports that most failures in tool generation are traceable to incorrect parameter values rather than to schema or URL errors. It then claims that a knowledge base of verified tools, queried by semantic similarity of parameter names and descriptions, can infer missing parameter values more effectively than a general-purpose LLM prompted to guess (33 vs 17 tools in a leave-one-API-out test). Together these results are presented as evidence that the pipeline generalizes across documentation styles and across application domains.","pith_inferences":["The paper's validity metric is the main thing to probe: if the GPT-4o judge accepts responses that return an error page or unrelated data, the 26/50 and 92 counts overstate the number of tools that actually perform their documented function.","The parameter-inference method presupposes that a pool of verified tools in the same domain already exists, so a cold-start application (first tool in a new domain) would need another source of example values.","The selection of authentication-free APIs is for evaluation convenience; the paper suggests keys could be injected as configuration, so the pipeline's transfer to authenticated APIs is plausible but untested.","A natural next experiment is to run ToolFactory on a set of well-structured OpenAPI documents and compare extraction quality against a schema-parser baseline, which would reveal how much of the benefit comes from the LLM extraction versus the validation and inference machinery."],"forward_implications":["Developers can generate AI-callable tools from REST API documentation in natural language without writing adapters, provided the documentation contains enough information to extract endpoints and parameters.","The 20-token soft prompt makes inference cheaper than in-context schema prompting, and the tiny trainable footprint means an 8B model can be fine-tuned quickly on a small dataset.","Because most failures are incorrect parameter values, the knowledge-base inference method is the key to raising tool recovery rates; the paper reports it doubled the number of tools whose missing parameters could be inferred relative to an LLM guessing directly.","The glyco case study demonstrates that the same pipeline can produce 92 validated tools spanning multiple databases, including searching, translation, visualization, correction, and cross-database ID conversion."],"supporting_citations":[{"why":"Supplies the base Llama 3 model that APILlama freezes and prompt-tunes for API information extraction.","marker":"[7]"},{"why":"Supplies the soft prompt tuning method used to compress the extraction instruction from about 572 tokens to 20 trainable virtual tokens.","marker":"[8]"},{"why":"Supplies the GPT-4o structured-output mode used to annotate the benchmark dataset and as a comparison baseline for extraction accuracy.","marker":"[15]"},{"why":"Provides the prior ToolLLM approach built on RapidAPI schemas, which ToolFactory contrasts against as being unable to transfer to research APIs.","marker":"[6]"},{"why":"Provides the RESTGPT baseline that relies on OpenAPI specifications, which ToolFactory aims to go beyond by handling natural-language documentation.","marker":"[17]"}],"fun_headline_variants":["LLM reads API docs, writes JSON, builds working tools","ToolFactory: auto-extract tool specs from messy API pages","From REST docs to callable tools without hand adapters","Missing API info? ToolFactory's KB fills the gaps","Frozen 8B model + soft prompt turns docs into tools"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A tool is counted as valid when the API returns HTTP 200 and a GPT-4o evaluator decides the response is not empty and not an error message, and the same model family that generated the ground-truth annotations is used in that evaluator; if this judgment accepts responses that do not actually perform the documented function, the recovery rates overstate real tool correctness.","fun_headline_variants_meta":{"raw":{"variants":["LLM reads API docs, writes JSON, builds working tools","ToolFactory: auto-extract tool specs from messy API pages","From REST docs to callable tools without hand adapters","Missing API info? ToolFactory's KB fills the gaps","Frozen 8B model + soft prompt turns docs into tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000436,"raw_usage":{"total_tokens":2223,"prompt_tokens":956,"completion_tokens":1267,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1182}},"tokens_in":572,"tokens_out":1267,"duration_ms":11932,"temperature":1.0,"reasoning_tokens":1182,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T05:30:55.547163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 26 tools that APILlama generated and that passed validation, call each one with the parameter values APILlama extracted, and independently verify that the response contains the specific data described for that endpoint (for example, the requested card information, weather data, or database record). If a large fraction of these calls return error pages, empty payloads, or data for the wrong entity while still passing the GPT-4o check, the paper's recovery claim would be disproven. A second check would re-run the leave-one-API-out parameter-inference experiment with ground-truth parameter values checked by human experts rather than by the GPT-4o judge.","supporting_citations":[{"cited_title":"The llama 3 herd of models,","cited_arxiv_id":null,"evidence_quote":"Supplies the base Llama 3 model that APILlama freezes and prompt-tunes for API information extraction."},{"cited_title":"The power of scale for parameter-efficient prompt tuning,","cited_arxiv_id":null,"evidence_quote":"Supplies the soft prompt tuning method used to compress the extraction instruction from about 572 tokens to 20 trainable virtual tokens."},{"cited_title":"Openai structured outputs","cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-4o structured-output mode used to annotate the benchmark dataset and as a comparison baseline for extraction accuracy."},{"cited_title":"Toolllm: Facilitating large language models to master 16000+ real-world apis,","cited_arxiv_id":null,"evidence_quote":"Provides the prior ToolLLM approach built on RapidAPI schemas, which ToolFactory contrasts against as being unable to transfer to research APIs."},{"cited_title":"Restgpt: Connecting large language models with real-world restful apis,","cited_arxiv_id":null,"evidence_quote":"Provides the RESTGPT baseline that relies on OpenAPI specifications, which ToolFactory aims to go beyond by handling natural-language documentation."}],"review_version":1}