{"id":"51db4a97-0413-4ff4-96f5-1f6a4dede466","arxiv_id":"2506.19998","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Doc2Agent automatically converts unstructured REST API documentation into validated, Python-based tools for AI agents, reporting a 55% relative WebArena improvement over direct API calling.","lead":"Doc2Agent is a pipeline that reads plain-language API documentation and automatically produces Python tools an AI agent can call, testing and fixing each tool along the way. The authors report a 55% relative success-rate gain on WebArena at one-tenth the cost versus direct API calling, but the comparison uses a modified evaluator and does not re-run the prior baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 55.1% WebArena gain is computed across two different evaluators: Doc2Agent uses the new API-specified evaluator while the API-based baseline uses the published vanilla evaluator; without a same-protocol rerun, the headline improvement is not established.","rationale":"The reader's weakest assumption identifies the exact point on which the central claim depends: the 55.1% relative improvement requires that Doc2Agent and the direct API baseline be measured under the same evaluation protocol. The paper instead compares Doc2Agent under a newly introduced API-specified evaluator with the API-based agent under the vanilla evaluator, and admits that the baseline logs are unavailable for re-scoring. This is not a mere reporting detail; the new evaluator is described as simultaneously stricter (LLM re-check of intent to remove false positives) and more permissive (functional equivalence for URL/program_html checks), so the net effect on any given agent is empirically unknown. The hybrid agent's modest improvement under the new evaluator does not bound the baseline's behavior, because the baseline has different action traces and error patterns. A concrete rerun of the baseline under the same evaluator would settle whether the headline delta is real or an artifact. I do not see an equally load-bearing concern elsewhere: the cost claim is plausible given token savings, the tool-validation rates are internally consistent with the pipeline, and the glycan evaluation is weaker but secondary to the WebArena claim. The paper's own limitation section flags related evaluation issues, which supports the reader's skepticism. Therefore the reader's REJECT verdict should stand unchanged until the same-protocol comparison is provided.","tokens_in":23894,"tokens_out":4094,"duration_ms":44093,"concrete_test":"Re-run the direct API-calling baseline (same CodeAct/GPT-4o setup with raw API documentation and no generated tools) on the same WebArena environments and tasks, score its logs with the exact API-specified evaluator defined in Appendix E, and recompute the delta row of Table 2. If the re-scored baseline average remains near 29.2 and per-task deltas stay positive and comparable, the headline claim is supported; if the re-scored baseline changes materially (for example, rises by more than a few absolute points on average, or per-task deltas shrink or flip sign), the 55.1% relative improvement is an artifact of mixing evaluation protocols.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Table 2 (Section 4.2), the 'Δ vs Direct API Calling' row compares Doc2Agent's average of 45.3 under the API-specified evaluator with the API-based agent's published average of 29.2 under the vanilla WebArena evaluator. The paper explicitly states that the prior agent could not be re-evaluated because its logs are unavailable, and the modified evaluator was applied only to Doc2Agent and the hybrid agent. This is the load-bearing issue: the 55.1% relative improvement conflates the tool-generation effect with a change in evaluation protocol. The direction of the bias is not obvious a priori; the API-specified evaluator both tightens intent confirmation (removing false positives like the Task 22 example in Appendix E.1) and relaxes URL/program_html checks through LLM-based functional equivalence, so per-task scores could move in either direction. The hybrid agent's 'consistent overall performance with minor improvements' does not tell us how the direct API baseline would score under the same protocol, because the two agents interact with the environment differently and have different false-positive/negative profiles. Until both agents are scored under an identical protocol, the claimed 55.1% relative improvement cannot be separated from evaluator effects, making the central performance claim unsupported as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Doc2Agent, a pipeline that takes unstructured REST API documentation and automatically generates, validates, refines, and deploys Python-based tools for LLM agents. The pipeline combines direct, JSON-schema-based tool generation with target-oriented tool generation, an LLM-based validation loop, iterative refinement by a code agent, and a parameter-value database built from previously validated API responses. The authors evaluate Doc2Agent on 167 real-world API documentation pages (744 endpoints), WebArena, and glycoscience research APIs. They report 443 validated real-world tools, a 55.1% relative improvement over direct API calling on WebArena at 10% of the cost, and a domain-specific glycomaterial research agent built entirely from automatically generated tools.","tokens_in":24139,"tokens_out":7908,"duration_ms":75783,"significance":"If the quantitative claims held, Doc2Agent would address a real bottleneck in tool-agent deployment: unstructured, heterogeneous API documentation and missing parameter values. The pipeline design is coherent, the authors release code, and the paper is candid about several limitations, including stateful-API validation, attempts by the code agent to bypass validation, and the lack of a suitable benchmark for open-domain API usage. However, the central WebArena improvement claim is not supported as stated because it compares results under two different evaluation protocols, and the research-agent result relies on a filtered, LLM-judged evaluation. These issues currently reduce the contribution to a promising but unverified pipeline.","major_comments":[{"comment":"The headline 55.1% relative improvement is computed across two different evaluation protocols. The Δ row compares Doc2Agent's average of 45.3 under the new 'API-Specified Evaluation' with the API-based agent's published average of 29.2 under the vanilla WebArena evaluator. Section 4.2 states that the prior agent could not be re-evaluated because its logs are unavailable, and that the new evaluator was applied only to Doc2Agent and the hybrid agent. Appendix E.1 shows that the two protocols can disagree per task in both directions (Task 22 is a false positive under vanilla, while Task 395 requires functional-equivalence reasoning to be counted as a success), so the aggregate difference cannot be attributed to the tool-generation pipeline. To support the improvement claim, the direct-API baseline must be run under the identical API-specified evaluator; otherwise the comparison should be reported only as a cross-protocol observation, and the 55.1% claim should be removed from the abstract and Section 1.","section":"§4.2, Table 2"},{"comment":"No measure of variability is reported for any WebArena cell. Each score appears to be a single run or an aggregation with unspecified replication, and no error bars, standard deviations, or significance tests are provided. Given the small per-site differences in some rows, the claimed gains (e.g., 95.6% on CMS) cannot be distinguished from run-to-run noise. The authors should report multiple independent runs with confidence intervals or explicitly state the number of runs underlying each entry in Table 2.","section":"§4.2, Table 2"},{"comment":"The tool-validation protocol uses an LLM-generated expected response to decide whether an API response is valid, yet Section 2.2 asserts 'strong agreement with human judgment' without providing the human study, sample size, or agreement measure. Because the same model families are used for tool generation and for validation, the pass rates in Table 1 may partly reflect self-consistency in the LLM's expectations rather than genuine API correctness. The authors should add a human-annotated validation sample with reported agreement statistics or use an independent judge before treating the Table 1 pass rates as verified.","section":"§2.2, Table 1"},{"comment":"The glycoscience agent evaluation is not yet a reliable demonstration of domain adaptability. The 50 tasks are generated by GPT-4o from the tool descriptions, the success judgments are made by an LLM judge, and the reported percentages are computed over a 'Filtered' set defined as the union of tasks successfully completed by at least one agent. Constructing the denominator from the systems under test inflates relative rankings and makes the absolute success rates impossible to interpret. The authors should report unfiltered success rates, release the task set, and use independently verified or pre-specified ground-truth answers.","section":"§4.3, Table 3"},{"comment":"The claim that Doc2Agent generates verified tools for 59.5% of real-world APIs is a denominator error. Table 1 lists 443 validated tools from 744 real-world endpoints, which is 59.5% of endpoints, not of the 167 collected API documentation pages; the same ambiguity affects the 81.5% research-API figure, which is not derivable from the 70/131 entries in Table 1. The abstract and Section 1 should state that the rate is per endpoint, or provide the correct per-documentation calculation.","section":"§1, Table 1"}],"minor_comments":[{"comment":"Typo: 'programmic' should be 'programmatic' in the first paragraph.","section":"§1"},{"comment":"Typo: 'circumstanses' should be 'circumstances' in the sentence about unexpected situations during automatic testing.","section":"§2.1"},{"comment":"Typo: 'argentic services' should be 'agentic services' in the Related Works section.","section":"§5"},{"comment":"The cost comparison ($0.12 vs. $1.20 per task) does not state which costs are included; clarify whether it covers only inference tokens and whether tool generation and validation overhead are excluded.","section":"§4.2"},{"comment":"Table 3's header is ambiguous: it is unclear whether the numeric columns after 'Filtered' are per-cell success rates or per-row totals; add explicit column headers and units.","section":"Table 3"},{"comment":"The 'Tool Generation Example' text refers to the Pokemon TCG API, but the displayed code is the OSRM general_request function; align the example label with the code shown.","section":"Appendix F"}],"recommendation":"major_revision","confidential_remarks":"The paper's tool-generation contribution is useful, but the central WebArena comparison is not yet credible because of the evaluator mismatch. I would be willing to reconsider a revised version that either reruns the direct-API baseline under the API-specified evaluator or removes the improvement claim, and that adds validation for the modified evaluator. The author-provided limitation section is a strength, and the reported problems appear correctable within a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the pipeline is real and the write-up is unusually honest, but the 55.1% WebArena claim does not hold up in the current form. The comparison mixes Doc2Agent under a new \"API-specified\" evaluator with the prior API baseline under the vanilla WebArena evaluator, and the authors say they could not re-run the baseline. That is a load-bearing flaw in the central performance claim, and I would want it fixed before believing the number.\n\nWhat is new: AutoTools relies on RapidAPI's standardized specs, and SkillWeaver learns from practice, but Doc2Agent goes from free-text HTML docs to executable Python tools, validates them by actually calling the endpoints, and iteratively fixes them. The parameter database built from responses of previously validated tools is a nice trick, and it beats GPT-4o guessing on the glycan APIs in the leave-one-API-out test (33 vs 17). Deploying via MCP is sensible. The dataset of 167 real-world API docs, mostly semi-organized, is useful to the community.\n\nSoft spots: (1) The 55.1% delta is computed across evaluators. The API-specified evaluator tightens intent confirmation but relaxes URL checks via LLM equivalence, so the bias direction is not obvious. Until the baseline is re-scored under the identical protocol, the headline improvement is not established. Also, there are no error bars or significance tests anywhere, and per-task gaps like 39.6 vs 29.4 are within noise. (2) The validation loop is partly self-referential: the same class of LLM that generates the tools also approves them. The code agent's observed \"cheating\" by suppressing exceptions is exactly the failure mode you worry about with this design, so the 59.5% pass rate is a pipeline diagnostic, not ground truth. (3) The glycan success rates are LLM-judged on LLM-generated tasks, and the \"filtered\" evaluation is a union set, making it optimistic. The limitations section says much of this, which I respect, but it does not fix the numbers.\n\nWho this is for: people building tool agents or API wrappers. It is an engineering contribution, not a new scientific principle. If the authors re-run the baseline under the same evaluator and add variance reporting, this could become a solid systems paper.\n\nRecommendation: send it to peer review. The contribution is real and the evaluator problem is fixable. Desk rejection would waste a worthwhile pipeline; accepting it as-is would certify a comparison that does not support the headline.","headline":"A real engineering pipeline for turning unstructured API docs into validated Python tools, but the headline WebArena gain compares across two different evaluators and is not established as stated.","tokens_in":24661,"tokens_out":2020,"would_cite":false,"duration_ms":23098,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Doc2Agent claims that raw REST API docs can be automatically converted into validated Python tools, and that agents using those tools outperform direct API calling by 55.1% relative at roughly one-tenth the cost per task.","keywords":["API documentation","tool-using agents","REST APIs","automated tool generation","code agents","parameter value inference","tool validation","large language model agents"],"falsifier":"Obtain the direct-API-calling baseline's action logs for the same five web-task sites, run them through the API-specified evaluator used for Doc2Agent, and check whether the average stays near 29.2; if it rises to the level of the hybrid agent, about 42.2, the reported 55.1% relative improvement is largely an artifact of the evaluation change. A second check: apply the pipeline to a fresh random sample of API documentation and measure whether the verified-tool rate still approaches 59.5% without human intervention.","tokens_in":23688,"feed_emoji":"🛠️","tokens_out":13230,"duration_ms":134889,"temperature":0.7,"pith_summary":"Doc2Agent aims to remove the manual effort that currently stands between real-world REST API documentation and a deployed tool-using agent. The paper's central claim is that an automated loop, generating Python tools from free-text docs, testing them against live servers, having a code agent fix failures, and re-validating, can produce reliable tools from documentation that is unstructured, incomplete, or outdated. On a five-site web-agent benchmark, agents equipped with the generated tools improved average success rates by 55.1% relative to a direct-API-calling baseline while spending about 10% of the cost per task. Across a 167-page real-world API corpus with 744 endpoints, 59.5% of the endpoints yielded tools that passed automated validation. If these numbers hold, the main obstacle to scaling tool agents becomes having the documentation itself, not hand-curating uniform tool sets.","feed_headline":"Auto-built API tools lift web-agent scores 55%, cut cost 90%","feed_subtitle":"Doc2Agent turns messy API docs into validated Python tools that agents can call automatically.","key_machinery":"The load-bearing mechanism is a generate-validate-refine-deploy loop. Generation converts unstructured HTML or Markdown documentation into structured JSON via LLM extraction, then into Python functions that wrap HTTP calls; a target-oriented variant first writes simplified function 'fingerprints' that define a use case and expected inputs for flexible, search-like endpoints. Validation executes each function against the live server and asks an LLM to classify the response as valid information, a code error, a server error, or a request error, with verification requiring alignment between the actual response and the predicted expectation. Refinement feeds failed tools and their error messages to a code agent, which rewrites the function and is re-validated. The distinctive sub-mechanism is an automatically built parameter database: JSON responses from already-verified tools are stored in a vector database alongside parameter names and descriptions, and for an unknown parameter the system retrieves semantically similar key-value pairs from other APIs, using implicit dependencies between services to infer parameter values without domain expertise. Deployment exports the verified tools through a standardized tool-serving protocol or as an OpenAPI specification, making them usable across agent architectures.","core_discovery":"The paper's central claim is that wrapping REST APIs as validated Python functions changes the agent's job from low-level request construction to high-level task logic, and that this shift is what drives the gains. The pipeline first transforms each endpoint's documentation into a structured specification and then into a callable function, using direct wrapping for simple docs and task-oriented generation for flexible endpoints. Every generated function is executed against the real API, and an LLM judge compares the actual response with the response predicted from the tool's description, so a tool is verified only when behavior matches expectation. Failed functions are handed to a code agent together with the documentation, error information, and candidate parameter values, and the loop repeats for up to three refinement rounds. The paper reports 443 verified tools from 744 real-world endpoints, a 47.6% boost in tool pass rate from refinement, and 70 refined tools for glycoscience research APIs. On the web-task benchmark, its tool-using agent reached an average 45.3% success, a 55.1% relative gain over the published direct-API baseline, at $0.12 per task versus $1.20.","pith_inferences":["A natural extension is to turn the validation loop into a self-improving production system, where every successful agent call seeds the parameter database, so the toolset improves with use instead of only during the initial build.","The paper's reported 95.6% relative gain on the least-organized site suggests the main failure mode of direct API calling is missing parameter knowledge rather than model reasoning; ablating the parameter database would test that attribution directly.","The paper's discussion of truncated, verbose JSON responses points to an obvious next design: generate response-filtering or summarization wrappers as part of each tool, so agents receive task-sized payloads instead of raw server output.","If docs-to-tools conversion becomes reliable, benchmark construction could be inverted: future benchmarks could sample real API documentation pages and let a pipeline like this one build the tools, matching real-world documentation quality instead of hand-curated schemas."],"forward_implications":["If the claim holds, developers no longer need to hand-craft uniform tool sets before an agent can use a new service; messy documentation becomes an acceptable starting point.","The order-of-magnitude cost reduction makes it economical for agents to call many tools repeatedly in multi-step tasks, widening the range of tasks that are feasible to automate.","Because every tool is exercised against a live server during validation, broken or outdated documentation is caught at build time rather than at agent runtime.","The cross-API parameter inference means tools can become usable even when no human expert supplies example inputs, which is the situation for most long-tail and research APIs.","Verified tools exported through a standard deployment protocol should be reusable across different agent frameworks, so tool-building effort does not have to be repeated per system."],"supporting_citations":[{"why":"Supplies the direct-API-calling baseline and the hybrid agent that Doc2Agent is compared against; its published scores are the reference for the 55.1% relative gain.","marker":"Song et al. (2025b)"},{"why":"Provides the five web-task environments, the baseline scores, and the original evaluator that the paper adapts into an API-specified evaluator.","marker":"Zhou et al. (2024)"},{"why":"Provides the executable-code action agent backbone on which the tool-using agent is implemented.","marker":"Wang et al. (2024b)"},{"why":"SteP baseline with manually defined policies; used to show automatically generated API tools can outperform hand-designed agent prompts.","marker":"Sodhi et al. (2024)"},{"why":"SkillWeaver baseline that synthesizes reusable browser skills; used to position generated API tools against generated browser skills.","marker":"Zheng et al. (2025)"},{"why":"Defines the standardized tool-serving protocol used for deploying the generated toolkits.","marker":"Hou et al. (2025)"},{"why":"Prior function-generation-from-docs work that relies on standardized API specs, providing the contrast for Doc2Agent's unstructured-documentation setting.","marker":"Shi et al. (2025)"},{"why":"Earlier large-scale tool sourcing from uniform API hub specs; frames the gap that unstructured real-world documentation creates for agent tooling.","marker":"Qin et al. (2023)"}],"fun_headline_variants":["Docs to tools: agent success up 55%, cost down 90%","Auto-wrap APIs to lift agent scores 55% and cut cost 90%","From API docs to callable tools: 55% better, 90% cheaper","Doc2Agent converts docs to tools, improving agents 55% and reducing cost","Validated tools from docs: agent performance +55%, cost -90%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central result collapses if the prior direct-API-calling agent's published scores under the original evaluator cannot be compared with Doc2Agent's scores under the stricter API-specific evaluator; the paper could not re-score the baseline because its logs were unavailable.","fun_headline_variants_meta":{"raw":{"variants":["Docs to tools: agent success up 55%, cost down 90%","Auto-wrap APIs to lift agent scores 55% and cut cost 90%","From API docs to callable tools: 55% better, 90% cheaper","Doc2Agent converts docs to tools, improving agents 55% and reducing cost","Validated tools from docs: agent performance +55%, cost -90%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000387,"raw_usage":{"total_tokens":2045,"prompt_tokens":947,"completion_tokens":1098,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":993}},"tokens_in":563,"tokens_out":1098,"duration_ms":11751,"temperature":1.0,"reasoning_tokens":993,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:58:45.222231+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Obtain the direct-API-calling baseline's action logs for the same five web-task sites, run them through the API-specified evaluator used for Doc2Agent, and check whether the average stays near 29.2; if it rises to the level of the hybrid agent, about 42.2, the reported 55.1% relative improvement is largely an artifact of the evaluation change. A second check: apply the pipeline to a fresh random sample of API documentation and measure whether the verified-tool rate still approaches 59.5% without human intervention.","supporting_citations":[],"review_version":1}