{"id":"6ca67276-7035-4aec-aa19-db11a4ee5dd7","arxiv_id":"2507.05316","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A modular pipeline of rules and LLMs converts HTML API documentation into OpenAPI specifications, evaluated on hundreds of APIs and deployed in an enterprise setting.","lead":"OASBuilder is a system that reads messy HTML API documentation and automatically writes machine-readable OpenAPI specifications using web scraping, rule-based segmentation, and large language models. It matters because converting API documentation by hand is slow and error-prone, and standardized specs are what let AI agents and automation tools call web services.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Segmentation assumes title-marked operation blocks despite stated sole assumption of signatures; on title-less or interleaved docs every downstream stage is corrupted.","rationale":"OASBuilder is a plausible, concretely described engineering system, and the internal experiments give some evidence that the pipeline works on a selection of real documentation pages. The paper's stated novelty and central claim, however, depend on the first pipeline stage: correct segmentation of a webpage into per-operation blocks. That stage is the least evidenced component. The paper itself contains an internal inconsistency between Section 2's 'sole assumption' (signature or request example only) and Section 3.1's dependence on title elements for boundary determination. This is a correctness risk internal to the description, not a disagreement with community consensus. The reported end-to-end metrics cannot detect silent operation loss before the LLM stages, and the 86% scalability number is only a weak lower bound on content retention. The reader's CONDITIONAL verdict and the identification of segmentation as the weakest assumption are therefore well founded. My stress-test does not move the verdict; it sharpens the concern and points to a direct experimental check that would settle it.","tokens_in":16299,"tokens_out":4277,"duration_ms":50977,"concrete_test":"Construct a held-out corpus of 50 documentation pages that violate the title-and-contiguity assumption (flat heading-less sections, tabbed or interleaved operation blocks, repeated signatures across tabs). Manually annotate ground-truth operation spans and run OASBuilder's scraping module; compute operation-level boundary precision/recall and the fraction of pages with zero discovered operations. If boundary F1 on this adversarial set is substantially lower than on the paper's corpus, the generalization claim is unsupported. A lighter check is to instrument the existing 291-URL run to log segmentation failures and report how many operations were dropped before LLM generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines operation boundaries as spanning from 'the first title preceding its initial instance' to 'the first title marking the start of the next operation,' and assumes all instances of an operation appear sequentially. Section 2, however, states that OASBuilder's 'sole assumption is the presence of an operation signature or a request example.' These are inconsistent: on documentation pages with signatures but no heading/title element before each operation, the boundary rule is undefined. The paper reports no direct segmentation-quality metrics: no precision/recall of detected operation spans, no count of pages where zero operations were found, and no analysis of interleaved, tabbed, or repeated operation blocks. Appendix A.2's fallback for non-consecutive minimal ancestors assumes an ordered ancestor chain ending at the next request, which still fails for flat DOM structures or signatures repeated across tabs. Since the LLM stages in Sections 3.2 and 3.3 receive only the segmented block as input, an erroneous boundary propagates missing parameters, misassociated operations, and hallucinated fields into the final OAS. Table 2's request recall of 0.85-0.86 is computed only over operations that survived to a valid OAS, so it cannot expose operations lost upstream. The scalability result—86% of 291 outputs containing at least one operation and one parameter—is too weak to validate segmentation robustness, because a page reduced to a single operation would still pass. The central generalization claim therefore rests on an untested and under-specified segmentation assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"OASBuilder is a multi-stage system that scrapes HTML API documentation pages, segments them into operation blocks by locating operation signatures and request examples, uses parallel LLM calls with in-context examples to generate a demonstrative OAS from cURL commands and a descriptive OAS from HTML parameter tables, merges the two partial specifications, and enriches missing metadata with additional LLM prompts. The paper evaluates syntactic validity on 50 pages covering 189 operations and on a larger set of 291 URLs, and reports semantic parameter-level fidelity on 108 manually labeled operations using five open LLMs. The central claims are that OASBuilder generalizes across hundreds of APIs, produces valid OpenAPI specifications capturing most of the information in the original documentation, and has saved thousands of hours of manual effort in an enterprise deployment.","tokens_in":16510,"tokens_out":6323,"duration_ms":71566,"significance":"If the claims hold, OASBuilder addresses a real and important bottleneck: enabling AI agents to use enterprise REST APIs by automatically converting unstructured HTML documentation into OpenAPI specifications. The paper's strengths include a detailed and plausible pipeline description, evaluation across multiple open LLMs, a direct GPT-4 comparison on the full-HTML task, and evidence of real deployment. The strongest evidence is the request-side parameter precision and recall (0.94-0.96 and 0.55-0.86 across models in Table 2), which suggests the approach can capture much of the practical information needed for tool use. However, the evidence is substantially weakened by the absence of a released dataset or benchmark, the lack of comparison to prior OAS extraction systems, the unmeasured risk of in-context example leakage, and the fact that the semantic metrics are computed only over operations that survived to a valid OAS. The generalization claim is plausible but not yet established at the level of confidence implied by the abstract.","major_comments":[{"comment":"The segmentation procedure is load-bearing and underspecified. Section 2 states that OASBuilder's 'sole assumption is the presence of an operation signature or a request example,' but Section 3.1 defines operation boundaries as spanning 'from the first title preceding its initial instance to the first title marking the start of the next operation' and assumes all instances of an operation are sequential. On a page with signatures or request examples but no preceding title element, the boundary rule is undefined, and the paper provides no count of how many evaluation pages required the title-based rule, how many operations were missed, or any precision/recall for detected operation spans. Since every downstream LLM stage receives only the segmented block, a boundary error can drop parameters or mis-associate operations without being visible in Table 2, which is computed only over outputs that yielded valid OAS. Please report segmentation-level metrics (detected operations vs. ground-truth operations per page, including zero-operation pages) and describe how 'title' is detected.","section":"Section 3.1 vs Section 2; Appendix A.2"},{"comment":"The semantic evaluation is computed 'based on the valid OASs for each model,' meaning the request/response recall in Table 2 measures fidelity only among operations that survived the pipeline to a valid OAS. Operations lost during segmentation or filtering are absent from the denominator, so the recall values cannot support the abstract claim that the generated specifications 'encapsulate most of the information from the original documentation.' This concern is compounded by the response-side recall, which is consistently 0.54-0.62 across models even on the surviving operations. Please report end-to-end recall over all ground-truth operations/pages, including cases where no operation was detected, and provide per-page or per-API breakdowns so that the scope of the 'most information' claim is clear.","section":"Section 4.2, Table 2"},{"comment":"The in-context examples are drawn from real-world APIs, with GitHub explicitly mentioned in Appendix A.3, and the prompt figures use Shopify, PayPal, and GitHub examples. The evaluation URL list in Appendix A.4 contains numerous docs.github.com, shopify.dev, and Paypal pages. If the in-context examples come from the same API families or the same pages as evaluation targets, the reported semantic scores may partly reflect format memorization or leakage rather than generalization. Please clarify the overlap between in-context example sources and evaluation URLs, or otherwise show that the results are not inflated by this overlap.","section":"Appendix A.3, Appendix A.4, Figures 4-6"},{"comment":"No comparison is made to existing automated OAS generation approaches such as AutoREST, D2Spec, or Androcec and Tomašić's GPT-3-based method, and the evaluation datasets are not released. The direct GPT-4-128K experiment in Section 4.1 is a useful sanity check for the value of decomposition, but it measures only syntactic validity, not extraction quality. Without any baseline or public benchmark, the 'generalizes well across hundreds of APIs' claim is hard to verify or to position relative to prior work. At minimum, release the URL list and the labeled 108-operation ground truth, and report results on any existing public REST API documentation benchmark.","section":"Section 4 (Experiments) and Section 5 (Related Work)"},{"comment":"The semantic ground truth is described as a 'manually labeled dataset comprising of 108 operations containing thousands of parameters and properties,' but the paper does not describe the annotation protocol, the number of annotators, or inter-annotator agreement. Because the evaluation matches generated parameters against manually labeled ones and compares description similarity, the absence of agreement metrics makes it difficult to distinguish systematic extraction quality from annotation subjectivity. Please add an annotation guideline summary and agreement statistics, or otherwise bound the labeling noise.","section":"Section 4.2"}],"minor_comments":[{"comment":"The parenthetical '(see Section 3.2 for more details)' appears in the discussion of descriptive OAS generation but should refer to Section 3.3, since the descriptive pipeline is the subject of that section.","section":"Section 3.3"},{"comment":"The enumeration in Algorithm 1 is inconsistent: it uses 'a.', 'b.', then 'iv.' for the ranking criteria, and reuses '1.' under a different heading. Please renumber the steps for readability.","section":"Appendix A.2, Algorithm 1"},{"comment":"The column header 'D esc.' is visually awkward and can be misread as a single token; use 'Desc.' instead.","section":"Table 2"},{"comment":"The Workday URL appears three times in the list with slightly different fragment suffixes; please deduplicate the entries or explain why each variant is needed.","section":"Appendix A.4"},{"comment":"The sentence 'Baselines were not included, as previous studies neither evaluated on a public benchmark nor provided their code or reproduction details' is a rationale for not comparing with prior systems, but the lack of a shared benchmark does not remove the need for a baseline comparison in this paper; consider rephrasing to acknowledge this limitation.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to the practical contribution and the evidence of real enterprise deployment, but the current manuscript does not yet provide enough reviewable evidence for the strength of the central claim. The two largest gaps are the absence of segmentation-level evaluation (which is load-bearing for the whole pipeline) and the lack of clarification about in-context example overlap with evaluation URLs. With segmentation metrics, an end-to-end recall denominator, and a released or clearly described dataset, I would be able to evaluate the generalization claim more fairly. The paper is within scope for a software-engineering venue and the proposed revisions are feasible within the manuscript's current scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: OASBuilder is a real engineering contribution. The multi-stage design—segment the page, generate OAS from cURL examples in parallel, separately parse the descriptive HTML, merge, and enhance—is more thoughtful than the single-prompt GPT approaches cited. The appendix prompts are concrete, the LLM comparison across five open models is useful, and the enterprise deployment gives credibility to the 'saves thousands of hours' claim. I buy that this works well on the kinds of docs it was tuned for.\n\nThe soft spots are all about the evidence for generalization. The biggest is the segmentation step. Section 2 says the 'sole assumption' is an operation signature or request example, but Section 3.1 defines operation boundaries using the 'first title preceding' the signature and the 'first title' of the next operation, and assumes all instances of an operation appear sequentially. Those are two extra assumptions, and they are not tested anywhere. There are no precision/recall numbers for detected operation spans, no count of pages where zero operations were found, and no discussion of title-less or interleaved layouts. Since every later LLM call is scoped to a segmented block, a bad boundary silently drops parameters or misassigns them. Table 2's request recall of 0.85–0.86 only covers operations that survived to a valid OAS, so it cannot see upstream losses. The 86% of 291 outputs with at least one operation and one parameter is too weak: a page reduced to one operation passes.\n\nThe evaluation is also thinner than the prose suggests. No code or dataset release, no inter-annotator agreement on the manual ground truth, and no real baseline (the GPT-4-128K direct-HTML run is a sanity check, not a comparison). The in-context examples come from GitHub and Shopify, which overlap with evaluation URLs; that doesn't invalidate the results but makes the 'generalizes across hundreds of APIs' claim less certain. And the 'first LLM-based automated system' phrasing doesn't survive the paper's own citation of Andročec and Tomašić (2023), though the modular architecture is genuinely different.\n\nNone of this is fatal. The pipeline is plausible and the syntactic validity numbers (89% valid OAS on 291 URLs) are decent. But the paper would be much stronger with released data, direct segmentation metrics, and a recall computed over all ground-truth operations, not just survivors. As it stands, I'd treat it as a promising system with a conditional generalization claim.\n\nThis is worth a serious referee: the idea is timely, the engineering is real, and the flaws are fixable in revision. I'd ask for data/code release and a segmentation-focused evaluation. For my own reading group, I'd probably spend 30 minutes on it as an example of how to (and how not to) evaluate LLM pipelines.","headline":"A well-built enterprise pipeline for turning HTML API docs into OpenAPI specs, but the generalization claim rests on an untested segmentation assumption and evaluation choices that flatter the results.","tokens_in":17128,"tokens_out":3756,"would_cite":true,"duration_ms":40218,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OASBuilder converts messy HTML API docs into valid OpenAPI specs.","keywords":["OpenAPI Specification","API documentation","large language models","in-context learning","REST APIs","JSON schema generation","HTML scraping","API specification generation"],"falsifier":"Run OASBuilder on a documentation page where two operations' signatures are interleaved or where request and response content is hidden in tabbed panels rather than sequential title-delimited blocks, then compare the per-operation parameter recall and valid-OAS rate against the reported ~0.85–0.86 recall and 89% validity; a large drop would refute the central generalization claim.","tokens_in":16087,"feed_emoji":"🧩","tokens_out":9441,"duration_ms":93352,"temperature":0.7,"pith_summary":"OASBuilder is a proposed end-to-end pipeline for turning unstructured HTML API documentation pages into OpenAPI specifications, the machine-readable format that AI agents and business automation need in order to call external APIs. The paper claims that segmenting a page into per-operation sections and running separate LLM passes on usage examples and descriptive tables lets the system handle pages far larger than a single model context window. Across 291 documentation URLs the authors report 89% valid OpenAPI outputs, and on a labeled set of 108 operations the request-parameter recall reaches 0.85–0.86 with high precision. If the claim holds, converting enterprise API documentation into LLM-callable tools no longer requires manual specification writing, which the paper says has saved thousands of human hours in deployment.","feed_headline":"LLM pipeline turns messy API docs into valid OpenAPI specs","feed_subtitle":"OASBuilder splits long API pages per operation, then merges partial LLM specs into valid OpenAPI documents.","key_machinery":"The load-bearing machinery is operation-scope segmentation paired with a dual-source generation-and-merge loop. A browser-automation step expands dynamically loaded content, finds operation signatures or request examples, and defines each operation's boundaries as the span from the first title before its first instance to the first title of the next operation. For every operation, one LLM pass converts standardized cURL commands into a partial OAS, while another LLM pass converts the surrounding descriptive HTML (selected by a minimal-ancestor heuristic, the smallest scope containing the signature and parameter table) into a second partial OAS; deeply nested request and response bodies are fragmented by a line threshold and converted to JSON schemas in parallel. The two partial documents are then merged with a fixed priority rule, and a final enhancement stage uses keyword-filtered LLM calls to fill missing descriptions, enums, defaults, and examples from grounded text.","core_discovery":"The central discovery is a decomposition strategy: rather than asking one LLM to read an entire documentation page, OASBuilder splits the page into operations by locating HTTP method signatures and cURL examples, then generates two partial OAS documents per operation — one from demonstrative examples and one from descriptive HTML tables — and merges them using a fixed priority rule. The merge keeps descriptions and required flags from the descriptive pass and takes types and locations from the demonstrative pass, on the grounds that each documentation style is more reliable for different attributes. The paper reports that this pipeline produces valid OAS for 89% of 291 URLs and captures most request-side parameters (recall 0.85–0.86, high precision), while response recall is lower because responses are deeply nested and often lack descriptive documentation. The authors take these results as evidence that the decomposition generalizes across diverse documentation formats and that the generated documents are useful first drafts rather than finished specifications.","pith_inferences":["The title-based segmentation assumption is the most likely boundary: documentation with interleaved operations, tabbed panels, or repeated signatures would corrupt the input to every later LLM step, so a natural extension is DOM-tree or visual clustering before segmentation.","The response-recall gap points to the next bottleneck: aligning response examples with descriptive property tables, rather than generating schemas from JSON alone, may recover the missing response properties.","The dual-source decomposition (examples versus descriptive text) should transfer to other schema-generation settings, such as GraphQL or gRPC documentation, where examples and reference tables carry complementary information."],"forward_implications":["Documentation pages too large for a single LLM context window become convertible by per-operation decomposition, with 89% of 291 tested URLs yielding valid OAS.","Request-side information is captured well enough (recall ~0.85–0.86, high precision) that the generated OAS can serve as a strong first draft, sharply reducing manual annotation effort.","Descriptive and demonstrative documentation are complementary sources of truth, and the paper's merge priority — descriptions/required from descriptive, types/locations from demonstrative — is a reusable design choice for spec generation.","LLM choice matters: code-oriented models produced the highest valid-OAS rates, while a single direct generation attempt with a large-context model succeeded on only 25% of pages.","The enhancement stage can add missing metadata (descriptions, enums, defaults, examples) from grounded parameter descriptions, further cutting the human effort left after the initial generation."],"supporting_citations":[{"why":"Closest prior LLM-based approach that converts preprocessed HTML into REST service descriptions; OASBuilder's decomposition and extraction steps are positioned against it.","marker":"(Andročcec and Tomašić, 2023)"},{"why":"D2Spec rule-based extraction work cited to show deterministic parsing is impractical and to motivate LLM-based descriptive generation.","marker":"(Yang et al., 2018)"},{"why":"AutoREST fixed-rule conversion from plain HTML documentation, an earlier baseline for automated OAS generation.","marker":"(Cao et al., 2017)"},{"why":"Documents LLM difficulty with large nested JSON structures, motivating the fragmentation of request/response schemas by line threshold.","marker":"(Shorten et al., 2024)"},{"why":"Provides the GPT-4 large-context model used as the direct single-shot generation comparison, which succeeded on 25% of webpages.","marker":"(OpenAI et al., 2024)"},{"why":"Supplies the granite-20b-code-instruct model used in the syntactic evaluation and the 291-URL scalability experiment.","marker":"(Mishra et al., 2024)"}],"fun_headline_variants":["LLM pipeline splits API docs to auto-generate OpenAPI specs","OASBuilder turns free-form HTML docs into valid OpenAPI specifications","Decompose docs, merge partial specs: OASBuilder yields valid OpenAPI","From messy HTML to machine-readable API specs with OASBuilder","Split-and-merge LLM framework generates OpenAPI specs from online docs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that every operation's documentation appears as one contiguous block marked by titles, with all instances of an operation occurring sequentially on the page; interleaved operations, repeated signatures, or tabbed panels would corrupt the segmentation that every later LLM step depends on.","fun_headline_variants_meta":{"raw":{"variants":["LLM pipeline splits API docs to auto-generate OpenAPI specs","OASBuilder turns free-form HTML docs into valid OpenAPI specifications","Decompose docs, merge partial specs: OASBuilder yields valid OpenAPI","From messy HTML to machine-readable API specs with OASBuilder","Split-and-merge LLM framework generates OpenAPI specs from online docs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1359,"prompt_tokens":894,"completion_tokens":465,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":371}},"tokens_in":510,"tokens_out":465,"duration_ms":5280,"temperature":1.0,"reasoning_tokens":371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:33:02.207740+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OASBuilder on a documentation page where two operations' signatures are interleaved or where request and response content is hidden in tabbed panels rather than sequential title-delimited blocks, then compare the per-operation parameter recall and valid-OAS rate against the reported ~0.85–0.86 recall and 89% validity; a large drop would refute the central generalization claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AutoREST fixed-rule conversion from plain HTML documentation, an earlier baseline for automated OAS generation."}],"review_version":1}