Pith. sign in

REVIEW 5 major objections 5 minor 22 references

OASBuilder: Generating OpenAPI Specifications from Online API Documentation with Large Language Models

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OASBuilder converts messy HTML API docs into valid OpenAPI specs.

desk verdict A well-built enterprise pipeline for turning HTML API docs into OpenAPI specs, but the generalization claim rests on an untested segmentation assumption and evaluation choices that flatter the results. read the letter →

arxiv 2507.05316 v1 pith:3C6O6VLO submitted 2025-07-07 cs.SE cs.AI

Koren Lazar , Matan Vetzler , Kiran Kate , Jason Tsay , David Boaz Himanshu Gupta , Avraham Shinnar , Rohith D Vallam , David Amid Esther Goldbraich
show 3 more authors
Guy Uziel Jim Laredo Ateret Anaby Tavor
This is my paper
classification cs.SEcs.AI
keywords OpenAPISpecificationdocumentationlargelanguagemodelsin-contextlearningRESTAPIsJSONschemagenerationHTMLscraping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OASBuilder is a proposed end-to-end pipeline for turning unstructured HTML API documentation pages into OpenAPI specifications, the machine-readable format that AI agents and business automation need in order to call external APIs. The paper claims that segmenting a page into per-operation sections and running separate LLM passes on usage examples and descriptive tables lets the system handle pages far larger than a single model context window. Across 291 documentation URLs the authors report 89% valid OpenAPI outputs, and on a labeled set of 108 operations the request-parameter recall reaches 0.85–0.86 with high precision. If the claim holds, converting enterprise API documentation into LLM-callable tools no longer requires manual specification writing, which the paper says has saved thousands of human hours in deployment.

What carries the argument

The load-bearing machinery is operation-scope segmentation paired with a dual-source generation-and-merge loop. A browser-automation step expands dynamically loaded content, finds operation signatures or request examples, and defines each operation's boundaries as the span from the first title before its first instance to the first title of the next operation. For every operation, one LLM pass converts standardized cURL commands into a partial OAS, while another LLM pass converts the surrounding descriptive HTML (selected by a minimal-ancestor heuristic, the smallest scope containing the signature and parameter table) into a second partial OAS; deeply nested request and response bodies are fragmented by a line threshold and converted to JSON schemas in parallel. The two partial documents are then merged with a fixed priority rule, and a final enhancement stage uses keyword-filtered LLM calls to fill missing descriptions, enums, defaults, and examples from grounded text.

What would settle it

Run OASBuilder on a documentation page where two operations' signatures are interleaved or where request and response content is hidden in tabbed panels rather than sequential title-delimited blocks, then compare the per-operation parameter recall and valid-OAS rate against the reported ~0.85–0.86 recall and 89% validity; a large drop would refute the central generalization claim.

Watch

Extended reading notes

Core claim

The central discovery is a decomposition strategy: rather than asking one LLM to read an entire documentation page, OASBuilder splits the page into operations by locating HTTP method signatures and cURL examples, then generates two partial OAS documents per operation — one from demonstrative examples and one from descriptive HTML tables — and merges them using a fixed priority rule. The merge keeps descriptions and required flags from the descriptive pass and takes types and locations from the demonstrative pass, on the grounds that each documentation style is more reliable for different attributes. The paper reports that this pipeline produces valid OAS for 89% of 291 URLs and captures most request-side parameters (recall 0.85–0.86, high precision), while response recall is lower because responses are deeply nested and often lack descriptive documentation. The authors take these results as evidence that the decomposition generalizes across diverse documentation formats and that the generated documents are useful first drafts rather than finished specifications.

Load-bearing premise

The pipeline assumes that every operation's documentation appears as one contiguous block marked by titles, with all instances of an operation occurring sequentially on the page; interleaved operations, repeated signatures, or tabbed panels would corrupt the segmentation that every later LLM step depends on.

Editorial extensions

If this is right

  • Documentation pages too large for a single LLM context window become convertible by per-operation decomposition, with 89% of 291 tested URLs yielding valid OAS.
  • Request-side information is captured well enough (recall ~0.85–0.86, high precision) that the generated OAS can serve as a strong first draft, sharply reducing manual annotation effort.
  • Descriptive and demonstrative documentation are complementary sources of truth, and the paper's merge priority — descriptions/required from descriptive, types/locations from demonstrative — is a reusable design choice for spec generation.
  • LLM choice matters: code-oriented models produced the highest valid-OAS rates, while a single direct generation attempt with a large-context model succeeded on only 25% of pages.
  • The enhancement stage can add missing metadata (descriptions, enums, defaults, examples) from grounded parameter descriptions, further cutting the human effort left after the initial generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The title-based segmentation assumption is the most likely boundary: documentation with interleaved operations, tabbed panels, or repeated signatures would corrupt the input to every later LLM step, so a natural extension is DOM-tree or visual clustering before segmentation.
  • The response-recall gap points to the next bottleneck: aligning response examples with descriptive property tables, rather than generating schemas from JSON alone, may recover the missing response properties.
  • The dual-source decomposition (examples versus descriptive text) should transfer to other schema-generation settings, such as GraphQL or gRPC documentation, where examples and reference tables carry complementary information.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. OASBuilder is a multi-stage system that scrapes HTML API documentation pages, segments them into operation blocks by locating operation signatures and request examples, uses parallel LLM calls with in-context examples to generate a demonstrative OAS from cURL commands and a descriptive OAS from HTML parameter tables, merges the two partial specifications, and enriches missing metadata with additional LLM prompts. The paper evaluates syntactic validity on 50 pages covering 189 operations and on a larger set of 291 URLs, and reports semantic parameter-level fidelity on 108 manually labeled operations using five open LLMs. The central claims are that OASBuilder generalizes across hundreds of APIs, produces valid OpenAPI specifications capturing most of the information in the original documentation, and has saved thousands of hours of manual effort in an enterprise deployment.

Significance. If the claims hold, OASBuilder addresses a real and important bottleneck: enabling AI agents to use enterprise REST APIs by automatically converting unstructured HTML documentation into OpenAPI specifications. The paper's strengths include a detailed and plausible pipeline description, evaluation across multiple open LLMs, a direct GPT-4 comparison on the full-HTML task, and evidence of real deployment. The strongest evidence is the request-side parameter precision and recall (0.94-0.96 and 0.55-0.86 across models in Table 2), which suggests the approach can capture much of the practical information needed for tool use. However, the evidence is substantially weakened by the absence of a released dataset or benchmark, the lack of comparison to prior OAS extraction systems, the unmeasured risk of in-context example leakage, and the fact that the semantic metrics are computed only over operations that survived to a valid OAS. The generalization claim is plausible but not yet established at the level of confidence implied by the abstract.

major comments (5)
  1. [Section 3.1 vs Section 2; Appendix A.2] The segmentation procedure is load-bearing and underspecified. Section 2 states that OASBuilder's 'sole assumption is the presence of an operation signature or a request example,' but Section 3.1 defines operation boundaries as spanning 'from the first title preceding its initial instance to the first title marking the start of the next operation' and assumes all instances of an operation are sequential. On a page with signatures or request examples but no preceding title element, the boundary rule is undefined, and the paper provides no count of how many evaluation pages required the title-based rule, how many operations were missed, or any precision/recall for detected operation spans. Since every downstream LLM stage receives only the segmented block, a boundary error can drop parameters or mis-associate operations without being visible in Table 2, which is computed only over outputs that yielded valid OAS. Please report segmentation-level metrics (detected operations vs. ground-truth operations per page, including zero-operation pages) and describe how 'title' is detected.
  2. [Section 4.2, Table 2] The semantic evaluation is computed 'based on the valid OASs for each model,' meaning the request/response recall in Table 2 measures fidelity only among operations that survived the pipeline to a valid OAS. Operations lost during segmentation or filtering are absent from the denominator, so the recall values cannot support the abstract claim that the generated specifications 'encapsulate most of the information from the original documentation.' This concern is compounded by the response-side recall, which is consistently 0.54-0.62 across models even on the surviving operations. Please report end-to-end recall over all ground-truth operations/pages, including cases where no operation was detected, and provide per-page or per-API breakdowns so that the scope of the 'most information' claim is clear.
  3. [Appendix A.3, Appendix A.4, Figures 4-6] The in-context examples are drawn from real-world APIs, with GitHub explicitly mentioned in Appendix A.3, and the prompt figures use Shopify, PayPal, and GitHub examples. The evaluation URL list in Appendix A.4 contains numerous docs.github.com, shopify.dev, and Paypal pages. If the in-context examples come from the same API families or the same pages as evaluation targets, the reported semantic scores may partly reflect format memorization or leakage rather than generalization. Please clarify the overlap between in-context example sources and evaluation URLs, or otherwise show that the results are not inflated by this overlap.
  4. [Section 4 (Experiments) and Section 5 (Related Work)] No comparison is made to existing automated OAS generation approaches such as AutoREST, D2Spec, or Androcec and Tomašić's GPT-3-based method, and the evaluation datasets are not released. The direct GPT-4-128K experiment in Section 4.1 is a useful sanity check for the value of decomposition, but it measures only syntactic validity, not extraction quality. Without any baseline or public benchmark, the 'generalizes well across hundreds of APIs' claim is hard to verify or to position relative to prior work. At minimum, release the URL list and the labeled 108-operation ground truth, and report results on any existing public REST API documentation benchmark.
  5. [Section 4.2] The semantic ground truth is described as a 'manually labeled dataset comprising of 108 operations containing thousands of parameters and properties,' but the paper does not describe the annotation protocol, the number of annotators, or inter-annotator agreement. Because the evaluation matches generated parameters against manually labeled ones and compares description similarity, the absence of agreement metrics makes it difficult to distinguish systematic extraction quality from annotation subjectivity. Please add an annotation guideline summary and agreement statistics, or otherwise bound the labeling noise.
minor comments (5)
  1. [Section 3.3] The parenthetical '(see Section 3.2 for more details)' appears in the discussion of descriptive OAS generation but should refer to Section 3.3, since the descriptive pipeline is the subject of that section.
  2. [Appendix A.2, Algorithm 1] The enumeration in Algorithm 1 is inconsistent: it uses 'a.', 'b.', then 'iv.' for the ranking criteria, and reuses '1.' under a different heading. Please renumber the steps for readability.
  3. [Table 2] The column header 'D esc.' is visually awkward and can be misread as a single token; use 'Desc.' instead.
  4. [Appendix A.4] The Workday URL appears three times in the list with slightly different fragment suffixes; please deduplicate the entries or explain why each variant is needed.
  5. [Section 4.1] The sentence 'Baselines were not included, as previous studies neither evaluated on a public benchmark nor provided their code or reproduction details' is a rationale for not comparing with prior systems, but the lack of a shared benchmark does not remove the need for a baseline comparison in this paper; consider rephrasing to acknowledge this limitation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are measured against manually labeled ground truth, not derived from the system's own assumptions, and no fitted parameter or self-cited theorem forces the result.

full rationale

OASBuilder's central claims are empirical: request-parameter recall, precision, F1, and valid-OAS rates are computed against a manually labeled dataset of 108 operations. The paper states that generation uses in-context learning rather than fine-tuning, so no model parameter is fitted to that ground truth; the in-context examples are fixed prompt fixtures, not outputs tuned on the evaluation set. The post-generation check that generated parameter names appear in the input is a grounding constraint, not a circular reduction. The segmentation rule in Section 3.1 does rely on operation signatures or request examples plus an assumption of sequential instances, and the apparent inconsistency with Section 2's 'sole assumption' statement is a robustness threat for title-less or interleaved documentation pages; however, that affects correctness and generalization, not circularity, because the output is not equal to the input by construction. The stated limitations—no public benchmark, no baselines, unreleased in-context examples, and lower response recall—are evaluation-transparency concerns, not evidence that a prediction reduces to its inputs. Self-citations and the product footnote are not load-bearing in the derivation chain.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical, formal, or ontological entities. Its central claim rests on domain assumptions about the structure of API documentation pages and on an unverified assumption about LLM pretraining exposure, plus one hand-chosen fragmentation threshold whose value is not reported.

free parameters (1)
  • JSON fragmentation line threshold = not specified (described as 'predefined')
    Section 3.2 says JSON structures are divided into fragments based on a predefined line threshold; the value is not reported yet it determines how deeply nested request and response schemas are split and therefore affects schema quality.
assumptions (4)
  • domain assumption API documentation pages contain recurring semantic components: operation signature, descriptive documentation, and demonstrative documentation.
    Section 2 and Figure 1 describe these components as the basis for the pipeline; if a page lacks them, OASBuilder has nothing to anchor on. The paper says its sole assumption is the presence of an operation signature or request example, but the richer enrichment stages assume more.
  • domain assumption All instances of a specific operation appear sequentially on the page and operation boundaries are marked by titles.
    Section 3.1 defines operation boundaries as the span from the first title preceding the first instance to the first title marking the next operation. Non-sequential or untitled layouts break segmentation and corrupt downstream LLM input.
  • domain assumption The LLM was exposed to OpenAPI specifications during pretraining.
    Section 3 states 'It is assumed that the LLM was exposed to OAS during its pretraining.' The in-context generation of OAS documents depends on this unverified capability assumption.
  • domain assumption LLMs can generalize over structural variations in HTML documentation.
    Section 3.3 argues that deterministic rule-based parsing is impractical and that LLMs can handle structural differences; this is an unverified capability assumption that the descriptive generation stage relies on.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OASBuilder: Generating OpenAPI Specifications from Online API Documentation with Large Language Models." pith.science (2026). https://pith.science/paper/3C6O6VLO

@misc{pith2026250705316,
  author       = {Pith},
  title        = {Pith review of: OASBuilder: Generating OpenAPI Specifications from Online API Documentation with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3C6O6VLO}},
  note         = {Machine review of arXiv:2507.05316}
}
read the original abstract

AI agents and business automation tools interacting with external web services require standardized, machine-readable information about their APIs in the form of API specifications. However, the information about APIs available online is often presented as unstructured, free-form HTML documentation, requiring external users to spend significant time manually converting it into a structured format. To address this, we introduce OASBuilder, a novel framework that transforms long and diverse API documentation pages into consistent, machine-readable API specifications. This is achieved through a carefully crafted pipeline that integrates large language models and rule-based algorithms which are guided by domain knowledge of the structure of documentation webpages. Our experiments demonstrate that OASBuilder generalizes well across hundreds of APIs, and produces valid OpenAPI specifications that encapsulate most of the information from the original documentation. OASBuilder has been successfully implemented in an enterprise environment, saving thousands of hours of manual effort and making hundreds of complex enterprise APIs accessible as tools for LLMs.

Figures

Figures reproduced from arXiv: 2507.05316 by the authors.

Figure 1
Figure 1. Typical structure of API operation documen [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. OASBuilder pipeline: The system first segments the webpage based on identified API operations. Then, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Example of an API documentation webpage taken from Shopify API in 25.11.2024 (taken from [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Example prompt for generating an OAS from a cURL command. The prompt includes two in-context [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Prompt example to generate a JSON schema from a given JSON object or array. This prompt is used to [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: An example of a prompt used for generating an OAS based on the descriptive documentation found in the [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Example of enhancement for generating descriptions. Added lines are highlighted in green. Original doc [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Example of enhancement for generating examples. Added lines are highlighted in green. Orig [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 11 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Darko Andro c ec and Matija Toma s i \'c . 2023. Using gpt-3 to automatically create restful service descriptions. In 2023 4th International Conference on Communications, Information, Electronic and Energy Systems (CIEES), pages 1--4. IEEE

    work page 2023

  4. [4]

    Mehdi Bahrami, Mehdi Assefi, Ian Thomas, Wei-Peng Chen, Shridhar Choudhary, and Hamid R Arabnia. 2020. Deep sas: A deep signature-based api specification learning approach. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 1994--2001. IEEE

    work page 2020

  5. [5]

    Mehdi Bahrami and Wei-Peng Chen. 2020. Automated web service specification generation through a transformation-based learning. In Services Computing--SCC 2020: 17th International Conference, Held as Part of the Services Conference Federation, SCF 2020, Honolulu, HI, USA, September 18--20, 2020, Proceedings 17, pages 103--119. Springer

    work page 2020

  6. [6]

    Hanyang Cao, Jean-R \'e my Falleri, and Xavier Blanc. 2017. Automated generation of rest api specification from plain html documentation. In Service-Oriented Computing: 15th International Conference, ICSOC 2017, Malaga, Spain, November 13--16, 2017, Proceedings, pages 453--461. Springer

    work page 2017

  7. [7]

    Harrison Chase. 2022. https://github.com/langchain-ai/langchain Langchain

    work page 2022

  8. [8]

    Danielsen and Alan Jeffrey

    Peter J. Danielsen and Alan Jeffrey. 2013. https://doi.org/10.1109/ICWS.2013.76 Validation and interactivity of web api documentation . In 2013 IEEE 20th International Conference on Web Services, pages 523--530

    work page doi:10.1109/icws.2013.76 2013

Show all 22 references
  1. [9]

    Paola Espinoza-Arias, Daniel Garijo, and Oscar Corcho. 2020. Mapping the web ontology language to the openapi specification. In International Conference on Conceptual Modeling, pages 117--127. Springer

    2020

  2. [10]

    A Shaji George and AS Hovan George. 2023. A review of chatgpt ai's impact on several business sectors. Partners universal international innovation journal, 1(1):9--23

    2023

  3. [11]

    Ruikai Huang, Manish Motwani, Idel Martinez, and Alessandro Orso. 2024. Generating rest api specifications through static analysis

    2024

  4. [12]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  5. [13]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne...

    2024 arXiv

  6. [14]

    Myeongsoo Kim, Qi Xin, Saurabh Sinha, and Alessandro Orso. 2022. https://doi.org/10.1145/3533767.3534401 Automated test generation for rest apis: no time to rest yet . In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2022, ...

    2022

  7. [15]

    Cox, Ruchir Puri, and Rameswar Panda

    Mayank Mishra, Matt Stallone, Gaoyuan Zhang, Yikang Shen, Aditya Prasad, Adriana Meza Soria, Michele Merler, Parameswaran Selvam, Saptha Surendran, Shivdeep Singh, Manish Sethi, Xuan-Hong Dang, Pengyuan Li, Kun-Lung Wu, Syed Zawad, Andrew Coleman, Matthew White, Mark Lewis, Ra...

    2024 arXiv

  8. [16]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  9. [17]

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong...

    2024 arXiv

  10. [18]

    Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Erika Cardenas, Akanksha Sharma, John Trengrove, and Bob van Luijt. 2024. https://arxiv.org/abs/2408.11061 Structuredrag: Json response formatting with large language models . Preprint, arXiv:2408.11061

    2024 arXiv

  11. [19]

    Sheikh Mohammed Sohan, Craig Anslow, and Frank Maurer. 2015. Spyrest: Automated restful api documentation using an http proxy server (n). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 271--276. IEEE

    2015

  12. [20]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  13. [21]

    Mandana Vaziri, Louis Mandel, Avraham Shinnar, J \'e r \^o me Sim \'e on, and Martin Hirzel. 2017. Generating chat bots from web api specifications. In Proceedings of the 2017 ACM SIGPLAN international symposium on new ideas, new paradigms, and reflections on programming and s...

    2017

  14. [22]

    Jinqiu Yang, Erik Wittern, Annie T. T. Ying, Julian Dolby, and Lin Tan. 2018. https://doi.org/10.1145/3196398.3196411 Towards extracting web api specifications from documentation . In Proceedings of the 15th International Conference on Mining Software Repositories, MSR '18, pa...

    2018

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.