OOPS: Automated generation of REST API specification via LLMs

Chen Chen; Fengxu Lin; Hao Chen; Wei Li; Yunchun Li

REVIEW 4 major objections 5 minor 3 references

OOPS: Automated generation of REST API specification via LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OOPS claims to be the first technology-agnostic LLM-based static analysis method that generates OpenAPI specifications directly from server source code, reporting average F1-scores above 92% across 12 real-world REST APIs in five languages

desk verdict OOPS is a genuine new LLM-agent approach to OpenAPI generation with a serious multi-language evaluation, but the unconditional acyclic-dependency assumption and the missing LLM-baseline comparisons keep the technology-agnostic claim conditional. read the letter →

arxiv 2601.12735 v2 pith:XHRTOMJ4 submitted 2026-01-19 cs.SE

Hao Chen , Yunchun Li , Chen Chen , Fengxu Lin , Wei Li This is my paper · ORCID

classification cs.SE

keywords OpenAPISpecificationRESTLLMagentworkflowdependencygraphstaticanalysishallucinationmitigationendpointextractiongeneration

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that OpenAPI specifications can be produced automatically from server source code by a large-language-model agent workflow, without per-framework rules or expert annotations. The method builds a dependency graph of files involved in routing, which lets the LLM handle large projects despite context limits, and it splits generation into request and response stages followed by a self-refine loop to suppress hallucinations. On 12 REST APIs in five programming languages and eight frameworks, it reports average F1-scores of 98.8% for endpoint inference, 97.2% for request parameters, 97.0% for responses, and 92.3% for parameter constraints, at token costs far below context limits. This matters because accurate OpenAPI descriptions are the foundation for API documentation, automated testing, vulnerability detection, and LLM tool integration.

What carries the argument

The central object is the API dependency graph, a directed acyclic graph whose nodes are files and whose edges say that an API entry in one file references a handler implemented in another file. The graph lets the workflow perform topological sorting over only the relevant files, so the LLM never needs the whole repository in its context; it is paired with multi-stage request/response generation and a self-refine loop that detects and fixes syntactic and semantic hallucination errors.

What would settle it

Run OOPS on a small but real REST API whose route files import each other in a cycle, or add a circular import to one of the 12 projects; if the topological sort fails, produces incomplete paths, or the workflow does not terminate, the technology-agnostic claim for such projects is false.

Watch

Extended reading notes

Core claim

The paper's central claim is that endpoint extraction can be decomposed into a two-level mechanism: first detect API entries as (path, handler, tag) triples, where the tag marks whether the handler is defined locally or imported, and then build a directed acyclic graph over files whose edges indicate that one file's route prefix references a handler in another file. Feeding each local entry's descendant subgraph to the LLM in topological order gives the model exactly the files needed to reconstruct the full request path and HTTP methods. The same file associations are then used by separate LLM agents to generate request and response schemas, with a syntax-fixing self-refine loop that repairs

Load-bearing premise

The method assumes the API dependency graph is acyclic, because endpoint extraction topologically sorts the graph, but real code can contain circular imports and mutually referencing files, and the paper does not say what happens then.

Editorial extensions

If this is right

If the reported results hold, a machine-readable OpenAPI contract can be derived from a server codebase alone, eliminating the need for annotations, expert rule-writing, or a running deployment.
Because context usage stays low (average input under 5.6K tokens, maximum 16.13K), the approach scales to repositories whose full contents would not fit in an LLM window.
The generated OpenAPI can feed the standard downstream pipeline—documentation, automated testing, and vulnerability scanning—directly from source rather than from a manually maintained spec.
The reported results on private, non-public projects indicate the method does not depend on the LLM having memorised the codebase, addressing a common data-leakage concern.
The self-refine loop and multi-stage generation are credited with suppressing both invalid JSON syntax and mixed-version OpenAPI structures, so the quality should transfer to any OpenAPI-consuming toolchain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper leaves implicit is drift detection: regenerate the OpenAPI spec from current source and diff it against the maintained spec to flag undocumented endpoints or outdated schemas.
The dependency-graph mechanism may transfer to other interface styles—GraphQL schemas, RPC definitions, or message contracts—since it only relies on representing routes and handlers as cross-file references.
The acyclicity assumption marks a likely boundary: projects with circular imports would need cycle handling (for example, collapsing strongly connected components) before the technology-agnostic claim can cover them; the paper does not report on that case.
The hallucination-mitigation claim can be tested directly by injecting known reference and syntax errors into LLM outputs and measuring whether the self-refine loop converges rather than oscillates; the paper reports success but not failure-mode counts.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a real contribution to LLM-based OAS generation, and the evaluation is more careful than most in this subfield, but the headline "technology-agnostic" claim is not fully established.

The core idea is genuinely new relative to Chaplia and LRASGen: instead of JavaScript-specific dependency libraries or human-crafted regexes, OOPS has an LLM agent build an API dependency graph to associate files. That is a legitimate design step, and the paper supports it with an original 12-project, 5-language, 8-framework evaluation.

What the paper does well: the multi-stage generation with self-refine is sensible; the ablation shows both main stages contribute; RQ3 across five base LLMs is useful; and the context/cost numbers are concrete and cheap ($0.41 average per project). The authors are also honest in Section 6 about data leakage and external validity. That is not common enough and should be credited.

Now the soft spots, in proportion. I checked the stress-test concern about cycles and it holds. The API dependency graph is defined as a DAG, and Algorithm 1 unconditionally calls topologicalSort on the subgraph. There is no cycle detection, no fallback, and no discussion. Circular imports or mutually referencing route handlers would break the ordering. The 12 evaluated projects have max dependency depth 3 and appear to be acyclic, so the evaluation does not exercise the failure case. I would call this a moderate gap rather than fatal—many real projects may be acyclic in the relevant sense—but the paper should either handle cycles or explicitly scope the claim.

Also: the two LLM-based baselines named in the introduction, Chaplia and LRASGen, are not compared against because no runnable implementations are available. That weakens the "better than state-of-the-art" relative claim, which is only supported against non-LLM baselines. The aggregate baseline F1 numbers in Table 4 also mix different project subsets, so the headline gaps are not apples-to-apples. No error bars or repeated runs; temperature 0 helps but does not make LLM outputs deterministic across API versions and platforms. The manually constructed ground truth is normal for this area, but it is still a threat.

Who is this for: researchers working on API specification generation, LLM agents for repository-level analysis, and empirical software engineering with LLMs. It deserves a serious referee. I would send it out, with an expectation of major revision around cycle handling, baseline comparison, and statistical rigor.

Referee Report

4 major / 5 minor

Summary. The paper presents OOPS, an LLM-agent workflow that generates OpenAPI Specifications (OAS) from REST API server source code through static analysis. The method first identifies the server-side technology, then extracts endpoint methods by detecting file-level API entries, constructing an API dependency graph for file association, and running an endpoint method extractor in topological order. It then generates request and response specifications using two LLM agents with multi-stage generation, self-refinement, and reference reconstruction, merging them into a complete OAS. The evaluation covers 12 projects across 5 languages and 8 frameworks, reporting average F1 scores of 98.8% for endpoint methods, 97.2% for request parameters, 97.0% for responses, and 92.3% for parameter constraints, with modest token usage. The paper claims to be the first technology-agnostic LLM-based static OAS generation method.

Significance. If the reported results are reliable, OOPS is a substantial step forward for OAS generation: it removes the need for technology-specific rules and human expert intervention, addresses LLM context-length limitations with a coarse-grained API dependency graph, and mitigates both syntactic and semantic hallucinations through multi-stage generation and self-refine. The evaluation breadth, including proprietary projects, and the clearly described ablation study strengthen the empirical contribution. The cost and token-overhead analysis is also valuable for practitioners. However, several unresolved issues—most importantly the unconditional acyclic-dependency assumption, the absence of comparison with the two named LLM-based baselines, and the manually constructed ground truth—mean that the central claims are not yet fully supported as stated.

major comments (4)

[Section 3.2, Algorithm 1] The API dependency graph is defined as a directed acyclic graph (G=(V,E)) and Algorithm 1 (lines 40-45) unconditionally calls topologicalSort on every subgraph. The paper gives no argument for acyclicity, no cycle detection, and no fallback behavior. Real-world projects can have circular imports or mutually referencing route handlers; the selected projects show a maximum dependency depth of 3 (Table 7) and likely happen to be acyclic, so the experiments do not exercise cyclic cases. Since the technology-agnostic claim rests on handling arbitrary server codebases, this is a load-bearing gap: for cyclic dependency graphs the topological order is undefined or the extraction may fail. Please add cycle handling and evaluate on at least one cyclic project, or explicitly qualify the scope of the claim.
[Section 4.3, Table 4] The RQ1 comparison omits the two LLM-based baselines named in Section 2.2 (Chaplia and Klym 2024; LRASGen), citing unavailability of runnable code. Yet the introduction and Section 5.1 conclude that 'compared to state-of-the-art methods such as APICARV, ExpressO, Respector, and LRASGen, OOPS not only exhibits better generalization but also achieves higher precision and recall.' That superiority claim is unsupported by the experiments as they stand. Please either include a reproduction/adaptation, report published numbers on overlapping benchmarks, or narrow the claim to the non-LLM baselines actually compared.
[Section 4.5 and Section 5.1] All reported F1 scores come from a single run with temperature set to 0. LLM APIs are not guaranteed to be deterministic even at zero temperature, and the pipeline involves hundreds of LLM calls (Table 7), so compound stochasticity is possible. Without repeated runs and variance/confidence intervals, the tight averages in Table 4 lack statistical support. Please report multiple runs or an explicit determinism verification.
[Section 4.2] Ground truth is manually constructed by two experts, with no inter-rater agreement reported and no statement about the experts' independence from the authors. Since every precision/recall/F1 value in Tables 4-6 is computed against this ground truth, subjectivity or inadvertent bias is a central validity threat. Please describe the expert selection process, provide the annotation protocol, report disagreement counts, and release the ground-truth OAS for inspection.

minor comments (5)

[Algorithm 1, line 30] The condition 'if f < V' is incoherent as written because V is a set. It should probably be 'if f not in V' or similar. Please correct.
[Table 4] Cells with precision and recall both 0.0 report 'N/A' for F1. Please clarify how such undefined values are handled when computing averages.
[General] The title 'OpenAI for OpenAPI' is not used consistently with the method name OOPS; consider aligning the title with the paper's stated method or explain the relationship.
[References] The Chaplia and Klym citation appears in text as 'Chaplia and Klym, 2024' while the bibliography entry is under 'Chaplia, O., Klym, H.'; ensure citation-key consistency and check that all references in the bibliography are cited.
[Section 6.3] The external validity threat about proprietary frameworks and less common languages is honestly stated, but the abstract's 'technology-agnostic' claim should be tempered to reflect that only commonly used public technologies were tested.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: OOPS is evaluated against an independently constructed ground truth, and no predicted quantity is defined by the method's own inputs or by a self-citation chain.

full rationale

OOPS presents an LLM-agent pipeline whose outputs (OAS documents) are scored against a ground truth manually constructed by two experts from the same repositories. That evaluation protocol is independent of the method: no OOPS output, confidence score, or fitted parameter enters the ground-truth construction, and no equation in Algorithm 1 or Algorithm 2 defines a predicted quantity as a function of the target. The API dependency graph is built from LLM-reported file references and used only to order files for extraction; it does not encode the endpoint paths or HTTP methods being predicted, so the endpoint-method extraction result is not equivalent to the graph by construction. The reader's DAG/topological-sort concern is an algorithmic robustness limitation, not a circularity: it concerns a class of inputs the method does not currently handle, rather than a derivation that reduces to its own assumptions. No load-bearing self-citation appears; all baselines and prior-work references are external. The data-leakage concern for the 8 open-source projects is acknowledged and mitigated by including 4 proprietary projects; even if this weakened external validity, it would be training-data contamination, not circularity in the derivation sense. Therefore no circular step is exhibited.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

OOPS relies less on fitted numeric parameters than on LLM behavior and structural assumptions. The main entries are the LLM backbone choice, the acyclic dependency-graph assumption, the expert ground truth, the conversion tool's fidelity, and limited data leakage. None of these is an invented physical or conceptual entity.

free parameters (1)

Base LLM model = GPT-5 mini (main); 5 LLMs tested in RQ3
The reported performance depends heavily on the chosen LLM (Table 6, e.g., endpoint F1 84.2 for GPT-5 nano vs 99.3 for Gemini 3 Flash). The choice is made by the authors rather than derived, and the central generalizability claim is not model-independent.

assumptions (5)

domain assumption The API dependency graph is acyclic, so topologicalSort is always defined.
Algorithm 1 (lines 40-45) calls topologicalSort on subgraphs of the API dependency graph; cyclic file dependencies would break this. Section 6 on threats does not mention cycles.
domain assumption LLMs reliably perform structured API-entry detection, file-dependency analysis, and spec generation from the given prompts.
All extraction and generation steps depend on LLM tool-calling and structured-output behavior; there is no formal guarantee, and RQ3 shows strong variability across models.
domain assumption The expert-constructed ground-truth OAS is correct.
Section 4.2 describes two experts manually building OAS and reconciling, but no inter-rater reliability or third-party validation is reported.
domain assumption Swagger Codegen conversion from Swagger 2.0 to OpenAPI 3.0.4 preserves semantic content.
Section 3.3 generates Swagger 2.0 and converts via Swagger Codegen; conversion errors would affect the final OAS but are not analyzed.
domain assumption LLM training-data leakage does not materially inflate scores on the 8 open-source projects.
Section 6.1 addresses leakage by adding 4 proprietary projects and checking no identical output, but semantic memorization is not fully excluded.

how reviews work

0 comments

Cite this review

Pith. "Pith review of OOPS: Automated generation of REST API specification via LLMs." pith.science (2026). https://pith.science/paper/XHRTOMJ4

@misc{pith2026260112735,
  author       = {Pith},
  title        = {Pith review of: OOPS: Automated generation of REST API specification via LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XHRTOMJ4}},
  note         = {Machine review of arXiv:2601.12735}
}

read the original abstract

REST APIs, based on the REpresentational State Transfer (REST) architecture, are the primary type of Web API. The OpenAPI Specification (OAS) serves as the de facto standard for describing REST APIs and is crucial for multiple software engineering tasks. Automated OAS generation can help developers identify and correct issues in manually maintained OAS, but existing approaches rely on technology-specific rules and human expert intervention. LLMs' powerful code understanding capabilities offer the potential to overcome these limitations, but introduce additional challenges such as context length limitations and hallucinations. To address these challenges, we propose OOPS, the first technology-agnostic approach that leverages LLM-based static analysis of server code for OAS generation. Through an LLM agent workflow comprising two key steps, endpoint method extraction and OAS generation, OOPS eliminates the need for technology-specific rules or human expert intervention. By constructing an API dependency graph, it establishes necessary file associations to address LLMs' context length limitations. By multi-stage generation and self-refine, it mitigates both syntactic and semantic hallucinations during OAS generation. We evaluated OOPS on 12 real-world REST APIs spanning 5 programming languages and 8 development frameworks. Experimental results demonstrate that OOPS accurately generates high-quality OAS for REST APIs implemented with diverse technologies, achieving an average F1-score exceeding 98% for endpoint method inference, 97% for both request parameter and response inference, and 92% for parameter constraint inference. The input tokens average below 5.6K with a maximum of 16.13K, while the output tokens average below 0.9K with a maximum of 7.63K.

Figures

Figures reproduced from arXiv: 2601.12735 by the authors.

**Figure 1.** Example of a Swagger 2.0 specification. 2.2. Research motivation In recent years, OAS generation methods based on static analysis (Huang et al., 2024; Lercher et al., 2024) have gained widespread attention due to their ability to operate without requiring developers to modify source code or deploy REST APIs. However, these methods target specific programming languages or development frameworks, making them unable to… view at source ↗

**Figure 3.** Overview of the OOPS approach. technology-specific manual rules, while mitigating both syntactic and semantic hallucinations of LLMs through a multi-stage generation and self-refine strategy. 3. Approach 3.1. Overview Our method overview is illustrated in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

**Figure 4.** An example of an API dependency graph. As the last step in the endpoint method extraction workflow, OOPS integrates path fragments from different files based on the API dependency graph to obtain complete request paths and HTTP methods, thereby generating the endpoint method list (➍). As shown in lines 40-51 of Algorithm 1, OOPS traverses API entries with Tag LOCAL. For each entry, OOPS constructs a subgraph of the… view at source ↗

Figures from the paper (1 more)

**Figure 5.** Figure 5: Common violations of the OAS schema observed in LLM-generated [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 3 linked inside Pith

[2]

Generating openapi specifications from online api doc- umentation with large language models, in: Proceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (V olume 6: Industry Track), pp. 237–253. Lercher, A., Bauer, C., Macho, C., Pinzger, M., 2025. Au- toguard: Reporting breaking changes of rest apis from java spring boo...

arXiv 2025
[2024]

Fuzz4all: Universal fuzzing with large language mod- els, in: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1–13. 17 Yandrapally, R., Sinha, S., Tzoref-Brill, R., Mesbah, A., 2023. Carving ui tests to generate api tests and api specification, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (...

arXiv 2023
[2025]

arXiv preprint arXiv:2504.16833

Lrasgen: Llm-based restful api specification genera- tion. arXiv preprint arXiv:2504.16833 . Dong, Y ., Jiang, X., Jin, Z., Li, G., 2024. Self-collaboration code generation via chatgpt. ACM Transactions on Software Engineering and Methodology 33, 1–38. Du, W., Li, J., Wang, Y ., Chen, L., Zhao, R., Zhu, J., Han, Z., Wang, Y ., Xue, Z., 2024. Vulnerability...

arXiv 2024

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2]

Generating openapi specifications from online api doc- umentation with large language models, in: Proceedings of the 63rd Annual Meeting of the Association for Computa- tional Linguistics (V olume 6: Industry Track), pp. 237–253. Lercher, A., Bauer, C., Macho, C., Pinzger, M., 2025. Au- toguard: Reporting breaking changes of rest apis from java spring boo...

arXiv 2025

[2] [2024]

Fuzz4all: Universal fuzzing with large language mod- els, in: Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1–13. 17 Yandrapally, R., Sinha, S., Tzoref-Brill, R., Mesbah, A., 2023. Carving ui tests to generate api tests and api specification, in: 2023 IEEE/ACM 45th International Conference on Software Engineering (...

arXiv 2023

[3] [2025]

arXiv preprint arXiv:2504.16833

Lrasgen: Llm-based restful api specification genera- tion. arXiv preprint arXiv:2504.16833 . Dong, Y ., Jiang, X., Jin, Z., Li, G., 2024. Self-collaboration code generation via chatgpt. ACM Transactions on Software Engineering and Methodology 33, 1–38. Du, W., Li, J., Wang, Y ., Chen, L., Zhao, R., Zhu, J., Han, Z., Wang, Y ., Xue, Z., 2024. Vulnerability...

arXiv 2024