{"id":"05666991-ab94-4e72-89b9-d97af4f82078","arxiv_id":"2601.12735","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"OOPS uses LLM agents and an API dependency graph to generate OpenAPI specs from REST API server code across multiple languages and frameworks, with reported F1 above 92% on 12 APIs.","lead":"The paper presents OOPS, an LLM-agent workflow that reads server source code and automatically writes an OpenAPI specification for a REST API without per-framework rules. It reports F1 scores above 92% across 12 APIs in 5 languages, with most scores in the high 90s.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"API dependency graph is assumed acyclic without cycle handling; topological sort in Algorithm 1 may fail on real projects, undermining the technology-agnostic claim.","rationale":"The reader's weakest assumption is exactly the acyclicity of the API dependency graph. I agree that this is the most load-bearing concern because it attacks the algorithm's correctness on a class of real-world inputs, not merely the strength of the empirical evidence. The paper's own description in Section 3.2 defines the graph as a DAG and relies on topological sorting without any cycle handling, yet provides no evidence that cycles are absent in non-tested projects. The 12 evaluated projects are all acyclic in the relevant dependency structure, so the published F1 scores do not address this failure mode. Other concerns, such as the lack of comparison with LLM-based baselines and the absence of error bars, are important but less directly tied to whether the method can work at all. A conditional verdict is appropriate because the issue is addressable (e.g., by adding cycle detection, breaking cycles, or using a partial order), but the current manuscript does not address it. My analysis does not move the verdict; it reinforces the condition.","tokens_in":27417,"tokens_out":5701,"duration_ms":61101,"concrete_test":"Construct a minimal REST API with two route-handler files that mutually refer to each other's handlers (e.g., file A's router delegates to a handler in file B, and file B's router delegates to a handler in file A). Run OOPS on this project and observe whether Algorithm 1's topologicalSort completes or raises an error, and whether the resulting OAS paths are correct. If the sort crashes or the paths are incorrect, the acyclicity assumption is confirmed as a load-bearing limitation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that OOPS is technology-agnostic and generalizes across server codebases rests on the API dependency graph being a DAG (Section 3.2, definition of G=(V,E)). Algorithm 1 builds this graph by adding an edge from each referenced file to the referencing file (lines 26-38), then unconditionally calls topologicalSort on every subgraph (lines 43-44). Real-world projects can contain circular imports or mutually referencing route handlers; in such cases, the graph will contain a cycle and topologicalSort will either fail, throw an exception, or produce an undefined order. The paper never discusses this possibility, provides no cycle detection, and reports no fallback behavior. The 12 evaluated projects show a maximum dependency depth of 3 (Table 7) and likely happen to be acyclic, so the experiments do not exercise cyclic cases; therefore the empirical evidence does not support the claim that OOPS works 'without human expert intervention' on arbitrary technologies. If a cycle occurs, the endpoint method extractor may receive files in a nonsensical order or the process may terminate early, directly preventing OAS generation. This is a concrete algorithmic gap rather than a mere variability in LLM performance.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents OOPS, an LLM-agent workflow that generates OpenAPI Specifications (OAS) from REST API server source code through static analysis. The method first identifies the server-side technology, then extracts endpoint methods by detecting file-level API entries, constructing an API dependency graph for file association, and running an endpoint method extractor in topological order. It then generates request and response specifications using two LLM agents with multi-stage generation, self-refinement, and reference reconstruction, merging them into a complete OAS. The evaluation covers 12 projects across 5 languages and 8 frameworks, reporting average F1 scores of 98.8% for endpoint methods, 97.2% for request parameters, 97.0% for responses, and 92.3% for parameter constraints, with modest token usage. The paper claims to be the first technology-agnostic LLM-based static OAS generation method.","tokens_in":27678,"tokens_out":5218,"duration_ms":57368,"significance":"If the reported results are reliable, OOPS is a substantial step forward for OAS generation: it removes the need for technology-specific rules and human expert intervention, addresses LLM context-length limitations with a coarse-grained API dependency graph, and mitigates both syntactic and semantic hallucinations through multi-stage generation and self-refine. The evaluation breadth, including proprietary projects, and the clearly described ablation study strengthen the empirical contribution. The cost and token-overhead analysis is also valuable for practitioners. However, several unresolved issues—most importantly the unconditional acyclic-dependency assumption, the absence of comparison with the two named LLM-based baselines, and the manually constructed ground truth—mean that the central claims are not yet fully supported as stated.","major_comments":[{"comment":"The API dependency graph is defined as a directed acyclic graph (G=(V,E)) and Algorithm 1 (lines 40-45) unconditionally calls topologicalSort on every subgraph. The paper gives no argument for acyclicity, no cycle detection, and no fallback behavior. Real-world projects can have circular imports or mutually referencing route handlers; the selected projects show a maximum dependency depth of 3 (Table 7) and likely happen to be acyclic, so the experiments do not exercise cyclic cases. Since the technology-agnostic claim rests on handling arbitrary server codebases, this is a load-bearing gap: for cyclic dependency graphs the topological order is undefined or the extraction may fail. Please add cycle handling and evaluate on at least one cyclic project, or explicitly qualify the scope of the claim.","section":"Section 3.2, Algorithm 1"},{"comment":"The RQ1 comparison omits the two LLM-based baselines named in Section 2.2 (Chaplia and Klym 2024; LRASGen), citing unavailability of runnable code. Yet the introduction and Section 5.1 conclude that 'compared to state-of-the-art methods such as APICARV, ExpressO, Respector, and LRASGen, OOPS not only exhibits better generalization but also achieves higher precision and recall.' That superiority claim is unsupported by the experiments as they stand. Please either include a reproduction/adaptation, report published numbers on overlapping benchmarks, or narrow the claim to the non-LLM baselines actually compared.","section":"Section 4.3, Table 4"},{"comment":"All reported F1 scores come from a single run with temperature set to 0. LLM APIs are not guaranteed to be deterministic even at zero temperature, and the pipeline involves hundreds of LLM calls (Table 7), so compound stochasticity is possible. Without repeated runs and variance/confidence intervals, the tight averages in Table 4 lack statistical support. Please report multiple runs or an explicit determinism verification.","section":"Section 4.5 and Section 5.1"},{"comment":"Ground truth is manually constructed by two experts, with no inter-rater agreement reported and no statement about the experts' independence from the authors. Since every precision/recall/F1 value in Tables 4-6 is computed against this ground truth, subjectivity or inadvertent bias is a central validity threat. Please describe the expert selection process, provide the annotation protocol, report disagreement counts, and release the ground-truth OAS for inspection.","section":"Section 4.2"}],"minor_comments":[{"comment":"The condition 'if f < V' is incoherent as written because V is a set. It should probably be 'if f not in V' or similar. Please correct.","section":"Algorithm 1, line 30"},{"comment":"Cells with precision and recall both 0.0 report 'N/A' for F1. Please clarify how such undefined values are handled when computing averages.","section":"Table 4"},{"comment":"The title 'OpenAI for OpenAPI' is not used consistently with the method name OOPS; consider aligning the title with the paper's stated method or explain the relationship.","section":"General"},{"comment":"The Chaplia and Klym citation appears in text as 'Chaplia and Klym, 2024' while the bibliography entry is under 'Chaplia, O., Klym, H.'; ensure citation-key consistency and check that all references in the bibliography are cited.","section":"References"},{"comment":"The external validity threat about proprietary frameworks and less common languages is honestly stated, but the abstract's 'technology-agnostic' claim should be tempered to reflect that only commonly used public technologies were tested.","section":"Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The 'first technology-agnostic' claim should be weighed carefully against LRASGen's stated scope, which the authors themselves describe as supporting multiple languages via regex-based file location. The missing LLM-baseline comparison is not merely a statistical issue; it affects the paper's positioning and novelty claim. If the editors prioritize novelty, this needs to be addressed directly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution to LLM-based OAS generation, and the evaluation is more careful than most in this subfield, but the headline \"technology-agnostic\" claim is not fully established.\n\nThe core idea is genuinely new relative to Chaplia and LRASGen: instead of JavaScript-specific dependency libraries or human-crafted regexes, OOPS has an LLM agent build an API dependency graph to associate files. That is a legitimate design step, and the paper supports it with an original 12-project, 5-language, 8-framework evaluation.\n\nWhat the paper does well: the multi-stage generation with self-refine is sensible; the ablation shows both main stages contribute; RQ3 across five base LLMs is useful; and the context/cost numbers are concrete and cheap ($0.41 average per project). The authors are also honest in Section 6 about data leakage and external validity. That is not common enough and should be credited.\n\nNow the soft spots, in proportion. I checked the stress-test concern about cycles and it holds. The API dependency graph is defined as a DAG, and Algorithm 1 unconditionally calls topologicalSort on the subgraph. There is no cycle detection, no fallback, and no discussion. Circular imports or mutually referencing route handlers would break the ordering. The 12 evaluated projects have max dependency depth 3 and appear to be acyclic, so the evaluation does not exercise the failure case. I would call this a moderate gap rather than fatal—many real projects may be acyclic in the relevant sense—but the paper should either handle cycles or explicitly scope the claim.\n\nAlso: the two LLM-based baselines named in the introduction, Chaplia and LRASGen, are not compared against because no runnable implementations are available. That weakens the \"better than state-of-the-art\" relative claim, which is only supported against non-LLM baselines. The aggregate baseline F1 numbers in Table 4 also mix different project subsets, so the headline gaps are not apples-to-apples. No error bars or repeated runs; temperature 0 helps but does not make LLM outputs deterministic across API versions and platforms. The manually constructed ground truth is normal for this area, but it is still a threat.\n\nWho is this for: researchers working on API specification generation, LLM agents for repository-level analysis, and empirical software engineering with LLMs. It deserves a serious referee. I would send it out, with an expectation of major revision around cycle handling, baseline comparison, and statistical rigor.","headline":"OOPS is a genuine new LLM-agent approach to OpenAPI generation with a serious multi-language evaluation, but the unconditional acyclic-dependency assumption and the missing LLM-baseline comparisons keep the technology-agnostic claim conditional.","tokens_in":28156,"tokens_out":2820,"would_cite":true,"duration_ms":35436,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OOPS claims to be the first technology-agnostic LLM-based static analysis method that generates OpenAPI specifications directly from server source code, reporting average F1-scores above 92% across 12 real-world REST APIs in five languages","keywords":["OpenAPI Specification","REST API","LLM agent workflow","API dependency graph","static analysis","hallucination mitigation","endpoint extraction","specification generation"],"falsifier":"Run OOPS on a small but real REST API whose route files import each other in a cycle, or add a circular import to one of the 12 projects; if the topological sort fails, produces incomplete paths, or the workflow does not terminate, the technology-agnostic claim for such projects is false.","tokens_in":27317,"feed_emoji":"🤖","tokens_out":4471,"duration_ms":51474,"temperature":0.7,"pith_summary":"The paper tries to show that OpenAPI specifications can be produced automatically from server source code by a large-language-model agent workflow, without per-framework rules or expert annotations. The method builds a dependency graph of files involved in routing, which lets the LLM handle large projects despite context limits, and it splits generation into request and response stages followed by a self-refine loop to suppress hallucinations. On 12 REST APIs in five programming languages and eight frameworks, it reports average F1-scores of 98.8% for endpoint inference, 97.2% for request parameters, 97.0% for responses, and 92.3% for parameter constraints, at token costs far below context limits. This matters because accurate OpenAPI descriptions are the foundation for API documentation, automated testing, vulnerability detection, and LLM tool integration.","feed_headline":"OOPS turns server code into API specs at 97% F1","feed_subtitle":"A dependency graph lets one LLM workflow span five languages and eight frameworks without per-project rules.","key_machinery":"The central object is the API dependency graph, a directed acyclic graph whose nodes are files and whose edges say that an API entry in one file references a handler implemented in another file. The graph lets the workflow perform topological sorting over only the relevant files, so the LLM never needs the whole repository in its context; it is paired with multi-stage request/response generation and a self-refine loop that detects and fixes syntactic and semantic hallucination errors.","core_discovery":"The paper's central claim is that endpoint extraction can be decomposed into a two-level mechanism: first detect API entries as (path, handler, tag) triples, where the tag marks whether the handler is defined locally or imported, and then build a directed acyclic graph over files whose edges indicate that one file's route prefix references a handler in another file. Feeding each local entry's descendant subgraph to the LLM in topological order gives the model exactly the files needed to reconstruct the full request path and HTTP methods. The same file associations are then used by separate LLM agents to generate request and response schemas, with a syntax-fixing self-refine loop that repairs","pith_inferences":["A natural extension the paper leaves implicit is drift detection: regenerate the OpenAPI spec from current source and diff it against the maintained spec to flag undocumented endpoints or outdated schemas.","The dependency-graph mechanism may transfer to other interface styles—GraphQL schemas, RPC definitions, or message contracts—since it only relies on representing routes and handlers as cross-file references.","The acyclicity assumption marks a likely boundary: projects with circular imports would need cycle handling (for example, collapsing strongly connected components) before the technology-agnostic claim can cover them; the paper does not report on that case.","The hallucination-mitigation claim can be tested directly by injecting known reference and syntax errors into LLM outputs and measuring whether the self-refine loop converges rather than oscillates; the paper reports success but not failure-mode counts."],"forward_implications":["If the reported results hold, a machine-readable OpenAPI contract can be derived from a server codebase alone, eliminating the need for annotations, expert rule-writing, or a running deployment.","Because context usage stays low (average input under 5.6K tokens, maximum 16.13K), the approach scales to repositories whose full contents would not fit in an LLM window.","The generated OpenAPI can feed the standard downstream pipeline—documentation, automated testing, and vulnerability scanning—directly from source rather than from a manually maintained spec.","The reported results on private, non-public projects indicate the method does not depend on the LLM having memorised the codebase, addressing a common data-leakage concern.","The self-refine loop and multi-stage generation are credited with suppressing both invalid JSON syntax and mixed-version OpenAPI structures, so the quality should transfer to any OpenAPI-consuming toolchain."],"fun_headline_variants":["LLM agent auto-writes OpenAPI specs from server code","OOPS: 97% accurate API specs without per-project rules","One LLM workflow specs APIs across 5 languages, 8 frameworks","Server code to OpenAPI: OOPS hits 97% F1 automatically","Dependency graph lets LLM spec APIs with no tech-specific rules"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method assumes the API dependency graph is acyclic, because endpoint extraction topologically sorts the graph, but real code can contain circular imports and mutually referencing files, and the paper does not say what happens then.","fun_headline_variants_meta":{"raw":{"variants":["LLM agent auto-writes OpenAPI specs from server code","OOPS: 97% accurate API specs without per-project rules","One LLM workflow specs APIs across 5 languages, 8 frameworks","Server code to OpenAPI: OOPS hits 97% F1 automatically","Dependency graph lets LLM spec APIs with no tech-specific rules"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1366,"prompt_tokens":831,"completion_tokens":535,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":441}},"tokens_in":575,"tokens_out":535,"duration_ms":5284,"temperature":1.0,"reasoning_tokens":441,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T09:41:01.651647+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run OOPS on a small but real REST API whose route files import each other in a cycle, or add a circular import to one of the 12 projects; if the topological sort fails, produces incomplete paths, or the workflow does not terminate, the technology-agnostic claim for such projects is false.","supporting_citations":[],"review_version":1}