{"id":"c086ad7d-70e2-45df-8c0b-a29640cd2aa8","arxiv_id":"2509.05540","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"RestTSLLM combines Test Specification Language with few-shot prompting to generate REST API integration tests, and a comparative study finds Claude 3.5 Sonnet the best among eight LLMs.","lead":"This paper presents RestTSLLM, a pipeline that uses a formal test specification language and large language models to generate REST API integration tests from OpenAPI definitions. It compares eight LLMs on six .NET projects and reports that Claude 3.5 Sonnet produced the highest-quality tests across success rate, coverage, and mutation score.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper never isolates TSL: every condition runs the full RestTSLLM pipeline, so the central design claim that the TSL intermediate representation enables effective REST API test generation is not actually tested.","rationale":"I considered several candidate concerns. The mislabeled TOPSIS formula in Section 4.3 is real but not decisive for the ranking claim: Claude 3.5 Sonnet ranks first on every individual metric in Table 3, so any monotone aggregation would still place it first. Project representativeness, discussed in Section 4.2 and Section 6, is acknowledged and limits generalization, but the central claim is not explicitly about enterprise-scale APIs. The absence of repeated runs is a genuine threat to the precise ranking, but Claude's dominance across all metrics and the low overall failure rate make the qualitative feasibility claim relatively robust. The most load-bearing gap is the missing TSL ablation. The paper's novelty and title are precisely about combining TSL and LLM, and Section 3 makes a causal design claim that the intermediate TSL step helps the LLM perform more efficiently, yet the evaluation never varies that step. The observed results could be entirely due to few-shot prompting plus strong LLM code generation. This does not invalidate the reported data, but it does mean the central contribution is underdetermined by the experiment. Because the reader already returned CONDITIONAL, my finding does not change the verdict, but it sharpens the condition: before accepting the TSL claim, the authors should run the direct-prompting control.","tokens_in":17763,"tokens_out":6975,"duration_ms":63373,"concrete_test":"Run a controlled ablation on the same six projects with the same eight LLMs: (A) full RestTSLLM (OpenAPI -> TSL -> xUnit), (B) direct few-shot prompting from OpenAPI to xUnit with the same system prompt and examples, and optionally (C) a fixed, non-LLM-generated TSL produced by a deterministic category-partition converter. Compare success rate, branch coverage, and mutation score over at least five repeated runs per cell, with temperature and seeds held constant. If B or C is statistically indistinguishable from A, the TSL stage is not the active ingredient and the central claim needs to be weakened to: the prompt engineering, not TSL, drives performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty of RestTSLLM is the TSL intermediate step. Section 3 argues that separating scenario definition (Prompt 1) from implementation (Prompt 2) improves the LLM's effectiveness via decomposed prompting. But the experiment in Section 4 contains no condition without TSL. Every LLM is evaluated only on the full OpenAPI -> TSL -> xUnit pipeline, so the reported success rate, coverage, and mutation score reflect the joint effect of (i) the intermediate language, (ii) few-shot/decomposed prompting, and (iii) the underlying LLM code-generation ability. A direct OpenAPI -> xUnit condition, or a condition with a different intermediate representation, would be needed to attribute the results to TSL. Without such a control, the claim that TSL is what enables contextually coherent test generation is a design hypothesis, not an empirical finding. The limitations acknowledged in Section 6, such as project context, prompt dependency, and randomness, do not address this missing control. Since the paper's title and contribution center on combining TSL with LLMs, this gap is load-bearing: the approach might work equally well without TSL, and the current data cannot decide.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RestTSLLM, an approach that uses Test Specification Language (TSL) as an intermediate representation together with decomposed few-shot prompting to generate executable xUnit integration tests for REST APIs from OpenAPI specifications. The study evaluates eight LLMs on six small open-source .NET projects, measuring success rate, branch coverage, mutation score, and a \"calculated score\" that the authors label as TOPSIS. The authors report that Claude 3.5 Sonnet performs best across all metrics, followed by Deepseek R1, Qwen 2.5 32b, and Sabiá 3, and conclude that general-purpose LLMs combined with TSL can generate contextually coherent REST API integration tests.","tokens_in":17956,"tokens_out":3268,"duration_ms":28102,"significance":"The paper provides a reproducible empirical comparison of eight LLMs for REST API integration test generation, with public artifacts, an automated multi-model execution script, and a clear failure categorization. If the TSL-specific claim were supported, the decomposed-prompting design would be a useful methodological contribution to LLM-based test generation. The study also offers useful evidence on the feasibility of black-box test generation from OpenAPI specifications. However, the central attribution of the results to TSL is not empirically tested, and several methodological choices (single runs, no statistical significance testing, no external baselines, and a mislabeled aggregation method) substantially reduce the confidence in the model ranking and in the generalizability claims.","major_comments":[{"comment":"The paper never empirically isolates the TSL component. Every experimental condition runs the full OpenAPI-to-TSL-to-xUnit pipeline, so the reported success rate, coverage, and mutation score reflect the joint effect of the intermediate language, the decomposed/few-shot prompting, and the LLM's code-generation ability. A direct OpenAPI-to-xUnit condition, or a condition with an alternative intermediate representation, is required to support the claim that TSL 'improves the LLM's effectiveness' (Section 3) and the title's emphasis on combining TSL with LLMs. As presented, the role of TSL is a design hypothesis rather than an empirical finding.","section":"Section 3; Section 4; Table 3"},{"comment":"Each model/project combination appears to have been executed only once, with temperature set to 1, as acknowledged in Section 6 under 'Result Randomness', and no significance tests or confidence intervals are reported. The differences among models in Table 3 are small (overall scores from 62.2% to 70.9%, with several adjacent models within 1-2 points), so the ranking of models, including the claim that Claude 3.5 Sonnet 'outperformed all other models across every metric', may be within run-to-run noise. Repeated runs, variance reporting, or statistical testing are needed before RQ2 can be answered as stated.","section":"Section 4.6; Section 5.2; Table 3"},{"comment":"The formula labeled TOPSIS is, by the authors' own description, a weighted arithmetic mean of the three metrics with equal weights w=33.33%. TOPSIS requires computing distances to the positive-ideal and negative-ideal solutions after normalization; the presented formula is not TOPSIS. This mislabel matters because the 'calculated score' is the basis for the final model ranking in Table 3. The authors should either implement actual TOPSIS (including normalization and ideal/anti-ideal distance computation) or rename the aggregation as a weighted average.","section":"Section 4.3, Eq. (S)"},{"comment":"The evaluation includes no baseline comparison against established REST API testing tools such as RESTler, RESTTestGen, bBOXRT, or RESTest, which are cited in the related work. Without such a baseline, the significance of the reported success, coverage, and mutation values cannot be assessed, and the suggestion that the approach compares favorably with 'traditional black-box tools' (Section 2) is not substantiated by the experiments. The paper should either add such a comparison or explicitly restrict its claims to a within-LLM comparative study.","section":"Section 2; Section 5"}],"minor_comments":[{"comment":"The word 'denotated' should be 'denoted'.","section":"Section 4.3"},{"comment":"The spelling of the model name is inconsistent between 'Sabiá' and the unaccented 'Sabia'; please standardize.","section":"Abstract and Table 1"},{"comment":"The table uses decimal commas (e.g., '70,9%') while the text uses decimal points; standardize the decimal format throughout.","section":"Table 3"},{"comment":"The column 'T' reports average numbers of tests with fractional values (e.g., 38.3). Clarify that this is an average across projects and, if possible, also report integer total counts per model.","section":"Table 3"},{"comment":"The statement that tests were 'manually copied' into duplicated projects should be elaborated, since this manual step is a potential source of experimental error and could affect reproducibility.","section":"Section 4.6"},{"comment":"The listings are not explicitly referenced in the running text; add references such as 'as shown in Listing 1'.","section":"Section 3, Listings 1 and 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the venue, and the artifact availability is a strength. However, the central claim that TSL enables the observed performance is untested by the current design, and the statistical and baselining issues would need to be addressed in a revision. The TOPSIS mislabel should also be corrected because it affects the methodological soundness of the ranking. I do not see evidence of circularity or fabrication; the concerns are about experimental design and interpretation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on RestTSLLM. The paper builds something genuinely useful: a reproducible pipeline that takes an OpenAPI spec, has an LLM write test scenarios in TSL, then converts those to xUnit integration tests, and it compares eight LLMs on six small .NET projects with success rate, branch coverage, and mutation score. The artifacts are complete, costs are reported, and the failure analysis is honest. For a practitioner deciding which model to try, the comparison table is worth having.\n\nBut the central design claim is not actually tested. The paper argues that the TSL intermediate representation improves LLM effectiveness by separating scenario reasoning from implementation, yet every condition runs the full OpenAPI -> TSL -> xUnit pipeline. There is no condition that skips TSL and asks the LLM to generate xUnit directly, and no condition with a different intermediate representation. So the reported metrics reflect the joint effect of the intermediate language, the decomposed prompting, and the base model's coding ability. The data cannot support the statement that TSL is what enables contextually coherent tests. That is a hypothesis, not a finding. This is exactly the kind of ablation a good reviewer would require.\n\nOther soft spots are smaller. There is no comparison against non-LLM REST testing tools like RESTler, so the paper doesn't show the approach beats existing automated generators, only that LLMs can produce mostly passing tests. The experiments are a single run per model and project; with temperature 1, the rankings could be noise. The 'TOPSIS' label is wrong - the formula given is a simple weighted arithmetic mean with equal weights. And the six projects are small and simple; the authors themselves flag that.\n\nNone of that kills the paper, because the comparative evaluation of LLMs on this task is a legitimate contribution on its own. The qualitative failure analysis and cost data are useful. But if I were refereeing, I would ask for an ablation, at least one direct OpenAPI-to-xUnit condition, and ideally a baseline from a traditional tool, plus repeated runs. Without the ablation, the specific contribution of TSL is unsupported.\n\nThis is a paper I'd want a serious editor to referee, but with a clear request for major revision. I wouldn't cite the TSL claim as established; I would cite the empirical comparison and the pipeline as a useful data point.","headline":"The paper is a useful, reproducible comparison of eight LLMs on REST API test generation, but it never tests whether the TSL intermediate step is actually the thing that helps.","tokens_in":18470,"tokens_out":2822,"would_cite":true,"duration_ms":24152,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a pipeline combining Test Specification Language (TSL) with decomposed few-shot prompting lets general-purpose LLMs generate executable REST API integration tests from OpenAPI specifications, and that Claude 3.5…","keywords":["REST API testing","integration testing","test generation","Large Language Models","Test Specification Language (TSL)","OpenAPI","prompt engineering","mutation testing"],"falsifier":"Repeat the exact RestTSLLM pipeline on a held-out set of large industrial REST APIs—with hundreds of endpoints, asynchronous flows, and strict authentication—using the same eight models and prompts. If Claude 3.5 Sonnet no longer ranks first, or if the best model's calculated score drops far below the roughly 71% seen here because generated tests frequently fail or fail to compile, then the paper's central generalization is falsified.","tokens_in":17565,"feed_emoji":"🧪","tokens_out":7866,"duration_ms":66718,"temperature":0.7,"pith_summary":"The paper introduces RestTSLLM, a prompt-driven pipeline that turns REST API documentation into executable integration tests without fine-tuning any model. Its key move is to split the work into two steps with Test Specification Language (TSL) in between: first the LLM converts the OpenAPI specification into declarative test scenarios, then it converts those scenarios into xUnit (.NET) test code. The authors run this pipeline on six open-source .NET REST APIs with eight general-purpose LLMs and score the results by success rate, branch coverage, and mutation score. They report that every model produced mostly executable tests, with Claude 3.5 Sonnet ranking first on all metrics and being the only model with zero failed tests. The point of the study is to show that scenario design and input generation—historically the hard parts of integration testing—can be automated cheaply through prompting alone.","feed_headline":"Claude 3.5 Sonnet beats 7 LLMs at writing REST API tests","feed_subtitle":"A TSL intermediate step let all eight models generate executable integration tests; Sonnet led in every metric.","key_machinery":"The central machinery is a two-stage decomposed prompting chain with Test Specification Language (TSL) as the bridge. TSL is a high-level, declarative YAML-style format for test cases: each case declares an id, group, name, endpoint, method, preconditions, request body, and expected response. The pipeline uses four prompts: a behavior prompt that sets the model as an experienced tester; an example showing OpenAPI converted to TSL; an example showing TSL converted to xUnit code; and action prompts that run the same two conversions on the target specification. Because TSL expresses scenarios without code, the first conversion isolates business-rule understanding while the second isolates syntax and framework concerns—the decomposition is what lets general-purpose models produce coherent output. When models truncate long responses, a loop segments generation by OpenAPI tags carried into the TSL, preserving completeness of the test suite.","core_discovery":"RestTSLLM's central claim is that a general-purpose LLM, given a well-structured prompt plus a few worked examples, can perform the whole REST API integration-testing chain: reading OpenAPI, deciding what scenarios matter (happy paths, validation failures, edge cases), generating valid input data, and emitting compilable xUnit tests. The intermediate TSL representation is not decorative: it forces the model to settle business logic before syntax, and the examples teach the model to preserve TSL-defined inputs while making small adjustments, such as generating a unique email dynamically to keep tests repeatable. In the evaluation on six small open-source .NET projects, all eight models achieved success rates above 95%, and the calculated score—an equal-weight TOPSIS combination of success rate, branch coverage, and mutation score—placed Claude 3.5 Sonnet first at 70.9%, followed by Deepseek R1, Qwen 2.5 32b, and Sabiá 3, with all models within 8.7 percentage points. The paper also reports that only 2.38% of the 1,635 generated tests failed, with most failures traced to property-length boundary errors and misreadings of the specification.","pith_inferences":["The paper's failure taxonomy suggests a direct ablation: add explicit few-shot examples for property-length boundaries, required versus optional fields, and authentication errors, and the 39 failed tests should shrink; this is an extension because the paper does not run that ablation.","Because results were captured on a single date with temperature set to 1, the leaderboard is version-sensitive; a fair inference is that the durable contribution is the pipeline itself, not the specific model version that won.","The gap between near-universal success rates and much lower mutation scores hints that specification-driven tests mostly check the documented contract rather than implementation edges, so adding implementation-aware oracles may improve fault detection more than generating additional test cases.","The prompts were primarily written in Portuguese while the OpenAPI specifications stayed in English, so an obvious testable extension is whether the TSL-bridge benefit persists for other languages and for specifications with heavier natural-language business rules."],"forward_implications":["If the central claim holds, teams can generate a first-pass integration test suite from an OpenAPI file with no model fine-tuning and no per-API training data.","The TSL-bridge result implies test scenario design can be decoupled from test code generation, so teams could swap xUnit for another framework by changing only the second prompt.","Because costs per project stayed below $0.80 even for the most expensive model, per-regression automation with LLMs is economically plausible.","The six recurring failure categories give a concrete repair agenda: boundary-value property lengths, authentication handling, required-field semantics, and response deserialization, which together account for most of the 39 failed tests.","The comparison suggests all eight evaluated models are viable for this task, with Claude 3.5 Sonnet the current best option according to the paper's calculated score."],"supporting_citations":[{"why":"Supplies the category-partition method that defines how TSL scenarios and input-data categories are structured.","marker":"[39]"},{"why":"Provides decomposed prompting, the technique that splits OpenAPI-to-TSL from TSL-to-code.","marker":"[24]"},{"why":"Is the prior LLM approach that enriches OpenAPI specifications, which RestTSLLM contrasts with by generating executable tests directly.","marker":"[26]"},{"why":"Is the closest baseline using small fine-tuned models, highlighting RestTSLLM's choice of general-purpose models with no fine-tuning.","marker":"[25]"},{"why":"Documents black-box REST API testing tools and coverage challenges that motivate the need for LLM-based scenario generation.","marker":"[17]"},{"why":"Surveys LLM-based testing and identifies integration testing as an underexplored gap this study targets.","marker":"[53]"},{"why":"Defines TOPSIS, the multi-criteria method used to combine success rate, coverage, and mutation score into one calculated score.","marker":"[30]"},{"why":"Provides Stryker.NET, the mutation-testing tool that produces the mutation score metric.","marker":"[47]"},{"why":"Motivates combining coverage with mutation score because the gap between them can identify weak tests.","marker":"[22]"},{"why":"Defines the xUnit/.NET integration-test target technology shown in the example prompts.","marker":"[32]"}],"fun_headline_variants":["Sonnet tops 8 LLMs for REST API test generation","TSL + LLM pipeline: Sonnet leads in API test generation","RestTSLLM: Sonnet outperforms in automated API tests","Best LLM for REST API tests? Claude 3.5 Sonnet wins","Automated REST testing: TSL and LLMs, Sonnet on top"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results rest on the assumption that six small, simple, open-source .NET example projects fairly represent the REST APIs that teams actually test in industry, so the same success rates and model ranking would hold for large enterprise systems.","fun_headline_variants_meta":{"raw":{"variants":["Sonnet tops 8 LLMs for REST API test generation","TSL + LLM pipeline: Sonnet leads in API test generation","RestTSLLM: Sonnet outperforms in automated API tests","Best LLM for REST API tests? Claude 3.5 Sonnet wins","Automated REST testing: TSL and LLMs, Sonnet on top"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000761,"raw_usage":{"total_tokens":3441,"prompt_tokens":1072,"completion_tokens":2369,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":2273}},"tokens_in":688,"tokens_out":2369,"duration_ms":13461,"temperature":1.0,"reasoning_tokens":2273,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:22:53.048578+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the exact RestTSLLM pipeline on a held-out set of large industrial REST APIs—with hundreds of endpoints, asynchronous flows, and strict authentication—using the same eight models and prompts. If Claude 3.5 Sonnet no longer ranks first, or if the best model's calculated score drops far below the roughly 71% seen here because generated tests frequently fail or fail to compile, then the paper's central generalization is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents black-box REST API testing tools and coverage challenges that motivate the need for LLM-based scenario generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines TOPSIS, the multi-criteria method used to combine success rate, coverage, and mutation score into one calculated score."},{"cited_title":"2025.Stryker .NET - Configuration","cited_arxiv_id":null,"evidence_quote":"Provides Stryker.NET, the mutation-testing tool that produces the mutation score metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates combining coverage with mutation score because the gap between them can identify weak tests."},{"cited_title":"2025.Integration tests in ASP.NET Core","cited_arxiv_id":null,"evidence_quote":"Defines the xUnit/.NET integration-test target technology shown in the example prompts."}],"review_version":2}