Pith. sign in

REVIEW 4 major objections 4 minor 59 references

SAINT: Service-level Integration Test Generation with Program Analysis and LLM-based Agents

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Combining static analysis with LLM-based agents yields service-level tests that match or beat a leading white-box REST API tester on code coverage while requiring no OpenAPI specification.

desk verdict SAINT is a well-engineered LLM+static-analysis test generation paper whose comparative claims rest on a single-seed EvoMaster baseline; the core idea is promising but the empirical edge is not yet proven. read the letter →

arxiv 2511.13305 v2 pith:S3PLKSO2 submitted 2025-11-17 cs.SE

Rangeet Pan , Raju Pavuluri , Ruikai Huang , Rahul Krishna , Tyler Stennett , Alessandro Orso , Saurabh SInha This is my paper · ORCID
classification cs.SE
keywords service-leveltestingRESTAPILLM-basedagentsstaticanalysisendpointmodeloperationdependencygraphscenario-basedtestscodecoverage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SAINT is a new white-box approach to service-level testing of enterprise Java applications that builds a structured endpoint model and an operation dependency graph from source code, then uses LLM-based agents to generate two kinds of tests: endpoint-focused tests aimed at code and database coverage, and scenario-based tests that exercise coherent multi-endpoint use cases. The paper's central claim is that this combination of static analysis with agentic LLM workflows produces tests that are more effective and more aligned with what developers actually want than existing fuzzing-based or specification-driven tools. Across eight Java applications, SAINT matched or outperformed a leading white-box REST API test generator on code coverage (differences from -0.9% to +50.5%), while generating scenario-based tests that more than 90% of surveyed developers said they would use. An ablation study attributes the largest coverage gains to the operation dependency graph and inter-parameter dependency extraction, with the repair and coverage-augmentation agents contributing less, and small 8-billion-parameter models often matching much larger models.

What carries the argument

The central objects are the endpoint model and the operation dependency graph (ODG). The endpoint model is a structured summary of each service endpoint: its class, method signature, path, HTTP method, parameters with kinds and value constraints, inter-parameter dependency relations (such as Requires, AllOrNone, OnlyOne), database operations reachable from the endpoint, and response schema. The ODG is a directed graph whose nodes are endpoints carrying functional summaries and whose edges encode three ordering constraints: resource dependency, producer-consumer dependency, and database dependency. Together these structures convert raw application code into a compact semantic map that lets LL

What would settle it

Run the same baseline tool on the four applications that have API specifications using ten or more different random seeds under the same one-hour budget, and compare SAINT's coverage to the full distribution; if the baseline's median or best run closes the reported coverage gaps, the central effectiveness claim does not generalize.

Watch

Extended reading notes

Core claim

The central claim is that combining static program analysis with agentic LLM workflows produces service-level tests that are both more effective and more developer-aligned than existing fuzzing-based or spec-driven tools. The paper demonstrates this with a system that constructs an endpoint model—an 8-tuple capturing service class, method signature, path, HTTP method, parameter list, inter-parameter dependencies, reachable database operations, and response schema—plus an operation dependency graph encoding resource, producer-consumer, and database ordering constraints. LLM-based agents then plan, act, and reflect to generate endpoint-focused tests that maximize code and database coverage, an

Load-bearing premise

The comparison with the baseline white-box tester used a single one-hour run with one randomly chosen seed per application, so the reported coverage advantage assumes that one run represents the baseline's typical stochastic behavior.

Editorial extensions

If this is right

  • For applications without OpenAPI specifications—a common situation in enterprise codebases—SAINT can still produce compilable, executable tests, removing a key blocker for automated service-level testing.
  • Endpoint-focused tests reach code coverage comparable to or better than a leading white-box REST API tester, with differences ranging from -0.9% to +50.5%, while issuing far fewer HTTP requests per application.
  • Scenario-based tests surface multi-endpoint business workflows: 42.6% of generated scenarios span multiple endpoint classes, and 66% of surveyed developers would add them to a regression suite with little or no change.
  • Smaller 8-billion-parameter models match or beat much larger models on coverage, suggesting the static-analysis scaffolding—not raw model size—drives effectiveness.
  • The operation dependency graph's partial ordering is the largest single contributor to coverage in the ablation, ahead of inter-parameter dependency extraction, value constraints, the repair agent, and the coverage-augmentation agent.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the baseline comparison was run only once per application with one random seed, the reported coverage advantages could shift under re-runs; a multi-seed evaluation would show whether the +50.5% and +22.0% gaps are stable.
  • Because the endpoint model's fields (inter-parameter dependencies, value constraints, database operations) are language-independent in structure, porting the static analysis to Python or Node.js services could extend the same agentic loop beyond Java with minimal changes.
  • The scenario generator appears to find functional relationships beyond structural dependencies, as in the cancer-hotspot example; this makes the approach a plausible tool for discovering undocumented business flows, not just for generating tests.
  • A natural next experiment is manual inspection of the 5xx-triggering requests to confirm they correspond to real faults rather than merely server error pages; the current metric counts server failures, not verified bugs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces SAINT, a white-box service-level testing approach for Java REST applications that combines static analysis (via CLDK) with LLM-based agentic workflows. SAINT constructs an endpoint model and an operation dependency graph, then generates endpoint-focused tests (supported by repair and coverage-augmentation agents) and scenario-based tests (supported by a plan-act-reflect test-generation agent). The evaluation covers eight Java applications, four with OpenAPI specifications and four without; for those with specifications, SAINT is compared against EvoMaster in terms of code coverage and fault triggering. Additional evidence includes a developer survey on scenario-based tests and an ablation study of SAINT's components. The central claim is that combining static analysis with agentic LLM workflows enables more effective, functional, and developer-aligned service-level test generation without requiring OpenAPI specifications.

Significance. If the empirical claims hold, SAINT would be a meaningful advance: it is the first white-box LLM-agentic framework for REST API testing that does not depend on OpenAPI specifications, and it demonstrably handles legacy Java frameworks and a proprietary application. The paper includes a promised artifact, an ablation study quantifying the contributions of ODG, IPD extraction, value constraints, and agents, and repeated runs for one model (Devstral) to bound stochastic variability. The scenario-based test direction and developer survey address an underexplored aspect of API testing. However, the headline comparison against EvoMaster currently rests on a single EvoMaster run per application, and the fault-detection methodology may overcount failures. These issues make the results conditional rather than conclusive.

major comments (4)
  1. [§4.1 (Experiment Setup), RQ1/Finding 2] The EvoMaster comparison is based on a single one-hour run per application with a randomly chosen seed. EvoMaster is an evolutionary tool whose coverage and fault counts vary across seeds; the reported advantages (+50.5% line coverage on Feature-service, +22.0% on Genome-Nexus) and the LanguageTool deficit (-19.3%) could be seed artifacts. The paper reports repeated SAINT runs with Devstral (§7) but no repeated EvoMaster runs. Please provide multi-seed results (e.g., 5–10 seeds) with medians and confidence intervals, or a statistical test, before claiming SAINT 'matches or considerably outperforms' EvoMaster.
  2. [§4.2.4, Table 4] The fault-detection methodology is underspecified and likely overcounts. The text states that the implementation was 'enhanced' to 'automatically learn new regex patterns for unseen cases,' and it notes that Feature-service returns an entire HTTP page with stack trace on every fault. Without deduplication, pattern-precision analysis, and manual validation, the high SAINT counts (e.g., 76–179 vs. EvoMaster's 37 for Feature-service) cannot be interpreted as distinct faults. Please specify how learned regexes are generated and validated, report unique request–response pairs vs. unique root causes, and discuss whether 5xx responses to intentionally invalid inputs are counted as faults.
  3. [§3.4/RQ2, Table 2] The scenario-based generation results are purely descriptive and have no comparison baseline. 'Effectiveness' is supported by scenario counts, sequence lengths, and a 41-participant survey from the authors' organization, but not by comparison with existing scenario-based tools (e.g., RESTler, LogiAgent/RESTGPT) even on the four OpenAPI applications. Since the paper positions SAINT against LogiAgent and claims 'developer-aligned' scenario generation, some comparative evidence, or at least per-run variance and an acknowledgment of the survey's selection bias, is needed.
  4. [§4.1, Table 1 (external validity)] Most open-source subjects are public and likely present in LLM training data, so LLM-generated parameter values and scenarios may partly reflect memorization rather than reasoning from code. The proprietary App-X mitigates this concern but does not isolate the effect for the open-source benchmark. A concrete robustness test would be to run SAINT on a synthetic or renamed Java service and compare coverage and scenario quality; if that is infeasible, the paper should explicitly state this threat and temper the generalization claims.
minor comments (4)
  1. [§3.4] The cross-reference 'Listing ??' is unresolved; the PetClinic test case is shown in Figure 3, so the text should refer to that figure or to a properly numbered listing.
  2. [Abstract and §3.1] Typographical issues: 'we presentsaint' in the abstract, and 'IPD definitionsa' in §3.1. Also, the model name appears as 'Devstral-24B' and 'Devstral Small' in different places; unify the naming.
  3. [Table 2] Several reported values are fractional (e.g., '# of scenarios' 4.0 and 4.5, sequence length 2.8). If these are averages over two runs, state this explicitly; otherwise, counts should be integers.
  4. [Figure 8 and §4.2.1] The figure legend and the text use 'application coverage' and 'reachability coverage' without consistently stating whether these are line or branch coverage. Clarify the mapping and specify which coverage measure the '-0.9% to +50.5%' range refers to.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SAINT's claims are empirical and measured against external benchmarks; concerns about baselines and training data are validity threats, not definitional reductions.

full rationale

SAINT is an empirical systems paper rather than a derivation from first principles. Its central claims—code coverage, fault triggering, scenario quality, and developer acceptance—are evaluated by running the tool on eight applications and comparing against an external baseline (EvoMaster), with coverage measured by JaCoCo and reachability computed from static call chains. The endpoint model and ODG are constructed before test generation and are inputs to the process; their contribution is assessed by an ablation study (RQ5), not assumed. There is no equation in the paper that equates a fitted parameter with a predicted outcome, no target metric is used to define the endpoint model, and no result is derived from the paper's own definitions. The threats-to-validity section explicitly acknowledges LLM stochasticity and reports repeated Devstral runs (standard deviation 0.0–5.2% branch, 0.1–5.1% line), which supports the stability of the reported coverage. The single-seed EvoMaster comparison is a legitimate experimental-validity concern, and possible LLM training-data overlap with open-source subjects is a contamination risk, but neither is a circularity in the sense of a claim reducing to its inputs by construction. Self-citations that appear (e.g., prior LLM-based test generation work and RAFT-like dependency analysis) are related-work or implementation choices, not load-bearing evidence for the central empirical findings.

Assumptions & free parameters 4 free parameters · 4 assumptions · 5 invented entities

The paper introduces several internal constructs (endpoint model, ODG, three agents) and relies on assumptions about CLDK's coverage, LLM extraction accuracy, metric validity, and survey representativeness. There are no fitted numerical parameters in a derivation sense, but the experimental configuration includes hand-chosen temperature, run counts, action limits, and baseline budgets that affect the reported results. The proprietary App-X provides some external grounding, but most invented entities are only internally validated.

free parameters (4)
  • LLM decoding temperature = 0.2
    Chosen by hand (§4.1) to stabilize outputs while allowing diversity; affects the variance of generated tests.
  • Number of LLM runs per model = 2, except Devstral run 10 times
    Chosen for cost; limits confidence in coverage estimates and comparisons (§4.1, §7).
  • Agent action limit = 2 actions per planning step
    Set to control computational cost (§3.3.3); could affect how many faulty requests are repaired.
  • EvoMaster time budget = 1 hour per application, single seed
    Experimental config for the baseline; a single random seed may not represent EvoMaster's typical performance (§4.1).
assumptions (4)
  • domain assumption CLDK static analysis can identify endpoints, reachable code, and database operations across Jakarta, Spring, Struts, Stripes, and JDK HttpServer Java frameworks.
    Used throughout §3.1; if endpoint identification is incomplete, coverage results are understated and tests are missing.
  • domain assumption LLM prompts with in-context examples extract correct endpoint parameters, value constraints, IPDs, operation summaries, and scenarios.
    Invoked in §3.1–3.4; no independent verification of extracted artifacts beyond downstream test success and ablation.
  • domain assumption JaCoCo line/branch coverage on statically reachable code and response-code patterns measure test effectiveness and faults.
    §4.1, §4.2.4; fault counting uses modified regex patterns that may overcount unique faults for verbose error pages.
  • domain assumption The surveyed 41 employees represent target enterprise developers.
    §4.2.3; single organization, no external control group, potential institutional bias.
invented entities (5)
  • Endpoint model (8-tuple E)
    purpose: Captures syntactic and semantic information about each endpoint to drive LLM-based request generation.
    Internal data structure; validated only through aggregate system performance and ablation, no external benchmark.
  • Operation dependency graph (ODG)
    purpose: Captures resource, producer-consumer, and database ordering constraints between endpoints for scenario generation.
    Ablation shows partial-order construction contributes +17.1% line coverage, but the graph itself is not externally validated.
  • Repair agent
    purpose: Fixes 4xx HTTP requests via a plan-act-reflect loop.
    Ablation shows small or even negative contribution (+2.2% line, -1.3% DB); no isolated external evaluation.
  • Coverage-augmentation agent
    purpose: Generates additional requests to cover unreached lines reachable from an endpoint.
    Ablation shows +0.4% line and +5.2% DB coverage; internal validation only.
  • Test-generation agent
    purpose: Reifies atomic scenario blocks into concrete HTTP requests and executable JUnit tests.
    Validated via scenario-based test coverage, compilation, and developer survey; no separate external benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SAINT: Service-level Integration Test Generation with Program Analysis and LLM-based Agents." pith.science (2026). https://pith.science/paper/S3PLKSO2

@misc{pith2026251113305,
  author       = {Pith},
  title        = {Pith review of: SAINT: Service-level Integration Test Generation with Program Analysis and LLM-based Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S3PLKSO2}},
  note         = {Machine review of arXiv:2511.13305}
}
read the original abstract

Enterprise applications are typically tested at multiple levels, with service-level testing playing an important role in validating application functionality. Existing service-level testing tools, especially for RESTful APIs, often employ fuzzing and/or depend on OpenAPI specifications which are not readily available in real-world enterprise codebases. Moreover, these tools are limited in their ability to generate functional tests that effectively exercise meaningful scenarios. In this work, we present SAINT, a novel white-box testing approach for service-level testing of enterprise Java applications. SAINT combines static analysis, large language models (LLMs), and LLM-based agents to automatically generate endpoint and scenario-based tests. The approach builds two key models: an endpoint model, capturing syntactic and semantic information about service endpoints, and an operation dependency graph, capturing inter-endpoint ordering constraints. SAINT then employs LLM-based agents to generate tests. Endpoint-focused tests aim to maximize code and database interaction coverage. Scenario-based tests are synthesized by extracting application use cases from code and refining them into executable tests via planning, action, and reflection phases of the agentic loop. We evaluated SAINT on eight Java applications, including a proprietary enterprise application. Our results illustrate the effectiveness of SAINT in coverage, fault detection, and scenario generation. Moreover, a developer survey provides strong endorsement of the scenario-based tests generated by SAINT. Overall, our work shows that combining static analysis with agentic LLM workflows enables more effective, functional, and developer-aligned service-level test generation.

Figures

Figures reproduced from arXiv: 2511.13305 by the authors.

Figure 1
Figure 1. Overview of our approach. We present a new white-box technique, called saint, for service￾level testing that combines static program analysis with the power of large language models (LLMs)—leveraging their planning, rea￾soning, and reflection capabilities through agentic workflows to enhance test generation effectiveness. Our approach (shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. A test synthesized by saint capturing these con￾straints in a realistic scenario. challenges in scenario-based testing. It constructs a coherent multi￾endpoint test that maintains semantic consistency across endpoints with consistent use of owner ID 1 across the test (as seen in 1 ). It also enforces state-dependent constraints (as seen in 2 , where the edit operations assume that the pet was successfully created be… view at source ↗
Figure 4
Figure 4. formally defines the endpoint model. An endpoint E, corresponding to a method, is represented as an 8-tuple consisting of service class name 𝑐, method signature𝑚, endpoint path 𝑝, HTTP Symbol Type Description API endpoint (E) ≡ (c, m, p, H, Π, I, D, R) 𝑐 Σ + Fully qualified name of the class containing the endpoint method. 𝑚 Σ + Signature of the API endpoint method 𝑝 Σ ∗ Endpoint path 𝐻 H HTTP method. H = {𝐺𝐸𝑇 , 𝑃𝑂𝑆… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Construction of the endpoint model and ODG via [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: The workflow for generating endpoint-focused and scenario-based tests. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Sample test scenario extracted by saint. the generated information for a block to process subsequent blocks. The final step composes the test fragments together to create an executable JUnit test case for the scenario. 3.4.1 Generation of test scenarios and related end…
Figure 8
Figure 8. Figure 8: Application and reachability coverage [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Scenario and test quality assessment (left) and test [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

59 extracted references · 5 canonical work pages

  1. [1]

    Anastasios Antoniadis, Nikos Filippakis, Paddy Krishnan, Raghavendra Ramesh, Nicholas Allen, and Yannis Smaragdakis. 2020. Static analysis of Java enterprise applications: frameworks and caches, the elephants in the room. InProceed- ings of the 41st ACM SIGPLAN conference on programming language design and implementation. 794–807

    2020

  2. [2]

    Andrea Arcuri. 2018. EvoMaster: Evolutionary Multi-context Automated System Test Generation. In2018 IEEE 11th International Conference on Software Test- ing, Verification and Validation (ICST). 394–397. doi:10.1109/ICST.2018.00046 arXiv:1901.04472 [cs]

    arXiv 2018

  3. [3]

    Andrea Arcuri. 2019. RESTful API Automated Test Case Generation with Evo- Master.ACM Transactions on Software Engineering and Methodology (TOSEM)28, 1, Article 3 (jan 2019), 37 pages. doi:10.1145/3293455

    doi:10.1145/3293455 2019

  4. [4]

    Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, Montreal, QC, Canada, 748–758. doi:10.1109/ ICSE.2019.00083

    arXiv 2019

  5. [5]

    David F Bacon and Peter F Sweeney. 1996. Fast static analysis of C++ virtual function calls. InProceedings of the 11th ACM SIGPLAN conference on Object- oriented programming, systems, languages, and applications. 324–341

    1996

  6. [6]

    Cesare Bartolini, Antonia Bertolino, Eda Marchetti, and Andrea Polini. 2009. WS-TAXI: A WSDL-based Testing Tool for Web Services. InProceedings of the 2009 International Conference on Software Testing Verification and Validation. IEEE Computer Society, 326–335. doi:10.1109/ICST.2009.28

    work page doi:10.1109/icst.2009.28 2009

  7. [7]

    Asma Belhadi, Man Zhang, and Andrea Arcuri. 2024. Random Testing and Evolutionary Testing for Fuzzing GraphQL APIs.ACM Trans. Web18, 1, Article 14 (Jan. 2024), 41 pages. doi:10.1145/3609427

    doi:10.1145/3609427 2024

  8. [8]

    CodeLLM-Devkit

    cldk 2025. CodeLLM-Devkit. https://github.com/codellm-devkit/python-sdk

    2025

Show all 59 references
  1. [9]

    Davide Corradini, Zeno Montolli, Michele Pasqua, and Mariano Ceccato. 2024. DeepREST: Automated Test Case Generation for REST APIs Exploiting Deep Reinforcement Learning. InProceedings of the 39th IEEE/ACM International Confer- ence on Automated Software Engineering. Associati...

    2024

  2. [10]

    DayTrader8 Sample

    daytrader 2025. DayTrader8 Sample. https://github.com/OpenLiberty/sample. daytrader8

    2025

  3. [11]

    Sida Deng, Rubing Huang, Man Zhang, Chenhui Cui, Dave Towey, and Rongcun Wang. 2025. LRASGen: LLM-based RESTful API Specification Generation.arXiv preprint arXiv:2504.16833(2025)

    2025 arXiv

  4. [12]

    Gherkin Reference

    gherkinsyntax 2025. Gherkin Reference. https://cucumber.io/docs/gherkin/ reference

    2025

  5. [13]

    Ruikai Huang, Manish Motwani, Idel Martinez, and Alessandro Orso. 2024. Gen- erating REST API Specifications through Static Analysis. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). Arti- cle 107, 13 pages. doi:10.1145/3597503.3639137

    2024

  6. [14]

    JaCoCo. 2025. JaCoCo Agent. https://www.eclemma.org/jacoco/trunk/doc/agent. html. [Online; accessed Nov-2025]

    2025

  7. [15]

    Jakarta EE

    jakarta 2025. Jakarta EE. https://jakarta.ee/

    2025

  8. [16]

    JavaParser. 2025. JavaParser. https://github.com/javaparser/. [Online; accessed Nov-2025]

    2025

  9. [17]

    JDK HttpServer

    jdkhttpserver 2025. JDK HttpServer. https://docs.oracle.com/en/java/javase/21/ docs/api/jdk.httpserver/com/sun/net/httpserver/HttpServer.html

    2025

  10. [18]

    MyBatis JPetStore

    jpetstore 2025. MyBatis JPetStore. https://github.com/mybatis/jpetstore-6

    2025

  11. [19]

    Stefan Karlsson, Adnan Čaušević, and Daniel Sundmark. 2021. Automatic Property-based Testing of GraphQL APIs. In2021 IEEE/ACM International Confer- ence on Automation of Software Test (AST). 1–10. doi:10.1109/AST52587.2021.00009

    2021

  12. [20]

    Myeongsoo Kim, Davide Corradini, Saurabh Sinha, Alessandro Orso, Michele Pasqua, Rachel Tzoref-Brill, and Mariano Ceccato. 2023. Enhancing REST API Testing with NLP Techniques. InProceedings of the 32nd ACM SIGSOFT Interna- tional Symposium on Software Testing and Analysis. As...

    2023

  13. [21]

    Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2023. Adaptive REST API Testing with Reinforcement Learning. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering. IEEE Press, 446–458. doi:10.1109/ASE56229.2023.00218

    2023

  14. [22]

    Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2025. LlamaRestTest: Effective REST API Testing with Small Language Models. doi:10.48550/arXiv. 2501.08598 arXiv:2501.08598 [cs]

    2025 doi arXiv

  15. [23]

    Myeongsoo Kim, Tyler Stennett, Dhruv Shah, Saurabh Sinha, and Alessandro Orso. 2024. Leveraging Large Language Models to Improve REST API Testing. doi:10.48550/arXiv.2312.00894 arXiv:2312.00894 [cs]

    2024 doi arXiv

  16. [24]

    Myeongsoo Kim, Tyler Stennett, Saurabh Sinha, and Alessandro Orso. 2025. A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM- Driven Inputs. doi:10.48550/arXiv.2411.07098 arXiv:2411.07098 [cs]

    2025 doi arXiv

  17. [25]

    LanguageTool

    languagetool 2025. LanguageTool. https://github.com/languagetool-org/ languagetool

    2025

  18. [26]

    Tri Le, Thien Tran, Duy Cao, Vy Le, Tien Nguyen, and Vu Nguyen. 2024. KAT: Dependency-aware Automated API Testing with Large Language Models. In2024 IEEE Conference on Software Testing, Verification and Validation (ICST). 82–92. doi:10.1109/ICST60714.2024.00017 arXiv:2407.10227 [cs]

    2024

  19. [27]

    Jia Li, Jiacheng Shen, Yuxin Su, and Michael R. Lyu. 2025. LLM-assisted Mutation for Whitebox API Testing. doi:10.48550/arXiv.2504.05738 arXiv:2504.05738 [cs]

    2025 doi

  20. [28]

    Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: Model-based RESTful API Testing with Execution Feedback. doi:10.48550/arXiv.2204.12148 arXiv:2204.12148 [cs]

    2022 doi arXiv

  21. [29]

    Alberto Martin-Lopez, Andrea Arcuri, Sergio Segura, and Antonio Ruiz-Cortés

  22. [30]

    Alberto Martin-Lopez, Sergio Segura, Carlos Müller, and Antonio Ruiz-Cortés

  23. [31]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. A Cata- logue of Inter-parameter Dependencies in RESTful Web APIs. InService-Oriented Computing: 17th International Conference, ICSOC 2019, Toulouse, France, October 28–31, 2019, Proceedings(Toulouse, France)....

    2019 doi

  24. [32]

    Specification and automated analysis of inter-parameter dependencies in web APIs.IEEE Transactions on Services Computing15, 4 (2021), 2342–2355

    2021

  25. [33]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2022. Online Testing of RESTful APIs: Promises and Challenges. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. Association for C...

    2022

  26. [34]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2021. RESTest: Automated Black-Box Testing of RESTful Web APIs. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. Associ- ation for Computing Machinery, 682–685. doi:10....

    2021

  27. [35]

    Vikram Nitin, Shubhi Asthana, Baishakhi Ray, and Rahul Krishna. 2022. Cargo: Ai-guided dependency analysis for migrating monolithic applications to microser- vices architecture. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–12

    2022

  28. [36]

    Srinivas Nidhra and Jagruthi Dondeti. 2012. Black box and white box testing techniques-a literature review.International Journal of Embedded Systems and Applications (IJESA)2, 2 (2012), 29–50

    2012

  29. [37]

    OpenRouter. 2025. OpenRouter. https://openrouter.ai. [Online; accessed Nov- 2025]

    2025

  30. [38]

    OpenAPI Specification

    openapispec 2025. OpenAPI Specification. https://spec.openapis.org/oas/latest. html

    2025

  31. [39]

    Spring PetClinic Sample Application

    petclinic 2025. Spring PetClinic Sample Application. https://github.com/spring- projects/spring-petclinic

    2025

  32. [40]

    Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha

  33. [41]

    REST-assured

    restassured 2025. REST-assured. https://rest-assured.io

    2025

  34. [42]

    Diptikalyan Saha, Devika Sondhi, Swagatam Haldar, and Saurabh Sinha. 2025. REST API Functional Tester. InProceedings of the 18th Innovations in Software Engineering Conference. Association for Computing Machinery, Article 8, 11 pages. doi:10.1145/3717383.3717388

    2025

  35. [43]

    Marty Pitt, Dilip Krishnan, and Adrian Kelly. 2020. SpringFox. https://github. com/springfox/springfox. [Online; accessed Nov-2025]

    2020

  36. [44]

    Spring Team. 2013. Spring PetClinic. https://github.com/spring-projects/spring- petclinic A sample Spring-based application

    2013

  37. [45]

    springdoc. 2025. springdoc-openapi. https://github.com/springdoc/springdoc- openapi. [Online; accessed Nov-2025]

    2025

  38. [46]

    spring 2025. Spring. https://spring.io/

    2025

  39. [47]

    Stripes Framework

    stripes 2025. Stripes Framework. https://github.com/StripesFramework/stripes

    2025

  40. [48]

    Apache Struts

    struts 2025. Apache Struts. https://struts.apache.org/

    2025

  41. [49]

    Tyler Stennett, Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2025. AutoRestTest: A Tool for Automated REST API Testing Using LLMs and MARL. doi:10.48550/arXiv.2501.08600 arXiv:2501.08600 [cs]

    2025 doi arXiv

  42. [50]

    Swagger. 2025. Swagger Core. https://github.com/swagger-api/swagger-core. [Online; accessed Nov-2025]

    2025

  43. [51]

    Tree-sitter

    treesitter 2025. Tree-sitter. https://tree-sitter.github.io/tree-sitter

    2025

  44. [52]

    Supplementary Material

    supplementary 2025. Supplementary Material. https://github.com/aster-test- generation/saint

    2025

  45. [53]

    WALA. 2025. WALA. https://github.com/wala/WALA. [Online; accessed Nov- 2025]

    2025

  46. [54]

    Ke Zhang, Chenxi Zhang, Chong Wang, Chi Zhang, YaChen Wu, Zhenchang Xing, Yang Liu, Qingshan Li, and Xin Peng. 2025. LogiAgent: Automated Logical Testing for REST Systems with LLM-Based Multi-Agents. doi:10.48550/arXiv.2503.15079 arXiv:2503.15079 [cs]

    2025 doi arXiv

  47. [55]

    Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTEST- GEN: Automated Black-Box Testing of RESTful APIs. In2020 IEEE 13th Interna- tional Conference on Software Testing, Validation and Verification (ICST). IEEE, Porto, Portugal, 142–152. doi:10.1109/icst463...

    2020

  48. [56]

    Man Zhang and Andrea Arcuri. 2023. Open Problems in Fuzzing RESTful APIs: A Comparison of Tools.ACM Trans. Softw. Eng. Methodol.32, 6, Article 144 (Sept. 2023), 45 pages. doi:10.1145/3597205

    2023 doi

  49. [58]

    Man Zhang and Andrea Arcuri. 2021. Adaptive hypermutation for search-based system test generation: A study on REST APIs with EvoMaster.ACM Transactions on Software Engineering and Methodology (TOSEM)31, 1 (2021), 1–52

    2021

  50. [2021]

    In2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE)

    Black-box and white-box test case generation for RESTful APIs: Enemies or allies?. In2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE). IEEE, 231–241

  51. [2025]

    In ACM/IEEE International Conference on Software Engineering

    ASTER: Natural and Multi-language Unit Test Generation with LLMs. In ACM/IEEE International Conference on Software Engineering

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.