Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Utilizing API Response for Test Refinement

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A black-box REST API tester can learn constraints from 4xx response messages and reach high operation coverage with one to three orders of magnitude fewer requests than search-based tools.

desk verdict The core idea—extracting constraints from natural-language 4xx messages—is genuinely new, but the paper's own Table 4 undermines the 'fewer requests than all SOTA tools' claim, since EvoMaster uses fewer requests than ASTRA. read the letter →

arxiv 2501.18145 v1 pith:ASMMQVAA submitted 2025-01-30 cs.SE

Devika Sondhi , Ananya Sharma , Diptikalyan Saha This is my paper · ORCID
classification cs.SE
keywords RESTAPItestingblack-boxtestrefinementresponsemessageanalysisconstraintinferencelargelanguagemodelssearch-basedOpen
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most automated REST API testing either trusts the OpenAPI specification, which is often incomplete, or learns by making tens of thousands of requests, which is too expensive for industry use. This paper argues that the error message itself carries the missing information: a 4xx response names the parameter, operation, or data value that caused rejection. The proposed tool, ASTRA, reads each failure message, classifies it into one of 14 constraint categories, and adds the inferred constraint to an extended specification model before generating the next round of tests. The result, the paper reports, is a testing loop that converges after 6 to 186 requests per benchmark API and produces a higher share of valid 2xx tests than state-of-the-art search-based tools while reaching comparable or better operation coverage.

What carries the argument

The load-bearing mechanism is the feedback loop around an extended specification model $Spec := \langle O,S\rangle$: execute generated tests, log unique 4xx responses, classify each response into one of 14 constraint categories using an LLM, have an agent extract the constrained entities and values, add the resulting constraint object to the model, and regenerate tests until an iteration introduces no new failure. Parameter-selection constraints are compiled into SMT constraints so the generator can enumerate compliant parameter scenarios, and data constraints are turned into natural-language prompts for a small language model that proposes realistic values, verified by a constraint solver. This loop is what lets ASTRA learn from a handful of responses instead of tens of thousands.

What would settle it

Take ASTRA and one benchmark API, and rewrite the API's error responses to a fixed generic string such as 'Bad request' while keeping the status codes unchanged. If the valid-test ratio and request count stay roughly the same, then response-message content is not the source of ASTRA's learning; if they degrade sharply, the central claim is confirmed.

Watch

Extended reading notes

Core claim

ASTRA's central claim is that natural-language 4xx responses can be converted into machine-readable test constraints, so that each failed request teaches the tester how to pass. The paper derives a taxonomy of 14 response categories from 1,663 unique messages, covering operation-level prerequisites, parameter-selection rules, data-value restrictions, and nested dependencies. An LLM classifies each failure and an agent extracts the entities and values needed to form a constraint, which is stored in an extended specification model. Test generation then re-runs with the new constraints, producing operation sequences, parameter scenarios, and data values that avoid the previously rejected request. On 11 public benchmark APIs, the paper reports that ASTRA achieves the highest operation coverage on 9 APIs and the highest successful (2xx) operation coverage on 9 APIs, and that it does so with 925 total requests versus 188,676 for MOREST, 343 for EvoMaster, and 429,204 for ARAT-RL across the benchmark set.

Load-bearing premise

The pipeline collapses if API error messages cannot be mapped to the 14 predefined constraint categories or the language model cannot identify the parameter or operation the message refers to.

Editorial extensions

If this is right

  • ASTRA reaches 100% operation coverage on most of the 11 benchmark APIs and matches or exceeds the best baseline on 9 of them, while its total request count is 925 compared with 188,676 for MOREST, 343 for EvoMaster, and 429,204 for ARAT-RL.
  • When the baselines are given the same request budget as ASTRA, their operation coverage drops sharply, which the paper reads as evidence that reinforcement-learning and search-based exploration need much more data to learn the same constraints.
  • The iterative loop shifts the response mix away from 4xx toward 2xx and 5xx over time; the 5xx responses expose defects, and ASTRA reports 33 unique defects, 13 with stack traces, versus 29 for MOREST without a hit limit.
  • Blank or uninformative 4xx responses, about 22% of benchmark requests, are handled by fallback heuristics such as treating a 404 on an identifier parameter as a missing producer operation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that its request budget makes functional API testing economical for pay-per-call or rate-limited services, where tens of thousands of requests would be infeasible; the paper frames the cost motivation but does not quantify the savings.
  • A natural extension is to feed ASTRA's learned constraints into a white-box or coverage-guided phase, since the paper notes its branch coverage trails EvoMaster when no request limit is imposed.
  • The approach's dependence on informative error text could be tested head-to-head by running ASTRA against a mirrored API whose messages are replaced with generic 'Bad request' strings; this would isolate how much of the learning comes from the message content rather than the status code.
  • The 14-category taxonomy was built from responses collected from APIs that overlap heavily with the 11 evaluation benchmarks, so transfer to unseen API domains with novel error phrasing is an open question rather than an established property.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ASTRA, a black-box REST API testing tool that iteratively learns constraints from 4xx response messages using an LLM-based agent, updates an extended specification model, and generates new test cases. The evaluation compares ASTRA with MOREST, EvoMaster, and ARAT-RL on 11 public APIs, reporting operation coverage, 2xx coverage, request counts, and defect detection. The central claims are that ASTRA reduces 4xx responses, achieves high coverage, and does so with fewer API requests than state-of-the-art search-based tools.

Significance. If the claims hold, ASTRA would be a valuable contribution to black-box API testing, particularly for cost-sensitive industrial settings where request budgets are limited. The paper contributes a specification model that can represent constraints not expressible in OpenAPI, a taxonomy of 14 constraint categories, and an iterative greedy refinement algorithm. The evaluation is based on live APIs and the artifacts appear to be released. However, the significance is currently weakened by an overgeneralized efficiency claim and by the overlap between the corpora used to build the taxonomy and the evaluation benchmarks.

major comments (3)
  1. [Abstract, Section 1, Table 4] The abstract and Section 1 claim that ASTRA obtains high coverage with 'a lesser number of API requests' compared with state-of-the-art search-based API testing tools. Table 4 reports ASTRA total operation hits of 925 against EvoMaster's 343, and on 9 of the 11 benchmarks EvoMaster makes fewer requests than ASTRA (e.g., gestao 60 vs. 186, market 26 vs. 132). The hit-limit experiment in Section 3.2.2 does not repair this discrepancy because for EvoMaster the cap binds on only two benchmarks (scs and ncs) where EvoMaster's unprotected hits exceed ASTRA's; elsewhere EvoMaster is already under ASTRA's budget. The request-efficiency claim must be scoped to the two baselines for which it holds, or the paper must discuss why EvoMaster's lower request count does not undermine the headline comparison.
  2. [Section 2.3.1, Tables 1 and 3] The 14-category taxonomy and the blank-response heuristics (Section 2.3) were derived from a corpus of responses that includes 17 APIs listed in Table 1, and the 11 evaluation benchmarks in Table 3 overlap with this corpus substantially (petstore, gestao, market, user, problem, langtool, scs, ncs, restcountries, person). Consequently, the statement in Section 3.4 that 'we could map each failure in the benchmark to one of the defined categories' partly reflects the fact that the categories were constructed from these very benchmarks. The 95.5% BERT classification accuracy is reported on a random split of the same corpus and is not a measure of performance on genuinely unseen APIs. The external validity of the taxonomy and the classifier should be evaluated on a held-out set of APIs that were not used in the construction.
  3. [Section 3.2.2, Table 4] For EvoMaster, the hit limit is imposed via 'maxActionEvaluations', but the paper compares 'operation hits' across tools without establishing that an EvoMaster action evaluation corresponds one-to-one with an HTTP request. If action evaluations include internal or filtered actions that do not result in HTTP calls, the request counts in Table 4 are not comparable across tools. The authors should define the counting unit precisely and, if needed, instrument both tools to count actual HTTP requests.
minor comments (6)
  1. [Section 2.1.3] In the ProducerConsumer example, 'consParam=deleteorder.path.orderId' is missing a capital 'O' in 'deleteOrder'; this is inconsistent with the rest of the text.
  2. [Section 2.3.1] The BERT classifier accuracy of 95.5% is reported from a single 80:20 split without the number of seeds or standard deviation; please report the variance or use repeated stratified splits.
  3. [Section 3.3.2] The word 'significant' is used without a statistical test; the reader cannot tell whether the changes in Fig. 6 are due to random variation.
  4. [Section 3.2.2] Please clarify how the average of three ASTRA runs is converted into an integer hit budget for the baseline tools.
  5. [Section 3.4] The threat to validity section acknowledges the non-exhaustive category list but does not address the overlap between the taxonomy-construction corpus and the evaluation benchmarks; a sentence on this would be helpful.
  6. [Section 6] The anonymous repository link is good; please ensure the version of the code and the exact tool configurations are pinned for reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

The constraint taxonomy and blank-response heuristics are derived from a corpus overlapping the evaluation benchmarks, making the reported generalizability partly self-confirming; the request-efficiency claim is additionally internally inconsistent with the paper's own Table 4.

  1. fitted input called prediction [Section 2.3.1 (Constraint Inference), Table 1, Table 2, and Section 3.4 (Threats to Validity)]
    "We collected response messages from over 17 APIs (Table 1) when run with API testing tools MOREST, EvoMaster... On extracting the unique responses, we obtained a dataset of 1663 responses. These response messages were studied to identify 14 different types of categories. ... The list of classification categories may not be exhaustive, however, we could map each failure in the benchmark to one of the defined categories."

    Table 1 lists 17 APIs used to build the taxonomy, and it substantially overlaps Table 3's 11 evaluation benchmarks (petstore, person, gestao, market, user, problem, langtool, scs, ncs, rest countries). The 14 constraint categories are therefore fitted to the same response distribution on which ASTRA is later evaluated. The Section 3.4 claim that every benchmark failure maps to one of these categories is a consequence of that derivation, not an independent validation. This does not force the coverage or request-count numbers, which come from live executions, but it inflates the apparent generality of the constraint-inference component.

full rationale

The central ASTRA loop is genuinely iterative and evaluated against live external APIs, so there is no fitted constant whose value is recovered as a prediction. However, the paper's own Section 2.3.1 shows that the 14-category constraint taxonomy was induced from a response-message corpus that includes most of the same benchmark APIs used in Section 3. The later assertion that all benchmark failures map to the categories is therefore partly self-confirming rather than an external test. Separately, the headline claim of 'a lesser number of API requests' is contradicted by Table 4, where EvoMaster makes 343 total requests versus ASTRA's 925; this is an internal-consistency problem in the reported evidence, not a circularity, but it weakens the central efficiency claim. Overall, the derivation is not circular in the strict sense of Eq. X = Eq. Y, but the taxonomy-overlap issue warrants a moderate score of 4.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

No new physical entities are introduced. The central claim rests on a hand-built taxonomy of error-message categories, a trained BERT classifier, and LLM behavior, none of which are externally fixed benchmarks. The taxonomy and classifier are partly fitted to the same APIs used in the evaluation, which is the main circularity burden in the ledger.

free parameters (3)
  • Number of constraint categories in taxonomy = 14
    The taxonomy size was chosen by hand after studying 1,663 unique responses from 17 APIs (Section 2.3.1). The same APIs largely overlap with the evaluation benchmarks, so this is a hand-fitted discrete model of the test distribution.
  • BERT classifier parameters = Trained on 1,663 unique responses, 95.5% test accuracy
    The fallback classifier is trained on response data that includes the benchmark APIs, and its weights are fitted to that corpus. It is not an independent external benchmark.
  • Number of data scenarios per parameter scenario = Unspecified 'predefined number'
    Section 2.3 says a predefined number of data scenarios are generated per parameter scenario, but the number is never reported. This affects total request count and coverage, and is a hidden degree of freedom in the evaluation.
assumptions (5)
  • ad hoc to paper API response error messages can be mapped to a finite taxonomy of 14 constraint categories.
    Introduced in Section 2.3.1 after a study of 17 APIs. Completeness and generalizability to unseen APIs is assumed, and the taxonomy overlaps with the evaluation benchmarks.
  • domain assumption A 4xx response indicates an invalid test input rather than an API defect, so reducing 4xx responses increases test quality.
    Used throughout Sections 1 and 2.3. Some 4xx responses could indicate genuine API issues, such as authorization requirements or valid business rule rejections.
  • domain assumption LLM-based zero-shot classification and entity extraction (Mistral Large-2) are reliable enough to infer correct constraints.
    Assumed in Sections 2.3.1 and 3.4. Only indirect evidence via end-to-end metrics is given; no component-level accuracy for entity extraction is reported.
  • domain assumption The OpenAPI specification is the only available oracle for operation and parameter structure.
    Assumed for the black-box setup. A more complete internal specification would change the problem setting.
  • ad hoc to paper Producer-consumer relationships can be inferred from noun phrases in error messages (e.g., 'Order Not Found').
    Used in Sections 2.1.3 and 2.3.1 for the LLM agent. No systematic evaluation of inference accuracy across APIs is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Utilizing API Response for Test Refinement." pith.science (2026). https://pith.science/paper/ASMMQVAA

@misc{pith2026250118145,
  author       = {Pith},
  title        = {Pith review of: Utilizing API Response for Test Refinement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ASMMQVAA}},
  note         = {Machine review of arXiv:2501.18145}
}
read the original abstract

Most of the web services are offered in the form of RESTful APIs. This has led to an active research interest in API testing to ensure the reliability of these services. While most of the testing techniques proposed in the past rely on the API specification to generate the test cases, a major limitation of such an approach is that in the case of an incomplete or inconsistent specification, the test cases may not be realistic in nature and would result in a lot of 4xx response due to invalid input. This is indicative of poor test quality. Learning-based approaches may learn about valid inputs but often require a large number of request-response pairs to learn the constraints, making it infeasible to be readily used in the industry. To address this limitation, this paper proposes a dynamic test refinement approach that leverages the response message. The response is used to infer the point in the API testing flow where a test scenario fix is required. Using an intelligent agent, the approach adds constraints to the API specification that are further used to generate a test scenario accounting for the learned constraint from the response. Following a greedy approach, the iterative learning and refinement of test scenarios are obtained from the API testing system. The proposed approach led to a decrease in the number of 4xx responses, taking a step closer to generating more realistic test cases with high coverage that would aid in functional testing. A high coverage was obtained from a lesser number of API requests, as compared with the state-of-the-art search-based API Testing tools.

Figures

Figures reproduced from arXiv: 2501.18145 by the authors.

Figure 1
Figure 1. Sample Response 1 these techniques may be effective in obtaining good coverage, in the industry scenario, making a large number of API requests incurs a cost that may not be practical [19, 42]. For such tools to be widely used in the practical scenario, the key is to make minimal API requests, while maximizing the testing effectiveness. This effectiveness may be measured through metrics such as operation coverage, d… view at source ↗
Figure 2
Figure 2. Sample Response 2 prodParam=placeOrder.200.id, consOp=deleteOrder, consParam=deleteorder.path.orderId) and adds it to the specification model. On the subsequent run, ASTRA derives a sequence of operations [placeOrder, deleteOrder] from the constraint and injects the id produced by placeOrder to deleteOrder. This results in a successful 200 response, as can be seen in the figure. To summarize, ASTRA’s approach broadl… view at source ↗
Figure 3
Figure 3. Architecture Design for ASTRA • opname is a unique name for the operation. This is typically presented in OpenAPI specification as operationid. • path is the relative URL for accessing the operation endpoint. • tag is an optional set of keywords to tag the operation. • type is the HTTP method for RESTful operation that can take values: POST (create), GET (read), PUT (update/replace), PATCH (update/modify), and DELET… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Prompt for categorization errors at the API’s end and usually reveal defects in the API. By learning from the 4xx failures, any subsequent 5xx would be a step closer to revealing defects in the API. The algorithm converges when no new failure is logged from the run. Th…
Figure 5
Figure 5. Figure 5: Line and Branch Coverage of 4 tools across 11 APIs, with and without OpHit Limit [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Trend of Response Code with each Iteration of Refinement with ASTRA [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Response Code Distribution [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WFC/WFD: Web Fuzzing Commons, Dataset and Guidelines to Support Experimentation in REST API Fuzzing

    cs.SE 2025-09 conditional novelty 6.0 of 10

    The paper introduces standardized schemas for REST API authentication and fault reporting, a 36-API Docker-backed dataset, and a six-fuzzer comparison with guidelines for fair experimentation.

Reference graph

Works this paper leans on

48 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [1]

    [n. d.]. EclEmma - JaCoCo Java Code Coverage Library — eclemma.org. https://www.eclemma.org/jacoco/. [Accessed 20-10-2023]

    work page 2023

  2. [2]

    RestGo repository

    2022. RestGo repository. https://github.com/codingsoo/REST_Go. [Online; accessed 19-August-2022]

    work page 2022

  3. [3]

    Andrea Arcuri. 2017. Many independent objective (MIO) algorithm for test suite generation. InInternational symposium on search based software engineering . Springer, 3–17

    work page 2017

  4. [4]

    Andrea Arcuri. 2017. RESTful API automated test case generation. In 2017 IEEE International Conference on Software Quality, Reliability and Security (QRS) . IEEE, 9–20. 1https://anonymous.4open.science/r/ASTRA-B96F , Vol. 1, No. 1, Article . Publication date: January 2025. Utilizing API Response for Test Refinement 19

    work page 2017

  5. [5]

    Andrea Arcuri. 2018. Evomaster: Evolutionary multi-context automated system test generation. In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) . IEEE, 394–397

    work page 2018

  6. [6]

    Andrea Arcuri. 2018. Test suite generation with the Many Independent Objective (MIO) algorithm. Information and Software Technology 104 (2018), 195–206

    work page 2018

  7. [7]

    Andrea Arcuri. 2019. RESTful API automated test case generation with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 28, 1 (2019), 1–37

    work page 2019

  8. [8]

    Andrea Arcuri. 2020. Automated black-and white-box testing of restful apis with EvoMaster.IEEE Software 38, 3 (2020), 72–78

    work page 2020

Show all 48 references
  1. [9]

    Andrea Arcuri and Juan P Galeotti. 2021. Enhancing search-based testing with testability transformations for existing APIs. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021), 1–34

    2021

  2. [10]

    Andrea Arcuri, Juan Pablo Galeotti, Bogdan Marculescu, and Man Zhang. 2021. EvoMaster: A search-based system test generation tool. Journal of Open Source Software 6, 57 (2021), 2153

    2021

  3. [11]

    Vaggelis Atlidakis. [n. d.]. Testing of Cloud Services. ([n. d.])

  4. [12]

    Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. Restler: Stateful rest api fuzzing. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 748–758

    2019

  5. [13]

    Antonia Bertolino, Guglielmo De Angelis, Antonio Guerriero, Breno Miranda, Roberto Pietrantuono, and Stefano Russo. 2020. DevOpRET: Continuous reliability testing in DevOps. Journal of Software: Evolution and Process (2020), e2298

    2020

  6. [14]

    Davide Corradini, Amedeo Zampieri, Michele Pasqua, and Mariano Ceccato. 2021. Empirical comparison of black-box test case generation tools for RESTful APIs. In 2021 IEEE 21st International Working Conference on Source Code Analysis and Manipulation (SCAM). IEEE, 226–236

    2021

  7. [15]

    Davide Corradini, Amedeo Zampieri, Michele Pasqua, and Mariano Ceccato. 2021. Restats: A test coverage tool for RESTful APIs. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME) . IEEE, 594–598

    2021

  8. [16]

    Davide Corradini, Amedeo Zampieri, Michele Pasqua, Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato

  9. [17]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs.CL]

    2019 arXiv

  10. [18]

    Hamza Ed-Douibi, Javier Luis Cánovas Izquierdo, and Jordi Cabot. 2018. Automatic generation of test cases for REST APIs: A specification-based approach. In 2018 IEEE 22nd international enterprise distributed object computing conference (EDOC). IEEE, 181–190

    2018

  11. [19]

    Antonio Gamez-Diaz, Pablo Fernandez, Antonio Ruiz-Cortés, Pedro J Molina, Nikhil Kolekar, Prithpal Bhogill, Mad- hurranjan Mohaan, and Francisco Méndez. 2019. The role of limitations and SLAs in the API industry. In Proceedings of the 2019 27th ACM Joint Meeting on European So...

    2019

  12. [20]

    Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API data fuzzing. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 725–736

    2020

  13. [21]

    Google. [n. d.]. Google Maps Platform Documentation Google for Developers — developers.google.com. https: //developers.google.com/maps/documentation

  14. [22]

    Gestao Hospital. 2022. Gestao Swagger Specification. https://github.com/ValchanOficial/GestaoHospital. [Online; accessed 19-August-2022]

    2022

  15. [23]

    Stefan Karlsson, Adnan Čaušević, and Daniel Sundmark. 2020. QuickREST: Property-based test generation of OpenAPI- described RESTful APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). IEEE, 131–141

    2020

  16. [24]

    Myeongsoo Kim, Davide Corradini, Saurabh Sinha, Alessandro Orso, Michele Pasqua, Rachel Tzoref-Brill, and Mariano Ceccato. 2023. Enhancing REST API Testing with NLP Techniques. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (Seat...

    2023

  17. [25]

    Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2023. Adaptive REST API Testing with Reinforcement Learning. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 446–458

    2023

  18. [26]

    Nuno Laranjeiro, João Agnelo, and Jorge Bernardino. 2021. A black box tool for robustness testing of REST services. IEEE Access 9 (2021), 24738–24754

    2021

  19. [27]

    Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: Model-based RESTful API Testing with Execution Feedback. arXiv preprint arXiv:2204.12148 (2022)

    2022 arXiv

  20. [28]

    Alberto Martin-Lopez, Sergio Segura, Carlos Müller, and Antonio Ruiz-Cortés. 2021. Specification and automated analysis of inter-parameter dependencies in web APIs.IEEE Transactions on Services Computing 15, 4 (2021), 2342–2355. , Vol. 1, No. 1, Article . Publication date: Jan...

    2021

  21. [29]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2019. Test coverage criteria for RESTful web APIs. In Proceedings of the 10th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 15–21

    2019

  22. [30]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2021. RESTest: automated black-box testing of RESTful web APIs. In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis . 682–685

    2021

  23. [31]

    Phil McMinn. 2004. Search-based software test data generation: a survey. Software testing, Verification and reliability 14, 2 (2004), 105–156

    2004

  24. [32]

    Changhai Nie and Hareton Leung. 2011. A survey of combinatorial testing. ACM Computing Surveys (CSUR) 43, 2 (2011), 1–29

    2011

  25. [33]

    OpenAPI. 2023. OpenAPI Specification. https://www.openapis.org/

    2023

  26. [34]

    Petstore. 2022. PetStore Swagger Specification. https://petstore.swagger.io/. [Online; accessed 19-August-2022]

    2022

  27. [35]

    Stefano Russo. [n. d.]. Assessing Black-box Test Case Generation Techniques for Microservices. InQuality of Information and Communications Technology: 15th International Conference, QUATIC 2022, Talavera de la Rina, Spain, September 12-14, 2022, Proceedings. Springer Nature, 46

    2022

  28. [36]

    Sergio Segura, José A Parejo, Javier Troya, and Antonio Ruiz-Cortés. 2017. Metamorphic testing of RESTful web APIs. IEEE Transactions on Software Engineering 44, 11 (2017), 1083–1099

    2017

  29. [37]

    Dimitri Stallenberg, Mitchell Olsthoorn, and Annibale Panichella. 2021. Improving Test Case Generation for REST APIs Through Hierarchical Clustering. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 117–128

    2021

  30. [38]

    Stripe. [n. d.]. Error codes — stripe.com. https://stripe.com/docs/error-codes

  31. [39]

    Language Tool. 2022. Language Tool. https://github.com/languagetool-org/languagetool. [Online; accessed 19-August- 2022]

    2022

  32. [40]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  33. [41]

    Chung-Hsuan Tsai, Shi-Chun Tsai, and Shih-Kun Huang. 2021. REST API Fuzzing by Coverage Level Guided Blackbox Testing. arXiv preprint arXiv:2112.15485 (2021)

    2021 arXiv

  34. [42]

    Prasang Upadhyaya, Magdalena Balazinska, and Dan Suciu. 2016. Price-optimal querying with data apis. Proceedings of the VLDB Endowment 9, 14 (2016), 1695–1706

    2016

  35. [43]

    Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. Resttestgen: automated black-box testing of restful apis. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) . IEEE, 142–152

    2020

  36. [44]

    Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTESTGEN: Automated black-box testing of RESTful APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST) . IEEE, 142–152

    2020

  37. [45]

    Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial Testing of RESTful APIs. In ACM/IEEE International Conference on Software Engineering (ICSE)

    2022

  38. [46]

    Yelp. [n. d.]. Getting Started — docs.developer.yelp.com. https://docs.developer.yelp.com/docs

  39. [47]

    Youtube. [n. d.]. YouTube Data API - Errors Google for Developers — developers.google.com. https://developers. google.com/youtube/v3/docs/errors. , Vol. 1, No. 1, Article . Publication date: January 2025

    2025

  40. [2022]

    Software Testing, Verification and Reliability (2022), e1808

    Automated black-box testing of nominal and error scenarios in RESTful APIs. Software Testing, Verification and Reliability (2022), e1808

    2022

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.