Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

WFC/WFD: Web Fuzzing Commons, Dataset and Guidelines to Support Experimentation in REST API Fuzzing

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes WFC (schema libraries for authentication and fault reporting) and WFD (36 authenticated REST APIs with Docker scaffolding), arguing these remove three chronic obstacles to fair REST API fuzzing experiments: authentication

desk verdict Useful standard-and-dataset paper whose engineering artifacts matter more than its own head-to-head fuzzer results, which are too confounded to interpret. read the letter →

arxiv 2509.01612 v1 pith:ZSHTGW4M submitted 2025-09-01 cs.SE

Omur Sahin , Man Zhang , Andrea Arcuri This is my paper · ORCID
classification cs.SE
keywords RESTAPIfuzzingauthenticationspecificationfuzzerbenchmarkfaultcatalogempiricalcomparisonOpenreproducibilitytestgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

REST API fuzzing research keeps tripping on three practical problems: how a fuzzer learns to log in, how different tools report which faults they found, and which APIs everyone should agree to test on. This paper tries to turn each problem into a standardized, reusable artifact. WFC is an open-source library of JSON Schema definitions: one schema declaratively describes authentication information (static credentials or dynamic token extraction from login endpoints), and another defines a fuzz-report format with a catalog of fault types. WFD is a corpus of 36 open-source JVM REST APIs, each with Docker Compose setup, coverage tooling, and WFC-formatted authentication configs, making it the largest open-source REST API selection used in academic fuzzing experiments to date.

What carries the argument

The central objects are two schema definitions packaged as WFC: auth.yaml, a declarative schema for authentication configuration (static credentials, login endpoint payloads, token extraction via JSON Pointer, cookie or header injection), and report.yaml, a schema for fuzzer output with a catalog of fault-type codes for oracles found in the literature. The second machinery piece is WFD, a corpus of 36 JVM REST APIs with Docker Compose scaffolding, JaCoCo instrumentation, mitmproxy HTTP logging, and WFC auth files for the 15 APIs that require login. Together they let a fuzzer be pointed at an API with the same declarative auth config and produce reports that other tools and HTML viewers can r

What would settle it

Re-run the 15 auth-requiring APIs in WFD after giving RESTler and Schemathesis their native script-based authentication setups (and keeping the other tools as-is). If the coverage and fault-detection gap between EvoMaster and those two tools collapses or reverses, the paper's implicit claim that WFD exposes intrinsic tool weaknesses would not hold. A second check: feed each non-EvoMaster fuzzer pre-fixed OpenAPI schemas that resolve basePath/server URL ambiguities, to see if the 'schema robustness' explanation for their near-zero results is correct.

Watch

Extended reading notes

Core claim

The authors claim that the three major barriers to rigorous REST API fuzzing comparisons are plumbing problems rather than research gaps: there is no standard way to tell a fuzzer how to authenticate, no standard taxonomy or format for reporting detected fault types, and no shared, scaffolded corpus of real APIs for experiments. WFC attacks the first two by publishing JSON Schemas: an auth schema that declaratively describes static credentials and dynamic token endpoints (including cookie vs. Authorization-header extraction), and a report schema that assigns unique IDs to automated oracles (e.g., F100 for HTTP 500, F101 for schema-validation mismatches, F204–F206 for access-policy violations

Load-bearing premise

The empirical comparison assumes the large performance gap reflects the fuzzers' intrinsic capabilities, even though only EvoMaster received authentication credentials in WFC format and received crash fixes before the runs, while the other tools ran without their native authentication configurations.

Editorial extensions

If this is right

  • If fuzzers adopt WFC, authentication configuration written once for an API can be reused across tools, removing a major source of setup effort and vendor lock-in.
  • WFD gives future studies a common 36-API corpus with working scaffolding, making cherry-picked API subsets harder to justify and enabling community-driven superset growth.
  • A standardized fault report format allows comparisons that go beyond HTTP 500 counts, capturing security-relevant faults such as access-policy violations.
  • The comparison guidelines imply that fuzzer evaluations should measure coverage on the final generated executable tests, not just during the fuzzing session, because the session-time numbers overstate what practitioners receive.
  • The reserved fault-code range (900–999) lets new experimental oracles be reported in WFC format before they are formally added to the catalog.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If RESTler and Schemathesis were given WFC-equivalent credentials in their native script-based form on the 15 auth-requiring WFD APIs, the reported EvoMaster margin could shrink considerably; that is a directly testable re-run.
  • The roughly 20% assertion-failure rate in EvoMaster's generated Python tests points toward flaky-assertion and test-order research, and suggests mutation testing could become a standard quality gate for REST API fuzzers.
  • The community-superset selection rule the paper advocates for REST APIs could transfer to other software testing domains, making 'add new subjects rather than drop old ones' a general expectation for empirical evaluations.
  • WFD's JVM-only composition leaves open whether the findings generalize to non-Java REST stacks; extending the corpus with Python, Go, or Node.js APIs would test that scope.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces Web Fuzzing Commons (WFC), a set of JSON schemas and open-source libraries for declarative authentication configuration and standardized fault reporting for REST API fuzzers, and Web Fuzzing Dataset (WFD), a collection of 36 JVM-based open-source REST APIs with Docker scaffolding, authentication configs, and experiment scripts. To demonstrate usefulness, the authors run six fuzzers (ARAT-RL, EmRest, EvoMaster, LLamaRestTest, RESTler, Schemathesis) on all APIs for one hour each with 10 repetitions, measuring 2xx endpoint coverage, 500-status fault endpoints, and JaCoCo coverage. EvoMaster outperforms the others dramatically, but the authors acknowledge confounds: only EvoMaster received WFC authentication credentials and crash fixes, EmRest ran on different hardware and skipped two APIs. The paper discusses methodological pitfalls (cherry-picking APIs, measuring coverage during fuzzing versus on generated tests, lack of executable tests from many fuzzers, flaky assertions) and provides guidelines for future comparisons.

Significance. If adopted, WFC's authentication schema and fault catalog could reduce setup effort and improve comparability across fuzzers; WFD is the largest open-source REST API benchmark with scaffolding to date. The most valuable contribution may be the methodological discussion in Section 6.2 on coverage measurement and test generation, which is well illustrated with data. However, the empirical demonstration is weakened by confounds: the only fuzzer that supports WFC is the authors' own EvoMaster, and the comparison gives EvoMaster both authenticated access and targeted bug fixes. Thus the paper does not yet establish that WFD enables fair cross-tool comparisons, only that WFC works for EvoMaster. The guidelines are sensible, but they would be more convincing if followed in the paper's own showcase.

major comments (3)
  1. [Section 6, 'Authentication information was provided...'] The comparison is not fair: WFC auth configs were provided only to EvoMaster, and EvoMaster crashes were fixed before the runs, while RESTler/Schemathesis did not receive their native auth scripts and other tools' crashes were not fixed. This contradicts the claim in §6.1 that 'all tools get exactly the same information as input.' The headline margins in Tables 4–6 (EvoMaster 57.2% avg 2xx vs. next 23.6%) are likely driven by these asymmetries. Since the paper presents WFD as facilitating fair comparisons (Abstract, §5.2), the authors must either supply equivalent auth to all tools (e.g., native auth for RESTler/Schemathesis) or explicitly restrict the claim to a demonstration of EvoMaster using WFC. This is load-bearing for the central value proposition.
  2. [Section 6, EmRest hardware and missing APIs] EmRest was run on a different machine (MacBook Pro M4) than all other tools (Windows 11 server), and could not run genome-nexus and ocvn due to missing Docker images. Tables 4–6 still include zeros for these two APIs under EmRest, biasing the aggregated ranks and effect sizes. Missing data should be excluded or reported separately. This is a second confound in the headline comparison.
  3. [Section 6.1, 'As the author of EvoMaster...'] The usefulness demonstration is circular: WFD is an extension of the authors' EMB, and the study fixes bugs only in EvoMaster. The paper acknowledges this bias, but the consequence is that the claim 'any fuzzer can benefit from WFC and WFD' (Abstract) is unsupported. To substantiate this, the authors should integrate WFC into at least one independent fuzzer (or provide a detailed integration guide with a case study). Without this, the evaluation only shows that EvoMaster can use WFC/WFD, not that the artifacts are generally usable.
minor comments (4)
  1. [Section 6.2, Table 7] The pairwise comparison labels X and Y are not identified in the text; although the paper says 'which tools they are is not relevant', readers are left guessing (EvoMaster is X and Schemathesis is Y). Please name them for transparency.
  2. [Section 5.2, Table 3] The 'Total' row reports 6,465 source files and 657,162 LOCs, but summing the visible rows appears inconsistent. Please verify the totals.
  3. [Section 2.1] Typo: 'Representational Sate Transfer' should be 'Representational State Transfer'.
  4. [Section 6.1, Table 9] The fault catalog includes F100/F101 etc., but only EvoMaster's results are shown. It would be helpful to state explicitly that the other fuzzers do not yet produce WFC-format reports, so the table does not permit cross-tool fault comparisons.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: WFC/WFD are self-contained contributions; the disclosed EvoMaster-centric evaluation is a validity threat, not a circular derivation.

full rationale

The paper's central artifacts — the WFC schemas for declarative authentication and standardized fault reports, and the WFD corpus of 36 APIs — are constructed from external specifications, prior literature, and open-source projects, not from the empirical results they are used to produce. The empirical demonstration uses EvoMaster, the authors' own tool and currently the only WFC supporter, and this is explicitly disclosed: "Authentication information was provided in the WFC format presented in this paper, which is currently supported only by EvoMaster. Technically, this gives EvoMaster a possibly unfair advantage over other tools that have their own custom methods for authentication, like RESTler and Schemathesis" (Section 6). Similarly, the authors admit differential crash fixing: "we made sure that, if there was any crash, we fixed them before finalizing the experiments" (Section 6.1). These are serious threats to the fairness of the tool comparison, and the paper's own discussion narrows the claims accordingly. However, they are experimental confounds, not circularity: no reported quantity is defined in terms of an input, no fitted parameter is renamed as a prediction, and no load-bearing conclusion rests solely on a self-citation. EvoMaster's inclusion is additionally justified by independent studies ([57], [75]), and the paper explicitly disclaims the goal of determining the best fuzzer: "Our goal is not to study which fuzzer technique is best." WFD is transparently described as an updated version of the authors' own EMB corpus, so there is no hidden renaming or undisclosed reuse. The self-citations are contextual and corroborated by independent evidence; they do not reduce the central claims to their own assumptions.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

The paper's contributions are artifacts and an evaluation protocol; it introduces no fitted numeric parameters. Its main axiomatic commitments are the representativeness of the 36-API corpus, the methodological claim that generated tests are the correct evaluation unit, and the completeness of the WFC authentication schema. The fault catalog is a new taxonomy with no independent evidence.

assumptions (3)
  • domain assumption The 36 open-source JVM REST APIs in WFD are a representative sample of real-world REST APIs for evaluating fuzzers.
    Section 5.2 argues that Java-only is a reasonable compromise, but the paper itself warns that results may not generalize to other languages or industrial APIs.
  • domain assumption Black-box fuzzer comparisons should be based on the executable test suites generated by the fuzzer, not on coverage collected during the fuzzing session.
    Section 6.2 makes this methodological claim; it is a stance that not all researchers in the field share, and it is load-bearing for the paper's critique of prior studies.
  • ad hoc to paper The WFC authentication schema covers the authentication mechanisms needed for real-world REST APIs.
    Section 4.1 states the schema was designed from mechanisms encountered in EvoMaster's industrial collaborations and in the selected open-source APIs, and invites extension for unhandled mechanisms, conceding incompleteness.
invented entities (1)
  • WFC fault catalog codes (F100, F101, F200-F206, with 900-999 reserved for experimental oracles)
    purpose: Assign unique identifiers to automated oracles so that fault detections from different fuzzers can be compared in a standardized way.
    This is a new taxonomy proposed by the paper; there is no external validation of its completeness or orthogonality, and only EvoMaster currently implements it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of WFC/WFD: Web Fuzzing Commons, Dataset and Guidelines to Support Experimentation in REST API Fuzzing." pith.science (2026). https://pith.science/paper/ZSHTGW4M

@misc{pith2026250901612,
  author       = {Pith},
  title        = {Pith review of: WFC/WFD: Web Fuzzing Commons, Dataset and Guidelines to Support Experimentation in REST API Fuzzing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZSHTGW4M}},
  note         = {Machine review of arXiv:2509.01612}
}
read the original abstract

Fuzzing REST APIs is an important research problem, with practical applications and impact in industry. As such, a lot of research work has been carried out on this topic in the last few years. However, there are three major issues that hinder further progress: how to deal with API authentication; how to catalog and compare different fault types found by different fuzzers; and what to use as case study to facilitate fair comparisons among fuzzers. To address these important challenges, we present Web Fuzzing Commons (WFC) and Web Fuzzing Dataset (WFD). WFC is a set of open-source libraries and schema definitions to declaratively specify authentication info and catalog different types of faults that fuzzers can automatically detect. WFD is a collection of 36 open-source APIs with all necessary scaffolding to easily run experiments with fuzzers, supported by WFC. To show the usefulness of WFC/WFD, a set of experiments is carried out with EvoMaster, a state-of-the-art fuzzer for Web APIs. However, any fuzzer can benefit from WFC and WFD. We compare EvoMaster with other state-of-the-art tools such as ARAT-RL, EmRest, LLamaRestTest, RESTler, and Schemathesis. We discuss common pitfalls in tool comparisons, as well as providing guidelines with support of WFC/WFD to avoid them.

Figures

Figures reproduced from arXiv: 2509.01612 by the authors.

Figure 1
Figure 1. Snippet of Petstore - OpenAPI 3.0 in YAML format [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Authentication configuration in YAML for the API called [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Beginning of a test case generated by EvoMaster in Java, in which authentication information has been exploited to make dynamic calls. 1 auth : 2 − name : admin 3 loginEndpointAuth : 4 payloadRaw : ”{\” usernameOrEmail \ ”: \”admin \” , \” password \ ”: \” bar123 \”}” 5 − name : u s e r 6 loginEndpointAuth : 7 payloadRaw : ”{\” usernameOrEmail \ ”: \” u s e r \” , \” password \ ”: \” bar123 \”}” 8 9 authTemplate : 1… view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Authentication configuration in YAML for the API called [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Test case generated by EvoMaster in Python, in which authentication information has been exploited to make dynamic calls. on, and all industrial APIs used by all of our industrial partners now and in the past (e.g., currently Fortune 500 enterprises such as Meituan and…
Figure 6
Figure 6. Figure 6: Snippet of JSON Schema definition for our authentication configurations. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Snippet of JSON Schema definition for our test reports. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Screenshot of files generated as output of [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Results of opening the index.html file in an HTTP server, e.g., by clicking on webreport.command. are provided [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Results of opening the Endpoints tab, providing details per endpoint [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Results of opening the details for a specific endpoint. [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Code of a test case, visualized in the WFC Report. [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: List of usability requirements for selection of tools in empirical comparisons of REST APIs. 5.4 Fuzzer Usability Requirements Regardless of which criteria are used to select fuzzers for comparisons, those selected fuzzers must be usable. Being open-source, and with a…
Figure 14
Figure 14. Figure 14: Custom configurations to run experiments with ARAT-RL. [PITH_FULL_IMAGE:figures/full_fig_p023_14.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Using OAI Overlay to Enhance REST API Fuzzing

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Native Overlay support in EvoMaster lets testers supply named examples that improve black-box REST API fuzzing results on industrial systems without vendor-specific formats.

Reference graph

Works this paper leans on

91 extracted references · 73 canonical work pages · cited by 1 Pith paper

  1. [1]

    APIs.guru

    [n.d.]. APIs.guru. https://apis.guru/. Online, Accessed August 6, 2024

    2024

  2. [2]

    RapidAPI

    [n.d.]. RapidAPI. https://rapidapi.com/. Online, Accessed August 6, 2024

    2024

  3. [3]

    Dario Amoroso d’Aragona, Alexander Bakhtin, Xiaozhou Li, Ruoyu Su, Lauren Adams, Ernesto Aponte, Francis Boyle, Patrick Boyle, Rachel Koerner, Joseph Lee, et al . 2024. A dataset of microservices-based open-source projects. In Proceedings of the 21st International Conference on Mining Software Repositories. 504--509

    2024

  4. [4]

    Andrea Arcuri. 2017. RESTful API Automated Test Case Generation. In IEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 9--20

    2017

  5. [5]

    Andrea Arcuri. 2018. Test suite generation with the Many Independent Objective (MIO) algorithm. Information and Software Technology104 (2018), 195--206

    2018

  6. [6]

    Andrea Arcuri. 2019. RESTful API Automated Test Case Generation with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM)28, 1 (2019), 3

    2019

  7. [7]

    Andrea Arcuri. 2020. Automated Black-and White-Box Testing of RESTful APIs With EvoMaster. IEEE Software 38, 3 (2020), 72--78

    2020

  8. [8]

    Arcuri and L

    A. Arcuri and L. Briand. 2014. A Hitchhiker’s Guide to Statistical Tests for Assessing Randomized Algorithms in Software Engineering. Software Testing, Verification and Reliability (STVR)24, 3 (2014), 219--250

    2014

Show all 91 references
  1. [9]

    Galeotti

    Andrea Arcuri and Juan P. Galeotti. 2019. SQL Data Generation to Enhance Search-Based System Testing. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO ’19). Association for Computing Machinery, New York, NY, USA, 1390–1398. https://doi.org/10.1145/3...

    2019

  2. [10]

    Andrea Arcuri and Juan P Galeotti. 2020. Handling SQL databases in automated system test generation. ACM Transactions on Software Engineering and Methodology (TOSEM)29, 4 (2020), 1--31

    2020

  3. [11]

    Andrea Arcuri and Juan P Galeotti. 2020. Testability transformations for existing APIs. In 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST). IEEE, 153--163

    2020

  4. [13]

    Andrea Arcuri and Juan P Galeotti. 2021. Enhancing Search-based Testing with Testability Transformations for Existing APIs. ACM Transactions on Software Engineering and Methodology (TOSEM)31, 1 (2021), 1--34

    2021

  5. [14]

    Andrea Arcuri, Philip Garrett, Juan Pablo Galeotti, and Man Zhang. 2025. Widening The Adoption of Web API Fuzzing: Docker, GitHub Action and Python Support for EvoMaster. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering. 1084--1088

    2025

  6. [15]

    Arcuri, A

    A. Arcuri, A. Poth, and O. Rrjolli. 2025. Introducing Black-Box Fuzz Testing for REST APIs in Indus- try: Challenges and Solutions. In IEEE International Conference on Software Testing, Verification and Validation (ICST)

    2025

  7. [16]

    Andrea Arcuri, Omur Sahin, and Man Zhang. 2025. Fuzzing for Detecting Access Policy Violations in REST APIs. In IEEE International Symposium on Software Reliability Engineering (ISSRE)

    2025

  8. [17]

    Andrea Arcuri, Man Zhang, and Juan Pablo Galeotti. 2024. Advanced White-Box Heuristics for Search- Based Fuzzing of REST APIs. ACM Transactions on Software Engineering and Methodology (TOSEM) (2024). https://doi.org/10.1145/3652157

    2024 doi

  9. [18]

    Andrea Arcuri, Man Zhang, Amid Golmohammadi, Asma Belhadi, Juan P Galeotti, Bogdan Marculescu, and Susruthan Seran. 2023. EMB: A curated corpus of web/enterprise applications and library support for software testing research. In 2023 IEEE Conference on Software Testing, Verifi...

    2023

  10. [19]

    Andrea Arcuri, Man Zhang, Susruthan Seran, Asma Belhadi, Juan Pablo Galeotti, Bogdan, Amid Gol- mohammadi, Onur Duman, Agustina Aldasoro, Philip, Alberto Mart ´ ın L´ opez, Hernan Ghianni,¨Om¨ ur S ¸ahin, Annibale Panichella, Kyle Niemeyer, and Marcello Maugeri. 2025. WebFuzzi...

    2025 doi

  11. [20]

    Andrea Arcuri, Man Zhang, Susruthan Seran, Juan Pablo Galeotti, Amid Golmohammadi, Onur Duman, Agustina Aldasoro, and Hernan Ghianni. 2025. Tool report: EvoMaster—black and white box search-based fuzzing for REST, GraphQL and RPC APIs. Automated Software Engineering32, 1 (2025), 1--11

    2025

  12. [21]

    Andrea Arcuri, Man Zhang, ¨Om¨ urS ¸ahin, Amid Golmohammadi, Asma Belhadi, Susruthan Seran, Onur Duman, Juan Pablo Galeotti, Gebremariam Mesfin Assres, and Hernan Ghianni. 2025. WebFuzzing/Dataset: v4.0.0. https://doi.org/10.5281/zenodo.16949375

    2025 doi

  13. [22]

    Andrea Arcuri and ¨Om¨ urS ¸ahin. 2025. WebFuzzing/Commons: v0.1.0. https://doi.org/10.5281/zeno do.16948758

    2025 doi

  14. [23]

    Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. In ACM/IEEE International Conference on Software Engineering (ICSE). 748–758

    2019

  15. [24]

    Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2020. Checking security properties of cloud service rest apis. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 387--397

    2020

  16. [25]

    Asma Belhadi, Man Zhang, and Andrea Arcuri. 2023. Random Testing and Evolutionary Testing for Fuzzing GraphQL APIs. ACM Transactions on the Web(2023)

    2023

  17. [26]

    Marcel B¨ ohme, L´ aszl´ o Szekeres, and Jonathan Metzman. 2022. On the reliability of coverage-based fuzzer benchmarking. In Proceedings of the 44th International Conference on Software Engineering. 1621--1633

    2022

  18. [27]

    Sujit Kumar Chakrabarti and Prashant Kumar. 2009. Test-the-rest: An approach to testing restful web-services. In Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns, 2009. COMPUTATIONWORLD’09. Computation World:. IEEE, 302--308

    2009

  19. [28]

    Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: a practical mutation testing tool for Java. In ACM Int. Symposium on Software Testing and Analysis (ISSTA). ACM, 449--452

    2016

  20. [29]

    Davide Corradini, Zeno Montolli, Michele Pasqua, and Mariano Ceccato. 2024. DeepREST: Automated Test Case Generation for REST APIs Exploiting Deep Reinforcement Learning. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1383--1394

    2024

  21. [30]

    Davide Corradini, Zeno Montolli, Michele Pasqua, and Mariano Ceccato. 2024. Replication package for the paper ”DeepREST: Automated Test Case Generation for REST APIs Exploiting Deep Reinforcement Learning”. https://doi.org/10.5281/zenodo.13745242

    2024 doi

  22. [31]

    Davide Corradini, Michele Pasqua, and Mariano Ceccato. 2023. Automated black-box testing of mass assignment vulnerabilities in RESTful APIs. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2553--2564

    2023

  23. [32]

    Davide Corradini, Amedeo Zampieri, Michele Pasqua, and Mariano Ceccato. 2021. Restats: A test coverage tool for RESTful APIs. In 2021 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 594--598

    2021

  24. [33]

    Davide Corradini, Amedeo Zampieri, Michele Pasqua, Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2022. Automated black-box testing of nominal and error scenarios in RESTful APIs. Software Testing, Verification and Reliability(2022), e1808

    2022

  25. [34]

    Alix Decrop, Sara Eraso, Xavier Devroey, and Gilles Perrouin. 2025. A Public Benchmark of REST APIs. In 2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR). IEEE, 421--433

    2025

  26. [35]

    Gelei Deng, Zhiyi Zhang, Yuekang Li, Yi Liu, Tianwei Zhang, Yang Liu, Guo Yu, and Dongjin Wang

  27. [36]

    Sergio Di Meglio, Luigi Libero Lucio Starace, and Sergio Di Martino. 2023. Starting a New REST API Project? A Performance Benchmark of Frameworks and Execution Environments.. In IWSM-Mensura. 30

    2023

  28. [37]

    H. Do, S. Elbaum, and G. Rothermel. 2005. Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact. Empirical Software Engineering10, 4 (2005), 405--435

    2005

  29. [38]

    Wenlong Du, Jian Li, Yanhao Wang, Libo Chen, Ruijie Zhao, Junmin Zhu, Zhengguang Han, Yijun Wang, and Zhi Xue. 2024. Vulnerability-oriented testing for restful apis. In 33rd USENIX Security Symposium (USENIX Security 24). USENIX Association, 739--755

    2024

  30. [39]

    Mat´ uˇ s Ferech and Pavel Tvrd ´ ık. 2023. Efficient fuzz testing of web services. In2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS). IEEE, 291--300

    2023

  31. [40]

    Roy Thomas Fielding. 2000. Architectural styles and the design of network-based software architectures. Ph.D. Dissertation. University of California, Irvine

    2000

  32. [41]

    Myles Foley and Sergio Maffeis. 2025. APIRL: Deep Reinforcement Learning for REST API Fuzzing. In Thirty-ninth Conference on Artificial Intelligence (AAAI 2025)

    2025

  33. [42]

    Fraser and A

    G. Fraser and A. Arcuri. 2012. Sound Empirical Evidence in Software Testing. In ACM/IEEE International Conference on Software Engineering (ICSE). 178--188

    2012

  34. [43]

    Gordon Fraser and Andrea Arcuri. 2014. A large-scale evaluation of automated unit test generation using EvoSuite. ACM Transactions on Software Engineering and Methodology (TOSEM)24, 2 (2014), 8

    2014

  35. [44]

    Gregory Gay and Ren´ e Just. 2020. Defects4J as a challenge case for the search-based software engineering community. In International Symposium on Search Based Software Engineering. Springer, 255--261

    2020

  36. [45]

    Hernan Ghianni, Man Zhang, Juan P Galeotti, and Andrea Arcuri. 2025. Search-Based Fuzzing For RESTful APIs That Use MongoDB. arXiv preprint arXiv:2507.20848(2025)

    2025 arXiv

  37. [46]

    Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2023. .NET/C# instrumentation for search-based software testing. Software Quality Journal(2023), 1--27

    2023

  38. [47]

    Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2023. Testing RESTful APIs: A Survey. ACM Transactions on Software Engineering and Methodology(aug 2023). https://doi.org/10.1145/3617175

    2023 doi

  39. [48]

    P´ eter Gyimesi, B´ ela Vancsics, Andrea Stocco, Davood Mazinanian, Arp´ ad Besz´ edes, Rudolf Ferenc, and Ali Mesbah. 2019. BugsJS: a benchmark of JavaScript bugs. In 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE, 90--101

    2019

  40. [49]

    Zac Hatfield-Dodds and Dmitry Dygalo. 2022. Deriving Semantics-Aware Fuzzers from Web API Schemas. In 2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 345--346

    2022

  41. [50]

    Ahmad Hazimeh, Adrian Herrera, and Mathias Payer. 2020. Magma: A ground-truth fuzzing benchmark. Proceedings of the ACM on Measurement and Analysis of Computing Systems4, 3 (2020), 1--29

    2020

  42. [51]

    Monica Hutchins, Herb Foster, Tarak Goradia, and Thomas Ostrand. 1994. Experiments on the effectiveness of dataflow-and control-flow-based test adequacy criteria. In Proceedings of 16th International conference on Software engineering. IEEE, 191--200

    1994

  43. [52]

    Yue Jia and Mark Harman. 2011. An analysis and survey of the development of mutation testing. IEEE Transactions on Software Engineering (TSE)37, 5 (2011), 649--678

    2011

  44. [53]

    Ren´ e Just, Darioush Jalali, and Michael D Ernst. 2014. Defects4J: A database of existing faults to enable controlled testing studies for Java programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis. 437--440

    2014

  45. [54]

    Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2023. Adaptive rest api testing with reinforcement learning. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 446--458

    2023

  46. [55]

    Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2025. LlamaRestTest: Effective REST API Testing with Small Language Models. In ACM Symposium on the Foundations of Software Engineering (FSE)

    2025

  47. [56]

    Myeongsoo Kim, Tyler Stennett, Saurabh Sinha, and Alessandro Orso. 2025. A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs. ACM/IEEE International Conference on Software Engineering (ICSE)(2025). 31

    2025

  48. [57]

    Myeongsoo Kim, Qi Xin, Saurabh Sinha, and Alessandro Orso. 2022. Automated Test Generation for REST APIs: No Time to Rest Yet. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2022). Association for Computing Machinery, New...

    2022

  49. [58]

    George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018. Evaluating fuzz testing. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security. 2123--2138

    2018

  50. [59]

    Nuno Laranjeiro, Jo˜ ao Agnelo, and Jorge Bernardino. 2021. A black box tool for robustness testing of REST services. IEEE Access 9 (2021), 24738--24754

    2021

  51. [60]

    Tri Le, Thien Tran, Duy Cao, Vy Le, Tien N Nguyen, and Vu Nguyen. 2024. KAT: Dependency-aware automated API testing with large language models. In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 82--92

    2024

  52. [61]

    Yuwei Li, Shouling Ji, Yuan Chen, Sizhuang Liang, Wei-Han Lee, Yueyao Chen, Chenyang Lyu, Chunming Wu, Raheem Beyah, Peng Cheng, et al . 2021. {UNIFUZZ}: A holistic and pragmatic {Metrics-Driven} platform for evaluating fuzzers. In 30th USENIX Security Symposium (USENIX Securi...

    2021

  53. [62]

    Zhuo Li, Yuechen Wu, Lei Ma, Xiaofei Xie, Yingfeng Chen, and Changjie Fan. 2022. GBGallery: A benchmark and framework for game testing. Empirical Software Engineering27, 6 (2022), 1--27

    2022

  54. [63]

    Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: Model-based RESTful API Testing with Execution Feedback. In ACM/IEEE International Conference on Software Engineering (ICSE)

    2022

  55. [64]

    Chenyang Lyu, Jiacheng Xu, Shouling Ji, Xuhong Zhang, Qinying Wang, Binbin Zhao, Gaoning Pan, Wei Cao, Peng Chen, and Raheem Beyah. 2023. MINER: A Hybrid Data-Driven Approach for REST API Fuzzing. In 32nd USENIX Security Symposium (USENIX Security 23). 4517--4534

    2023

  56. [65]

    Bogdan Marculescu, Man Zhang, and Andrea Arcuri. 2022. On the Faults Found in REST APIs by Automated Test Generation. ACM Transactions on Software Engineering and Methodology (TOSEM)31, 3 (2022), 1--43

    2022

  57. [66]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cort´ es. 2021. RESTest: Automated Black-Box Testing of RESTful Web APIs. In ACM Int. Symposium on Software Testing and Analysis (ISSTA). ACM, 682--685

    2021

  58. [67]

    Jonathan Metzman, L´ aszl´ o Szekeres, Laurent Simon, Read Sprabery, and Abhishek Arya. 2021. FuzzBench: an open fuzzer benchmarking platform and service. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations ...

    2021

  59. [68]

    Miao Miao, Sriteja Kummita, Eric Bodden, and Shiyi Wei. 2025. Program Feature-Based Benchmarking for Fuzz Testing. Proceedings of the ACM on Software Engineering2, ISSTA (2025), 527--549

    2025

  60. [69]

    Sam Newman. 2021. Building microservices. ” O’Reilly Media, Inc.”

    2021

  61. [70]

    Jiradet Ounjai, Valentin W¨ ustholz, and Maria Christakis. 2023. Green fuzzer benchmarking. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1396--1406

    2023

  62. [71]

    Alexander Poth, Olsi Rrjolli, and Andrea Arcuri. 2025. Technology adoption performance evaluation applied to testing industrial REST APIs. Automated Software Engineering32, 1 (2025), 5

    2025

  63. [72]

    RV Rajesh. 2016. Spring Microservices. Packt Publishing Ltd

    2016

  64. [73]

    Diptikalyan Saha, Devika Sondhi, Swagatam Haldar, and Saurabh Sinha. 2025. Rest API Functional Tester. In Proceedings of the 18th Innovations in Software Engineering Conference. 1--11

    2025

  65. [74]

    June Sallou, Thomas Durieux, and Annibale Panichella. 2024. Breaking the silence: the threats of using llms in software engineering. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results. 102--106

    2024

  66. [75]

    Hassan Sartaj, Shaukat Ali, and Julie Marie Gjøby. 2024. REST API Testing in DevOps: A Study on an Evolving Healthcare IoT Application. arXiv:cs.SE/2410.12547 https://arxiv.org/abs/2410.12547 32

    2024 arXiv

  67. [76]

    Susruthan Seran, Man Zhang, Onur Duman, and Andrea Arcuri. 2025. Handling Web Service Interactions in Fuzzing with Search-Based Mock-Generation. ACM Transactions on Software Engineering and Methodology (2025)

    2025

  68. [77]

    Devika Sondhi, Ananya Sharma, and Diptikalyan Saha. 2025. Utilizing API Response for Test Refinement. arXiv preprint arXiv:2501.18145(2025)

    2025 arXiv

  69. [78]

    Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTESTGEN: Automated Black- Box Testing of RESTful APIs. In IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE

    2020

  70. [79]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering 50, 4 (2024), 911--936

    2024

  71. [80]

    Yu Wang and Yue Xu. 2024. Beyond REST: Introducing APIF for Comprehensive API Vulnerability Fuzzing. In Proceedings of the 27th International Symposium on Research in Attacks, Intrusions and Defenses. 435--449

    2024

  72. [81]

    Dylan Wolff, Marcel B¨ ohme, and Abhik Roychoudhury. 2025. Fuzzing: On Benchmarking Outcome as a Function of Benchmark Properties. ACM Transactions on Software Engineering and Methodology(2025)

    2025

  73. [82]

    Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial Testing of RESTful APIs. In ACM/IEEE International Conference on Software Engineering (ICSE)

    2022

  74. [83]

    Lixin Xu, Huayao Wu, Zhenyu Pan, Tongtong Xu, Shaohua Wang, Xintao Niu, and Changhai Nie

  75. [84]

    Ke Zhang, Chenxi Zhang, Chong Wang, Chi Zhang, YaChen Wu, Zhenchang Xing, Yang Liu, Qingshan Li, and Xin Peng. 2025. LogiAgent: Automated Logical Testing for REST Systems with LLM-Based Multi-Agents. arXiv preprint arXiv:2503.15079(2025)

    2025 arXiv

  76. [85]

    Man Zhang and Andrea Arcuri. 2021. Adaptive Hypermutation for Search-Based System Test Generation: A Study on REST APIs with EvoMaster. ACM Transactions on Software Engineering and Methodology (TOSEM) 31, 1 (2021)

    2021

  77. [86]

    Man Zhang and Andrea Arcuri. 2023. Open Problems in Fuzzing RESTful APIs: A Comparison of Tools. ACM Transactions on Software Engineering and Methodology (TOSEM)(may 2023). https: //doi.org/10.1145/3597205

    2023 doi

  78. [87]

    Man Zhang, Andrea Arcuri, Yonggang Li, Yang Liu, and Kaiming Xue. 2023. White-Box Fuzzing RPC- Based APIs with EvoMaster: An Industrial Case Study. ACM Transactions on Software Engineering and Methodology 32, 5 (2023), 1--38

    2023

  79. [88]

    Man Zhang, Andrea Arcuri, Yonggang Li, Yang Liu, Kaiming Xue, Zhao Wang, Jian Huo, and Weiwei Huang. 2025. Fuzzing microservices: A series of user studies in industry on industrial systems with evomaster. Science of Computer Programming(2025), 103322

    2025

  80. [89]

    Man Zhang, Andrea Arcuri, Piyun Teng, Kaiming Xue, and Wenhao Wang. 2024. Seeding and Mocking in White-Box Fuzzing Enterprise RPC APIs: An Industrial Case Study. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2024--2034

    2024

  81. [90]

    Man Zhang, Asma Belhadi, and Andrea Arcuri. 2023. JavaScript SBST Heuristics To Enable Effective Fuzzing of NodeJS Web APIs. ACM Transactions on Software Engineering and Methodology(2023). 33 Table 5: Average results, out of 10 runs, for the number of endpoints in which at lea...

    2023

  82. [2023]

    In 32nd USENIX Security Symposium (USENIX Security 23)

    {NAUTILUS}: Automated {RESTful}{API} Vulnerability Detection. In 32nd USENIX Security Symposium (USENIX Security 23). 5593--5609

  83. [2025]

    Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1978--2000

    Effective REST APIs Testing with Error Message Analysis. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1978--2000

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.