Pith. sign in

REVIEW 2 major objections 4 minor 35 references

A standard Overlay format lets testers feed domain examples into REST API fuzzers without vendor lock-in, and industrial runs show it raises successful endpoint coverage.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 20:02 UTC pith:UFA2SEZR

load-bearing objection Solid industrial feasibility study of Overlay as a portable way to feed examples into REST fuzzers; the Table 1 gains are real but do not isolate the standard from the hand-written examples. the 2 major comments →

arxiv 2607.04325 v1 pith:UFA2SEZR submitted 2026-07-05 cs.SE

Using OAI Overlay to Enhance REST API Fuzzing

Omur Sahin , Man Zhang , Alexander Poth , Olsi Rrjolli , Andreas Faes , Piyun Teng , Kaiming Xue , Wenjuan Ma
show 1 more author
Andrea Arcuri
This is my paper
classification cs.SE
keywords RESTAPIfuzzingblack-boxOpenAPIOverlaytestingexamples
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Black-box REST API fuzzers usually start from an OpenAPI schema alone, which is often too thin to produce the right parameter combinations or realistic identifiers. Testers therefore need a way to inject extra example values, yet custom per-fuzzer formats create lock-in and extra learning cost. This paper shows that the OpenAPI Initiative Overlay standard can carry those examples as named transformations on the schema. After native Overlay support was added to a mature fuzzer, engineers at five companies of widely different sizes wrote Overlay files for one of their own APIs and ran matched ten-minute sessions. In every case the Overlay-augmented runs matched or beat baseline coverage of successful responses and, on three APIs, also found additional faults. The work therefore argues that Overlay is already a practical, standards-based channel for domain knowledge in industrial black-box fuzzing.

Core claim

Industrial experiments on five live APIs demonstrate that supplying domain examples through OAI Overlay files measurably improves the black-box coverage and fault-finding power of a REST fuzzer relative to the same fuzzer run without those examples.

What carries the argument

OAI Overlay actions that insert named “examples” (and same-name combinations) into an OpenAPI schema; the fuzzer samples those named values, treats each named combination-plus-status-code as an archive target, and surfaces their use in an interactive report.

Load-bearing premise

That a single ten-minute run performed by the same engineers who wrote the Overlay files is enough to credit the Overlay mechanism itself, rather than the quality of the hand-chosen examples or the particular APIs chosen.

What would settle it

Re-run the identical five APIs for many independent longer sessions (or with other fuzzers that only consume a post-merge schema) and show that the coverage and fault gains disappear once the same examples are supplied without Overlay.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any fuzzer that already understands OpenAPI examples can immediately accept Overlay-supplied data without inventing a proprietary hint language.
  • Testers can keep example data in separate, version-controlled Overlay files that survive schema regenerations.
  • Named-example combinations give a lightweight way to express multi-parameter constraints that the schema itself does not capture.
  • Multiple Overlay files can be organised by feature area and applied together, scaling beyond a single monolithic file.
  • Generated test reports can highlight which supplied examples actually appeared, aiding review with product owners.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same Overlay mechanism could later carry “links” or workflow sketches, turning it into a broader channel for domain knowledge beyond scalar examples.
  • Because Overlay is an independent standard, the same files could be shared across fuzzers, security scanners and mock servers once more tools add native support.
  • The modest gains on the smallest and on the buggy-fuzzer APIs suggest Overlay is most valuable precisely when the fuzzer’s own search is already competent but still data-starved.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes using the OAI Overlay standard to inject named 'examples' into OpenAPI schemas so that black-box REST API fuzzers can be steered with domain-specific test data without permanently editing the schema or inventing a proprietary seed format. The authors implement native Overlay support (including multi-parameter named-example combinations and archive retention) in EvoMaster via an open-source JVM library, and evaluate it on five industrial APIs from five enterprises of different sizes and geographies. Table 1 reports that 10-minute Overlay-augmented runs improve 2xx (or success-code) endpoint coverage relative to baseline EvoMaster on four of the five APIs, with practitioner feedback that Overlay is usable once learned. The central claim is that Overlay is a viable, maintainable industrial solution for supplying such hints.

Significance. If the result holds, the work offers a practical, standards-based answer to a recurring industrial pain point: how to give black-box REST fuzzers domain knowledge without vendor lock-in or schema pollution. Strengths that should be credited include the open-source overlay-jvm library, native multi-example combination handling and web-report tracing inside EvoMaster, and a genuinely multi-enterprise industrial evaluation (Fortune-500 firms plus a three-person startup) that is rare in the REST-fuzzing literature. Even if the quantitative gains are modest and confounded, the engineering artifact and the demonstration that practitioners can write Overlay files for real APIs remain useful contributions to the tooling ecosystem.

major comments (2)
  1. [§5, Table 1] §5 and Table 1: the only quantitative evidence is five single 10-minute with/without Overlay runs. There are no repeated trials, no variance estimates, and no statistical tests. Given that two APIs already saturate or are limited by unrelated fuzzer bugs (A3 PATCH media-type fault, A4 full coverage without Overlay), the directional improvements cannot be treated as robust evidence that Overlay itself improves fuzzing effectiveness.
  2. [Abstract, §1, §5] Abstract, §1 and §5: the design confounds the Overlay delivery mechanism with the quality of the hand-crafted examples. The authors themselves note that, from the fuzzer’s perspective, Overlay is equivalent to manually editing examples into a schema copy (already shown useful in prior work [2]). Without a third arm that injects identical example values by a non-Overlay route (schema mutation or proprietary seed file), the claim that “Overlay is a viable solution” cannot isolate the standard as the causal factor; the observed lift is most parsimoniously attributed to the examples themselves.
minor comments (4)
  1. [Table 1] Table 1 column headers (B-2xx / O-2xx) are never defined in the caption; a one-sentence clarification that they count endpoints returning at least one 2xx (or success code for A2) would help.
  2. [§4] §4: the probability P used for named-example combination sampling is mentioned only as “e.g., P = 0.5”; stating the actual default used in the experiments would improve reproducibility.
  3. [§4, Figure 4] Figure 4 is referenced but its content is not described in the text; a short caption explaining what the “Examples” view shows would make the usability claim more concrete.
  4. [§2, §4] Several self-citations to concurrent or in-press EvoMaster papers ([2], [3], [25]) are used to justify baseline behaviour; ensuring that the essential claims are self-contained would help readers who cannot access those works.

Circularity Check

0 steps flagged

No circularity: empirical with/without Overlay comparison uses independent practitioner-written inputs and does not define metrics in terms of the claimed result.

full rationale

The paper is an industrial feasibility study of native Overlay support in EvoMaster, not a derivation of a mathematical or predictive claim. The load-bearing evidence is Table 1 (Section 5): five single 10-minute black-box runs comparing baseline EvoMaster against the same fuzzer plus Overlay files that inject named examples. Those Overlay files are external artifacts written by the industrial partners; the 2xx/success-code and fault counts are measured outputs of the runs, not quantities defined from the Overlay content or fitted parameters. Self-citations (e.g., to prior EvoMaster papers [2,3,6,25]) supply the baseline tool and reporting infrastructure; they do not supply the Overlay result itself, nor do they invoke uniqueness theorems or ansätze that force the observed gains. From the fuzzer’s perspective the authors themselves note that Overlay is equivalent to manually editing examples into a schema copy—an already-known technique—so the experiment simply re-uses that technique under a standardized delivery format. No equation, coverage metric, or “prediction” reduces by construction to its own inputs. The design may confound the value of the examples with the value of the Overlay standard (a validity concern), but that is not circularity under the stated criteria. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The work is an empirical tool-extension study rather than a formal derivation. It rests on the existing OpenAPI and Overlay specifications, the prior capabilities of EvoMaster, and the assumption that the five chosen industrial APIs and the hand-written Overlay files are representative enough to support a feasibility claim. No free parameters are fitted; no new physical or mathematical entities are postulated.

axioms (3)
  • domain assumption OpenAPI 3.1 named examples and the OAI Overlay 1.1.0 transformation rules behave as specified by their standards documents.
    The entire technique is defined by applying those standards; correctness of the merge library and of EvoMaster’s sampling logic depends on them (§3–4).
  • ad hoc to paper A 10-minute black-box run of EvoMaster is long enough for differences caused by example injection to become visible.
    All comparisons use a fixed 10-minute budget chosen by the authors (§5); longer or shorter budgets could change the observed deltas.
  • domain assumption The five industrial APIs selected by the partner companies are sufficiently representative of industrial REST APIs for a feasibility claim.
    External validity rests on this selection; the paper itself notes threats to generalization (§6).

pith-pipeline@v1.1.0-grok45 · 15777 in / 2327 out tokens · 25920 ms · 2026-07-11T20:02:42.862336+00:00 · methodology

0 comments
read the original abstract

REST APIs are widely used in industry. Therefore, a lot of research has been focused on how to automatically generate test cases for REST APIs, with few different open-source fuzzers existing in the literature. For a thorough testing, especially in black-box scenarios, just relying on the information provided in the OpenAPI schemas is not enough. Testers typically need to provide extra input data to help steer the fuzzers in the right direction. Dedicated formats specific to each different fuzzer would work, but they would create a vendor lock-in, as well as increasing cognitive load. The OpenAPI Initiative (OAI) standard Overlay might be a solution to this problem. Such standard enables to define transformations on the OpenAPI schemas, where testers can provide input data in Overlay files where such data is provided as ``examples'' entries. In this paper, we have extended the state-of-the-art fuzzer EvoMaster to support Overlay files natively. Experiments are carried out in industry on five APIs from five enterprises from around the world (e.g., Belgium, China, Germany and T\"urkiye), including two Fortune500 enterprises as well as a 3-man startup. Our industrial results show that Overlay is a viable solution to better enable black-box fuzzing of REST APIs in industry.

Figures

Figures reproduced from arXiv: 2607.04325 by Alexander Poth, Andrea Arcuri, Andreas Faes, Kaiming Xue, Man Zhang, Olsi Rrjolli, Omur Sahin, Piyun Teng, Wenjuan Ma.

Figure 1
Figure 1. Figure 1: Artificial example of an OpenAPI schema defining one [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overlay file definition used to add four ‘‘examples’’ entries. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Schema result of applying Overlay transformation in Figure 2 on the schema defined in Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Screenshot of WFC Web Report opened on the ‘‘Examples’’ view. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 2 canonical work pages

  1. [1]

    Andrea Arcuri. 2017. RESTful API Automated Test Case Generation. InIEEE International Conference on Software Quality, Reliability and Security (QRS). IEEE, 9--20

    2017

  2. [2]

    Arcuri, A

    A. Arcuri, A. Poth, and O. Rrjolli. 2025. Introducing Black-Box Fuzz Testing for REST APIs in Indus- try: Challenges and Solutions. InIEEE International Conference on Software Testing, Verification and Validation (ICST)

    2025

  3. [3]

    Andrea Arcuri, Alexander Poth, Olsi Rrjolli, Philip Garrett, and Juan P Galeotti. 2026. Fuzzing REST APIs in Industry: Necessary Features and Open Problems.arXiv preprint arXiv:2604.01759(2026)

    arXiv 2026

  4. [4]

    Andrea Arcuri, Omur Sahin, and Man Zhang. 2025. Fuzzing for Detecting Access Policy Violations in REST APIs. InIEEE International Symposium on Software Reliability Engineering (ISSRE). 7https://github.com/WebFuzzing/EvoMaster 8

    2025

  5. [5]

    2026.WebFuzzing/EvoMaster: v5.0.2.https://doi.org/10.5281/zenodo.18262322

    Andrea Arcuri, Man Zhang, Susruthan Seran, Asma Belhadi, Juan Pablo Galeotti, Bogdan, Amid Golmo- hammadi, Onur Duman, Philip, Agustina Aldasoro, ¨Om¨ urS ¸ahin, Mohsen Taheri Shalmani, Franco Castagna, Alberto Mart´ ın L´ opez, Hernan Ghianni, aszyrej, Miguel Rodriguez, Franco Nicol´ as Castagna, Lucas Mas Roca, IvaK, Annibale Panichella, Kyle Niemeyer, ...

    work page doi:10.5281/zenodo.18262322 2026

  6. [6]

    Andrea Arcuri, Man Zhang, Susruthan Seran, Juan Pablo Galeotti, Amid Golmohammadi, Onur Duman, Agustina Aldasoro, and Hernan Ghianni. 2025. Tool report: EvoMaster—black and white box search-based fuzzing for REST, GraphQL and RPC APIs.Automated Software Engineering32, 1 (2025), 1--11

    2025

  7. [7]

    Andrea Arcuri, Man Zhang, Susruthan Seran, Juan P. Galeotti, Amid Golmohammadi, Philip Garrett, Omur Sahin, Mohsen Taher Shalmani, Franco Castagna, Iva Kertusha, Fanny Febriani Susilo, Guru Prasad Bhandari, Gebremariam Assres, and Jefferson Seide Moll´ eri. 2026. EvoMaster at REST League 2026 Tool Competition. In19th Search-Based and Fuzz Testing Workshop...

    2026

  8. [8]

    Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. InACM/IEEE International Conference on Software Engineering (ICSE). 748–758

    2019

  9. [9]

    Davide Corradini, Seung Yeob Shin, and Domenico Bianculli. 2026. Automated REST API Black-box Test Generation in Practice: An Experience Report from Industry. In19th IEEE International Conference on Software Testing, Verification and Validation (ICST). Institute of Electrical and Electronics Engineers (IEEE)

    2026

  10. [10]

    Myles Foley and Sergio Maffeis. 2025. APIRL: Deep Reinforcement Learning for REST API Fuzzing. In Thirty-ninth Conference on Artificial Intelligence (AAAI 2025)

    2025

  11. [11]

    Philip Garrett, Juan P Galeotti, Andrea Arcuri, Alexander Poth, and Olsi Rrjolli. 2025. Generating REST API Tests With Descriptive Names.arXiv preprint arXiv:2512.01690(2025)

    arXiv 2025

  12. [12]

    Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2022. Testing RESTful APIs: A Survey.ACM Transactions on Software Engineering and Methodology(2022)

    2022

  13. [13]

    Zac Hatfield-Dodds and Dmitry Dygalo. 2022. Deriving Semantics-Aware Fuzzers from Web API Schemas. In2022 IEEE/ACM 44th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 345--346

    2022

  14. [14]

    Stefan Karlsson, Adnan Causevic, and Daniel Sundmark. 2020. QuickREST: Property-based Test Generation of OpenAPI Described RESTful APIs. InIEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE

    2020

  15. [15]

    Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2023. Adaptive rest api testing with reinforcement learning. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 446--458

    2023

  16. [16]

    Myeongsoo Kim, Tyler Stennett, Saurabh Sinha, and Alessandro Orso. 2025. A Multi-Agent Approach for REST API Testing with Semantic Graphs and LLM-Driven Inputs.ACM/IEEE International Conference on Software Engineering (ICSE)(2025)

    2025

  17. [17]

    Myeongsoo Kim, Qi Xin, Saurabh Sinha, and Alessandro Orso. 2022. Automated Test Generation for REST APIs: No Time to Rest Yet. InProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis(Virtual, South Korea)(ISSTA 2022). Association for Computing Machinery, New York, NY, USA, 289–301.https://doi.org/10.1145/3533767.3534401

    doi:10.1145/3533767.3534401 2022

  18. [18]

    Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: Model-based RESTful API Testing with Execution Feedback. InACM/IEEE International Conference on Software Engineering (ICSE)

    2022

  19. [19]

    Yi Liu, Yuekang Li, Yang Liu, Ruiyuan Wan, Runchao Wu, and Qingkun Liu. 2022. Morest: industry practice of automatic restful API testing. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1--5

    2022

  20. [20]

    Bogdan Marculescu, Man Zhang, and Andrea Arcuri. 2022. On the Faults Found in REST APIs by Automated Test Generation.ACM Transactions on Software Engineering and Methodology (TOSEM)31, 3 (2022), 1--43. 9

    2022

  21. [21]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cort´ es. 2019. Test coverage criteria for RESTful web APIs. InProceedings of the 10th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation. 15--21

    2019

  22. [22]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cort´ es. 2021. RESTest: Automated Black-Box Testing of RESTful Web APIs. InACM Int. Symposium on Software Testing and Analysis (ISSTA). ACM, 682--685

    2021

  23. [23]

    Alexander Poth, Olsi Rrjolli, and Andrea Arcuri. 2025. Technology adoption performance evaluation applied to testing industrial REST APIs.Automated Software Engineering32, 1 (2025), 5

    2025

  24. [24]

    Thomas Rooijakkers, Anne Nijsten, Cristian Daniele, Erieke Weitenberg, Ringo Groenewegen, and Arthur Melissen. 2025. WuppieFuzz: Coverage-Guided, Stateful REST API Fuzzing.arXiv preprint arXiv:2512.15554(2025)

    arXiv 2025

  25. [25]

    Omur Sahin, Man Zhang, and Andrea Arcuri. 2025. WFC/WFD: Web Fuzzing Commons, Dataset and Guidelines to Support Experimentation in REST API Fuzzing. arXiv:2509.01612 [cs.SE] https: //arxiv.org/abs/2509.01612

    Pith/arXiv arXiv 2025

  26. [26]

    Omur Sahin, Man Zhang, and Andrea Arcuri. 2026. Enhancing REST API Fuzzing with Access Policy Violation Checks and Injection Attacks.arXiv preprint arXiv:2604.00702(2026)

    arXiv 2026

  27. [27]

    Hassan Sartaj, Shaukat Ali, and Julie Marie Gjøby. 2025. Rest api testing in devops: A study on an evolving healthcare iot application.ACM Transactions on Software Engineering and Methodology(2025)

    2025

  28. [28]

    Emanuele Viglianisi, Michael Dallago, and Mariano Ceccato. 2020. RESTTESTGEN: Automated Black- Box Testing of RESTful APIs. InIEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE

    2020

  29. [29]

    Huayao Wu, Lixin Xu, Xintao Niu, and Changhai Nie. 2022. Combinatorial Testing of RESTful APIs. In ACM/IEEE International Conference on Software Engineering (ICSE)

    2022

  30. [30]

    Lixin Xu, Huayao Wu, Zhenyu Pan, Tongtong Xu, Shaohua Wang, Xintao Niu, and Changhai Nie

  31. [31]

    Effective REST APIs Testing with Error Message Analysis.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1978--2000

    2025

  32. [32]

    Man Zhang and Andrea Arcuri. 2023. Open Problems in Fuzzing RESTful APIs: A Comparison of Tools.ACM Transactions on Software Engineering and Methodology (TOSEM)(may 2023). https: //doi.org/10.1145/3597205

    work page doi:10.1145/3597205 2023

  33. [33]

    Man Zhang, Andrea Arcuri, Yonggang Li, Yang Liu, and Kaiming Xue. 2023. White-Box Fuzzing RPC- Based APIs with EvoMaster: An Industrial Case Study.ACM Transactions on Software Engineering and Methodology32, 5 (2023), 1--38

    2023

  34. [34]

    Man Zhang, Andrea Arcuri, Yonggang Li, Yang Liu, Kaiming Xue, Zhao Wang, Jian Huo, and Weiwei Huang. 2025. Fuzzing microservices: A series of user studies in industry on industrial systems with evomaster. Science of Computer Programming(2025), 103322

    2025

  35. [35]

    Man Zhang, Andrea Arcuri, Piyun Teng, Kaiming Xue, and Wenhao Wang. 2024. Seeding and Mocking in White-Box Fuzzing Enterprise RPC APIs: An Industrial Case Study. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 2024--2034. 10

    2024