REVIEW 3 major objections 1 minor 1 cited by
MASTOR generates semantic test oracles for RESTful APIs by extracting source context and using challenger review to catch business logic faults.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-06-27 12:42 UTC pith:D2GUMV2R
load-bearing objection MASTOR's multi-agent setup with parallel paths and challenger review beats the cited baselines on mutation score, but the gains rest on an unexamined assumption about import-closure context quality. the 3 major comments →
MASTOR: A Multi-Agent Approach to Semantic Test Oracle Generation for RESTful APIs
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
MASTOR consists of a source extraction agent that assembles context from transitive import closures, followed by two oracle-generation paths: one for status and field oracles on individual operations and one for behavioral consistency oracles across operation sequences. Both paths incorporate a challenger-agent review that supplies improvement hints for regeneration, after which normalization removes structurally invalid outputs. On 296 operations across 13 projects totaling over 250,000 lines of code, the system produced 10,022 oracles that achieved 75.4 percent average mutation score and translated into executable assertions and readable descriptions.
What carries the argument
Multi-agent pipeline with a source extraction agent, parallel single- and multi-operation oracle generators, and a challenger reviewer agent that iterates on weaknesses.
Load-bearing premise
The source extraction agent can assemble a context from imports that is complete enough for the other agents to capture all relevant semantic associations and business logic without introducing critical omissions or hallucinations.
What would settle it
Apply the generated oracles to a fresh set of known semantic faults injected into the same 13 projects and measure whether detection rates fall substantially below the reported 75.4 percent mutation score.
If this is right
- Oracles translate directly into executable JUnit and Postman assertions that can be run in existing test suites.
- Multi-operation oracles capture consistency across sequences that single-operation checks miss.
- The approach yields 30 percentage points higher mutation score than direct prompting and 49 points higher than SATORI on the same 50 operations.
- Normalization step removes invalid oracles after generation, raising the proportion of usable outputs.
Where Pith is reading between the lines
- The same extraction-plus-review structure could be adapted to generate oracles for stateful protocol testing beyond REST.
- Pairing MASTOR outputs with existing coverage-guided fuzzers might reduce the manual effort needed to write assertions for complex business flows.
- If the challenger review loop scales with larger codebases, the method could lower the barrier to maintaining semantic tests as APIs evolve.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MASTOR, a multi-agent framework for generating semantic test oracles for RESTful APIs. It consists of a source analysis phase that builds context for each operation via a source extraction agent analyzing transitive import closures, followed by two parallel oracle-generation paths (single-operation for status/field oracles and multi-operation for behavioral consistency oracles) that incorporate challenger-agent review and normalization. Evaluated on 13 open-source projects comprising 296 operations, MASTOR generates 10,022 oracles and achieves an average mutation score of 75.4%; on a 50-operation subset it outperforms Direct Prompting (69.9% vs. 39.8%) and SATORI (69.9% vs. 20.5%).
Significance. If the results hold, MASTOR would represent a meaningful step forward in automated REST API testing by moving beyond status-code and schema checks to oracles that target semantic faults and cross-operation consistency, using source-code-derived context and multi-agent collaboration. The concrete mutation scores, baseline comparisons, and translation to executable assertions provide empirical grounding that strengthens the contribution.
major comments (3)
- [Abstract] Abstract: the central performance claims (75.4% mutation score and the 30.1/49.4 pp baseline gains) rest on the source extraction agent producing a context from 'transitive import closure of relevant source files' that is sufficient to capture state dependencies and business logic; no verification mechanism, completeness metric, or hallucination-mitigation strategy is described, leaving open systematic under-detection of certain fault classes.
- [Abstract] Abstract: the baseline comparison is reported only on '50 selected operations'; without explicit selection criteria or representativeness argument, it is impossible to determine whether the reported improvements are robust or affected by post-hoc subset choice.
- [Abstract] Abstract: the mutation-score evaluation provides no details on the oracle validation process or the rationale for chosen mutation operators, which directly affects whether the 75.4% figure can be taken as representative of semantic coverage.
minor comments (1)
- The translation steps (ToJUnit, ToPostmanAssertify, ToReadable) are mentioned but not described; a brief methods paragraph or reference would improve reproducibility.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback on our manuscript. We address each major comment point by point below, providing clarifications from the full paper where relevant and committing to revisions that improve transparency without altering the core claims or results.
read point-by-point responses
-
Referee: [Abstract] Abstract: the central performance claims (75.4% mutation score and the 30.1/49.4 pp baseline gains) rest on the source extraction agent producing a context from 'transitive import closure of relevant source files' that is sufficient to capture state dependencies and business logic; no verification mechanism, completeness metric, or hallucination-mitigation strategy is described, leaving open systematic under-detection of certain fault classes.
Authors: The source extraction agent constructs context by traversing the transitive import closure of relevant source files for each operation, which in the evaluated projects captures the necessary state dependencies and business logic as demonstrated by the achieved mutation scores. The multi-agent design incorporates challenger-agent review during oracle generation to identify and mitigate weaknesses, including those potentially arising from incomplete context. While the abstract is concise, the full manuscript details the extraction process in Section 3.1. We agree that an explicit discussion of limitations and mitigation would strengthen the presentation; we will revise the manuscript to add a dedicated paragraph on source extraction verification (including manual spot-checks performed during development) and how the challenger review serves as a hallucination-mitigation strategy. revision: yes
-
Referee: [Abstract] Abstract: the baseline comparison is reported only on '50 selected operations'; without explicit selection criteria or representativeness argument, it is impossible to determine whether the reported improvements are robust or affected by post-hoc subset choice.
Authors: The 50 operations were chosen from the 296 total to enable feasible manual inspection and baseline execution while covering a range of operation complexities, HTTP methods, and projects from both WFD and PRAB datasets. The full paper (Section 5.3) provides the list of selected operations and notes that they were sampled to include both simple and state-dependent endpoints. We acknowledge that the abstract lacks explicit criteria; we will revise both the abstract and evaluation section to state the selection criteria (stratified sampling by project and operation type) and include a brief representativeness argument supported by statistics comparing the subset to the full benchmark. revision: yes
-
Referee: [Abstract] Abstract: the mutation-score evaluation provides no details on the oracle validation process or the rationale for chosen mutation operators, which directly affects whether the 75.4% figure can be taken as representative of semantic coverage.
Authors: Mutation analysis was conducted by translating generated oracles into executable assertions (via ToJUnit and ToPostmanAssertify) and applying a set of mutation operators targeting semantic faults, including status-code alterations, field-value changes, and sequence-inconsistency injections, chosen because they align with common REST API fault classes reported in the literature. The 75.4% average mutation score reflects the proportion of mutants killed across the 13 projects. The full manuscript (Section 5.2) describes the process and operators in detail. We agree the abstract is too brief on this point; we will revise the abstract and add a short rationale subsection in the evaluation to explicitly list the mutation operators and validation steps, ensuring the metric's interpretation is clear. revision: yes
Circularity Check
No significant circularity; empirical results measured against external benchmarks and baselines
full rationale
The paper describes a multi-agent system for oracle generation and reports mutation scores (75.4%) plus baseline comparisons (Direct Prompting, SATORI) on 13 projects. These metrics are obtained by executing generated oracles on independent mutant suites and external methods; no equations, fitted parameters, self-definitions, or self-citation chains reduce the reported performance to quantities defined by the method itself. The source extraction and challenger review steps are procedural descriptions without self-referential closure in the evaluation chain.
Axiom & Free-Parameter Ledger
read the original abstract
Existing automated RESTful API testing approaches commonly rely on simple checks (e.g., HTTP status codes, schema conformance), which are insufficient for detecting semantic faults, business logic violations, and state-dependent inconsistencies. To address this, we propose MASTOR, a Multi-Agent approach for generating Semantic Test Oracles for RESTful APIs based on implementation source code. MASTOR consists of two phases: source analysis and oracle generation. The former employs a source extraction agent to construct a source context for each endpoint operation by analyzing a transitive import closure of relevant source files. The latter employs two parallel oracle-generation paths over the collected contexts: a single-operation path producing status and field oracles per operation, and a multi-operation path generating behavioral consistency oracles for operation sequences by leveraging cross-operation semantic associations. Both paths apply a challenger-agent review, where a dedicated reviewer identifies weaknesses and issues improvement hints to guide targeted regeneration, followed by oracle normalization to filter out structurally invalid oracles. We evaluated MASTOR on a benchmark of 13 open-source RESTful API projects (296 operations, 251,303 lines of code) from the WFD and PRAB datasets. MASTOR achieved an average mutation score of 75.4%, generating 10,022 oracles. These oracles were translated into executable assertions via ToJUnit and ToPostmanAssertify, and into human-readable descriptions via ToReadable. In a baseline comparison on 50 selected operations, MASTOR outperformed Direct Prompting by 30.1 percentage points (69.9% vs. 39.8%) and SATORI by 49.4 percentage points (69.9% vs. 20.5%).
Figures
Forward citations
Cited by 1 Pith paper
-
MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models
MANGO uses Generator, Assessor, and Judge agents to create reusable atomic tasks and fine-grained oracles from natural language, evaluated on LIBERO_10 and RoboCasa benchmarks for comparable failure detection with bet...
Reference graph
Works this paper leans on
-
[1]
Aliyun. 2026. Qwen3.6-Plus. https://www.aliyun.com/product/tongyi
2026
-
[2]
Alonso, Michael D
Juan C. Alonso, Michael D. Ernst, Sergio Segura, and Antonio Ruiz-Cortés. 2025. Test Oracle Generation for REST APIs.ACM Transactions on Software Engineering and Methodology35, 1 (2025), 19:1–19:37
2025
-
[3]
Alonso, Alberto Martin-Lopez, Sergio Segura, Gabriele Bavota, and Antonio Ruiz-Cortés
Juan C. Alonso, Alberto Martin-Lopez, Sergio Segura, Gabriele Bavota, and Antonio Ruiz-Cortés. 2025. SATORI: Static Test Oracle Generation for REST APIs. InProceedings of the 40th IEEE/ACM International Conference on Automated Software Engineering. 1364–1376
2025
-
[4]
Alonso, Sergio Segura, and Antonio Ruiz-Cortés
Juan C. Alonso, Sergio Segura, and Antonio Ruiz-Cortés. 2023. AGORA: Automated Generation of Test Oracles for REST APIs. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1542–1553
2023
-
[5]
Andrea Arcuri, Man Zhang, and Juan Galeotti. 2024. Advanced White-Box Heuristics for Search-Based Fuzzing of REST APIs.ACM Transactions on Software Engineering and Methodology33, 6 (2024), 142:1–142:36
2024
-
[6]
Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. RESTler: Stateful REST API Fuzzing. InProceedings of the 41st International Conference on Software Engineering. 748–758
2019
-
[7]
Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo
Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The Oracle Problem in Software Testing: A Survey.IEEE Transactions on Software Engineering41, 5 (2015), 507–525
2015
- [8]
- [9]
-
[10]
Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java. InProceedings of the 25th ACM SIGSOFT International Symposium on Software Testing and Analysis. 449–452
2016
-
[11]
Alix Decrop, Sara Eraso, Xavier Devroey, and Gilles Perrouin. 2025. A Public Benchmark of REST APIs. InProceedings of the IEEE/ACM 22nd International Conference on Mining Software Repositories. 421–433
2025
-
[12]
DeepSeek. 2025. DeepSeek-V4-Pro. https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro
2025
-
[13]
DeepSeek. 2025. Models & Pricing | DeepSeek API Docs. https://api-docs.deepseek.com/quick_start/pricing/
2025
-
[14]
Sida Deng. 2025. MASTOR: Multi-Agent Semantic Test Oracle Generation for RESTful APIs. https://github.com/ Alysrazorr/MASTOR
2025
-
[15]
Sida Deng, Rubing Huang, Man Zhang, Chenhui Cui, Dave Towey, and Rongcun Wang. 2026. LRASGen: LLM- based RESTful API Specification Generation.ACM Transactions on Software Engineering and Methodology(2026). https://doi.org/10.1145/3810241 J. ACM, Vol. 88, No. 8, Article 888. Publication date: June 2026. MASTOR: A Multi-Agent Approach to Semantic Test Oracl...
- [16]
-
[17]
Ernst, Jeff H
Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao. 2007. The Daikon System for Dynamic Detection of Likely Invariants.Science of Computer Programming69, 1 (2007), 35–45
2007
-
[18]
Leo Fernandes, João Vitor Souza Rocha, Igor Wiese, João Lucas Correia, Ismael Trindade Fraga, Renato Torres Pinto, Erick Barboza, Baldoino Fonseca, and Márcio Ribeiro. 2025. Case Study: Applying Fuzzing to REST APIs in a Large-Scale Industrial Setting. InProceedings of the IEEE 36th International Symposium on Software Reliability Engineering Workshops (IS...
2025
-
[19]
2000.Architectural Styles and the Design of Network-Based Software Architectures
Roy Thomas Fielding. 2000.Architectural Styles and the Design of Network-Based Software Architectures. Ph. D. Dissertation. University of California, Irvine
2000
-
[20]
Josue Capistran García, Jorge Octavio Ocharán Hernández, Juan Carlos Peréz Arriaga, and Hector Javier Limón Riaño
-
[21]
InProceedings of the 2023 Mexican International Conference on Computer Science
Advances in Web API Testing: A Systematic Mapping Study. InProceedings of the 2023 Mexican International Conference on Computer Science. 1–8
2023
-
[22]
Hernan Ghianni, Man Zhang, Juan P. Galeotti, and Andrea Arcuri. 2025. Search-Based Fuzzing For RESTful APIs That Use MongoDB.arXiv preprint2507.20848 (2025)
-
[23]
Patrice Godefroid, Bo-Yuan Huang, and Marina Polishchuk. 2020. Intelligent REST API Data Fuzzing. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 725–736
2020
-
[24]
Amid Golmohammadi. 2023. Enhancing White-Box Search-Based Testing of RESTful APIs. InProceedings of the IEEE 34th International Symposium on Software Reliability Engineering Workshops. 9–12
2023
-
[25]
Amid Golmohammadi, Man Zhang, and Andrea Arcuri. 2023. Testing RESTful APIs: A Survey.ACM Transactions on Software Engineering and Methodology33, 1 (2023), 27:1–27:41
2023
- [26]
-
[27]
Barbara Hayes-Roth. 1985. A Blackboard Architecture for Control.Artificial Intelligence26, 3 (1985), 251–321
1985
-
[28]
Ruikai Huang, Manish Motwani, Idel Martinez, and Alessandro Orso. 2024. Generating REST API Specifications through Static Analysis. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. https://doi.org/10.1145/3597503.3639137
-
[29]
Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing.IEEE Transactions on Software Engineering37, 5 (2011), 649–678
2011
-
[30]
Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2025. LlamaRestTest: Effective REST API Testing with Small Language Models.Proceedings of the ACM on Software Engineering2, FSE (2025), 465–488
2025
-
[31]
Myeongsoo Kim, Tyler Stennett, Dhruv Shah, Saurabh Sinha, and Alessandro Orso. 2024. Leveraging Large Language Models to Improve REST API Testing. InProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results. 37–41
2024
- [32]
-
[33]
Alexander Lercher, Christian Macho, Clemens Bauer, and Martin Pinzger. 2024. Generating Accurate OpenAPI Descriptions from Java Source Code. https://doi.org/10.48550/arXiv.2410.23873
-
[34]
Jiaxian Lin, Tianyu Li, Yang Chen, Guangsheng Wei, Jiadong Lin, Sen Zhang, and Hui Xu. 2023. foREST: A Tree-based Black-box Fuzzing Approach for RESTful APIs. InIn Proceedings of the 34th IEEE International Symposium on Software Reliability Engineering (ISSRE). 695–705
2023
-
[35]
Yi Liu, Yuekang Li, Gelei Deng, Yang Liu, Ruiyuan Wan, Runchao Wu, Dandan Ji, Shiheng Xu, and Minli Bao. 2022. Morest: Model-Based RESTful API Testing with Execution Feedback. InProceedings of the 44th International Conference on Software Engineering. 1406–1417
2022
- [36]
-
[37]
OpenAPI Initiative. 2024. OpenAPI Specification v3.1.1. https://spec.openapis.org/oas/latest.html
2024
-
[38]
Qwen Team. 2025. Qwen3.6-35B-A3B. https://huggingface.co/Qwen/Qwen3.6-35B-A3B
2025
-
[39]
Thomas Rooijakkers, Anne Nijsten, Cristian Daniele, Erieke Weitenberg, Ringo Groenewegen, and Arthur Melissen
-
[40]
InProceedings of the 12th International Conference on Information Systems Security and Privacy
WuppieFuzz: Coverage-Guided, Stateful REST API Fuzzing. InProceedings of the 12th International Conference on Information Systems Security and Privacy. 221–231
- [41]
-
[42]
Dimitri Stallenberg, Mitchell Olsthoorn, and Annibale Panichella. 2021. Improving Test Case Generation for REST APIs Through Hierarchical Clustering. InProceedings of the 36th IEEE/ACM International Conference on Automated J. ACM, Vol. 88, No. 8, Article 888. Publication date: June 2026. 888:36 Deng et al. Software Engineering. 117–128
2021
-
[43]
Tyler Stennett, Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2025. AutoRestTest: A Tool for Automated REST API Testing Using LLMs and MARL. InProceedings of the IEEE/ACM 47th International Conference on Software Engineering: Companion Proceedings. 21–24
2025
-
[44]
WebFuzzing. 2025. Catwatch. https://github.com/WebFuzzing/Dataset/tree/main/jdk_8_maven/cs/rest/original/ catwatch
2025
-
[45]
WebFuzzing. 2025. Features-Service. https://github.com/WebFuzzing/Dataset/tree/main/jdk_8_maven/cs/rest/original/ features-service
2025
-
[46]
WebFuzzing. 2025. Genome-Nexus. https://github.com/WebFuzzing/Dataset/tree/main/jdk_8_maven/em/external/ rest/genome-nexus
2025
-
[47]
WebFuzzing. 2025. LanguageTool. https://github.com/WebFuzzing/Dataset/tree/main/jdk_8_maven/cs/rest/original/ languagetool
2025
-
[48]
WebFuzzing. 2025. Market. https://github.com/WebFuzzing/Dataset/tree/main/jdk_11_maven/em/external/rest/ market
2025
-
[49]
WebFuzzing. 2025. Person-Controller. https://github.com/WebFuzzing/Dataset/tree/main/jdk_21_maven/cs/rest/ person-controller
2025
-
[50]
WebFuzzing. 2025. ProxyPrint. https://github.com/WebFuzzing/Dataset/tree/main/jdk_8_maven/cs/rest/original/ proxyprint
2025
-
[51]
WebFuzzing. 2025. Restcountries. https://github.com/WebFuzzing/Dataset/tree/main/jdk_8_maven/cs/rest/original/ restcountries
2025
-
[52]
WebFuzzing. 2025. Swagger-Petstore. https://github.com/WebFuzzing/Dataset/tree/main/jdk_8_maven/cs/rest/ original/swagger-petstore
2025
-
[53]
WebFuzzing. 2025. Tracking-System. https://github.com/WebFuzzing/Dataset/tree/main/jdk_11_maven/cs/rest/ tracking-system
2025
-
[54]
WebFuzzing. 2025. User-Management. https://github.com/WebFuzzing/Dataset/tree/main/jdk_8_maven/cs/rest/ original/user-management
2025
-
[55]
White, Doug Burger, and Chi Wang
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. 2023. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. InProceedings of the 2nd Workshop on Agent Learning in Open-Endedness at NeurIPS 2023
2023
-
[56]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InProceedings of the 11th International Conference on Learning Representations
2023
-
[57]
Man Zhang and Andrea Arcuri. 2022. Adaptive Hypermutation for Search-Based System Test Generation: A Study on REST APIs with EvoMaster.ACM Transactions on Software Engineering and Methodology31, 1 (2022), 1–52
2022
-
[58]
Man Zhang and Andrea Arcuri. 2023. Open Problems in Fuzzing RESTful APIs: A Comparison of Tools.ACM Transactions on Software Engineering and Methodology32, 6 (2023), 144:1–144:45
2023
-
[59]
Man Zhang, Bogdan Marculescu, and Andrea Arcuri. 2019. Resource-based test case generation for RESTful web services. InIn Proceedings of the Genetic and Evolutionary Computation Conference (GECCO). 1426–1434
2019
-
[60]
operations
Tao Zheng, Jiang Shao, Jinqiao Dai, Shuyu Jiang, Xingshu Chen, and Changxiang Shen. 2024. RESTLess: Enhancing State-of-the-Art REST API Fuzzing With LLMs in Cloud Service Computing.IEEE Transactions on Services Computing 17, 6 (2024), 4225–4238. J. ACM, Vol. 88, No. 8, Article 888. Publication date: June 2026. MASTOR: A Multi-Agent Approach to Semantic Te...
2024
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.