Pith. sign in

REVIEW 5 major objections 4 minor 43 references

A lightweight LLM, fine-tuned with reinforcement learning, can write strict REST API test assertions from a single observed request–response pair, outperforming prompt-based generalist models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An RL fine-tuned lightweight LLM generates REST API test oracles from one traffic sample and beats prompt-engineered large models in industrial adoption.

T0 review reviewed 2026-07-31 challenge →

load-bearing objection Restor is a genuine RL-for-oracle application with strong production evidence, but the offline evaluation is less independent than it looks and needs another pass before the headline F1 is taken at face value. the 5 major comments →

arxiv 2607.23963 v1 pith:CY7QDYSI submitted 2026-07-27 cs.SE

RESTOR: Automated Test Oracle Generation for RESTful APIs via Reinforcement Learning

Xun Zhou , Zhen Dong , Mingyu Ren , Qiang Li , JunJie Li , Sifan Wang , Xiaolong Yu , Chaofeng Sha
show 1 more author
Xin Peng
This is my paper
classification cs.SE
keywords REST API testingtest oracle generationreinforcement learningGRPOassertion generationblack-box testingkey field identificationLLM fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Restor sets out to solve the 'cold-start' test oracle problem: generating meaningful, executable assertions for a REST API when there is no formal specification and only one captured request–response pair. The paper's central claim is that fine-tuning a lightweight language model with Group Relative Policy Optimization (GRPO), using execution feedback rather than a static ground-truth corpus, makes the model internalize a tester's 'common sense' about which fields matter and which value boundaries are valid. Trained this way, the model identifies business-critical fields with an F1 of 85.42% on an industrial test set, produces more exact-match assertions than either its base model or a much larger generalist model, and reduces missed validations to 28 cases. In production deployment, the framework raised the adoption rate of automatically generated test cases from 74.1% to above 96%, suggesting the approach is practically usable, not just benchmark-competitive.

Core claim

On its own terms, Restor's discovery is that RL fine-tuning with a carefully designed reward function can make a small model behave like a human QA engineer in a black-box, single-sample setting. The reward has three parts: an execution gate that rejects assertions that fail on the observed response; a field-identification term rewarding overlap with human-labeled key fields; and a semantic-accuracy term that rewards assertions that accept constructed positive variants and reject constructed negative variants. GRPO converts these outcome-level rewards into group-relative advantages, letting the model improve without a learned value function. The consequence is that the model learns to infer

What carries the argument

The central machinery is the composite reward R(A) = ρ_fail if the assertion fails execution on the original response, else clip(α·R_ident(A) + β·R_sem(A), -1, 1). R_ident measures precision/recall of the fields the assertion touches against the hand-labeled key set K; R_sem measures the differential kill rate over augmented samples: it rewards rejecting negative samples (S_neg) and accepting positive samples (S_pos). These augmented samples are constructed from a single traffic trace by human annotators who define natural-language constraints and three valid plus three invalid value variants per key field. The training algorithm GRPO samples a group of candidate assertions, normalizes rewar

Load-bearing premise

The load-bearing premise is that the positive and negative samples hand-constructed by human annotators from one observed response correctly represent the full boundary of valid and invalid values for each key field; if those samples miss legitimate value variations, the model is rewarded for rejecting behavior that actually occurs in production.

What would settle it

Take an enum-like field and observe its values across a longer production window; if the Restor-trained model rejects a value that occurs legitimately but was never included in S_neg, the reward proxy has misrepresented the field's true valid set. Concretely, run the deployed assertions on held-out traffic collected after training and count false-positive failures on fields whose valid values expanded.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Cold-start REST API testing becomes practical: assertions can be generated for newly deployed endpoints with only one observed traffic sample.
  • A lightweight, RL-tuned model can replace much larger general-purpose models for oracle generation, reducing latency and cost in CI/CD pipelines.
  • Generated oracles filter dynamic noise fields (timestamps, trace IDs), reducing flaky tests that fail on every run.
  • More precise key-field selection and semantically strict assertions lower manual QA editing effort, as reflected in the reported rise in adoption rate.
  • The reward design directly targets missed validations and false positives, and the paper reports fewer 'not validated' fields than baselines.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: The same GRPO + positive/negative sample reward could be applied to other output formats and languages (e.g., Java/JUnit), since the training signal is execution feedback rather than code syntax.
  • Editorial: The manual construction of S_pos/S_neg is the likely scale bottleneck; a testable extension would generate these samples automatically where a spec does exist, then measure whether the model degrades.
  • Editorial: The reported degradation on payloads with >400 fields suggests an adaptive sampling or field-selection strategy during inference could extend the approach beyond the training distribution.
  • Editorial: Because the reward assumes a fixed valid/invalid boundary, the approach may need periodic re-training or re-annotation when APIs evolve — a longitudinal study of assertion validity after API changes would settle this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. Restor proposes a GRPO fine-tuning pipeline for a lightweight ByteDance LLM (Doubao-Seed-1.6-flash) that generates Python test assertions from a single black-box request-response pair, without OpenAPI specifications or historical logs. Training data is built from over 2,300 production traffic samples: three QA engineers label key fields and infer semantic constraints per field, then construct three positive and three negative value variants. A composite reward penalizes execution failure and rewards key-field identification and semantic kill rates over those variants. Offline evaluation on 229 held-out samples reports F1 = 85.42% for key-field identification and 663 Exact-Match assertions, outperforming prompt-based Doubao-flash and DeepSeek-V3.1-Terminus. An expert user study and a production deployment at ByteDance are also reported, with adoption rate rising from 74.1% to above 96%.

Significance. If the evaluation were fully independent, this would be a significant industrial demonstration that RL fine-tuning can distill oracle-generation 'common sense' into a small, low-latency model, and it would be, to my knowledge, the first production deployment of RL-tuned LLMs for REST API oracle generation. Strengths include the large real-world dataset, the statistical testing of the F1 comparison, the inclusion of a strong large-model baseline, the explicit threats-to-validity section, and the deployment evidence. However, several load-bearing gaps prevent a stronger assessment: the offline ground truth is produced by the same annotator-derived protocol that defines the training reward; the whole pipeline is a single training run; there is no SFT baseline or reward-component ablation; and the train/test split unit is ambiguous. These issues are addressable but require additional experimental work.

major comments (5)
  1. [§3.1.2–3.1.3, §5.1.2, §5.1.4] The RQ1 oracle-quality metric is not independent of the training reward. The same three-engineer majority-voting protocol that labels K and constructs S_pos/S_neg in §3.1.2–3.1.3 is used in §5.1.2 to establish ground-truth key fields and semantic constraints, and §5.1.4 uses expert review to categorize assertion correctness. Eq. (5) rewards exactly the behavior that the RQ1 Exact-Match category measures: accepting S_pos, rejecting S_neg, and selecting K. The prompt-based baselines never see S_pos/S_neg, so the reported F1=85.42% and 663 Exact-Match assertions may measure fit to the annotators' boundary rather than an independent property such as production oracle quality. Please add an evaluation basis that is not generated by the same annotation process—e.g., annotations by a different group/blinded, mutation of the service logic, or correlation with real regressions. RQ3 is a useful co
  2. [§4, §5.2.1] The paper states in §4 that 'the training and evaluation processes were conducted in a single run.' GRPO samples groups stochastically, and the final policy depends on initialization and rollout randomness. The Wilcoxon tests in §5.2.1 are over 229 endpoints but for a single model realization, so they do not quantify training variance. Please report multiple fine-tuning runs/seeds with mean±std for F1 and Exact-Match counts, or justify why a single run is sufficient.
  3. [§3.2, §5.1.3] The central claim is that GRPO fine-tuning, rather than supervised fine-tuning, produces the improvement. The comparison against the untuned Doubao-flash can only attribute the gain to 'some fine-tuning on the augmented data'; it cannot isolate GRPO from SFT. Add an SFT baseline trained on the same K/S_pos/S_neg data (or an equivalent supervised objective on the same data), and ideally ablate the reward components (R_ident vs R_sem, and the coefficients) to support the RL-specific conclusion.
  4. [§5.1.2, §7.1] The split is described as 'randomly partitioned into training, validation, and testing sets in an 8:1:1 ratio,' while each API contributes 1–3 traces. If the random split is at the trace level, the same endpoint can appear in both training and testing, which would contradict the 'held-out test set' / '229 unseen API samples' phrasing and weaken the data-leakage mitigation in §7.1 ('physically excluding test set API endpoints'). Please state explicitly whether the split is by endpoint or by trace, and if it is by trace, redo the evaluation with endpoint-disjoint splits.
  5. [Appendix A, §1, §5.1.2] The paper's central setup is a 'single observed request-response pair' with no historical logs, but the system prompt in Appendix A instructs the model to 'combine it with some historical traffic, and generate assertions,' and §5.1.2 says each API contributes 1–3 traces. Please clarify what exactly is in the model input. If multiple traces are used, the 'single-traffic cold-start' claim should be revised; if not, the prompt wording should be corrected.
minor comments (4)
  1. [Table 2] For begin_time, Restor is shown as Partial Match with 'val >= 0', but the ground-truth constraint is 'non-negative Unix timestamp', which requires an integer check as well. As printed, DeepSeek's Partial Match ('int(val) >= 0') is stronger than Restor's, which may confuse readers; please reconcile the example with the classification.
  2. [§3.2.2, §4] The reward coefficients α, β, w_prec, w_rec, w_neg, and w_pos are never given numerical values. Since no code or model is released, these values are needed for reproducibility.
  3. [§5.3, §7.1] §7.1 says RQ2 uses traffic from newly developed features 'not yet integrated into the traffic recording platform,' but §5.3 describes the study as sampled from the internal test generation platform. Please clarify the provenance of the RQ2 cases.
  4. [Fig. 4 and Fig. 6] Raw counts of Exact Match / Partial Match / etc. are hard to interpret without normalization by the total number of assertions generated per model or by the number of key fields. Raw counts conflate coverage with verbosity; consider reporting per-model rates as well.

Circularity Check

0 steps flagged

No significant circularity: the training reward and held-out evaluation share an annotation protocol, but no claimed prediction reduces to its inputs by construction.

full rationale

Restor is an ML training/evaluation paper rather than a formal derivation, and no step reduces a claimed prediction to its inputs by construction. The training reward (§3.2.2, Eqs. 4–5) is defined on expert-annotated key fields K and constructed positive/negative samples S_pos/S_neg; the RQ1 evaluation (§5.1.2, §5.1.4) uses the same annotation protocol for held-out APIs. This is a proximity of training objective and evaluation metric, not a logical circle: the 229 test APIs are excluded from the training corpus (§7.1, 'physically excluding test set API endpoints from the training corpus'), and the test ground-truth constraints are independently elicited from experts for unseen APIs, so the reported F1=85.42% and Exact-Match counts measure generalization to new annotator-derived constraints rather than being forced by the reward. The paper's own limitation in §5.3.3 (black-box settings cannot infer 'proprietary, complex business rules') is a validity caveat, not evidence of circularity. The only self-citations (e.g., [40]) appear in related-work context and are not load-bearing for the central claim, and no uniqueness theorem or ansatz is imported from the authors' prior work. Production adoption (RQ3) provides an additional independent, if observational, grounding. I find no significant circularity.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The central claim rests on human expert labels, hand-set reward weights, and the assumption that constructed positive/negative samples are exhaustive. No new physical or conceptual entities are introduced. The training pipeline is a standard ML/RL configuration with several unspecified coefficients.

free parameters (6)
  • reward coefficients alpha, beta = unspecified
    Eq. (4) balances key-field identification R_ident and semantic accuracy R_sem; values are not reported and no sensitivity analysis is given.
  • identification weights w_prec, w_rec = unspecified
    R_ident = w_prec*precision + w_rec*recall; these hand-set weights directly shape how much precision vs recall is rewarded.
  • semantic weights w_neg, w_pos = unspecified
    R_sem = w_neg*KillRate(S_neg) - w_pos*KillRate(S_pos); values are not reported but determine the strictness/leniency trade-off.
  • failure penalty rho_fail = -1
    Set in Eq. (4) as a fixed penalty for assertions that fail on the original response.
  • GRPO hyperparameters: lr, KL beta, clip epsilon, epochs, group size G = lr=1e-6, beta=0.001, epsilon=0.2, epochs=2, G=8
    Reported in Section 4 without ablation; these are hand-chosen training configurations that affect convergence and final policy quality.
  • per-field augmentation count = 3 positive + 3 negative variants
    Section 3.1.3 replaces each key field value with three valid and three invalid variants; this fixed count determines the reward signal density.
axioms (6)
  • domain assumption GRPO improves LLM reasoning and transfers to assertion generation
    The paper adopts GRPO from DeepSeekMath [36] without evidence in the testing domain and without comparing to SFT/PPO baselines.
  • domain assumption Majority-vote of three QA engineers gives reliable ground truth for key fields and constraints
    Set K and S_pos/S_neg are derived from expert annotation (§3.1.2, §3.1.3); if annotations are biased, both reward and evaluation skew in the same direction.
  • domain assumption A high-quality oracle must pass all constructed positive samples and reject all constructed negative samples
    Eq. (5) defines R_sem via kill rates on S_pos/S_neg; this assumes the constructed samples capture the full valid/invalid boundary, which the paper later concedes is not true for some domain rules (§5.3.3).
  • ad hoc to paper Dynamic fields such as timestamps and log IDs are always noise and must be excluded from assertions
    Section 3.1.2 excludes systime/log_id from K by design. This is a deliberate choice that may be wrong for APIs where time-related invariants matter.
  • domain assumption A single traffic sample plus domain knowledge is sufficient for a useful oracle
    The entire cold-start setting assumes one request-response pair carries enough information to infer meaningful constraints; if not, the task is ill-posed.
  • domain assumption Adoption rate with peer review is a valid proxy for oracle quality
    Section 7.1 argues that committed code passes peer review, but adoption can still reflect convenience, habit, or lack of scrutiny.

reviewed 2026-07-31 · how reviews work

0 comments
Cite this review

Pith. "Pith review of RESTOR: Automated Test Oracle Generation for RESTful APIs via Reinforcement Learning." pith.science (2026). https://pith.science/paper/CY7QDYSI

@misc{pith2026260723963,
  author       = {Pith},
  title        = {Pith review of: RESTOR: Automated Test Oracle Generation for RESTful APIs via Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CY7QDYSI}},
  note         = {Machine review of arXiv:2607.23963}
}
Share X Bluesky LinkedIn Reddit HN
abstract

Modern REST API testing faces a critical challenge in defining reliable test oracles, particularly in agile industrial environments where formal specifications (e.g., OpenAPI) are frequently missing or outdated, and historical execution logs are unavailable for newly deployed endpoints. In this paper, we present Restor (Reinforcement Enhanced Single-Traffic Oracle generator for REST APIs), a framework that generates executable test assertions from a single observed request-response pair in a black-box setting. Unlike existing approaches that rely on rule-based templates or massive training logs, Restor utilizes a novel data augmentation pipeline to fine-tune a lightweight Large Language Model (LLM) via Group Relative Policy Optimization (GRPO). This training process enables the model to internalize testing "common sense" by optimizing a reward function that jointly encourages: (i) the selection of stable, semantically meaningful fields for validation and the avoidance of dynamic noise (e.g., timestamps or trace IDs); (ii) the generation of robust assertions that withstand logic variations. We evaluate Restor on an industrial dataset comprising over 2,300 API traces across 246 real-world services. Comprehensive experiments demonstrate that Restor significantly outperforms prompt-engineered baselines and generalist models, achieving a superior $F_1$ score of 85.42% in key field identification and increasing the proportion of semantically accurate assertions. Furthermore, deployment in a production CI/CD workflow at ByteDance confirms its practical value: the system raised the adoption rate of automatically generated test cases from 74.1% to over 96%, substantially reducing manual Quality Assurance (QA) effort while ensuring high execution stability.

Figures

Figures reproduced from arXiv: 2607.23963 by Chaofeng Sha, Junjie Li, Mingyu Ren, Qiang Li, Sifan Wang, Xiaolong Yu, Xin Peng, Xun Zhou, Zhen Dong.

Figure 1
Figure 1. Figure 1: Motivating Example. Left: Response example of [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the Restor framework. The workflow comprises two primary phases: (1) Dataset Con￾struction (top) and (2) Model Training (bottom). As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of Key Field Identification Performance. The visualization integrates violin plots to show [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparative Analysis of Assertion Accuracy. This distribution quantifies the semantic correctness of [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of Key Field Identification Metrics in User Study. The figure integrates violin plots (density), [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of Assertion Accuracy Categories. The chart quantifies the semantic correctness of the [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Temporal trends of adoption rate. The deployment of the fine-tuned model (starting early November) [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Adoption rates (top) and task volumes (bottom) distributed by Business Lines. Usage is concentrated [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 1 canonical work pages

  1. [1]

    OpenAPI Specification

    2025. OpenAPI Specification. https://www.openapis.org. Accessed: November 2025

    2025

  2. [2]

    Alonso, Michael D

    Juan C. Alonso, Michael D. Ernst, Sergio Segura, and Antonio Ruiz-Cortés. 2025. Test Oracle Generation for REST APIs.ACM Trans. Softw. Eng. Methodol.(March 2025). doi:10.1145/3726524 Just Accepted

    work page doi:10.1145/3726524 2025

  3. [3]

    Juan C Alonso, Alberto Martin-Lopez, Sergio Segura, Gabriele Bavota, and Antonio Ruiz-Cortés. 2025. SATORI: Static Test Oracle Generation for REST APIs.arXiv preprint arXiv:2508.16318(2025)

    Pith/arXiv arXiv 2025

  4. [4]

    Juan C Alonso, Sergio Segura, and Antonio Ruiz-Cortés. 2023. AGORA: automated generation of test oracles for REST APIs. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1018–1030. Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA094. Publication date: October 2026. RESTOR: Automated Test Oracle Generati...

    2023

  5. [5]

    Vaggelis Atlidakis, Patrice Godefroid, and Marina Polishchuk. 2019. Restler: Stateful rest api fuzzing. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 748–758

    2019

  6. [6]

    ByteDance. 2025. Doubao-Seed-1.6-flash Large Language Model. https://console.volcengine.com/ark/region:ark+cn- beijing/model/detail?Id=doubao-seed-1-6-flash. Accessed: 2026-01-13

    2025

  7. [7]

    Xiaobao Cai, Zhen Dong, Yongjiang Wang, Abhishek Tiwari, and Xin Peng. 2024. Reproducing Timing-dependent GUI Flaky Tests in Android Apps via A Single Event Delay. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)

    2024

  8. [8]

    Junjie Chen, Chenyao Suo, Jiajun Jiang, Peiqi Chen, and Xingjian Li. 2023. Compiler test-program generation via memoized configuration search. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2035–2047

    2023

  9. [9]

    Tianyi Chen, Kihong Heo, and Mukund Raghothaman. 2021. Boosting static analysis accuracy with instrumented test executions. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1154–1165

    2021

  10. [10]

    Tianyi Chen, Qidi Wang, Zhen Dong, Liwei Shen, and Xin Peng. 2023. Enhancing Robotic Program Synthesis Through Environmental Context. InAdvances in Neural Information Processing Systems 37 (NeurIPS)

    2023

  11. [11]

    Yiru Chen, Chenxi Zhang, Zhen Dong, Dingyu Yang, Xin Peng, Jiayu Ou, Hong Yang, Zheshun Wu, Xiaoun Qu, and Wei Li. 2023. Dynamic Graph Neural Networks-based Alert Link Prediction for Online Service Systems. InProceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)

    2023

  12. [12]

    DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. arXiv:2412.19437 [cs.CL] https://arxiv.org/abs/2412.19437

    Pith/arXiv arXiv 2024

  13. [13]

    DeepSeek-AI. 2025. DeepSeek-V3.1-Terminus: A Specialized Large Language Model. https://huggingface.co/deepseek- ai/DeepSeek-V3.1-Terminus. Version 3.1-Terminus

    2025

  14. [14]

    Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K Lahiri. 2022. Toga: A neural method for test oracle generation. InProceedings of the 44th International Conference on Software Engineering. 2130–2141

    2022

  15. [15]

    Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020. Time-travel Testing of Android Apps. In Proceedings of the 42nd IEEE/ACM International Conference on Software Engineering (ICSE)

    2020

  16. [16]

    Andréa Doreste, Matteo Biagiola, and Paolo Tonella. 2024. Adversarial testing with reinforcement learning: A case study on autonomous driving. In2024 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 293–304

    2024

  17. [17]

    2000.Architectural styles and the design of network-based software architectures

    Roy Thomas Fielding. 2000.Architectural styles and the design of network-based software architectures. University of California, Irvine

    2000

  18. [18]

    Gregory Gay, Sanjai Rayadurgam, and Mats PE Heimdahl. 2014. Improving the accuracy of oracle verdicts through automated model steering. InProceedings of the 29th ACM/IEEE international conference on Automated software engineering. 527–538

    2014

  19. [19]

    Wunan Guo, Zhen Dong, Liwei Shen, Wei Tian, Ting Su, and Xin Peng. 2022. Detecting and Fixing Data Loss Issues in Android Apps. InProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)

    2022

  20. [20]

    Zac Hatfield-Dodds and Dmitry Dygalo. 2022. Deriving semantics-aware fuzzers from web api schemas. InProceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings. 345–346

    2022

  21. [21]

    Jie He, Ezio Bartocci, Dejan Ničković, Haris Isakovic, and Radu Grosu. 2022. Deepstl: from english requirements to signal temporal logic. InProceedings of the 44th International Conference on Software Engineering. 610–622

    2022

  22. [22]

    Yibo He, Jiaming Huang, Hao Yu, and Tao Xie. 2024. An empirical study on focal methods in deep-learning-based approaches for assertion generation.Proceedings of the ACM on Software Engineering1, FSE (2024), 1750–1771

    2024

  23. [23]

    Soneya Binta Hossain and Matthew Dwyer. 2024. Togll: Correct and strong test oracle generation with llms.arXiv preprint arXiv:2405.03786(2024)

    Pith/arXiv arXiv 2024

  24. [24]

    Dmytro Humeniuk, Foutse Khomh, and Giuliano Antoniol. 2024. Reinforcement learning informed evolutionary search for autonomous systems testing.ACM Transactions on Software Engineering and Methodology33, 8 (2024), 1–45

    2024

  25. [25]

    Myeongsoo Kim, Saurabh Sinha, and Alessandro Orso. 2025. Llamaresttest: Effective rest api testing with small language models.Proceedings of the ACM on Software Engineering2, FSE (2025), 465–488

    2025

  26. [26]

    Jiageng Li, Zhen Dong, Chong Wang, Haozhen You, Cen Zhang, Yang Liu, and Xin Peng. 2025. LLM Based Input Space Partitioning Testing for Library APIs. InProceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE)

    2025

  27. [27]

    Chengjie Lu, Yize Shi, Huihui Zhang, Man Zhang, Tiexin Wang, Tao Yue, and Shaukat Ali. 2022. Learning configurations of operating environment of autonomous vehicles to maximize their collisions.IEEE Transactions on Software Engineering49, 1 (2022), 384–402

    2022

  28. [28]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2020. RESTest: Black-box constraint-based testing of RESTful web APIs. InInternational Conference on Service-Oriented Computing. Springer, 459–475. Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA094. Publication date: October 2026. ISSTA094:22 X. Zhou, Z. Dong, M. Ren, Q. Li, J. Li, S. ...

    2020

  29. [29]

    Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cortés. 2021. RESTest: automated black-box testing of RESTful web APIs. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 682–685

    2021

  30. [30]

    Facundo Molina, Pablo Ponzio, Nazareno Aguirre, and Marcelo Frias. 2021. Evospex: An evolutionary algorithm for learning postconditions. In2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1223–1235

    2021

  31. [31]

    Davide Molinelli, Alberto Martin-Lopez, Elliott Zackrone, Beyza Eken, Michael D Ernst, and Mauro Pezzè. 2025. Tratto: A Neuro-Symbolic Approach to Deriving Axiomatic Test Oracles.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1887–1909

    2025

  32. [32]

    Paulina Stevia Nouwou Mindom, Amin Nikanjam, and Foutse Khomh. 2023. A comparison of reinforcement learning frameworks for software testing tasks.Empirical Software Engineering28, 5 (2023), 111

    2023

  33. [33]

    Minxue Pan, An Huang, Guoxin Wang, Tian Zhang, and Xuandong Li. 2020. Reinforcement learning based curiosity- driven testing of android applications. InProceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis. 153–164

    2020

  34. [34]

    Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation.IEEE Transactions on Software Engineering50, 1 (2023), 85–105

    2023

  35. [35]

    Sergio Segura, Juan C Alonso, Alberto Martin-Lopez, Amador Durán, Javier Troya, and Antonio Ruiz-Cortés. 2022. Automated generation of metamorphic relations for query-based systems. InProceedings of the 7th International Workshop on Metamorphic Testing. 48–55

    2022

  36. [36]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300(2024)

    Pith/arXiv arXiv 2024

  37. [37]

    Jingling Sun, Ting Su, Junxin Li, Zhen Dong, Geguang Pu, Tao Xie, and Zhendong Su. 2021. Understanding and Finding System Setting-Related Defects in Android Apps. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)

    2021

  38. [38]

    Cody Watson, Michele Tufano, Kevin Moran, Gabriele Bavota, and Denys Poshyvanyk. 2020. On learning meaningful assert statements for unit test cases. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 1398–1409

    2020

  39. [39]

    Tianyi Wu, Liwei Shen, Zhen Dong, Xin Peng, and Wenyun Zhao. 2024. Synthesizing Programmatic Policy for Generalization within Task Domain. InProceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI)

    2024

  40. [40]

    Haozhen You, Jingjing Wang, Qiang Li, Xin Peng, and Zhen Dong. 2026. Industrial Practice of LLM-based Test Case Carving and Assertion Generation. InProceedings of the 35th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA)

    2026

  41. [41]

    Hao Yu, Yiling Lou, Ke Sun, Dezhi Ran, Tao Xie, Dan Hao, Ying Li, Ge Li, and Qianxiang Wang. 2022. Automated assertion generation via information retrieval and its integration with deep learning. InProceedings of the 44th International Conference on Software Engineering. 163–174

    2022

  42. [42]

    Chenxi Zhang, Zhen Dong, Xin Peng, Bicheng Zhang, and Miao Chen. 2024. Trace-based Multi-Dimensional Root Cause Localization of Performance Issues in Microservice Systems. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE)

    2024

  43. [43]

    Shaohua Zhang, Shuang Liu, Jun Sun, Yuqi Chen, Wenzhi Huang, Jinyi Liu, Jian Liu, and Jianye Hao. 2021. FIGCPS: Effective Failure-inducing Input Generation for Cyber-Physical Systems with Deep Reinforcement Learning. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 555–567. doi:10.1109/ASE51524. 2021.9678832 Received ...

    arXiv 2021

This paper was first reviewed by deepseek-v4-flash on July 31, 2026.