Pith. sign in

REVIEW 4 major objections 4 minor 33 references

Industrial Practice of LLM-Based Test Case Carving and Assertion Generation (Experience Paper)

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This experience paper claims that NL2Test, a pipeline combining a large language model with deterministic guardrails and recorded HTTP traffic, can automatically carve replayable API regression tests and generate business-aligned assertions

desk verdict Solid industrial experience paper: the integrated pipeline is genuinely new and the numbers are plausible, but single-company evidence, the verbatim-value assumption, and missing artifacts should keep it framed as practice with candid limitations rather than a generalizable result. read the letter →

arxiv 2607.24000 v2 pith:FJLRIYRP submitted 2026-07-27 cs.SE

Haozhen You , Zhen Dong , Jingjing Wang , Qiang Li , Xin Peng This is my paper
classification cs.SE
keywords testcarvingassertiongenerationAPIregressiontestingmicroserviceslargelanguagemodelstrafficcapturedatadependencyreconstructionindustrialdeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that traffic-grounded, LLM-assisted test generation can replace most manual work in building API regression tests for microservice systems. Given a natural-language scenario and a raw traffic capture, NL2Test filters noise, aligns scenario steps to API calls, reconstructs cross-request data dependencies from observed values, and synthesizes assertions over real response fields. The authors report 82.4% exact match against senior-QA ground truth on 51 scenarios, 98.0% functionally usable after minor edits, and an 85.4% engineer adoption rate across 3,196 generated tests in nine months of production use. If true, this means QA teams can shift from writing integration tests by hand to reviewing machine-generated drafts.

What carries the argument

The central object is a two-stage agent workflow: Test Script Carving and Assertion Generation. Carving uses rule-based filtering, an LLM step-to-endpoint alignment that outputs an inspectable mapping, dynamic data parameterization that validates candidate dependencies by searching earlier responses for the same value plus a semantic-consistency check, and deterministic template-based script assembly. Assertion generation uses an LLM to propose structured assertion candidates, a deterministic validator that checks response paths, repairs unique field-name matches, and normalizes expected-value forms, and a constrained code-generation agent that only translates validated candidates into execu

What would settle it

Take a scenario where a server responds with an opaque signed token or an ID that is subsequently rewritten (e.g., displayed ID vs. internal ID) so the same business object never appears verbatim in two requests. Run NL2Test on that capture: if it cannot produce a replayable parameterized test (or silently hard-codes the value), the core dependency-reconstruction claim fails. A cheaper check is the paper's own U2 case, where repetitive same-endpoint calls under 300+ background requests should cause step-alignment failure.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a decomposition into narrow, inspectable LLM tasks interleaved with deterministic validation—rather than a single end-to-end generation step—makes LLM-based test carving reliable enough for industrial regression suites. The load-bearing result is the exact-match rate of 42/51 and the adoption rate of 2730/3196, which the authors attribute to grounding every generated artifact in recorded traffic: request sequences are carved from the capture, dependencies are accepted only when a value appears in an earlier response and passes a semantic-consistency check, and assertions are kept only when their target fields exist in the recorded response

Load-bearing premise

The pipeline assumes every dynamic value needed for replay appears verbatim as a value in some earlier response in the same capture; if a value is transformed, encrypted, or server-generated without appearing in the trace, the dependency cannot be reconstructed and the test will not replay.

Editorial extensions

If this is right

  • QA engineers can expect a machine-generated draft that is directly runnable for the majority of standard HTTP-based microservice scenarios, with manual effort reduced to post-editing assertions.
  • Adoption is uneven by domain: business lines with highly dynamic real-time content and non-standard binary/async protocols show much lower acceptance, implying the approach is best suited to stable HTTP APIs with observable state.
  • The 85.4% adoption rate implies that generated tests pass maintainability review and expose real regressions when merged, as teams reported functional logic errors, missing fields, and type mismatches.
  • Because regeneration is only needed when contracts or workflows change, the generated tests become stable regression assets rather than one-off outputs.
  • The pipeline's token cost (average ~42.6k tokens, ~265 seconds per case) is acceptable in integration-test contexts but scales with trace size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct consequence the paper leaves implicit: the same value-consistency dependency check could serve as a coverage metric for trace completeness—if a candidate dynamic value cannot be found in any earlier response, that is evidence the capture is missing a step.
  • The approach might transfer to other record-and-replay settings, such as mobile UI automation, where a natural-language user story and a captured network trace are both available; the main adaptation would be the noise-filtering rules for each platform's protocol overhead.
  • A testable extension is to feed the rejected candidates (like the 'odidId' case) back as negative few-shot examples, since the paper's own failure analysis suggests the semantic-consistency check is too conservative when field names are obscure.
  • The reported 'trust latency' pattern—six months of low adoption followed by a surge—suggests that organizational rollout time, not just technical accuracy, determines whether such tools take hold; measuring that lag in other companies would be a natural follow-up.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents NL2Test, an LLM-based pipeline that generates executable API regression tests from a natural-language scenario description and a recorded HTTP traffic capture. The pipeline combines LLM agents for semantic interpretation (step-to-endpoint alignment, dynamic-data identification, assertion intent extraction, constrained code generation) with deterministic algorithms for request filtering, dependency validation via value consistency, assertion-path validation, and script assembly. The authors evaluate NL2Test on 51 industrial regression scenarios from one large consumer-facing Internet company, reporting an exact-match rate of 82.4% (42/51), a functionally usable draft rate of 98.0% (50/51), and a 9-month production deployment that generated 3,196 test cases with an 85.4% adoption rate. The core claim is that traffic-grounded generation with deterministic guardrails can largely automate API regression-test carving and oracle generation in complex microservice environments.

Significance. If the reported results hold, the paper makes a valuable practical contribution: it offers a concrete architecture for combining LLM semantic reasoning with deterministic validation to produce replayable, assertion-rich API regression tests from noisy industrial traffic. The paper is candid about failure modes (e.g., U2, U3, S2) and distills design lessons that are likely useful to other practitioners. The deployment data, including adoption rates and expert feedback, provides rare industrial evidence. However, the empirical claims rest on a single enterprise, a single-source ground truth, and an effectiveness metric that treats missing parameter dependencies as 'minor' post-edits. These issues limit the strength of the central claim until addressed with additional transparency and robustness analysis.

major comments (4)
  1. [§4.2, Table 1] The 'functionally usable draft' rate of 98.0% (50/51) counts partial-match cases that contain defects, including missing parameter dependencies (P=1) such as U3. The paper's core value proposition is replayability through dynamic dependency binding; a missing dependency means the generated script may hard-code a value from the recorded trace and fail against fresh data. The claim that such cases are 'usable with minor post-edits' is not justified without quantifying the repair effort. Please report, for each partial-match case, whether the script actually replays in a fresh environment and provide a concrete measure of post-edit effort (e.g., number of edited lines or time-to-repair).
  2. [§4.1.1, §7] The ground-truth scripts were manually written and reviewed by senior QA engineers from the same team that designed and deployed the tool. Exact match is measured against this single-source ground truth, with no inter-rater reliability or independent validation. Because the evaluation metric is functional equivalence to these scripts, the reported 82.4% exact-match rate could partly reflect the authors' own expectations rather than an objective oracle. Please provide evidence of ground-truth reliability, such as multiple independent annotators, a measure of annotation agreement, or a validation that the ground-truth scripts actually expose real regressions.
  3. [§3.1.3] Dependency reconstruction relies on the exact runtime value appearing verbatim in an earlier response, followed by a semantic-consistency check. The paper's own U3 failure (odidId not linked to userId) shows the fragility of this mechanism, and transformed values (signed tokens, encrypted IDs, server-generated references) are not handled at all. The claim that NL2Test 'reconstructs dynamic data dependencies' is therefore limited to verbatim-value scenarios. Please state this scope explicitly and report, across the 51 scenarios, how often dynamic parameters were successfully bound versus hard-coded, and how many partial-match cases had a missing dependency. Without this, the reader cannot assess the replayability that is central to the approach.
  4. [§4.3, Table 2] The deployment adoption rate of 85.4% and the expert feedback are self-reported and lack a precise definition of 'adoption.' It is unclear whether 'adopted' means merged without modification, merged with modifications, or merely accepted by an engineer during review. The paper also does not report how many of the 3,196 generated tests were actually executed in CI and passed, nor does it compare against a manual baseline beyond anecdotal quotes. Please define 'adoption' operationally and report execution/pass rates separated from merge decisions. This would substantially strengthen the maintainability claim.
minor comments (4)
  1. [§4.1.2] The LLM is referred to as 'Enterprise-LLM' without any specification of model family, parameter count, or inference configuration. Moreover, the paper does not report the number of repeated runs or the variance of the exact-match rate due to LLM stochasticity, even though a single run per scenario is implied. Please report whether the model was run with a fixed seed/temperature and, if possible, provide variance across at least a few runs.
  2. [Table 1] The formatting of Table 1 is difficult to parse: several rows appear to have missing separators (e.g., '442 /✓' in C1, '194 (0/1/0)✓ ∗' in U3). Please ensure that the Outcome and Defects columns are clearly delimited and that the symbols are explained in the caption.
  3. [§7] The Threats to Validity section is brief and focuses on external validity. It would be helpful to also acknowledge internal validity risks, such as the single-source ground truth and the lack of independent verification, and construct validity, such as the operationalization of 'usable' and 'adopted.'
  4. [§9] The Data Availability section states that data is not public due to proprietary concerns. While this is understandable for an industry paper, please make available at least the anonymized scenario descriptions, the generated scripts (sanitized), and the ground-truth skeletons to the extent legally possible. This would allow the community to partially reproduce the evaluation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pipeline is evaluated against manually written ground truth, and the key design choices are explicit deterministic checks rather than fitted outputs presented as predictions.

full rationale

I walked the derivation chain from inputs (NL scenario + traffic capture) to claimed results (82.4% exact match, 98.0% usable drafts, 85.4% adoption) and found no step where an output is identical to an input by construction. The dependency-reconstruction mechanism in §3.1.3 is the closest structural risk: a dependency is accepted only after searching previous responses for the same value and then applying semantic consistency. This is an explicit algorithm definition and a stated assumption, not a self-definitional prediction; the paper also reports the U3 failure case where the value was present but the semantic check refused the link. The benchmark uses ground-truth scripts manually written and reviewed by senior QA engineers (§4.1.1), and the few-shot examples are explicitly excluded from the evaluation benchmark (§3.1.2, §3.2.1), so the evaluation is not generated by the tool itself. Assertion validation (§3.2.2) removes or repairs invalid candidates rather than introducing the expected outcome from the target claim. The production adoption metric (2730/3196, 85.4%) is an external engineering decision, not a fitted parameter relabeled as a result. The authors' self-citations appear only in background and related-work contexts (e.g., [6, 11, 18, 19, 30, 34]) and are not load-bearing for any central conclusion; no uniqueness theorem or ansatz is imported from prior work by the same authors. The paper's own threats-to-validity section acknowledges single-enterprise external-validity limits and the dependence on traffic-observable API workflows, but these are limitations rather than circular reductions. Residual reviewer concerns about same-team evaluation and internal model disclosure are correctness/evidence concerns, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim carries no fitted numeric parameters, but it rests on hand-chosen filter, few-shot, and dependency-validation rules, and on five domain assumptions about traffic completeness, value visibility, LLM capability, ground-truth quality, and internal metric reliability.

free parameters (3)
  • Rule-based traffic filtering categories
    The categories filtered as noise (static assets, CORS/protocol, infrastructure/periodic) and the conservative keep-when-unsure policy are hand-chosen. They affect every scenario and were not fit to the benchmark, but they determine how much signal the LLM sees.
  • Few-shot example selection policy
    Few-shot examples for alignment and assertion intent are hand-selected from manually verified development cases by decision pattern and excluded from evaluation. This design choice influences outputs and is not independently benchmarked.
  • Dynamic-dependency value-consistency strictness
    A dependency is accepted only if the recorded value appears verbatim in a previous response and semantics match. This hand-chosen strictness maximizes precision but caused the U3 odidId miss.
assumptions (5)
  • domain assumption A representative traffic capture exists for every scenario and contains the business requests plus enough response structure.
    §2.1 assumes two artifacts: an NL scenario description and a traffic capture; §7 acknowledges incomplete captures are not studied.
  • domain assumption Dynamic replay values appear unchanged in earlier responses.
    §3.1.3 value-consistency search; the U3 failure demonstrates the boundary of this assumption.
  • domain assumption The undisclosed Enterprise-LLM can reliably map steps to endpoints and propose assertion intents under the given prompts.
    §3.1.2 and §3.2.1 rely on LLM semantic judgment; no public model or version is given, so external replication cannot validate this.
  • domain assumption Ground-truth scripts written by senior QA engineers correctly encode intended business behavior.
    §4.1.1 uses manually reviewed ground truth; no inter-rater reliability or independent audit is reported.
  • domain assumption Adoption rate and person-day savings are accurate measurements of maintainability and value.
    §4.3 and §4.4 rely on self-reported internal metrics; no independent audit is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Industrial Practice of LLM-Based Test Case Carving and Assertion Generation (Experience Paper)." pith.science (2026). https://pith.science/paper/FJLRIYRP

@misc{pith2026260724000,
  author       = {Pith},
  title        = {Pith review of: Industrial Practice of LLM-Based Test Case Carving and Assertion Generation (Experience Paper)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FJLRIYRP}},
  note         = {Machine review of arXiv:2607.24000}
}
read the original abstract

Enterprise regression testing for microservice systems is often constrained by incomplete or outdated documentation. In practice, QA engineers frequently rely on real execution traffic to reconstruct business scenarios; however, turning raw traffic into replayable regression tests with stable validation logic remains labor-intensive and error-prone. This paper presents NL2Test, an end-to-end approach and tool that generates executable API regression tests from (i) a natural-language scenario description and (ii) a traffic capture recorded while executing the scenario. NL2Test addresses two coupled tasks: test case carving, which extracts a minimal replayable request sequence and reconstructs data dependencies so that dynamic values are bound from their responses rather than hard-coded; and assertion generation, which produces assertions aligned with business intent while avoiding non-deterministic fields and hallucinated paths. To improve reliability, NL2Test uses LLMs for semantic interpretation and constrained code synthesis, and uses deterministic algorithms for request filtering, dependency confirmation via value consistency, and assertion-path validation. We evaluate NL2Test on 51 industrial regression scenarios extracted from a large consumer-facing Internet company. NL2Test achieves an exact-match rate of 82.4% (42/51), and produces a functionally usable draft in 98.0% (50/51) of scenarios when allowing minor post-edits. In a 9-month production deployment starting in March 2025, NL2Test generated 3,196 test cases with an overall code adoption rate of 85.4%. These results indicate that traffic-grounded generation with deterministic guardrails can substantially reduce manual effort while improving regression automation in complex microservice environments.

Figures

Figures reproduced from arXiv: 2607.24000 by the authors.

Figure 1
Figure 1. Example of traffic-grounded test carving and assertion generation. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of Test Script Carving. 3.1.1 Traffic Filtering. We first reduce the raw traffic through rule-based filtering before invoking the LLM. A practical lesson from our experience is that many irrelevant requests can be identified without semantic reasoning. We mainly filter the following categories of requests: Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA008. Publication date: October 2026 [PITH_FULL… view at source ↗
Figure 3
Figure 3. Overview of assertion generation and deterministic validation. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Adoption Rate of Generated Cases by Business Line. [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Monthly Active Users Proportion Trend. 4.4.2 Proportion of LLM-Generated Test Cases. The tool successfully shifted from being an auxiliary utility to the dominant method for test creation, eventually accounting for 57% of all new test cases generated organization-wide …
Figure 6
Figure 6. Figure 6: Case Penetration Trend (March-Dec). Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA008. Publication date: October 2026 [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Review history (2 revisions) →

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 2 linked inside Pith

  1. [1]

    Nadia Alshahwan, Arianna Blasi, Kinga Bojarczuk, Andrea Ciancone, Natalija Gucevska, Mark Harman, Michal Krolikowski, Rubmary Rojas, Dragos Martac, Simon Schellaert, Kate Ustiuzhanina, Inna Harper, Yue Jia, and Will Lewis. 2024. Enhancing Testing at Meta with Rich-State Simulated Populations. InProceedings of the 46th International Conference on Software ...

    arXiv 2024

  2. [4]

    Marcin Andrzejewski, Nina Dubicka, Jędrzej Podolak, Marek Kowal, and Jakub Siłka. 2025. Automated Test Generation Using Large Language Models.Data10, 10 (2025), 156. doi:10.3390/data10100156

    doi:10.3390/data10100156 2025

  3. [5]

    Moritz Beller, Hongyu Li, Vivek Nair, Vijayaraghavan Murali, Imad Ahmad, Jürgen Cito, Drew Carlson, Ari Aye, and Wes Dyer. 2023. Learning to Learn to Predict Performance Regressions in Production at Meta. In2023 IEEE/ACM International Conference on Automation of Software Test (AST). 56–67. doi:10.1109/AST58925.2023.00010

    arXiv 2023

  4. [7]

    Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. ChatUniTest: A Framework for LLM-Based Test Generation. InCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering(Porto de Galinhas, Brazil)(FSE 2024). Association for Computing Machinery, New York, NY, USA, 572–576. doi:...

    arXiv 2024

  5. [8]

    Yiru Chen, Chenxi Zhang, Zhen Dong, Dingyu Yang, Xin Peng, Jiayu Ou, Hong Yang, Zheshun Wu, Xiaojun Qu, and Wei Li. 2023. Dynamic Graph Neural Networks-Based Alert Link Prediction for Online Service Systems. In2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). 79–90. doi:10.1109/ASE56229.2023.00177

    arXiv 2023

  6. [9]

    Amirhossein Deljouyi and Andy Zaidman. 2023. Generating Understandable Unit Tests through End-to-End Test Scenario Carving. In2023 IEEE 23rd International Working Conference on Source Code Analysis and Manipulation (SCAM). 107–118. doi:10.1109/SCAM59687.2023.00021

    arXiv 2023

  7. [10]

    Elizabeth Dinella, Gabriel Ryan, Todd Mytkowicz, and Shuvendu K. Lahiri. 2022. TOGA: a neural method for test oracle generation. InProceedings of the 44th International Conference on Software Engineering(Pittsburgh, Pennsylvania)(ICSE ’22). Association for Computing Machinery, New York, NY, USA, 2130–2141. doi:10.1145/3510003.3510141

    arXiv 2022

  8. [11]

    Zhen Dong, Marcel Böhme, Lucia Cojocaru, and Abhik Roychoudhury. 2020. Time-travel testing of Android apps. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering(Seoul, South Korea)(ICSE ’20). Association for Computing Machinery, New York, NY, USA, 481–492. doi:10.1145/3377811.3380402

    arXiv 2020

Show all 33 references
  1. [12]

    Dwyer, and Matthew Jorde

    Sebastian Elbaum, Hui Nee Chin, Matthew B. Dwyer, and Matthew Jorde. 2009. Carving and Replaying Differential Unit Test Cases from System Test Cases.IEEE Transactions on Software Engineering35, 1 (2009), 29–45. doi:10.1109/ TSE.2008.103

    2009

  2. [13]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. LLM-Based Test- Driven Interactive Code Generation: User Study and Empirical Evaluation.IEEE Transactions on Software Engineering 50, 9 (2024), 2254–2268. doi:10.1109/TSE.2024.3428972

    2024

  3. [14]

    Erhu Feng, Wenbo Zhou, Zibin Liu, Le Chen, Yunpeng Dong, Cheng Zhang, Yisheng Zhao, Dong Du, Zhichao Hua, Yubin Xia, et al. 2025. Get Experience from Practice: LLM Agents with Record & Replay.arXiv preprint arXiv:2505.17716 (2025). doi:10.48550/arXiv.2505.17716

    2025 doi arXiv

  4. [15]

    Gordon Fraser and Andrea Arcuri. 2011. EvoSuite: automatic test suite generation for object-oriented software. InProceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering(Szeged, Hungary)(ESEC/FSE ’11). Association fo...

    2011

  5. [16]

    Alessio Gambi, Hemant Gouni, Daniel Berreiter, Vsevolod Tymofyeyev, and Mattia Fazzini. 2023. Action-Based Test Carving for Android Apps. In2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). 107–116. doi:10.1109/ICSTW58534.20...

    2023

  6. [17]

    Jiale Guo, Suizhi Huang, Mei Li, Dong Huang, Xingsheng Chen, Regina Zhang, Zhijiang Guo, Han Yu, Siu-Ming Yiu, Pietro Lio, et al . 2025. A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System.arXiv preprint arXiv:2510.09721(2...

    2025 doi

  7. [18]

    Wunan Guo, Zhen Dong, Liwei Shen, Wei Tian, Ting Su, and Xin Peng. 2022. Detecting and fixing data loss issues in Android apps. InProceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, South Korea)(ISSTA 2022). Association for Co...

    2022

  8. [19]

    Wunan Guo, Zhen Dong, Liwei Shen, Daihong Zhou, Bin Hu, Chen Zhang, and Hai Xue. 2025. Effectively Modeling UI Transition Graphs for Android Apps Via Reinforcement Learning. In2025 IEEE/ACM 33rd International Conference on Program Comprehension (ICPC). 13–24. doi:10.1109/ICPC6...

    2025

  9. [20]

    Mark Harman, Jillian Ritchey, Inna Harper, Shubho Sengupta, Ke Mao, Abhishek Gulati, Christopher Foster, and Hervé Robert. 2025. Mutation-Guided LLM-based Test Generation at Meta. InProceedings of the 33rd ACM International Conference on the Foundations of Software Engineering...

    2025

  10. [21]

    Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2024. From llms to llm-based agents for software engineering: A survey of current, challenges and future.arXiv preprint arXiv:2408.02479(2024). doi:10.48550/ arXiv.2408.02479

    2024 doi arXiv

  11. [22]

    Myeongsoo Kim, Tyler Stennett, Dhruv Shah, Saurabh Sinha, and Alessandro Orso. 2024. Leveraging Large Language Models to Improve REST API Testing. InProceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results(Lisbon, P...

    2024

  12. [23]

    Yinfeng Li, Chen Gao, Xiaoyi Du, Huazhou Wei, Hengliang Luo, Depeng Jin, and Yong Li. 2022. Automatically Discovering User Consumption Intents in Meituan. InProceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining(Washington DC, USA)(KDD ’22). Assoc...

    2022

  13. [24]

    Cristian Mascia, Antonio Guerriero, Luca Giamattei, Roberto Pietrantuono, and Stefano Russo. 2025. Microservices Per- formance Testing with Causality-enhanced Large Language Models. In2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineerin...

    2025

  14. [25]

    Facundo Molina, Alessandra Gorla, and Marcelo d’Amorim. 2025. Test Oracle Automation in the Era of LLMs.ACM Trans. Softw. Eng. Methodol.34, 5, Article 150 (May 2025), 24 pages. doi:10.1145/3715107

    2025 doi

  15. [26]

    Dezhi Ran, Hao Wang, Zihe Song, Mengzhou Wu, Yuan Cao, Ying Zhang, Wei Yang, and Tao Xie. 2024. Guardian: A Runtime Framework for LLM-Based UI Exploration. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis(Vienna, Austria)(ISSTA 202...

    2024

  16. [27]

    Harshit Saokar, Soteris Demetriou, Nick Magerko, Max Kontorovich, Josh Kirstein, Margot Leibold, Dimitrios Skarlatos, Hitesh Khandelwal, and Chunqiang Tang. 2023. ServiceRouter: Hyperscale and Minimal Cost Service Mesh at Meta. In 17th USENIX Symposium on Operating Systems Des...

    2023

  17. [28]

    Arkadii Sapozhnikov, Mitchell Olsthoorn, Annibale Panichella, Vladimir Kovalenko, and Pouria Derakhshanfar. 2024. TestSpark: IntelliJ IDEA’s Ultimate Test Generation Companion. InProceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion ...

    2024

  18. [29]

    Sumedh Sathaye, Patrick East, Reut Kovetz, Jennifer Minarik, Kelly Lisai, Arthur Lent, and Nicole Reineke. 2023. Creating CLI packages and API playbooks from codified graphical user experience designs. US Patent 11,740,878

    2023

  19. [30]

    Jingling Sun, Ting Su, Junxin Li, Zhen Dong, Geguang Pu, Tao Xie, and Zhendong Su. 2021. Understanding and finding system setting-related defects in Android apps. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis(Virtual, Denmark)(I...

    2021

  20. [31]

    Yutian Sun, Tim Meehan, Rebecca Schlussel, Wenlei Xie, Masha Basmanova, Orri Erling, Andrii Rosa, Shixuan Fan, Rongrong Zhong, Arun Thirupathi, Nikhil Collooru, Ke Wang, Sameer Agarwal, Arjun Gupta, Dionysios Logothetis, Kostas Xirogiannopoulos, Amit Dutta, Varun Gajjala, Rohi...

    2023

  21. [32]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision.IEEE Transactions on Software Engineering50, 4 (2024), 911–936. doi:10.1109/TSE.2024.3368208 Proc. ACM Softw. Eng., Vol...

    2024

  22. [33]

    Rahulkrishna Yandrapally, Saurabh Sinha, Rachel Tzoref-Brill, and Ali Mesbah. 2023. Carving UI Tests to Generate API Tests and API Specification. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). 1971–1982. doi:10.1109/ICSE48619.2023.00167

    2023

  23. [34]

    Chenxi Zhang, Zhen Dong, Xin Peng, Bicheng Zhang, and Miao Chen. 2024. Trace-based Multi-Dimensional Root Cause Localization of Performance Issues in Microservice Systems. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering(Lisbon, Portugal)(ICS...

    2024

  24. [35]

    Quanjun Zhang, Weifeng Sun, Chunrong Fang, Bowen Yu, Hongyan Li, Meng Yan, Jianyi Zhou, and Zhenyu Chen

  25. [2025]

    Exploring Automated Assertion Generation via Large Language Models.ACM Trans. Softw. Eng. Methodol.34, 3, Article 81 (Feb. 2025), 25 pages. doi:10.1145/3699598 Received 2026-01-30; accepted 2026-06-25 Proc. ACM Softw. Eng., Vol. 3, No. ISSTA, Article ISSTA008. Publication date...

    2025 doi

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.