Pith. sign in

REVIEW 3 major objections 6 minor 32 references

AI-Driven Tools in Modern Software Quality Assurance: An Assessment of Benefits, Challenges, and Future Directions

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that ReAct-style browser agents can generate and execute end-to-end regression tests on a sample web app with an 8.3% flaky-execution rate—on par with mature scripted suites—at under $0.005 per generated user story.

desk verdict A useful survey and a transparent but undersized experiment whose headline 8.3% flakiness claim doesn't survive separating mutated false-negatives from true flakiness. read the letter →

arxiv 2506.16586 v1 pith:F3PW6KPH submitted 2025-06-19 cs.SE cs.AI

Ihor Pysmennyi , Roman Kyslyi , Kyrylo Kleshch This is my paper · ORCID
classification cs.SEcs.AI
keywords qualityassuranceLLMAIagentsend-to-endtestingtestcasegenerationflakytestsReActmetamorphic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that LLM-based tools can take over a large share of software quality assurance work, from writing test cases to executing full end-to-end regression flows. Its central empirical claim is that a ReAct-style browser agent running generated test scenarios on a sample e-commerce app produced only 8.3% flaky executions across 48 runs—a rate the authors put on par with mature scripted suites—at a cost of roughly $0.005 per generated user story for the best model. The paper also documents a serious caveat: agents often 'correct' deliberately broken mutated scenarios back to a passing state, which would conceal real defects, so the authors argue that execution logs and generated artifacts need human- or machine-based verification and that guardrails such as token caps and checkpoint assertions are required.

What carries the argument

The load-bearing mechanism is the ReAct feedback loop, in which the language model repeatedly reasons about the current browser state, plans an action, executes it through a browser automation engine (Playwright), and observes the result before replanning. This closed loop lets the agent adapt to timing issues, unexpected popups, and UI changes that make scripted locator-based tests flaky, and it is what the paper credits for the 8.3% flaky rate. The proposed companion machinery is the LLM-as-a-judge validator with RAG context enrichment, which filters and corrects generated test cases, and mutation testing, which judges whether generated unit tests genuinely increase coverage by checking whether they catch seeded defects.

What would settle it

Re-run the same six flows on a multi-page application with injected faults, separating mutated from unmutated scenarios and logging whether the agent alters the mutated input before reporting a pass. If the agent consistently passes mutated scenarios, or if flakiness on unmutated flows does not reproduce in a larger sample, the parity claim fails; any run where the agent changes the test data should be reclassified from 'flaky' to 'false negative'.

Watch

Extended reading notes

Core claim

The central discovery, as the authors state it, is that a single LLM-driven workflow can close the loop between test design and test execution: state-of-the-art models generate runnable acceptance tests from user stories (with 100% acceptance-criteria coverage in the best case), and the same family of models, embedded in a ReAct-style agent, can execute those scenarios in a live browser. Across two of the evaluated models (gpt-4o and gpt-4o-mini), six flows, and four runs apiece, only 4 of 48 executions were flaky or failed—8.3%—which the authors compare directly to the industry flakiness statistics of large-scale production suites. The authors treat this as evidence that AI-agent-based end-to-end regression is a practical alternative or supplement to traditional automation, provided that the known failure modes—particularly the agents' tendency to repair mutated negative scenarios and the smaller model's infinite reasoning loop in the checkout flow—are controlled.

Load-bearing premise

The headline 8.3% flakiness figure rests on 48 executions of six flows on a single demo app, and it lumps failed executions of deliberately mutated scenarios into 'flaky' even though the paper itself shows agents often correct those mutations—so the failures may be false negatives, not instability.

Editorial extensions

If this is right

  • If the 8.3% figure generalizes, agent-based end-to-end regression can substitute for scripted suites on UI-heavy applications, eliminating the dominant maintenance cost of locator and xpath updates.
  • Test design effort shifts from scripting to prompt-writing and output review; the best model in Experiment 1 reached 100% acceptance-criteria coverage at about $0.005 per user story, making LLM-generated cases a cheap first-pass test asset.
  • Because agents can silently 'fix' intentionally broken scenarios, a pass/fail status is not enough: pipelines must verify execution logs and detect input alteration, or real bugs will be reported as green.
  • Pairing an observation-based unit-test generator with a mutation-testing feedback loop would retain only tests that add effective coverage, directly addressing the semantic-duplication problem the paper reports.
  • If adopted into continuous delivery, these techniques move defect discovery earlier in the software lifecycle and against the paper's cited multi-trillion-dollar annual cost of poor-quality software, but they require reskilling human QA into prompt engineers and governance owners.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 8.3% figure conflates two very different failure modes: the mutated flows that agents 'corrected' are false negatives (bugs missed), not flaky runs, while true flakiness appears only in non-mutated flows. Re-analysis of Table 3 suggests both models had 0% flakiness on the four unmutated flows, so the honest headline might be '0% flaky on normal flows, but 25–50% of mutated scenarios we
  • A natural next experiment is to vary the mutation severity and count how often the agent alters the input before reporting pass; that number is a direct measure of the false-negative risk the paper flags and can be benchmarked against human testers or scripted negative tests.
  • The infinite reasoning loop observed with the smaller model suggests cost and latency—not just correctness—should be part of any evaluation of agentic QA; a token/time cap could turn a $0.05 test into an availability incident, so the paper's recommended guardrails double as SLO protections.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper assesses the use of LLM-based tools across QA activities—static analysis, test-case generation, unit test generation, end-to-end automation, and metamorphic testing—and reports two experiments. Experiment 1 (Section 3.1) evaluates seven LLMs generating test cases from user stories for the Sauce Demo application. Experiment 2 (Section 3.2) uses two OpenAI models as ReAct-style browser agents to execute six login, sorting, and checkout flows, four times each (48 executions total), and reports an aggregate 8.3% 'flaky' execution rate. The Conclusions present the 8.3% figure as evidence that agentic end-to-end regression is on parity with mature scripted suites, while also acknowledging false-negative behavior in mutated scenarios.

Significance. If the headline empirical claim were robust, the paper would offer a useful data point for practitioners considering LLM-based end-to-end regression. The authors deserve credit for making code and execution logs publicly available, using a real public application, and including an explicit limitations section that anticipates the false-negative problem. However, the evidence base is too thin and the headline metric is conceptually muddled: the flakiness rate conflates nondeterministic failures with expected failures on mutated inputs, and four runs per scenario cannot sustain the comparison to Google's production statistics. The survey portions are competent but largely derivative; the value of the paper hinges on the experimental section, which currently overstates its support.

major comments (3)
  1. [3.2, Table 3] The 'Flaky failed executions' column mixes two distinct phenomena: nondeterministic failures of unchanged test flows and false negatives on mutated flows, where the agent 'corrects' the defect and reports a pass. Section 3.3 itself states that agents 'deviate from negative test scenarios by attempting to correct the flow to achieve a positive outcome, masking potential failures.' A run that masks a mutation is not a flaky result under the standard definition (variation in outcome of an unchanged test); it is an oracle/agent failure. Counting the three mutated-flow outcomes in Table 3 together with the one non-mutated 'Buys&Checkout' outcome produces the aggregate 4/48 = 8.3%. Excluding mutated flows, the observed rate is 1/32 = 3.1%. Therefore the Conclusions sentence asserting 'only 8.3% flaky runs, being on parity with mature scripted suites' is not supported as stated. Please reclassify these outcomes, present separate rates, and revise the conclusion accordingly.
  2. [3.2, Table 3] Even after reclassification, the comparison to Google's production flakiness statistics (reference [31]) is not meaningful without a shared measurement protocol. Each scenario was executed only four times per model, so a single altered outcome changes the reported rate by 25 percentage points, and the corresponding confidence intervals are extremely wide. Google's numbers describe unchanged tests running in large CI pipelines, not small-sample agent executions on a public demo application. The paper should report exact failure counts per scenario, provide confidence intervals or otherwise acknowledge the uncertainty, and either remove the parity claim or support it with a matched baseline that is actually run in this study.
  3. [2.2, Conclusions] The conclusion that transformer-based static analyzers boost F1 scores 'by more than 40 percentage points over a SonarQube baseline' is not supported by the material presented. Section 2.2 lists F1 scores from a separate study of CodeBERT-style models and cites a CodeScene blog post for the SonarQube baseline, but no quantitative baseline values or paired comparison are given in the paper. The supporting data in Table 2 concerns test-case generation, not static analysis. A direct comparison table is needed, or the quantitative claim should be dropped or attributed with the exact source numbers.
minor comments (6)
  1. [Abstract, Section 2.3] The phrase 'test suit' should be 'test suite' (it also appears in the list of verification tasks in the Introduction).
  2. [3.2, Fig. 5 discussion] The text refers to 'section 2.4' for the AI-agent end-to-end workflow, but that workflow is described in Section 2.5.
  3. [Table 3] The column header 'Flaky failed executions' is ambiguous even apart from the mutated-flow issue; rename it to 'Unexpected outcomes' or split it into 'Flaky failures' and 'False negatives' after reclassification.
  4. [2.2] The value '66% for IndexError' appears twice in the list of F1 scores; remove the duplicate.
  5. [Conclusions] The passage 'Furthermore, the unexpected behavior observed in agentic end-to-end testing. The tendency to correct mutated test cases to match expected results' is a sentence fragment and should be merged into one complete sentence.
  6. [3.1, Table 2] The meaning of 'coverage according to format' is unclear; define whether it measures adherence to the requested six-field test-case structure or coverage of acceptance criteria.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical claims are measured against external LLMs and a public demo application, with no fitted parameter or self-citation chain doing load-bearing work.

full rationale

The paper's quantitative claims are empirical measurements, not derivations from fitted parameters. The 8.3% flakiness figure in Section 3.2 is the observed proportion of executions labeled 'Flaky failed executions' in Table 3 across six flows and two models; it does not presuppose the parity conclusion it is used to support. The F1 and executability figures are quoted from independent external sources (e.g., refs. 8, 9, 13), and the ReAct and LLM-as-a-judge methodologies are adopted from external prior work (refs. 26, 14), not from a self-citation chain. The authors' GitHub repository (ref. 30) is data availability, not a load-bearing uniqueness or derivation claim. The paper's own Limitations section (Section 3.3) concedes that agents 'deviate from negative test scenarios by attempting to correct the flow to achieve a positive outcome, masking potential failures,' which raises a legitimate validity question about counting mutated-flow outcomes in the same flakiness bucket as ordinary flows; however, that is a data-classification and comparability caveat, not a circular step. No equation or result in the paper reduces by construction to its inputs.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It relies on the domain assumption that the chosen demo app and manual evaluation labels are representative and accurate.

assumptions (2)
  • domain assumption The Sauce Demo e-commerce application is representative of modern distributed enterprise applications.
    The paper generalizes from this single toy app to enterprise QA practice (Sections 2.1, 3.2).
  • domain assumption Manual labeling of test-case executability and coverage is accurate.
    The paper relies on human annotation to score generated test cases (Section 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of AI-Driven Tools in Modern Software Quality Assurance: An Assessment of Benefits, Challenges, and Future Directions." pith.science (2026). https://pith.science/paper/F3PW6KPH

@misc{pith2026250616586,
  author       = {Pith},
  title        = {Pith review of: AI-Driven Tools in Modern Software Quality Assurance: An Assessment of Benefits, Challenges, and Future Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3PW6KPH}},
  note         = {Machine review of arXiv:2506.16586}
}
read the original abstract

Traditional quality assurance (QA) methods face significant challenges in addressing the complexity, scale, and rapid iteration cycles of modern software systems and are strained by limited resources available, leading to substantial costs associated with poor quality. The object of this research is the Quality Assurance processes for modern distributed software applications. The subject of the research is the assessment of the benefits, challenges, and prospects of integrating modern AI-oriented tools into quality assurance processes. We performed comprehensive analysis of implications on both verification and validation processes covering exploratory test analyses, equivalence partitioning and boundary analyses, metamorphic testing, finding inconsistencies in acceptance criteria (AC), static analyses, test case generation, unit test generation, test suit optimization and assessment, end to end scenario execution. End to end regression of sample enterprise application utilizing AI-agents over generated test scenarios was implemented as a proof of concept highlighting practical use of the study. The results, with only 8.3% flaky executions of generated test cases, indicate significant potential for the proposed approaches. However, the study also identified substantial challenges for practical adoption concerning generation of semantically identical coverage, "black box" nature and lack of explainability from state-of-the-art Large Language Models (LLMs), the tendency to correct mutated test cases to match expected results, underscoring the necessity for thorough verification of both generated artifacts and test execution results. The research demonstrates AI's transformative potential for QA but highlights the importance of a strategic approach to implementing these technologies, considering the identified limitations and the need for developing appropriate verification methodologies.

Figures

Figures reproduced from arXiv: 2506.16586 by the authors.

Figure 1
Figure 1. Cost of software quality 2022 [1] Traditional QA contains chain of multiple labor-intensive prac￾tices, which are a common bottleneck for product release lifecycle and struggle to address complexity and scale of modern enterprise software systems, and SAAS platforms. Generating test cases re￾quires comprehensive system analysis for exploratory testing and equivalence partitioning and is frequently shortened to only … view at source ↗
Figure 2
Figure 2. Test analyses square [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Test-case generation and validation system [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Unit-test generation system [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: AI-agent based end-to-end automation testing [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Metamorphic testing example [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 7
Figure 7. Figure 7: GPT-4o end-to-end flow [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 8
Figure 8. Figure 8: GPT-4o-mini end-to-end flow [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: GPT-4o-mini end-to-end flow with reasoning loop [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 21 canonical work pages

  1. [31]

    Micco, J. (2016). Flaky Tests at Google and How We Mitigate Them . Google T esting Blog. Available at: https://testing.googleblog.com/2016/05/flaky- tests-at-google-and-how-we.html

    work page 2016

  2. [1]

    Krasner, H. (2022). The Cost of Poor Software Quality In The Us: A 2022 Re - port. Austin. Available at: https://www.it-cisq.org/wp-content/uploads/ sites/6/2022/11/CPSQ-Report-Nov-22-2.pdf

    work page 2022

  3. [2]

    Cordy, M., Rwemalika, R., Franci, A., Papadakis, M., Harman, M. (2022). FlakiMe: Laboratory-Controlled T est Flakiness Impact Assessment. Proceed- ings of the 44th International Conference on Software Engineering. New York, 982–994. https://doi.org/10.1145/3510003.3510194

    arXiv 2022

  4. [3]

    Gumhold, F. (2022). How a lack of quality assurance can lead to a loss of $400 million in 37 seconds – ERNI. Available at: https://www.betterask.erni/how-a- lack-of-quality-assurance-can-lead-to-a-loss-of-400-million-in-37-seconds/

    work page 2022

  5. [4]

    Newtown Square: Proj- ect Management Institute, Inc

    Project Management Institute, A Guide to the Project Management Body of Knowl- edge and The Standard for Project Management (2021). Newtown Square: Proj- ect Management Institute, Inc

    work page 2021

  6. [5]

    A., Deshpande, A

    Deshpande, S. A., Deshpande, A. N., Marathe, M. V., Garje, G. V. (2010). Im - proving Software Quality with Agile T esting. International Journal of Com - puter Applications, 1 (22), 68–73. https://doi.org/10.5120/440-673

    work page doi:10.5120/440-673 2010

  7. [6]

    University of Minnesota Software Engi- neering Center

    Introduction to Software testing (2025). University of Minnesota Software Engi- neering Center. Available at: https://www.coursera.org/learn/introduction- software-testing/lecture/ohzH2/introduction

    work page 2025

  8. [7]

    Available at: https://www.saucedemo.com/

    Swag Labs. Available at: https://www.saucedemo.com/

    work page

Show all 32 references
  1. [8]

    Vokhranov, I., Bulakh, B. (2024). Transformer-based models application for bug detection in source code. Technology Audit and Production Reserves, 5 (2 (79)), 6–15. https://doi.org/10.15587/2706-5448.2024.310822

    2024

  2. [9]

    Available at: https://codescene.com/blog/6x-improvement-over-sonarqube

    6X improvement over SonarQube – Raising the Maintainability bar. Available at: https://codescene.com/blog/6x-improvement-over-sonarqube

  3. [10]

    Marjanov, T., Pashchenko, I., Massacci, F. (2022). Machine Learning for Source Code Vulnerability Detection: What Works and What Isn’t There Yet. IEEE Security & Privacy, 20 (5), 60–76. https://doi.org/10.1109/msec.2022.3176058

    2022

  4. [11]

    Roman, A. (2018). A Study Guide to the ISTQB® Foundation Level 2018 Syllabus. Springer International Publishing. https://doi.org/10.1007/978-3-319-98740-8

    2018 doi

  5. [12]

    C., Klein, J., Bissyandé, T

    Plein, L., Ouédraogo, W. C., Klein, J., Bissyandé, T. F. (2024). Automatic Gen - eration of T est Cases based on Bug Reports: A Feasibility Study with Large Language Models. Proceedings of the 2024 IEEE/ACM 46th International Con- ference on Software Engineering: Companion Pro...

    2024

  6. [13]

    Ayon, Z. A. H., Husain, G., Bisoi, R., Rahman, W., Osborn, D. T. (2025). An efficient approach to represent enterprise web application structure using Large Language Model in the service of Intelligent Quality Engineering. ArXiv. https:// doi.org/10.48550/arXiv.2501.06837

    2025 doi arXiv

  7. [14]

    Li, D., Jiang, B., Huang, L., Beigi, A., Zhao, C., Tan, Z. et al. (2024). From gen- eration to judgment: Opportunities and challenges of LLM-as-a-judge . arXiv. https://doi.org/10.48550/arXiv.2411.16594

    2024 doi

  8. [15]

    Müller, M., Žunič, G. (2024). Browser Use: Enable AI to control your browser. GitHub. Available at: https://github.com/browser-use/browser-use

    2024

  9. [16]

    Martin, R. C. (2017). Clean Architecture: A Craftsman’s Guide to Software Structure and Design. Prentice Hall Press

    2017

  10. [17]

    Daka, E., Fraser, G. (2014). A Survey on Unit T esting Practices and Problems. 2014 IEEE 25th International Symposium on Software Reliability Engineering, 201–211. https://doi.org/10.1109/issre.2014.11

    2014 doi

  11. [18]

    Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., Wang, Q. (2024). Software T esting With Large Language Models: Survey, Landscape, and Vision. IEEE Transactions on Software Engineering, 50 (4), 911–936. https://doi.org/10.1109/ tse.2024.3368208

    2024

  12. [20]

    Alshahwan, N., Chheda, J., Finogenova, A., Gokkaya, B., Harman, M., Harper, I. et al. (2024). Automated Unit T est Improvement using Large Language Models at Meta. Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering. New Yo...

    2024

  13. [21]

    Foster, C., Gulati, A., Harman, M., Harper, I., Mao, K., Ritchey, J. et al. (2025). Mutation-guided LLM-based test generation at Meta . arXiv. https://doi.org/ 10.48550/arXiv.2501.12862

    2025 doi arXiv

  14. [22]

    (2024, September)

    Shin, J., Aleithan, R., Hemmati, H., Wang, S. (2024, September). Retrieval- augmented test generation: How far are we? arXiv. https://doi.org/10.48550/ arXiv.2409.12682

    2024 doi

  15. [23]

    Hu, R., Peng, C., Wang, X ., Gao, C. (2025). An LLM-based agent for reliable Docker environment configuration . arXiv. https://doi.org/10.48550/arXiv.2502.13681

    2025 doi

  16. [24]

    S., Hao, M., Leesatapornwongsa, T., Patana-anake, T., Do, T., Adityatama, J

    Gunawi, H. S., Hao, M., Leesatapornwongsa, T., Patana-anake, T., Do, T., Adityatama, J. et al. (2014). What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems. Proceedings of the ACM Symposium on Cloud Com - puting. New York. https://doi.org/10.1145/2670979.2670986

    2014

  17. [25]

    Vocke, H. (2018). The Practical Test Pyramid. Available at: https:// martinfowler. com/articles/practical-test-pyramid.html

    2018

  18. [26]

    Y ao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., Cao, Y. (2022). ReAct: Synergizing reasoning and acting in language models . arXiv. https://doi. org/10.48550/arXiv.2210.03629 INFORMA TION AND CONTROL SYSTEMS: INFORMA TION TECHNOLOGIES 54 TECHNOLOGY AUDIT AND P...

    2022 doi arXiv

  19. [27]

    Ahlgren, J., Berezin, M., Bojarczuk, K., Dulskyte, E., Dvortsova, I., George, J. et al. (2021). T esting Web Enabled Simulation at Scale Using Metamorphic T esting. 2021 IEEE/ACM 43rd International Conference on Software Engi - neering: Software Engineering in Practice (ICSE-S...

    2021

  20. [28]

    T., Wu, T., Guestrin, C., Singh, S

    Ribeiro, M. T., Wu, T., Guestrin, C., Singh, S. (2020). Beyond Accuracy: Be - havioral T esting of NLP Models with CheckList. Proceedings of the 58th An - nual Meeting of the Association for Computational Linguistics . Stroudsburg, 4902–4912. https://doi.org/10.18653/v1/2020.a...

    2020 doi

  21. [29]

    John-Mathews, J.-M. (2022). How to test Machine Learning Models? Metamor- phic testing. Available at: https://www.giskard.ai/knowledge/how-to-test-ml- models-4-metamorphic-testing

    2022

  22. [30]

    Available at: https://github.com/igor- pysmennyi-kpi/qa-ai-overview-paper-2025

    Data for paper AI-Driven Testing Tools in Modern Quality Assurance: Benefits, Challenges, and Future Directions . Available at: https://github.com/igor- pysmennyi-kpi/qa-ai-overview-paper-2025

    2025

  23. [32]

    Valmeekam, K., Olmo, A., Sreedharan, S., Kambhampati, S. (2022). Large Language Models Still Can’t Plan (A Benchmark for LLMs on Planning and Reasoning about Change). iNeurIPS 2022 Foundation Models for Deci - sion Making Workshop, New Orleans . Available at: https://openrevie...

    2022

  24. [33]

    Igor Sikorsky Kyiv Polytechnic Institute

    Lee, H.-P . (Hank), Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., Wilson, N. (2025). The Impact of Generative AI on Critical Thinking: Self-Re- ported Reductions in Cognitive Effort and Confidence Effects from a Survey of Knowledge Workers. Proceedings of th...

    2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.