{"id":"c774654e-9612-4b13-b4bd-4c76f563f72c","arxiv_id":"2506.16586","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AI models can generate and execute test cases on a demo app, but the reported flakiness rate masks a more serious false-negative problem.","lead":"This paper explores whether AI language models can generate software tests and run browser-based regression checks, using a small e-commerce demo. It found some success, but its headline '8.3% flaky executions' conflates nondeterministic failures with cases where the AI silently fixed injected bugs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 8.3% flakiness figure mixes mutated scenarios (where a pass is a false negative, not a flaky result) with ordinary flows, so the parity claim against scripted suites is not supported as stated.","rationale":"The central empirical claim is the Conclusions sentence that ReAct-style browser agents achieve only 8.3% flaky runs and are on parity with mature scripted suites. The load-bearing assumption is that the 'flaky failed executions' column in Table 3 is a valid flakiness measure. The reader's weakest_assumption identifies the same issue: mutated scenarios are counted in the same flakiness bucket, even though a mutated flow that passes because the agent repairs the defect is a false negative, not a flaky run. The paper itself acknowledges this correction behavior in Section 3.3, so this is not an external or adversarial interpretation. The concrete test separates mutated from non-mutated executions and reclassifies outcomes using the provided logs; this directly settles whether 8.3% survives scrutiny. Because the paper is transparent about its limitations, provides artifacts, and the remaining evidence supports at most a conditional acceptance, the reader's CONDITIONAL verdict remains appropriate. No change to the verdict is needed.","tokens_in":12395,"tokens_out":4975,"duration_ms":55697,"concrete_test":"Recompute Table 3 from the e2elogs directory in the GitHub repository [30]. For each mutated flow, classify every execution as: (a) mutation detected (failed as expected), (b) mutation missed (agent corrected the mutation and passed), or (c) genuinely flaky (same input produced inconsistent pass/fail outcomes across the 4 runs). Recount flakiness using only category (c) and report the false-negative rate separately. If the 8.3% figure changes or overlaps with the false-negative rate, revise the Conclusions to report flakiness for ordinary (non-mutated) flows and to disclose the mutation-miss rate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 reports 48 executions (6 flows × 2 models × 4 runs), and Table 3 labels all non-zero outcomes as 'Flaky failed executions'. Two of the three non-zero rows are mutated flows: 'Sorting mutated' has 25% for gpt-4o-mini and 50% for gpt-4o, while the only non-mutated non-zero row is 'Buys&Checkout' at 25% for gpt-4o-mini. The paper's own Limitations section (3.3) states that agents 'deviate from negative test scenarios by attempting to correct the flow to achieve a positive outcome, masking potential failures.' For a mutated flow, a run that the agent 'corrects' into a pass is not a flaky result; it is a false negative. Counting such outcomes in the same 'flaky failed executions' bucket inflates the headline 8.3% (4/48) and makes the comparison to Google's production flakiness statistics [31] invalid, since those statistics concern nondeterministic outcomes of unchanged tests. Excluding mutated flows, the observed flaky rate would be 1/32 = 3.1%, so the Conclusions sentence asserting 'only 8.3% flaky runs, being on parity with mature scripted suites' is quantitatively under-supported unless the mutated outcomes are reclassified and separated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper assesses the use of LLM-based tools across QA activities—static analysis, test-case generation, unit test generation, end-to-end automation, and metamorphic testing—and reports two experiments. Experiment 1 (Section 3.1) evaluates seven LLMs generating test cases from user stories for the Sauce Demo application. Experiment 2 (Section 3.2) uses two OpenAI models as ReAct-style browser agents to execute six login, sorting, and checkout flows, four times each (48 executions total), and reports an aggregate 8.3% 'flaky' execution rate. The Conclusions present the 8.3% figure as evidence that agentic end-to-end regression is on parity with mature scripted suites, while also acknowledging false-negative behavior in mutated scenarios.","tokens_in":12795,"tokens_out":6419,"duration_ms":67253,"significance":"If the headline empirical claim were robust, the paper would offer a useful data point for practitioners considering LLM-based end-to-end regression. The authors deserve credit for making code and execution logs publicly available, using a real public application, and including an explicit limitations section that anticipates the false-negative problem. However, the evidence base is too thin and the headline metric is conceptually muddled: the flakiness rate conflates nondeterministic failures with expected failures on mutated inputs, and four runs per scenario cannot sustain the comparison to Google's production statistics. The survey portions are competent but largely derivative; the value of the paper hinges on the experimental section, which currently overstates its support.","major_comments":[{"comment":"The 'Flaky failed executions' column mixes two distinct phenomena: nondeterministic failures of unchanged test flows and false negatives on mutated flows, where the agent 'corrects' the defect and reports a pass. Section 3.3 itself states that agents 'deviate from negative test scenarios by attempting to correct the flow to achieve a positive outcome, masking potential failures.' A run that masks a mutation is not a flaky result under the standard definition (variation in outcome of an unchanged test); it is an oracle/agent failure. Counting the three mutated-flow outcomes in Table 3 together with the one non-mutated 'Buys&Checkout' outcome produces the aggregate 4/48 = 8.3%. Excluding mutated flows, the observed rate is 1/32 = 3.1%. Therefore the Conclusions sentence asserting 'only 8.3% flaky runs, being on parity with mature scripted suites' is not supported as stated. Please reclassify these outcomes, present separate rates, and revise the conclusion accordingly.","section":"3.2, Table 3"},{"comment":"Even after reclassification, the comparison to Google's production flakiness statistics (reference [31]) is not meaningful without a shared measurement protocol. Each scenario was executed only four times per model, so a single altered outcome changes the reported rate by 25 percentage points, and the corresponding confidence intervals are extremely wide. Google's numbers describe unchanged tests running in large CI pipelines, not small-sample agent executions on a public demo application. The paper should report exact failure counts per scenario, provide confidence intervals or otherwise acknowledge the uncertainty, and either remove the parity claim or support it with a matched baseline that is actually run in this study.","section":"3.2, Table 3"},{"comment":"The conclusion that transformer-based static analyzers boost F1 scores 'by more than 40 percentage points over a SonarQube baseline' is not supported by the material presented. Section 2.2 lists F1 scores from a separate study of CodeBERT-style models and cites a CodeScene blog post for the SonarQube baseline, but no quantitative baseline values or paired comparison are given in the paper. The supporting data in Table 2 concerns test-case generation, not static analysis. A direct comparison table is needed, or the quantitative claim should be dropped or attributed with the exact source numbers.","section":"2.2, Conclusions"}],"minor_comments":[{"comment":"The phrase 'test suit' should be 'test suite' (it also appears in the list of verification tasks in the Introduction).","section":"Abstract, Section 2.3"},{"comment":"The text refers to 'section 2.4' for the AI-agent end-to-end workflow, but that workflow is described in Section 2.5.","section":"3.2, Fig. 5 discussion"},{"comment":"The column header 'Flaky failed executions' is ambiguous even apart from the mutated-flow issue; rename it to 'Unexpected outcomes' or split it into 'Flaky failures' and 'False negatives' after reclassification.","section":"Table 3"},{"comment":"The value '66% for IndexError' appears twice in the list of F1 scores; remove the duplicate.","section":"2.2"},{"comment":"The passage 'Furthermore, the unexpected behavior observed in agentic end-to-end testing. The tendency to correct mutated test cases to match expected results' is a sentence fragment and should be merged into one complete sentence.","section":"Conclusions"},{"comment":"The meaning of 'coverage according to format' is unclear; define whether it measures adherence to the requested six-field test-case structure or coverage of acceptance criteria.","section":"3.1, Table 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a decent literature sweep plus a small, transparent proof-of-concept on one demo e-commerce app. The survey part has real value. The experimental part does not support its own headline. The often-quoted 8.3% flaky-executions figure mixes ordinary nondeterministic failures with runs where the agent 'fixed' a mutated negative scenario and passed, which is a false negative, not flakiness. The stress-test note is right. Excluding mutated flows, the flaky rate is 1/32 = 3.1%, and with four runs per scenario the confidence interval is enormous. Comparing that to Google's production flakiness stats [31] is apples-to-oranges.\n\nWhat's new: the paper surveys LLM applications across verification and validation tasks—static analysis, test generation, unit generation, mutation testing, metamorphic testing, end-to-end agents—citing recent work including Meta's TestGen and mutation-guided generation. The two experiments generate test cases from user stories for Sauce Demo and run a ReAct-style browser agent on a handful of flows. The per-model cost/token/execution-time data is real, and the authors ship code and logs in a GitHub repo. That is reproducible.\n\nWhat the paper does well: it is candid about the mutation-correction behavior, admits the black-box and explainability problems, and proposes guardrails like token caps, checkpoint assertions, and LLM-as-a-judge with RAG. The limitations section is honest, which makes the overstated abstract and conclusions even more puzzling.\n\nSoft spots: (1) the flakiness classification in Table 3. Mutated flows that end in 'pass' are counted as flaky, but they are actually undetected mutants. That is a measurement validity problem, not a minor wrinkle. (2) n=4 per scenario per model is too noisy to support 'parity with mature scripted suites.' (3) Table 2's metrics—AC covered, exploratory testing, coverage according to format—are not precisely defined; 'exploratory testing over stories' looks like a human judgment with no inter-rater reliability. (4) The survey covers the right papers, and the citation pattern is fine.\n\nFor a serious venue, this would need the flakiness analysis redone, the mutated outcomes separated, and a much larger test matrix. As is, it is a borderline practice paper. If I were editing the journal, I would send it to review with a request for revision: the topic is current, the artifacts are there, and the core question—whether agentic regression testing is reliable—is worth a careful look.\n\nRecommendation: engage with it as a survey and reproducible proof-of-concept, but do not cite the 8.3% figure.","headline":"A useful survey and a transparent but undersized experiment whose headline 8.3% flakiness claim doesn't survive separating mutated false-negatives from true flakiness.","tokens_in":13168,"tokens_out":2548,"would_cite":false,"duration_ms":27015,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that ReAct-style browser agents can generate and execute end-to-end regression tests on a sample web app with an 8.3% flaky-execution rate—on par with mature scripted suites—at under $0.005 per generated user story.","keywords":["quality assurance","LLM","AI agents","end-to-end testing","test case generation","flaky tests","ReAct","metamorphic testing"],"falsifier":"Re-run the same six flows on a multi-page application with injected faults, separating mutated from unmutated scenarios and logging whether the agent alters the mutated input before reporting a pass. If the agent consistently passes mutated scenarios, or if flakiness on unmutated flows does not reproduce in a larger sample, the parity claim fails; any run where the agent changes the test data should be reclassified from 'flaky' to 'false negative'.","tokens_in":12216,"feed_emoji":"🤖","tokens_out":8633,"duration_ms":85284,"temperature":0.7,"pith_summary":"The paper argues that LLM-based tools can take over a large share of software quality assurance work, from writing test cases to executing full end-to-end regression flows. Its central empirical claim is that a ReAct-style browser agent running generated test scenarios on a sample e-commerce app produced only 8.3% flaky executions across 48 runs—a rate the authors put on par with mature scripted suites—at a cost of roughly $0.005 per generated user story for the best model. The paper also documents a serious caveat: agents often 'correct' deliberately broken mutated scenarios back to a passing state, which would conceal real defects, so the authors argue that execution logs and generated artifacts need human- or machine-based verification and that guardrails such as token caps and checkpoint assertions are required.","feed_headline":"Agentic AI runs web regression tests with 8.3% flaky rate","feed_subtitle":"Demo on a sample store shows LLM agents can generate and run end-to-end tests, but may also hide real bugs.","key_machinery":"The load-bearing mechanism is the ReAct feedback loop, in which the language model repeatedly reasons about the current browser state, plans an action, executes it through a browser automation engine (Playwright), and observes the result before replanning. This closed loop lets the agent adapt to timing issues, unexpected popups, and UI changes that make scripted locator-based tests flaky, and it is what the paper credits for the 8.3% flaky rate. The proposed companion machinery is the LLM-as-a-judge validator with RAG context enrichment, which filters and corrects generated test cases, and mutation testing, which judges whether generated unit tests genuinely increase coverage by checking whether they catch seeded defects.","core_discovery":"The central discovery, as the authors state it, is that a single LLM-driven workflow can close the loop between test design and test execution: state-of-the-art models generate runnable acceptance tests from user stories (with 100% acceptance-criteria coverage in the best case), and the same family of models, embedded in a ReAct-style agent, can execute those scenarios in a live browser. Across two of the evaluated models (gpt-4o and gpt-4o-mini), six flows, and four runs apiece, only 4 of 48 executions were flaky or failed—8.3%—which the authors compare directly to the industry flakiness statistics of large-scale production suites. The authors treat this as evidence that AI-agent-based end-to-end regression is a practical alternative or supplement to traditional automation, provided that the known failure modes—particularly the agents' tendency to repair mutated negative scenarios and the smaller model's infinite reasoning loop in the checkout flow—are controlled.","pith_inferences":["The paper's 8.3% figure conflates two very different failure modes: the mutated flows that agents 'corrected' are false negatives (bugs missed), not flaky runs, while true flakiness appears only in non-mutated flows. Re-analysis of Table 3 suggests both models had 0% flakiness on the four unmutated flows, so the honest headline might be '0% flaky on normal flows, but 25–50% of mutated scenarios we","A natural next experiment is to vary the mutation severity and count how often the agent alters the input before reporting pass; that number is a direct measure of the false-negative risk the paper flags and can be benchmarked against human testers or scripted negative tests.","The infinite reasoning loop observed with the smaller model suggests cost and latency—not just correctness—should be part of any evaluation of agentic QA; a token/time cap could turn a $0.05 test into an availability incident, so the paper's recommended guardrails double as SLO protections."],"forward_implications":["If the 8.3% figure generalizes, agent-based end-to-end regression can substitute for scripted suites on UI-heavy applications, eliminating the dominant maintenance cost of locator and xpath updates.","Test design effort shifts from scripting to prompt-writing and output review; the best model in Experiment 1 reached 100% acceptance-criteria coverage at about $0.005 per user story, making LLM-generated cases a cheap first-pass test asset.","Because agents can silently 'fix' intentionally broken scenarios, a pass/fail status is not enough: pipelines must verify execution logs and detect input alteration, or real bugs will be reported as green.","Pairing an observation-based unit-test generator with a mutation-testing feedback loop would retain only tests that add effective coverage, directly addressing the semantic-duplication problem the paper reports.","If adopted into continuous delivery, these techniques move defect discovery earlier in the software lifecycle and against the paper's cited multi-trillion-dollar annual cost of poor-quality software, but they require reskilling human QA into prompt engineers and governance owners."],"supporting_citations":[{"why":"Supplies the ReAct reasoning-acting loop that underpins the agentic end-to-end execution experiment.","marker":"[26]"},{"why":"Provides the industry flakiness statistics (Google's flaky-test data) against which the 8.3% figure is judged.","marker":"[31]"},{"why":"The Sauce Demo (Swag Labs) e-commerce application used as the system under test in both experiments.","marker":"[7]"},{"why":"The GitHub repository with generated test cases, execution logs, and scripts that implement the proof-of-concept.","marker":"[30]"},{"why":"The browser-use agent framework used to drive the browser in the end-to-end experiment.","marker":"[15]"},{"why":"The LLM-as-a-judge methodology used in the proposed test-case generation and validation pipeline.","marker":"[14]"},{"why":"Gives the 70–90% executability range for LLM-generated test cases that frames Experiment 1's results.","marker":"[13]"}],"fun_headline_variants":["LLM agents generate and run web tests, with only 8.3% flaky","Agentic QA: AI writes and runs acceptance tests, 8.3% flaky","LLM QA agents: 8.3% flaky, but they can patch mutated tests","AI test agents: 8.3% flaky — and they can hide real bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline 8.3% flakiness figure rests on 48 executions of six flows on a single demo app, and it lumps failed executions of deliberately mutated scenarios into 'flaky' even though the paper itself shows agents often correct those mutations—so the failures may be false negatives, not instability.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents generate and run web tests, with only 8.3% flaky","Agentic QA: AI writes and runs acceptance tests, 8.3% flaky","LLM QA agents: 8.3% flaky, but they can patch mutated tests","AI test agents: 8.3% flaky — and they can hide real bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001483,"raw_usage":{"total_tokens":5999,"prompt_tokens":1030,"completion_tokens":4969,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":4873}},"tokens_in":646,"tokens_out":4969,"duration_ms":36052,"temperature":1.0,"reasoning_tokens":4873,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:36:27.816604+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same six flows on a multi-page application with injected faults, separating mutated from unmutated scenarios and logging whether the agent alters the mutated input before reporting a pass. If the agent consistently passes mutated scenarios, or if flakiness on unmutated flows does not reproduce in a larger sample, the parity claim fails; any run where the agent changes the test data should be reclassified from 'flaky' to 'false negative'.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the industry flakiness statistics (Google's flaky-test data) against which the 8.3% figure is judged."},{"cited_title":"Available at: https://www.saucedemo.com/","cited_arxiv_id":null,"evidence_quote":"The Sauce Demo (Swag Labs) e-commerce application used as the system under test in both experiments."},{"cited_title":"Available at: https://github.com/igor- pysmennyi-kpi/qa-ai-overview-paper-2025","cited_arxiv_id":null,"evidence_quote":"The GitHub repository with generated test cases, execution logs, and scripts that implement the proof-of-concept."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The browser-use agent framework used to drive the browser in the end-to-end experiment."},{"cited_title":"An efficient approach to represent enterprise web application structure using Large Language Model in the service of Intelligent Quality Engineering","cited_arxiv_id":"2501.06837","evidence_quote":"Gives the 70–90% executability range for LLM-generated test cases that frames Experiment 1's results."}],"review_version":1}