{"id":"d0fb0582-8b15-4bfe-9a05-e8d4d119e91c","arxiv_id":"2501.06837","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A hierarchical DOM representation fed to an LLM generates executable web tests, with 90% and 70% success on two small login/signup suites.","lead":"This paper proposes a hierarchical format for describing web page structures so that large language models can generate and run automated tests. The authors report 90% and 70% test execution success on a demo e-commerce site and an internal healthcare app, but without comparing against any alternative approach.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 90%/70% success rates are not evidence for the hierarchical representation because no baseline or ablation is run; the Section 5.1.2 prompts already dictate exactly which tests to generate, so the representation's causal contribution is unmeasured.","rationale":"I agree with the Reader's weakest_assumption and the REJECT verdict. The load-bearing issue is causal attribution: the paper claims the hierarchical representation enables or enhances LLM test generation, but the only evidence is pass/fail counts on two small flows and qualitative relevance ratings. The instruction prompts in Section 5.1.2 already enumerate the required scenarios, so the observed outcomes can be explained without the representation. The earlier failures of raw DOM or chunking are asserted in Section 4.2.1 but never measured, and no artifacts are available for independent reconstruction. The proposed ablation--holding prompts, model, and harness constant while varying only the input representation--would settle whether the hierarchical format is actually responsible. Since this concern reinforces the existing rationale for rejection rather than introducing a new direction, the Reader's verdict should remain unchanged.","tokens_in":12637,"tokens_out":4077,"duration_ms":42688,"concrete_test":"Run an ablation on the same two applications with the same LLM, temperature, prompts, and Selenium harness from Section 5.1, varying only the input representation: (a) the proposed hierarchical format; (b) the raw DOM dump; (c) a flat element list with no hierarchy or navigation flow; (d) no site-structure input, containing only the instruction text. For each condition, generate and execute the same 10 tests per app and report pass rate, proportion of syntactically invalid scripts, and blind expert relevance ratings. If (b)-(d) match the reported 90%/70% pass rates or relevance, the hierarchical representation is not load-bearing; if (a) clearly dominates, the causal claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is causal: the hierarchical representation 'significantly enhances' LLM-driven test generation. The experiments never isolate that variable. Section 5.1.2 shows the Swag Labs prompt already instructs the LLM to 'Generate a detailed test plan for logging in... valid and invalid credentials... Create a Selenium test script... Execute...' and the MediBox prompt says 'Create and execute a minimum of 10 functional test scripts specifically for the user signup process.' Thus the LLM is told exactly what to test; the 90%/70% execution pass rates in Table 3 could plausibly come from prompt specificity, the simplicity of the login/signup flows, the Selenium harness, or general LLM coding ability, with no contribution from the proposed site representation. The §4.2.1 narrative that full DOM and chunking approaches 'induced problems' is anecdotal and not quantified, and no baseline, ablation, or released artifacts allow the reader to verify the comparison. Table 4's 'High' relevance ratings are qualitative and self-assessed. Because the central claim depends on the representation being the active ingredient, the absence of a counterfactual that keeps the prompts and execution harness fixed while varying only the input representation leaves the causal claim unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a five-phase pipeline for representing enterprise web applications as a hierarchical structure (page-level summaries, section contexts, element details, and navigation flow) to enable LLM-based in-context test generation, Selenium execution, and AI-generated reporting. The central claim is that this representation materially improves the quality and executability of generated tests, supported by reported 90% and 70% execution success rates on two applications (Swag Labs and MediBox), a claim of approximately 87% correctness for one intermediate LLM output, and an assertion of upwards of 50% time savings over traditional automation.","tokens_in":12826,"tokens_out":4331,"duration_ms":42043,"significance":"If the causal claim were established, the hierarchical representation would be a practical contribution to automated functional web testing, offering a structured alternative to raw DOM dumps or naive chunking. The paper's strengths are that it reports real Selenium executions against two actual applications, lists concrete failure modes (timeouts, NoneType errors, assertion failures), and describes an end-to-end pipeline rather than a single component. However, the experimental design does not isolate the representation's contribution, the sample is tiny (20 tests total), the relevance metrics are self-assessed qualitative labels, and no artifact or baseline is provided; as a result, the significance of the claimed effect is not currently supportable.","major_comments":[{"comment":"The central causal claim that the hierarchical representation 'significantly enhances' test generation is not supported because no baseline or ablation varies only the input representation while keeping prompts and execution harness fixed. The prompts reproduced in Section 5.1.2 already enumerate the exact scenarios to generate (e.g., 'Generate a detailed test plan for logging in... valid and invalid credentials' and 'Create and execute a minimum of 10 functional test scripts specifically for the user signup process'), so the observed 90% and 70% success rates could plausibly be driven by prompt specificity, the simplicity of the login and signup flows, the Selenium harness, or the LLM's general coding ability. The statement in Section 4.2.1 that full DOM and chunking approaches 'induced problems' is anecdotal and unquantified, so it cannot establish the representation's contribution.","section":"Section 5, Table 3 and Section 5.1.2"},{"comment":"The claim that 'the information received from LLM after this operation to be correct approximately 87% of the time' is given without any definition of correctness, sample size, evaluation procedure, or error bars. This number is not tied to any table or experiment in the paper and cannot be assessed; it should either be substantiated with a defined measurement or removed.","section":"Section 3.3"},{"comment":"The relevance criteria ('Instruction Relevance', 'Web Application Relevance', 'Data Mappings Relevance', 'Synthetic Data Contextuality') are reported only as qualitative 'High' labels with no rubric, independent raters, or inter-rater agreement. Combined with only 20 executed test cases and no release of prompts, code, or logs, the evaluation is insufficient to justify the broad conclusion that the representation approach significantly improves LLM-based test generation. A quantitative scoring scheme and a larger, more diverse set of applications or tasks would be needed.","section":"Section 5, Table 4 and Section 4.7"},{"comment":"The closed set PT = {login, signup, account, listing, detail, form, static} is asserted to cover 'any page from any web application', but no evidence is given for its completeness or for how the page-type labels were validated. Because page-type identification is a component of the representation, this unsupported universal claim weakens the paper's generalization argument; the authors should at least explain how the set was derived and test it against a broader corpus of pages.","section":"Section 3.2, item 4"}],"minor_comments":[{"comment":"The methodology section states it is 'divided into four phases', but Sections 3.1 through 3.5 describe five phases; please correct the count or renumber consistently.","section":"Section 3"},{"comment":"There are typos in the text: 'fulfilling thee processes' in Section 1 and 'out test generation approach' in Section 3.2 should read 'the' and 'our', respectively.","section":"Section 1 and Section 3.2"},{"comment":"The reference to 'Jen-tse et al.' is written as 'Jen-Tse et al. Jen-tse et al.' and appears inconsistently; please unify the citation format.","section":"Section 2"},{"comment":"Table 4 mixes quantitative success rates (90% and 70%) with qualitative labels ('High') in the same 'Success Rate' columns; consider separating these or renaming the columns to avoid conflation.","section":"Section 5, Table 4"},{"comment":"The platform name 'Flame' is introduced without any description; please define it at first use or remove the reference.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's evaluation is conducted by the authors on their own Atalgo platform and the MediBox application, and the qualitative relevance ratings are self-assessed; independent replication with released artifacts and an external baseline would materially strengthen the claims. The time-savings and 87% correctness statements appear to be unsupported assertions that should be either substantiated or deleted in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: the headline success rates (90%, 70%) don't actually support the paper's causal claim that the hierarchical representation improves LLM test generation. There is no baseline, no ablation, and the prompts in 5.1.2 already tell the LLM which tests to write. The representation's contribution is unmeasured.\n\nWhat is genuinely new: the exact encoding—unique element IDs, a closed page-type set (login, signup, account, listing, detail, form, static), page-wise DOM chunking, and navigation-flow metadata—is a concrete variant not in the cited literature. The paper also gives a sensible pipeline: scrape DOM elements, build page-level summaries, generate test cases, execute via Selenium, and report with LLM summaries. The execution results are real (20 tests, 16 passed), which shows feasibility: the encoding does not break the LLM. That is a legitimate practical data point.\n\nSoft spots, in order. The big one is the missing counterfactual: keeping the prompts and harness fixed while varying only the input representation would be necessary to support 'significantly enhances.' Instead, Section 4.2.1 gives anecdotal complaints about full-DOM and chunking approaches without numbers. Second, 10 tests per app with no error bars is small, and the relevance ratings in Table 4 are self-defined and qualitative. Third, the '87% correct' and '50% time savings' are stated without methodology. Fourth, no artifacts are released, so independent verification is impossible. These are load-bearing gaps for any comparative claim.\n\nThat said, I don't think the paper is incoherent or a scam. It's a plausible engineering report with a concrete representation that might be useful to practitioners building LLM-based test automation. The clear thinking is there—they identified real problems (sibling locators, context limits) and made reasonable choices. The writing is rough in places (typos, vague phrases), but the structure is understandable.\n\nWho this is for: a practitioner who wants a starting point for representing web apps to an LLM, not a researcher needing validated evidence. I'd bring it to a reading group as an example of how not to evaluate a technique, but I wouldn't cite it as evidence in my own work.\n\nRecommendation: send to peer review anyway. The method is concrete and the feasibility data is real, and a decent reviewer could push the authors to add baselines and release artifacts. It's a borderline but not a desk-reject.","headline":"A concrete encoding for web app structure, but the reported success rates don't get causal because no one ran the counterfactual.","tokens_in":13393,"tokens_out":2355,"would_cite":false,"duration_ms":21683,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A scraped DOM, reformatted as a typed page hierarchy with navigation flow, lets a large language model generate executable tests with 90% and 70% success on two applications.","keywords":["Large Language Model","In-Context Learning","Document Object Model","Generative AI","Hierarchical Representation","Enterprise Test Automation","Intelligent Quality Engineering"],"falsifier":"Run the identical prompt set and Selenium harness on both applications under four input conditions: the proposed hierarchy, raw DOM, naive page-wise chunking, and a flat element list without page types or navigation flow. If execution success and relevance ratings do not fall in the non-hierarchical conditions, the observed 90% and 70% cannot be attributed to the representation.","tokens_in":12376,"feed_emoji":"🧪","tokens_out":6085,"duration_ms":55491,"temperature":0.7,"pith_summary":"This paper claims that a web application can be summarized into a hierarchical text representation—page types, section contexts, element details with unique identifiers, and navigation flow—that lets a large language model understand the whole site within a prompt. The authors build a five-phase pipeline that scrapes the DOM, synthesizes the site structure, generates Selenium test cases from instructions, executes them, and summarizes results. On a login flow and a signup flow, they report 90% and 70% execution success rates and 'High' relevance across evaluation criteria. A sympathetic reader would take the claim to be that this representation, not the LLM's general coding skill alone, is what makes generated tests executable and relevant. If true, the approach offers a path toward autonomous end-to-end quality engineering for enterprise web applications without model fine-tuning.","feed_headline":"A page hierarchy lets LLMs write working web tests at 90% and 70%","feed_subtitle":"Scraped DOM structure with page types and navigation flow turns login and signup suites into executable Selenium scripts.","key_machinery":"The central object is the hierarchical site structure produced in the multi-page analysis phase: page-level summaries, section-level contexts, element-level details with unique IDs, and a navigation flow with priorities. A closed set of seven page types (login, signup, account, listing, detail, form, static) anchors the representation, while unique element identifiers prevent confusion between sibling elements whose locators look similar. The mechanism is that this structure fits within the LLM's context window page-wise while preserving cross-page relationships, so few-shot in-context learning can generate tests that follow both instructions and application flow.","core_discovery":"The paper claims that the bottleneck in using LLMs for web testing is not the model but how the application is presented. Its discovery is a representation that keeps the DOM's hierarchy intact: each element gets a unique identifier alongside its locator, pages are typed from a closed set (login, signup, account, listing, detail, form, static), elements are grouped into logical sections, and navigation flow and priority are encoded. Fed this structure page by page, an LLM produces test cases that are validated for structural integrity, uniqueness, and relevance, mapped to real or synthetic test data, and translated into Selenium actions. On ten login tests for an e-commerce app and ten signup tests for a healthcare app, 90% and 70% executed successfully, with relevance rated 'High' on instruction adherence, application fit, data mapping, and synthetic data contextuality. The authors also report that the LLM's correctness on their priority-prediction task was approximately 87%.","pith_inferences":["If the representation is the causal factor, then ablating it—raw DOM, naive chunking, or a flat element list—should drop execution success and instruction relevance; this is a direct testable consequence the paper does not report.","The closed set of seven page types may not cover flows like checkout wizards or dashboards; a natural extension is to expand or learn page types and measure whether success holds.","Because the method passes page-wise hierarchical chunks instead of the whole DOM, its gains could come partly from context-window management; separating that from hierarchical ordering would clarify the mechanism.","If the reported time savings generalize, the representation may be more valuable for keeping tests alive across application changes than for first-generation test creation, since the experiments do not directly test maintenance."],"forward_implications":["Login and signup test suites can be generated and executed largely automatically: nine of ten login tests and seven of ten signup tests passed on the first pipeline run.","The same representation drives test case generation, test data mapping, and result reporting, so generated suites and failure summaries stay tied to the actual page structure.","Page-wise DOM chunking with navigation flow avoids feeding the whole DOM, keeping inputs within the LLM context window while retaining site-level understanding.","The authors estimate time savings above 50% versus traditional automation once setup is complete, with the savings more pronounced in maintenance as the application scales."],"supporting_citations":[{"why":"Supplies the systematic literature review motivating automated test generation from requirements and the need to match generation approaches to requirement characteristics.","marker":"[15]"},{"why":"Provides a prior method for generating functional tests from UI test scenarios written in human-friendly languages, which the current work extends to full application representation.","marker":"[18]"},{"why":"Defines the Selenium web-testing practices and automation capabilities used as the execution layer for the generated test suites.","marker":"[19]"},{"why":"Shows how large language models can interpret restricted natural language requirements into functional test scripts, the closest prior step this representation builds on.","marker":"[22]"},{"why":"Highlights that generated test cases may not preserve intended semantics and can produce high false-alarm rates, motivating the structural validation step in the proposed pipeline.","marker":"[32]"}],"fun_headline_variants":["LLMs write passing web tests when fed a page hierarchy","Page hierarchy + LLM = 90% success on e-commerce tests","Structured DOM representation boosts LLM test generation to 90%","Web app hierarchy enables LLM-based testing at 90% and 70%","LLM test automation succeeds with hierarchical page representation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim attributes the test success to the hierarchical representation, but the experiments never compare it with the same LLM and execution harness using raw DOM, naive chunking, or flat element lists; on such simple flows those alternatives might perform equally well.","fun_headline_variants_meta":{"raw":{"variants":["LLMs write passing web tests when fed a page hierarchy","Page hierarchy + LLM = 90% success on e-commerce tests","Structured DOM representation boosts LLM test generation to 90%","Web app hierarchy enables LLM-based testing at 90% and 70%","LLM test automation succeeds with hierarchical page representation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1867,"prompt_tokens":947,"completion_tokens":920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":831}},"tokens_in":563,"tokens_out":920,"duration_ms":6365,"temperature":1.0,"reasoning_tokens":831,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:49:57.344162+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identical prompt set and Selenium harness on both applications under four input conditions: the proposed hierarchy, raw DOM, naive page-wise chunking, and a flat element list without page types or navigation flow. If execution success and relevance ratings do not fall in the non-hierarchical conditions, the observed 90% and 70% cannot be attributed to the representation.","supporting_citations":[{"cited_title":"Automated test case generation from requirements: A systematic literature review","cited_arxiv_id":null,"evidence_quote":"Supplies the systematic literature review motivating automated test generation from requirements and the need to match generation approaches to requirement characteristics."},{"cited_title":"Functional test generation from ui test scenarios using reinforcement learning for android applications","cited_arxiv_id":null,"evidence_quote":"Provides a prior method for generating functional tests from UI test scenarios written in human-friendly languages, which the current work extends to full application representation."},{"cited_title":"Web program testing using selenium python: Best practices and effective approaches","cited_arxiv_id":null,"evidence_quote":"Defines the Selenium web-testing practices and automation capabilities used as the execution layer for the generated test suites."},{"cited_title":"Automating test oracles from restricted natural language agile requirements","cited_arxiv_id":null,"evidence_quote":"Shows how large language models can interpret restricted natural language requirements into functional test scripts, the closest prior step this representation builds on."},{"cited_title":"AEON: A Method for Automatic Evaluation of NLP Test Cases","cited_arxiv_id":"2205.06439","evidence_quote":"Highlights that generated test cases may not preserve intended semantics and can produce high false-alarm rates, motivating the structural validation step in the proposed pipeline."}],"review_version":1}