REVIEW 4 major objections 5 minor 1 cited by
An efficient approach to represent enterprise web application structure using Large Language Model in the service of Intelligent Quality Engineering
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A scraped DOM, reformatted as a typed page hierarchy with navigation flow, lets a large language model generate executable tests with 90% and 70% success on two applications.
desk verdict A concrete encoding for web app structure, but the reported success rates don't get causal because no one ran the counterfactual. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the hierarchical site structure produced in the multi-page analysis phase: page-level summaries, section-level contexts, element-level details with unique IDs, and a navigation flow with priorities. A closed set of seven page types (login, signup, account, listing, detail, form, static) anchors the representation, while unique element identifiers prevent confusion between sibling elements whose locators look similar. The mechanism is that this structure fits within the LLM's context window page-wise while preserving cross-page relationships, so few-shot in-context learning can generate tests that follow both instructions and application flow.
What would settle it
Run the identical prompt set and Selenium harness on both applications under four input conditions: the proposed hierarchy, raw DOM, naive page-wise chunking, and a flat element list without page types or navigation flow. If execution success and relevance ratings do not fall in the non-hierarchical conditions, the observed 90% and 70% cannot be attributed to the representation.
Extended reading notes
Core claim
The paper claims that the bottleneck in using LLMs for web testing is not the model but how the application is presented. Its discovery is a representation that keeps the DOM's hierarchy intact: each element gets a unique identifier alongside its locator, pages are typed from a closed set (login, signup, account, listing, detail, form, static), elements are grouped into logical sections, and navigation flow and priority are encoded. Fed this structure page by page, an LLM produces test cases that are validated for structural integrity, uniqueness, and relevance, mapped to real or synthetic test data, and translated into Selenium actions. On ten login tests for an e-commerce app and ten signup tests for a healthcare app, 90% and 70% executed successfully, with relevance rated 'High' on instruction adherence, application fit, data mapping, and synthetic data contextuality. The authors also report that the LLM's correctness on their priority-prediction task was approximately 87%.
Load-bearing premise
The claim attributes the test success to the hierarchical representation, but the experiments never compare it with the same LLM and execution harness using raw DOM, naive chunking, or flat element lists; on such simple flows those alternatives might perform equally well.
Editorial extensions
If this is right
- Login and signup test suites can be generated and executed largely automatically: nine of ten login tests and seven of ten signup tests passed on the first pipeline run.
- The same representation drives test case generation, test data mapping, and result reporting, so generated suites and failure summaries stay tied to the actual page structure.
- Page-wise DOM chunking with navigation flow avoids feeding the whole DOM, keeping inputs within the LLM context window while retaining site-level understanding.
- The authors estimate time savings above 50% versus traditional automation once setup is complete, with the savings more pronounced in maintenance as the application scales.
Reading between the lines
- If the representation is the causal factor, then ablating it—raw DOM, naive chunking, or a flat element list—should drop execution success and instruction relevance; this is a direct testable consequence the paper does not report.
- The closed set of seven page types may not cover flows like checkout wizards or dashboards; a natural extension is to expand or learn page types and measure whether success holds.
- Because the method passes page-wise hierarchical chunks instead of the whole DOM, its gains could come partly from context-window management; separating that from hierarchical ordering would clarify the mechanism.
- If the reported time savings generalize, the representation may be more valuable for keeping tests alive across application changes than for first-generation test creation, since the experiments do not directly test maintenance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a five-phase pipeline for representing enterprise web applications as a hierarchical structure (page-level summaries, section contexts, element details, and navigation flow) to enable LLM-based in-context test generation, Selenium execution, and AI-generated reporting. The central claim is that this representation materially improves the quality and executability of generated tests, supported by reported 90% and 70% execution success rates on two applications (Swag Labs and MediBox), a claim of approximately 87% correctness for one intermediate LLM output, and an assertion of upwards of 50% time savings over traditional automation.
Significance. If the causal claim were established, the hierarchical representation would be a practical contribution to automated functional web testing, offering a structured alternative to raw DOM dumps or naive chunking. The paper's strengths are that it reports real Selenium executions against two actual applications, lists concrete failure modes (timeouts, NoneType errors, assertion failures), and describes an end-to-end pipeline rather than a single component. However, the experimental design does not isolate the representation's contribution, the sample is tiny (20 tests total), the relevance metrics are self-assessed qualitative labels, and no artifact or baseline is provided; as a result, the significance of the claimed effect is not currently supportable.
major comments (4)
- [Section 5, Table 3 and Section 5.1.2] The central causal claim that the hierarchical representation 'significantly enhances' test generation is not supported because no baseline or ablation varies only the input representation while keeping prompts and execution harness fixed. The prompts reproduced in Section 5.1.2 already enumerate the exact scenarios to generate (e.g., 'Generate a detailed test plan for logging in... valid and invalid credentials' and 'Create and execute a minimum of 10 functional test scripts specifically for the user signup process'), so the observed 90% and 70% success rates could plausibly be driven by prompt specificity, the simplicity of the login and signup flows, the Selenium harness, or the LLM's general coding ability. The statement in Section 4.2.1 that full DOM and chunking approaches 'induced problems' is anecdotal and unquantified, so it cannot establish the representation's contribution.
- [Section 3.3] The claim that 'the information received from LLM after this operation to be correct approximately 87% of the time' is given without any definition of correctness, sample size, evaluation procedure, or error bars. This number is not tied to any table or experiment in the paper and cannot be assessed; it should either be substantiated with a defined measurement or removed.
- [Section 5, Table 4 and Section 4.7] The relevance criteria ('Instruction Relevance', 'Web Application Relevance', 'Data Mappings Relevance', 'Synthetic Data Contextuality') are reported only as qualitative 'High' labels with no rubric, independent raters, or inter-rater agreement. Combined with only 20 executed test cases and no release of prompts, code, or logs, the evaluation is insufficient to justify the broad conclusion that the representation approach significantly improves LLM-based test generation. A quantitative scoring scheme and a larger, more diverse set of applications or tasks would be needed.
- [Section 3.2, item 4] The closed set PT = {login, signup, account, listing, detail, form, static} is asserted to cover 'any page from any web application', but no evidence is given for its completeness or for how the page-type labels were validated. Because page-type identification is a component of the representation, this unsupported universal claim weakens the paper's generalization argument; the authors should at least explain how the set was derived and test it against a broader corpus of pages.
minor comments (5)
- [Section 3] The methodology section states it is 'divided into four phases', but Sections 3.1 through 3.5 describe five phases; please correct the count or renumber consistently.
- [Section 1 and Section 3.2] There are typos in the text: 'fulfilling thee processes' in Section 1 and 'out test generation approach' in Section 3.2 should read 'the' and 'our', respectively.
- [Section 2] The reference to 'Jen-tse et al.' is written as 'Jen-Tse et al. Jen-tse et al.' and appears inconsistently; please unify the citation format.
- [Section 5, Table 4] Table 4 mixes quantitative success rates (90% and 70%) with qualitative labels ('High') in the same 'Success Rate' columns; consider separating these or renaming the columns to avoid conflation.
- [Section 5] The platform name 'Flame' is introduced without any description; please define it at first use or remove the reference.
Circularity Check
No significant circularity: the central test-execution results are externally grounded in Selenium pass/fail outcomes, and no load-bearing claim reduces to its inputs by definition or self-citation.
full rationale
The paper's central quantitative claims are the 90% and 70% Selenium execution success rates (Section 5.1.1, Table 3), which are empirical pass/fail outcomes of generated scripts executed against real web applications. The hierarchical representation is not defined in terms of these outcomes, and no fitted parameter is renamed as a prediction. The only potentially self-referential evaluation criterion is 'relevance of the test cases based on instruction provided' (Section 5.1.2), where test cases are compared against instructions that were also included in the LLM prompt; this is a weak, partially tautological metric, but it is not the load-bearing result and does not make the overall derivation circular. The paper contains no load-bearing self-citations, imports no uniqueness theorem from prior author work, and does not smuggle an ansatz in via citation. The main validity threat is that no baseline or ablation isolates the representation's causal contribution, so the claim that the representation 'significantly enhances' LLM test generation is under-supported; however, that is an experimental-design and correctness concern, not circularity of the kind where a result equals its input by construction. Accordingly, the appropriate circularity finding is 0.
Assumptions & free parameters
assumptions (4)
- ad hoc to paper For any page from any web application, the page type is a member of the closed set PT = {login, signup, account, listing, detail, form, static}.
- domain assumption Page-wise DOM chunking plus navigation-flow metadata preserves enough hierarchical context for LLM test generation.
- domain assumption Assigning unique element identifiers resolves locator ambiguity that misleads LLMs.
- domain assumption LLM in-context learning is the best approach for dynamic enterprise test automation compared with fine-tuning.
Cite this review
Pith. "Pith review of An efficient approach to represent enterprise web application structure using Large Language Model in the service of Intelligent Quality Engineering." pith.science (2026). https://pith.science/paper/ICN3KVGG
@misc{pith2026250106837,
author = {Pith},
title = {Pith review of: An efficient approach to represent enterprise web application structure using Large Language Model in the service of Intelligent Quality Engineering},
year = {2026},
howpublished = {\url{https://pith.science/paper/ICN3KVGG}},
note = {Machine review of arXiv:2501.06837}
}
read the original abstract
This paper presents a novel approach to represent enterprise web application structures using Large Language Models (LLMs) to enable intelligent quality engineering at scale. We introduce a hierarchical representation methodology that optimizes the few-shot learning capabilities of LLMs while preserving the complex relationships and interactions within web applications. The approach encompasses five key phases: comprehensive DOM analysis, multi-page synthesis, test suite generation, execution, and result analysis. Our methodology addresses existing challenges around usage of Generative AI techniques in automated software testing by developing a structured format that enables LLMs to understand web application architecture through in-context learning. We evaluated our approach using two distinct web applications: an e-commerce platform (Swag Labs) and a healthcare application (MediBox) which is deployed within Atalgo engineering environment. The results demonstrate success rates of 90\% and 70\%, respectively, in achieving automated testing, with high relevance scores for test cases across multiple evaluation criteria. The findings suggest that our representation approach significantly enhances LLMs' ability to generate contextually relevant test cases and provide better quality assurance overall, while reducing the time and effort required for testing.
Figures
Forward citations
Cited by 1 Pith paper
-
AI-Driven Tools in Modern Software Quality Assurance: An Assessment of Benefits, Challenges, and Future Directions
AI models can generate and execute test cases on a demo app, but the reported flakiness rate masks a more serious false-negative problem.
Reference graph
Works this paper leans on
-
[1]
Coral Calero. Quality in web engineering. In 2010 Seventh International Conference on the Quality of Information and Communications Technology. IEEE, 2010
work page 2010
-
[2]
Applications of automated model’s extraction in enterprise systems
Cristina Marinescu. Applications of automated model’s extraction in enterprise systems. In Proceedings of the 14th International Conference on Software Technologies. SCITEPRESS - Science and Technology Publications, 2019
work page 2019
-
[3]
Revolutionizing software testing: The impact of AI, ML, and IoT
Vukašin Jeremi´c, Rocsana Bucea-Manea- ¸ Tonis, Slavimir Vesi´c, and Hana Stefanovi´c. Revolutionizing software testing: The impact of AI, ML, and IoT. EdTech Journal, 3(1):12–15, 2023
work page 2023
-
[4]
Yogeshwar Kulkarni. Artificial intelligence in software testing.International Journal of Innovative Science and Research Technology, pages 616–619, 2024
work page 2024
-
[5]
Maurizio Leotta, Filippo Ricca, Alessandro Marchetto, and Dario Olianas. An empirical study to compare three web test automation approaches: NLP-based, programmable, and capture&replay. Journal of Software: Evolution and Process, 36(5), 2024
work page 2024
-
[6]
AI-powered software testing: The impact of large language models on testing methodologies
Vahit Bayrı and Ece Demirel. AI-powered software testing: The impact of large language models on testing methodologies. In 2023 4th International Informatics and Software Engineering Conference (IISEC), pages 1–4. IEEE, 2023
work page 2023
-
[7]
Chunhua Deming, Md Abul Khair, Suman Reddy Mallipeddi, and Aleena Varghese. Software testing in the era of AI: Leveraging machine learning and automation for efficient quality assurance. Asian Journal of Applied Science and Engineering, 10(1):66–76, 2021
work page 2021
-
[8]
A comprehensive enterprise system metamodel for quality assurance
Jan Svacina, Vincent Bushong, Dipta Das, and Tomas Cerny. A comprehensive enterprise system metamodel for quality assurance. In Lecture Notes in Electrical Engineering, pages 245–252. Springer Singapore, Singapore, 2021. 14 arXiv Template A PREPRINT
work page 2021
Show all 32 references
-
[9]
Natural language processing-based software testing: A systematic literature review
Mohamed Boukhlif, Mohamed Hanine, Nassim Kharmoum, Atenea Noriega, David Obeso, and Imran Ashraf. Natural language processing-based software testing: A systematic literature review. IEEE Access, pages 1–1, 2024
2024
-
[10]
Large language models for software engineering: Survey and open problems
Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M Zhang. Large language models for software engineering: Survey and open problems. In 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (IC...
2023
-
[11]
Software testing with large language models: Survey, landscape, and vision
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 50(4):911–936, 2024
2024
-
[12]
A conceptual framework for quality assurance of LLM-based socio-critical systems
Luciano Baresi, Matteo Camilli, Tommaso Dolci, and Giovanni Quattrocchi. A conceptual framework for quality assurance of LLM-based socio-critical systems. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 2314–2318. ACM, 2024
2024
-
[13]
LLM for test script generation and migration: Challenges, capabilities, and opportunities
Shengcheng Yu, Chunrong Fang, Yuchen Ling, Chentian Wu, and Zhenyu Chen. LLM for test script generation and migration: Challenges, capabilities, and opportunities. In 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), pages 206–217, 2023
2023
-
[14]
A case study on test case construction with large language models: Unveiling practical insights and challenges
Roberto Francisco de Lima Junior, Luiz Fernando Paes de Barros Presta, Lucca Santos Borborema, Vanderson Nogueira da Silva, Marcio Leal de Melo Dahia, and Anderson Carlos Sousa e Santos. A case study on test case construction with large language models: Unveiling practical ins...
2023 arXiv
-
[15]
Automated test case generation from requirements: A systematic literature review
Ahmad Mustafa, Wan Hazli Wan Kadir, Norazana Ibrahim, Muhammad Arif Shah, Muhammad Younas, Atif Ali Khan, Mahdi Zareei, and Faisal Alanazi. Automated test case generation from requirements: A systematic literature review. Computers Materials & Continua, 2021
2021
-
[16]
Requirement-based automated test case generation: Systematic literature review
M S Farooq. Requirement-based automated test case generation: Systematic literature review. Vfast Transactions on Software Engineering, 2022
2022
-
[17]
A semi-automated approach for requirement-based early validation of flight control platforms
Philipp Chrysalidis, Hauke Hoeber, and Frank Thielecke. A semi-automated approach for requirement-based early validation of flight control platforms. CEAS Aeronautical Journal, 2022
2022
-
[18]
Functional test generation from ui test scenarios using reinforcement learning for android applications
Yavuz Koroglu and Alper ¸ Sen. Functional test generation from ui test scenarios using reinforcement learning for android applications. Software Testing, Verification and Reliability, 2020
2020
-
[19]
Web program testing using selenium python: Best practices and effective approaches
Rusdiansyah Rusdiansyah. Web program testing using selenium python: Best practices and effective approaches. Sinkron, 2024
2024
-
[20]
Automated testing of web project functionality with using of error propagation analysis.Computer Systems and Information Technologies, 2023
Iryna Zasornova. Automated testing of web project functionality with using of error propagation analysis.Computer Systems and Information Technologies, 2023
2023
-
[21]
Automated functional testing pada api menggunakan keyword driven framework.Journal of Informatics and Communication Technology, 2021
Syifa Nurgaida Yutia. Automated functional testing pada api menggunakan keyword driven framework.Journal of Informatics and Communication Technology, 2021
2021
-
[22]
Automating test oracles from restricted natural language agile requirements
Maryam Imtiaz Malik, Muddassar Azam Sindhu, Akmal Saeed Khattak, Rabeeh Ayaz Abbasi, and Khalid Saleem. Automating test oracles from restricted natural language agile requirements. Expert Systems, 2020
2020
-
[23]
A bert-based transfer learning approach to text classification on software requirements specifications
Derya Kici, Garima Malik, Mucahit Cevik, Devang Parikh, and Ay¸ se Ba¸ sar. A bert-based transfer learning approach to text classification on software requirements specifications. Proceedings of the Canadian Conference on Artificial Intelligence, 2021
2021
-
[24]
User stories and natural language processing: A systematic literature review
Indra Kharisma Raharjana, Daniel Siahaan, and Chastine Fatichah. User stories and natural language processing: A systematic literature review. IEEE Access, 2021
2021
-
[25]
MuFBDTester: A mutation-based test sequence generator for FBD programs implementing nuclear power plant software
Lingjun Liu, Eunkyoung Jee, and Doo-Hwan Bae. MuFBDTester: A mutation-based test sequence generator for FBD programs implementing nuclear power plant software. Software Testing, Verification and Reliability, 2022
2022
-
[26]
Software test case generation using natural language processing (NLP): A systematic literature review
Halima Ayenew. Software test case generation using natural language processing (NLP): A systematic literature review. Artificial Intelligence Evolution, 2024
2024
-
[27]
An empirical study to compare three web test automation approaches: NLP-based, programmable, and capture&replay
Maurizio Leotta, Filippo Ricca, Alessandro Marchetto, and Dario Olianas. An empirical study to compare three web test automation approaches: NLP-based, programmable, and capture&replay. Journal of Software Evolution and Process, 2023
2023
-
[28]
Chunhui Wang, Fabrizio Pastore, Arda Göknil, and Lionel C. Briand. Automatic generation of acceptance test cases from use case specifications: An NLP-based approach. IEEE Transactions on Software Engineering, 2022
2022
-
[29]
Chen, Gunvant Chaudhari, Thienkhai Vu, Youngho Seo, Jared Narvid, and Jae Ho Sohn
Yeshwant Chillakuru, Shourya Munjal, Benjamin Laguna, Timothy L. Chen, Gunvant Chaudhari, Thienkhai Vu, Youngho Seo, Jared Narvid, and Jae Ho Sohn. Development and web deployment of an automated neuroradiology mri protocoling tool with natural language processing. BMC Medical ...
2021
-
[30]
Natural language processing for assessing quality indicators in free-text colonoscopy and pathology reports: Development and usability study
Jung Ho Bae, Hyun Wook Han, Sun Young Yang, Gyuseon Song, Soonok Sa, Goh Eun Chung, Ji Yeon Seo, Eun Hyo Jin, Heecheon Kim, and DongUk An. Natural language processing for assessing quality indicators in free-text colonoscopy and pathology reports: Development and usability stu...
2022
-
[31]
Tignanelli, Greg Silverman, Elizabeth Lindemann, A
Christopher J. Tignanelli, Greg Silverman, Elizabeth Lindemann, A. Trembley, Jon C. Gipson, Gregory J. Beilman, John Lyng, Raymond Finzel, Reed McEwan, Benjamin C. Knoll, S. V . Pakhomov, and Genevieve B. Melton. Natural language processing of prehospital emergency medical ser...
2020
-
[32]
Jen-tse Huang, Jianping Zhang, Wenxuan Wang, Pinjia He, Yuxin Su, and Michael R. Lyu. AEON: A method for automatic evaluation of NLP test cases. arXiv:2205.06439, 2022. 16
2022 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.