REVIEW 3 major objections 1 minor 54 references
An LLM can extract component hierarchies and route transitions from Vue.js source to build a usable page transition graph for model-based GUI testing.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.3
2026-06-29 04:20 UTC pith:EX3JYIAF
load-bearing objection LLMVue extracts PTGs from Vue.js source with GPT-4o and reports better coverage than random, but the validation of those graphs against LLM errors is missing. the 3 major comments →
LLM-Assisted Model-Based GUI Testing for Vue.js Web Applications
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
LLMVue uses an LLM to infer component hierarchies and route transitions from raw Vue.js source, then merges the extracted information into a single page transition graph; the graphs obtained from ten real projects exhibit high precision and recall together with low graph edit distance, and the same graphs, when used to direct test execution, produce measurably higher coverage and faster exploration than random walks within the same time limit.
What carries the argument
LLMVue framework that prompts an LLM (GPT-4o) to infer component hierarchies and route transitions from Vue.js source and merges them into a unified page transition graph (PTG).
Load-bearing premise
The large language model can correctly and completely recover component hierarchies and route transitions from Vue.js source code without systematic omissions or hallucinations that would produce an invalid PTG.
What would settle it
Manual inspection of the inferred PTG against actual runtime navigation behavior on a new set of Vue.js applications, counting any missing or spurious transitions that cause test execution to deviate from the real application.
If this is right
- The constructed PTG can serve directly as a model for automated GUI test generation in Vue.js applications.
- LLM-guided exploration achieves higher coverage than random exploration when both are limited to the same execution time.
- Graph construction metrics remain stable across ten distinct open-source Vue.js projects.
- Merging per-component and router-derived transitions into one PTG removes the manual normalization step previously required for model-based testing.
Where Pith is reading between the lines
- The same LLM extraction pattern could be applied to other single-file-component frameworks that separate routing from component logic.
- If the PTG is kept up to date by re-running the LLM after code changes, regression test suites could be regenerated automatically.
- The approach assumes source code is the sole input; extending it to combine source with runtime traces would address cases where dynamic routes are generated at runtime.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript proposes LLMVue, a framework that uses an LLM (GPT-4o) to extract a Page Transition Graph (PTG) from Vue.js source code. The approach infers component hierarchies and route transitions from single-file components and router configurations, then merges them into a unified executable PTG for model-based GUI testing. On ten open-source projects the constructed PTGs show high precision/recall and low graph edit distance relative to human references; LLM-guided testing also yields higher coverage and exploration efficiency than a random baseline under identical time limits. The work positions itself as the first application of LLMs to source-level PTG extraction for Vue.js.
Significance. If the empirical claims hold after the missing methodological details are supplied, the paper would constitute a modest but useful contribution: the first reported use of LLMs for automated model extraction in Vue.js GUI testing. The evaluation on ten real projects and the dual comparison (human PTG + random baseline) are positive features. The central risk is that the headline results rest entirely on the unverified fidelity of the LLM-generated PTGs; without explicit validation of that step the coverage/efficiency gains cannot be confidently attributed to model-based testing.
major comments (3)
- [Evaluation] The headline claims of high precision, recall, and low GED presuppose that the LLM outputs are faithful PTGs. The manuscript supplies no description of how the human reference graphs were constructed, no inter-annotator agreement figures, and no systematic audit for hallucinations or omissions in the extracted hierarchies and transitions.
- [Approach / LLMVue framework] The graph-merging algorithm that combines inferences across multiple SFCs and router files is not specified, nor are the prompts, error-handling strategies, or post-processing steps used with GPT-4o. These components are load-bearing for the PTG quality that underpins all subsequent testing results.
- [Results] The reported coverage and efficiency improvements versus the random baseline lack raw per-project data, statistical significance tests, and implementation details of the baseline itself (e.g., how actions were sampled and whether the same execution harness was used).
minor comments (1)
- [Abstract] The abstract states results on "ten open-source Vue.js projects" but does not name the projects or the selection criteria; adding this information would improve reproducibility.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript to supply the missing methodological details.
read point-by-point responses
-
Referee: [Evaluation] The headline claims of high precision, recall, and low GED presuppose that the LLM outputs are faithful PTGs. The manuscript supplies no description of how the human reference graphs were constructed, no inter-annotator agreement figures, and no systematic audit for hallucinations or omissions in the extracted hierarchies and transitions.
Authors: We agree the validation of human reference graphs requires more detail. The revised manuscript will add a subsection explaining that two authors independently built the reference PTGs from source code and documentation, with discrepancies resolved by consensus. We will report inter-annotator agreement (Cohen's kappa) and describe a manual audit comparing LLM outputs to references for hallucinations or omissions. revision: yes
-
Referee: [Approach / LLMVue framework] The graph-merging algorithm that combines inferences across multiple SFCs and router files is not specified, nor are the prompts, error-handling strategies, or post-processing steps used with GPT-4o. These components are load-bearing for the PTG quality that underpins all subsequent testing results.
Authors: We will expand the LLMVue framework section with a precise description of the graph-merging algorithm (node/edge union with conflict resolution based on router priority). The revised paper will include the exact GPT-4o prompts, error-handling (retries on parse errors), and post-processing steps (label normalization) either in the main text or an appendix. revision: yes
-
Referee: [Results] The reported coverage and efficiency improvements versus the random baseline lack raw per-project data, statistical significance tests, and implementation details of the baseline itself (e.g., how actions were sampled and whether the same execution harness was used).
Authors: We will add a table with raw per-project coverage and efficiency numbers. Statistical significance will be assessed using paired Wilcoxon tests with reported p-values. The baseline description will be expanded to confirm uniform action sampling from available elements and use of the identical execution harness. revision: yes
Circularity Check
No circularity: empirical evaluation against external references
full rationale
The paper reports an empirical study: LLM-generated PTGs are evaluated via precision, recall, and graph edit distance against human reference graphs on 10 open-source projects, plus coverage/efficiency gains versus a random baseline under fixed time. No equations, fitted parameters, self-definitional constructs, or load-bearing self-citations appear in the derivation chain. All headline metrics are direct measurements from the experiments rather than quantities that reduce to the inputs by construction. The work is self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
read the original abstract
Vue.js is a popular framework for building modern web applications. As Vue.js functionality and tooling support grow, ensuring its reliability (through automated testing) is becoming increasingly important. Although model-based testing has been successfully used to automate graphical user interface (GUI) testing on other platforms, its application to Vue.js remains challenging: Transition candidates, which are spread across router configurations and single-file components (SFCs), must be concretized and normalized into an executable page transition graph (PTG) for testing. To address this, we propose the LLMVue framework, which uses a large language model (LLM) to generate a PTG from Vue.js source code. LLMVue infers component hierarchies and route transitions, merging them into a unified PTG across multiple SFCs. We evaluated LLMVue on a collection of ten open-source Vue.js projects from GitHub, using GPT-4o as the LLM backbone. The constructed graphs demonstrate high precision and recall, with low graph edit distance. LLMVue -guided testing also significantly improves the coverage and exploration efficiency, compared to a random exploration baseline (with the same time constraints). To the best of our knowledge, this is the first use of LLMs for model-based GUI testing of Vue.js applications using source-level PTG extraction.
Figures
Reference graph
Works this paper leans on
-
[1]
Parsa Alian, Noor Nashid, Mobina Shahbandeh, and Ali Mesbah. 2024. Semantic constraint inference for web form test generation. InProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’24). 932–944
2024
-
[2]
Parsa Alian, Noor Nashid, Mobina Shahbandeh, Taha Shabani, and Ali Mesbah. 2025. Feature-driven end-to-end test generation. InProceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE’25). 450–462
2025
-
[3]
Nadia Alshahwan, Mark Harman, Alessandro Marchetto, Roberto Tiella, and Paolo Tonella. 2012. Crawlability metrics for web applications. InProceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation (ICST’12). IEEE, 151–160
2012
- [4]
-
[5]
Sebastian Balsam and Deepti Mishra. 2025. Web application testing—Challenges and opportunities.Journal of Systems and Software219 (2025), 112186
2025
-
[6]
Fevzi Belli, Mutlu Beyazıt, Christof J Budnik, and Tugkan Tuglular. 2017. Advances in model-based testing of graphical user interfaces.Advances in Computers107 (2017), 219–280
2017
-
[7]
Kamara Benjamin, Gregor Von Bochmann, Mustafa Emre Dincturk, Guy-Vincent Jourdan, and Iosif Viorel Onut. 2011. A strategy for efficient crawling of rich internet applications. InProceedings of the International Conference on Web Engineering (ICWE’11). 74–89
2011
-
[8]
Salah Boukhris, Anneliese Andrews, Ahmed Alhaddad, and Rinku Dewri. 2017. A case study of black box fail-safe testing in web applications.Journal of Systems and Software131 (2017), 146–167
2017
-
[9]
Islem Bouzenia and Michael Pradel. 2025. You name it, I run it: An LLM agent to execute tests of arbitrary projects. Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1054–1076
2025
-
[10]
Yige Chen, Sinan Wang, Yida Tao, and Yepang Liu. 2024. Model-based GUI testing for HarmonyOS apps. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE’24). 2411–2414
2024
-
[11]
Jianlei Chi, Xiaotian Wang, Yuhan Huang, Lechen Yu, Di Cui, Jianguo Sun, and Jun Sun. 2025. REACEEPT: Automated co-evolution of production and test code based on dynamic validation and large language models.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1234–1256
2025
-
[12]
Arilo C Dias Neto, Rajesh Subramanyan, Marlon Vieira, and Guilherme H Travassos. 2007. A survey on model-based testing approaches: A systematic review. InProceedings of the 1st ACM International Workshop on Empirical Assessment of Software Engineering Languages and Technologies (WEASELTech’07). 31–36
2007
-
[13]
Mustafa Emre Dincturk, Guy-Vincent Jourdan, Gregor V Bochmann, and Iosif Viorel Onut. 2014. A model-based approach for crawling rich internet applications.ACM Transactions on the Web8, 3 (2014), 1–39
2014
-
[14]
Jinhao Dong, Jun Sun, Wenjie Zhang, Jin Song Dong, and Dan Hao. 2025. ConTested: Consistency-aided tested code generation with LLM.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 596–617
2025
-
[15]
Xinbo Gao, Bing Xiao, Dacheng Tao, and Xuelong Li. 2010. A survey of graph edit distance.Pattern Analysis and Applications13, 1 (2010), 113–129
2010
-
[16]
Vahid Garousi, Ali Mesbah, Aysu Betin-Can, and Shabnam Mirshokraie. 2013. A systematic mapping study of web application testing.Information and Software Technology55, 8 (2013), 1374–1396. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. LLM-Assisted Model-Based GUI Testing for Vue.js Web Applications 111:23
2013
- [17]
-
[18]
Istanbul Code Coverage. 2025. nyc. https://github.com/istanbuljs/nyc. Accessed: 2025-12-24
2025
-
[19]
Nan Jiang, Xiaopeng Li, Shiqi Wang, Qiang Zhou, Soneya B Hossain, Baishakhi Ray, Varun Kumar, Xiaofei Ma, and Anoop Deoras. 2024. LEDEX: Training LLMs to better self-debug and explain code.Advances in Neural Information Processing Systems37 (2024), 35517–35543
2024
-
[20]
Tao Li, Chenhui Cui, Rubing Huang, Dave Towey, and Lei Ma. 2026. Large language models for automated web- form-test generation: An empirical study.ACM Transactions on Software Engineering and Methodology35, 3 (2026), 1–37
2026
- [21]
-
[22]
Jun-Wei Lin, Farn Wang, and Paul Chu. 2017. Using semantic similarity in crawling-based web application testing. In Proceedings of the 10th IEEE International Conference on Software Testing, Verification and Validation (ICST’17). 138–148
2017
-
[23]
Chenxu Liu, Junheng Wang, Wei Yang, Ying Zhang, and Tao Xie. 2026. Judge: Effective state abstraction for guiding automated web GUI testing.ACM Transactions on Software Engineering and Methodology35, 3 (2026), 1–40
2026
-
[24]
Jinwei Liu, Chao Li, Rui Chen, Shaofeng Li, Bin Gu, and Mengfei Yang. 2025. STRUT: Structured seed case guided unit test generation for C programs using LLMs.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 2113–2135
2025
-
[25]
Guilherme Ricken Mattiello and André Takeshi Endo. 2022. Model-based testing leveraged for automated web tests. Software Quality Journal30, 3 (2022), 621–649
2022
-
[26]
Atif M Memon. 2007. An event-flow model of GUI-based applications for testing.Software Testing, Verification and Reliability17, 3 (2007), 137–157
2007
-
[27]
Ali Mesbah. 2015. Advances in testing JavaScript-based web applications.Advances in Computers97 (2015), 201–235
2015
-
[28]
Ali Mesbah and Arie Van Deursen. 2009. Invariant-based automatic testing of AJAX user interfaces. InProceedings of the 2009 IEEE 31st International Conference on Software Engineering (ICSE’09). 210–220
2009
-
[29]
Ali Mesbah, Arie Van Deursen, and Stefan Lenselink. 2012. Crawling Ajax-based web applications through dynamic analysis of user interface state changes.ACM Transactions on the Web6, 1 (2012), 1–30
2012
-
[30]
Ali Mesbah, Arie Van Deursen, and Danny Roest. 2011. Invariant-based automatic testing of modern web applications. IEEE Transactions on Software Engineering38, 1 (2011), 35–53
2011
-
[31]
Microsoft. 2026. Playwright. https://github.com/microsoft/playwright. Accessed: 2026-01-01
2026
-
[32]
Mehdi Mirzaaghaei and Ali Mesbah. 2014. DOM-based test adequacy criteria for web applications. InProceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA’14). 71–81
2014
-
[33]
Zifan Nan, Zhaoqiang Guo, Kui Liu, and Xin Xia. 2025. Test intention guided LLM-based unit test generation. In Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE’25). 1026–1038
2025
-
[34]
2018.Getting to know Vue.js
Brett Nelson. 2018.Getting to know Vue.js. Apress
2018
-
[35]
OpenAI. 2026. GPT-4o. https://platform.openai.com/docs/models/gpt-4o. Accessed: 2026-01-01
2026
-
[36]
Giancarlo Pellegrino, Constantin Tschürtz, Eric Bodden, and Christian Rossow. 2015. jÄk: Using dynamic analysis to crawl and test modern web applications. InProceedings of the 18th International Symposium on Research in Attacks, Intrusions, and Defenses (RAID’15). 295–316
2015
-
[37]
RichestSoft. 2025. Top 10 Front End Frameworks for 2025. https://richestsoft.com/blog/10-most-popular-front-end- frameworks/. Accessed: 2025-12-01
2025
-
[38]
Fernando Pastor Ricós, Beatriz Marín, Tanja EJ Vos, Rick Neeft, and Pekka Aho. 2025. Delta GUI change detection using inferred models.Computer Standards and Interfaces92 (2025), 103925
2025
-
[39]
Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation.IEEE Transactions on Software Engineering50, 1 (2023), 85–105
2023
-
[40]
Ye Shang, Quanjun Zhang, Chunrong Fang, Siqi Gu, Jianyi Zhou, and Zhenyu Chen. 2025. A Large-Scale Empirical Study on Fine-Tuning Large Language Models for Unit Testing.Proceedings of the ACM on Software Engineering2, ISSTA (2025), 1678–1700
2025
-
[41]
Arie Van Deursen, Ali Mesbah, and Alex Nederlof. 2015. Crawl-based analysis of web applications: Prospects and challenges.Science of Computer Programming97 (2015), 173–180
2015
-
[42]
Vue.js. 2025. Vue. https://github.com/vuejs/. Accessed: 2025-12-01
2025
-
[43]
Vue.js. 2026. SFC Syntax Specification. https://vuejs.org/api/sfc-spec. Accessed: 2026-01-01
2026
-
[44]
Chong Wang, Jian Zhang, Yebo Feng, Tianlin Li, Weisong Sun, Yang Liu, and Xin Peng. 2025. Teaching code LLMs to use autocompletion tools in repository-level code generation.ACM Transactions on Software Engineering and Methodology34, 7 (2025), 1–27
2025
-
[45]
Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision.IEEE Transactions on Software Engineering50, 4 (2024), 911–936. J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018. 111:24 Tao Li, Chenhui Cui, Rubing Huang, Dave Towey, Shikai Guo, a...
2024
-
[46]
Siyi Wang, Sinan Wang, Yujia Fan, Xiaolei Li, and Yepang Liu. 2024. Leveraging large vision-language model for better automatic web GUI testing. InProceedings of the 2024 IEEE International Conference on Software Maintenance and Evolution (ICSME’24). 125–137
2024
-
[47]
Rahul Krishna Yandrapally and Ali Mesbah. 2022. Fragment-based test generation for web apps.IEEE Transactions on Software Engineering49, 3 (2022), 1086–1101
2022
-
[48]
Lin Yang, Chen Yang, Shutao Gao, Weijing Wang, Bo Wang, Qihao Zhu, Xiao Chu, Jianyi Zhou, Guangtai Liang, Qianxiang Wang, et al. 2024. On the evaluation of large language models in unit test generation. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE’24). 1607–1619
2024
-
[49]
2018.Testing Vue.js applications
Edd Yerburgh. 2018.Testing Vue.js applications. Simon and Schuster
2018
-
[50]
Xun Yuan and Atif M Memon. 2009. Generating event sequence-based test cases using GUI runtime state feedback. IEEE Transactions on Software Engineering36, 1 (2009), 81–95
2009
-
[51]
Haonan Zhang, Lizhi Liao, Zishuo Ding, Weiyi Shang, Nidhi Narula, Catalin Sporea, Andrei Toma, and Sarah Sajedi
-
[52]
InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE’24)
Towards a robust waiting strategy for web GUI testing for an industrial software system. InProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (ASE’24). 2065–2076
2065
-
[53]
Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Guoyin Wang, and Fei Wu. 2026. Instruction tuning for large language models: A survey.ACM Computing Surveys58, 7 (2026), 1–36
2026
-
[54]
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models.arXiv preprintarXiv:2303.18223 (2023). Received 18 June 2026 J. ACM, Vol. 37, No. 4, Article 111. Publication date: August 2018
work page internal anchor Pith review Pith/arXiv arXiv 2023
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.