Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models

Alexander Weinert; David Heidrich; Tim Rosenbach

REVIEW 4 major objections 5 minor 32 references

Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM-driven agent pair can find GUI inconsistencies that human testers miss.

desk verdict GERALLT is a real, well-described LLM GUI-testing system for a desktop engineering app, but its five-issue evaluation chain rests on author triage and in-group developer confirmation—worth a serious referee, not yet a settled result. read the letter →

arxiv 2505.17839 v1 pith:G4QH2KWO submitted 2025-05-23 cs.SE

Tim Rosenbach , David Heidrich , Alexander Weinert This is my paper · ORCID

classification cs.SE

keywords GUItestinglargelanguagemodelsexploratorydesktopapplicationacceptanceuserinterfaceconsistencytestautomationRCE

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an LLM-driven pair of agents can take over part of the exploratory GUI testing that is normally done by human testers, and that this can catch interface behavior the development team had not consciously noticed. The system, GERALLT, separates exploration from judgment: one LLM agent chooses actions on the desktop application while a second LLM agent compares screenshots before and after each action and flags anything inconsistent or unintuitive. Evaluated on the tool-integration wizard of the engineering software RCE, nine runs produced 752 actions and 72 evaluator flags; after manual triage these became seven true positives and five distinct issues, including an error message that calls a non-existent path invalid. The RCE developers confirmed the issues and described them as blind spots in their existing manual testing process. If the evidence holds, partial automation of acceptance testing is within reach for desktop software, not just web and mobile apps.

What carries the argument

The load-bearing mechanism is a two-agent prompting loop. The controller receives a prompt containing the testing task, a JSON widget-tree description of the current GUI, relevant RCE documentation, the set of possible actions on each widget, and a log of previous actions; it outputs a single action in a constrained JSON format. After the action is executed, the evaluator receives the previous screenshot, the current screenshot, and a natural-language description of the action, and returns a structured verdict of either problem or okay. A GUI parser built on a Windows automation package supplies the widget tree, while the evaluator stays deliberately local, judging only the immediate before/after pair rather than the full action log. This separation lets one agent explore broadly and the other critique each step independently.

What would settle it

A concrete check would be to run GERALLT against a desktop wizard with deliberately planted interface inconsistencies, such as an error message that reports an invalid non-existent path instead of complaining about an empty field, and have a blind panel classify the evaluator's flags; if planted issues are missed or the panel confirms few flags, the central claim fails.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that the controller/evaluator division of labor works: giving one LLM the goal of covering as many GUI elements as possible, along with a structured textual description of the current window, documentation, and an action history, produces enough valid exploration for a second LLM to find genuine interface issues by looking only at screenshot pairs and the action description. The paper reports that the RCE development team confirmed all five consolidated issues and agreed that these were not consciously noticed during their existing exploratory testing. The authors are careful to call this preliminary evidence rather than a proof that LLM-based GUI testing can replace human testers; their stated aim is to augment, not replace, the manual process.

Load-bearing premise

The evaluation assumes that the authors' manual separation of 72 evaluator flags into 7 true positives is unbiased, and that the confirming developers' agreement counts as independent ground truth.

Editorial extensions

If this is right

With two cooperating LLM agents, exploratory GUI testing of desktop software can be partially automated, shifting developer effort from walking through workflows to triaging flagged issues.
Because the evaluator needs only screenshots and a description of the last action, the judgement component is reusable across GUI frameworks once the parser and action executor are adapted.
The method can surface consistency problems such as a misleading error message for an empty path, not just crashes or functional failures.
Documentation already available to human testers can be folded into the controller prompt, allowing the agent to navigate an unfamiliar desktop application without custom scripting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorially, the 7-out-of-72 precision rate means a practical deployment would need a triage layer or evaluator prompt tuning; otherwise developers would face a large false-positive load.
Editorially, the approach's sensitivity to "unintuitive" behavior is likely proportional to how closely the GUI follows conventions the LLM has seen, so a highly domain-specific interface may need extra documentation or examples in the prompt.
Editorially, a sharper test would seed a wizard with deliberately planted inconsistencies and compare detection against a blind independent panel, avoiding reliance on the confirming developers' own judgement.
Editorially, the same architecture could be extended beyond clicks and text entry to drag-and-drop and other interactions, though the paper notes that would require implementation work rather than a conceptual change.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the thing to know: this is a real system, not a toy. GERALLT drives a desktop Eclipse RCP application (RCE) with two LLM agents—a controller that picks actions and an evaluator that screens before/after screenshots for unintuitive behavior. The architecture is clearly described, the prompts are in the paper, and the code is referenced on GitHub. The main novelty is the application to a real engineering desktop GUI, which prior LLM-testing work mostly avoided in favor of web or Android. That is a legitimate extension, and the paper is honest that it builds on GPT Droid and ASSISTGUI.

The strongest piece of evidence is Section 5.2. The evaluator flags that an empty 'working directory' field produces 'Invalid path to working directory' while an empty version field gets a specific 'must not be empty' message. That is a concrete, plausible UX inconsistency, and the developers confirmed it. Five issues were confirmed in total, and the paper calls this 'preliminary evidence,' which is the right level of claim.

Where it gets soft is the evaluation chain behind those five issues. Nine runs, 752 actions, 72 evaluator positives, manual triage by the authors down to 7 true positives, then consolidation to 5 issues, then developer confirmation. There are no triage criteria, no independent raters, no raw outputs or annotations, and no commit hash for the exact version. The confirming developers overlap with the authorship—Weinert is on this paper and on the RCE testing references. So the 72-to-5 reduction is effectively an in-group judgment loop. The stress-test note is on target. The paper acknowledges qualitative evaluation, but the abstract's 'is able to determine issues' leans entirely on that loop.

That said, these are addressable problems, not fatal ones. The system is real, the example is concrete, and the limitation section is unusually candid. For a serious referee, the main request would be: publish the raw evaluator outputs and triage annotations, have at least one independent rater redo the true/false split, and run a manual testing session as a comparison. The no-baseline issue is understandable for exploratory testing but still limits what the numbers mean.

Bottom line: I'd send this to peer review. It is a solid engineering paper with a weak evaluation section that can be fixed. Readers interested in LLM-based GUI testing or the RCE project will get value from it. I'd cite it as related work on desktop LLM testing, but I wouldn't rely on the five-issue claim until the triage is independently audited.

Referee Report

4 major / 5 minor

Summary. The paper presents GERALLT, an LLM-based system for exploratory GUI testing of a desktop engineering application (RCE). GERALLT combines a controller agent that explores the GUI with an evaluator agent that flags unintuitive or inconsistent behaviors after each action. The authors describe the system architecture, the controller and evaluator prompts, and a qualitative evaluation on the RCE tool integration wizard. In nine test runs, the controller performed 752 actions; the evaluator flagged 72 of them as problematic. The authors manually triaged these into seven true positives and then consolidated them into five unique issues, which the RCE development team confirmed as previously unnoticed 'blind spots.' The paper concludes that this is preliminary evidence that LLM-based systems can support exploratory GUI testing.

Significance. If the central claim holds, GERALLT would be a useful step toward automating exploratory acceptance testing for desktop GUI applications, an area less explored than mobile or web GUI testing. The paper's strengths include a concrete and reproducible architecture (the GitHub repository is referenced), a real-world case study rather than a toy application, a detailed example of a detected issue in Section 5.2, and an explicit discussion of limitations in Section 5.3. The evaluation, however, is currently too weak to establish the central claim: it relies on in-group manual triage and developer confirmation, lacks a baseline, and reports no raw data or precision metrics. The significance is therefore conditional on the evaluation being substantially strengthened.

major comments (4)

[Section 5.1] The central claim that GERALLT 'is able to determine issues with the interface' rests on the chain from 72 evaluator positives to 7 true positives to 5 unique issues. This chain depends entirely on the authors' manual triage and on confirmation by the RCE development team. The paper gives no criteria for the triage, no raw outputs, no inter-rater agreement, and the confirming developers overlap with the authorship (Weinert appears both in this paper and in RCE references [22,24]). To make the claim credible, the authors should make the triage auditable: publish the 72 evaluator outputs, the labels assigned, the consolidation rationale, and ideally have an independent rater (or at least a non-author developer) confirm the classification. Alternatively, the conclusion should be weakened to state that GERALLT produces candidate issues that may support developers, rather than that it determines real issues.
[Section 5.1, precision] The reported numbers imply a very low precision: out of 72 evaluator positives, only 7 were considered true positives (about 9.7%). The paper does not discuss this high false-positive rate or its acceptability for a tool intended to support, not replace, human testers. Since a support tool whose reports are mostly wrong would be quickly discarded, the authors should report precision explicitly and discuss how developers might handle the noise. They should also state whether the 7 true positives were all distinct or whether some duplicates were already present at the 72-output level.
[Section 5.1, baseline] The paper explicitly states that 'there exists no baseline of existing or known errors' and therefore performs a qualitative evaluation. However, a baseline is still possible and needed to support the claim that the LLM-based evaluator adds value: for example, comparing against a random or rule-based exploration of the GUI, or against the developers' own manual testing of the same feature. The absence of any comparison means the five issues could plausibly have been found by a much simpler approach, and the specific contribution of the LLM-based evaluator remains unquantified. At minimum, the authors should report how many issues the developers typically find in manual exploratory testing of the wizard (they cite [24] and could provide data) and whether GERALLT found anything beyond that.
[Section 5.2] The example issue is presented as evidence that GERALLT finds real problems, but the evaluator's criticism is a subjective UX judgment: a non-existent path 'cannot be invalid' and the error message should perhaps be more specific. The paper's validation of this as a real issue is the developers' agreement, which is subject to the overlap concern raised above. The authors should strengthen this by showing that the developers took a concrete action (e.g., filed a bug report or changed the code) or by providing independent confirmation from a user test. The screenshot pair in Figure 5 also does not by itself demonstrate that the behavior is confusing to users; a short user study or a developer quote would help.

minor comments (5)

[Section 1] The reference to 'Lui et al.' in the introduction is inconsistent with 'Liu et al.' used later; the spelling should be unified.
[Table 2] The evaluator prompt in Table 2 contains a duplicated word: 'The action performed between the two images was was: Click the 'Next' button'; this should be fixed.
[Abstract/Conclusion] The abstract says 'evaluate it on a real-world use case' but the evaluation covers a single wizard of one application; consider using 'pilot evaluation' to match the scope.
[Section 4.1] The paper mentions that the evaluator receives only the 'last action' via a dashed line in Figure 2, but the prompt in Table 2 includes the action description; this is clear, but an explicit sentence stating that the evaluator does not receive the full action log would help the reader.
[Section 5.3] The GitHub repository [27] is referenced but the README or a usage link is not described; if the repository is public, adding a short description of how to reproduce the experiment would improve reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central claim is an empirical evaluation, and self-citations are contextual rather than load-bearing.

full rationale

GERALLT's claimed contribution is an empirically demonstrated architecture, not a derived or fitted result. There is no equation or parameter that is calibrated against the reported issues; the controller and evaluator prompts are fixed system design, and the 752 actions, 72 evaluator positives, and 5 consolidated issues are observed outputs rather than predictions forced by construction. The manual triage in Section 5.1 and the developer confirmation in Sections 5.1-5.2 do not make the conclusion true by definition: the labels are assigned to observable GUI states and could in principle be wrong, so the evaluation is empirically checkable even though it is not independently audited. The paper's self-references are not load-bearing: the bachelor thesis [1] is cited for the choice of ChatGPT and as the basis of the work, and RCE references [22,23,24] provide background on the system-under-test and its manual-testing process; none of these citations supplies the paper's conclusion that GERALLT finds unintuitive interface behavior. The overlap between the confirming RCE developers and the paper's authors is a legitimate evaluation-independence concern, but it is a threat to empirical validity, not a circular reduction of the kind targeted by this pass.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No fitted numerical parameters or invented entities are involved; this is an empirical systems paper. The axioms above are the unproven domain assumptions the evaluation rests on: parser accuracy, evaluator visual judgment, developer ground truth, and unbiased manual triage.

assumptions (4)

domain assumption The PyWinAuto-based GUI parser produces a complete and accurate widget tree of the RCE GUI on Windows.
The controller's decisions depend on this structured GUI state (Section 4.2); no parser accuracy measurement is reported.
domain assumption The evaluator LLM can judge unintuitive or inconsistent GUI behavior from before/after screenshots and a short action description.
The entire issue-detection mechanism relies on this (Section 4.3); the prompt tells the model what to check, and no validation of the model's visual reasoning is given.
domain assumption The RCE developers' confirmation is a valid ground truth for UI issues.
The evaluation uses developer judgment instead of an objective baseline (Section 5); some developers are also authors of this paper or of the system-under-test references [22,24].
domain assumption The authors' manual classification of 72 evaluator positives into true and false positives is unbiased.
Section 5.1 reports the filtering but gives no criteria, rubric, or inter-rater reliability.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models." pith.science (2026). https://pith.science/paper/G4QH2KWO

@misc{pith2026250517839,
  author       = {Pith},
  title        = {Pith review of: Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G4QH2KWO}},
  note         = {Machine review of arXiv:2505.17839}
}

read the original abstract

One important step in software development is testing the finished product with actual users. These tests aim, among other goals, at determining unintuitive behavior of the software as it is presented to the end-user. Moreover, they aim to determine inconsistencies in the user-facing interface. They provide valuable feedback for the development of the software, but are time-intensive to conduct. In this work, we present GERALLT, a system that uses Large Language Models (LLMs) to perform exploratory tests of the Graphical User Interface (GUI) of a real-life engineering software. GERALLT automatically generates a list of potential unintuitive and inconsistent parts of the interface. We present the architecture of GERALLT and evaluate it on a real-world use case of the engineering software, which has been extensively tested by developers and users. Our results show that GERALLT is able to determine issues with the interface that support the software development team in future development of the software.

Figures

Figures reproduced from arXiv: 2505.17839 by the authors.

**Figure 2.** Architecture of GERALLT. The component “Previous Screenshot” holds the screenshot of the GUI taken during the last iteration. It is replaced by an updated screenshot after each iteration. The dashed line in the bottom right denotes that the evaluator only receives the last action performed on the GUI instead of the complete log. emulate the manual testing, which aims to explore all possible interactions of GUI eleme… view at source ↗

**Figure 3.** The appearance of the online help in the tool in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: The evaluation process for our system. (a) Screenshot before the performed action (b) Screenshot after the performed action [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Example error, where the evaluator criticized that the error message is too imprecise. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

32 extracted references · 31 canonical work pages

[24]

Automated and manual testing in the development of the research software rce,

R. Mischke, K. Schaffert, D. Schneider, and A. Wein- ert, “Automated and manual testing in the development of the research software rce,” in Computational Science – ICCS 2022, D. Groen, C. de Mulatier, M. Paszynski, V. V. Krzhizhanovskaya, J. J. Dongarra, and P. M. A. Sloot, Eds. Cham: Springer International Publishing, 2022, pp. 531–544

work page 2022
[1]

Developing an Interface between an LLM and the GUI of RCE,

T. Rosenbach, “Developing an Interface between an LLM and the GUI of RCE,” Bachelor Thesis, DHBW Mannheim, 2024

work page 2024
[2]

A survey of software testing practices in canada,

V. Garousi and J. Zhi, “A survey of software testing practices in canada,” Journal of Systems and Software, vol. 86, no. 5, pp. 1354–1376, May 2013

work page 2013
[3]

[Online]

ISO, ISO/IEC 25010:2023, ISO/IEC Std., 2023. [Online]. Available: https://www.iso.org/standard/78176.html

work page 2023
[4]

How can manual testing processes be opti- mized? developer survey, optimization guidelines, and case studies,

R. Haas, D. Elsner, E. Juergens, A. Pretschner, and S. Apel, “How can manual testing processes be opti- mized? developer survey, optimization guidelines, and case studies,” in ESEC/FSE, ser. ESEC/FSE ’21. ACM, Aug. 2021

work page 2021
[5]

Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions,

Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions,” in ICSE, ser. ICSE ’24. ACM, Apr. 2024, pp. 1–13

work page 2024
[6]

Gui-based software testing: An automated approach using gpt-4 and selenium webdriver,

D. Zimmermann and A. Koziolek, “Gui-based software testing: An automated approach using gpt-4 and selenium webdriver,” in A-TEST. IEEE, Sep. 2023, pp. 171–174

work page 2023
[7]

First experiments on automated execution of gherkin test specifications with collaborating llm agents,

S. Bergsmann, A. Schmidt, S. Fischer, and R. Ramler, “First experiments on automated execution of gherkin test specifications with collaborating llm agents,” in A- TEST, ser. A-TEST ’24. ACM, Sep. 2024, pp. 12–15

work page 2024

Show all 32 references

[8]

Evaluat- ing the effectiveness of neuroevolution for automated gui- based software testing,

D. Zimmermann, P. Deubel, and A. Koziolek, “Evaluat- ing the effectiveness of neuroevolution for automated gui- based software testing,” in ASEW. IEEE, Sep. 2023, pp. 119–126

2023
[9]

How effective are they? exploring large language model based fuzz driver generation,

C. Zhang, Y. Zheng, M. Bai, Y. Li, W. Ma, X. Xie, Y. Li, L. Sun, and Y. Liu, “How effective are they? exploring large language model based fuzz driver generation,” in ISSTA, ser. ISSTA ’24. ACM, Sep. 2024, pp. 1223–1235

2024
[10]

Large language models can connect the dots: Exploring model optimization bugs with domain knowledge-aware prompts,

H. Guan, G. Bai, and Y. Liu, “Large language models can connect the dots: Exploring model optimization bugs with domain knowledge-aware prompts,” in ISSTA, ser. ISSTA ’24. ACM, Sep. 2024, pp. 1579–1591

2024
[11]

Llm4fin: Fully automating llm-powered test case generation for fintech software acceptance test- ing,

Z. Xue, L. Li, S. Tian, X. Chen, P. Li, L. Chen, T. Jiang, and M. Zhang, “Llm4fin: Fully automating llm-powered test case generation for fintech software acceptance test- ing,” in ISSTA, ser. ISSTA ’24. ACM, Sep. 2024, pp. 1643–1655

2024
[12]

From text to transformation: A comprehensive review of large language models’ versatility,

P. Kaur, G. S. Kashyap, A. Kumar, M. T. Nafis, S. Kumar, and V. Shokeen, “From text to transformation: A comprehensive review of large language models’ versatility,” 2024. [Online]. Available: https://arxiv.org/ abs/2402.16142

2024 arXiv
[13]

Use of chatgpt as an assistant in the end-to-end test script generation for android apps,

B. Garc ´ ıa, M. Leotta, F. Ricca, and J. Whitehead, “Use of chatgpt as an assistant in the end-to-end test script generation for android apps,” in A-TEST, ser. A-TEST ’24. ACM, Sep. 2024, pp. 5–11

2024
[14]

Large language models for equivalent mutant detection: How far are we?

Z. Tian, H. Shu, D. Wang, X. Cao, Y. Kamei, and J. Chen, “Large language models for equivalent mutant detection: How far are we?” in ISSTA, ser. ISSTA ’24. ACM, Sep. 2024, pp. 1733–1745

2024
[15]

Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,

C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” in ISSTA, ser. ISSTA ’24. ACM, Sep. 2024, pp. 819–831

2024
[16]

Cref: An llm-based con- versational software repair framework for programming tutors,

B. Yang, H. Tian, W. Pian, H. Yu, H. Wang, J. Klein, T. F. Bissyand´ e, and S. Jin, “Cref: An llm-based con- versational software repair framework for programming tutors,” in ISSTA, ser. ISSTA ’24. ACM, Sep. 2024, pp. 882–894

2024
[17]

Face it yourselves: An llm-based two-stage strategy to localize configuration errors via logs,

S. Shan, Y. Huo, Y. Su, Y. Li, D. Li, and Z. Zheng, “Face it yourselves: An llm-based two-stage strategy to localize configuration errors via logs,” in ISSTA, ser. ISSTA ’24. ACM, Sep. 2024, pp. 13–25

2024
[18]

Guidelines for gui testing maintenance: a linter for test smell detection,

T. Fulcini, G. Garaccione, R. Coppola, L. Ardito, and M. Torchiano, “Guidelines for gui testing maintenance: a linter for test smell detection,” in A-TEST, ser. A-TEST ’22. ACM, Nov. 2022

2022
[19]

Chouette: An automated cross-platform ui crawler for improving app quality,

T. Wong, “Chouette: An automated cross-platform ui crawler for improving app quality,” in ASWE. IEEE, Sep. 2023, pp. 175–178

2023
[20]

An empirical study on the adop- tion of scripted gui testing for android apps,

R. Gu and J. M. Rojas, “An empirical study on the adop- tion of scripted gui testing for android apps,” in ASEW. IEEE, Sep. 2023, pp. 179–182

2023
[21]

ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation,

D. Gao, L. Ji, Z. Bai, M. Ouyang, P. Li, D. Mao, Q. Wu, W. Zhang, P. Wang, X. Guo, H. Wang, L. Zhou, and M. Z. Shou, “ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation,” Dec. 2023

2023
[22]

Rce: An in- tegration environment for engineering and science,

B. Boden, J. Flink, N. F¨ orst, R. Mischke, K. Schaffert, A. Weinert, A. Wohlan, and A. Schreiber, “Rce: An in- tegration environment for engineering and science,” Soft- wareX, vol. 15, p. 100759, Jul. 2021

2021
[23]

Orchestrating tool chains for model-based systems engineering with rce,

J. Flink, R. Mischke, K. Schaffert, D. Schneider, and A. Weinert, “Orchestrating tool chains for model-based systems engineering with rce,” in AERO. IEEE, Mar. 2022, pp. 1–9

2022
[25]

Getting started with LLM prompt engineering,

S. Peckham, J. Day, and DianaHbr, “Getting started with LLM prompt engineering,” https://learn.microsoft.com/ en-us/ai/playbook/technology-guidance/generative-ai/ working-with-llms/prompt-engineering, accessed De- cember 13, 2024

2024
[26]

GitHub - pywinauto/pywinauto: Windows GUI automation with python (based on text properties),

pywinauto, “GitHub - pywinauto/pywinauto: Windows GUI automation with python (based on text properties),” Oct. 2019

2019
[27]

Gerallt,

“Gerallt,” https://github.com/DLR-SC/GERALLT
[28]

ChatGPT,

OpenAI, “ChatGPT,” Available at chat.openai.com
[29]

Pywinauto,

PyWinAuto Contributors, “Pywinauto,” Available at https://github.com/pywinauto/pywinauto
[30]

Dogtail,

“Dogtail,” https://gitlab.com/dogtail/dogtail/
[31]

Selenium,

“Selenium,” https://www.selenium.dev/
[32]

“Appium,” https://develop.kde.org/docs/apps/tests/ appium/

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [24]

Automated and manual testing in the development of the research software rce,

R. Mischke, K. Schaffert, D. Schneider, and A. Wein- ert, “Automated and manual testing in the development of the research software rce,” in Computational Science – ICCS 2022, D. Groen, C. de Mulatier, M. Paszynski, V. V. Krzhizhanovskaya, J. J. Dongarra, and P. M. A. Sloot, Eds. Cham: Springer International Publishing, 2022, pp. 531–544

work page 2022

[2] [1]

Developing an Interface between an LLM and the GUI of RCE,

T. Rosenbach, “Developing an Interface between an LLM and the GUI of RCE,” Bachelor Thesis, DHBW Mannheim, 2024

work page 2024

[3] [2]

A survey of software testing practices in canada,

V. Garousi and J. Zhi, “A survey of software testing practices in canada,” Journal of Systems and Software, vol. 86, no. 5, pp. 1354–1376, May 2013

work page 2013

[4] [3]

[Online]

ISO, ISO/IEC 25010:2023, ISO/IEC Std., 2023. [Online]. Available: https://www.iso.org/standard/78176.html

work page 2023

[5] [4]

How can manual testing processes be opti- mized? developer survey, optimization guidelines, and case studies,

R. Haas, D. Elsner, E. Juergens, A. Pretschner, and S. Apel, “How can manual testing processes be opti- mized? developer survey, optimization guidelines, and case studies,” in ESEC/FSE, ser. ESEC/FSE ’21. ACM, Aug. 2021

work page 2021

[6] [5]

Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions,

Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Make LLM a Testing Expert: Bringing Human-like Interaction to Mobile GUI Testing via Functionality-aware Decisions,” in ICSE, ser. ICSE ’24. ACM, Apr. 2024, pp. 1–13

work page 2024

[7] [6]

Gui-based software testing: An automated approach using gpt-4 and selenium webdriver,

D. Zimmermann and A. Koziolek, “Gui-based software testing: An automated approach using gpt-4 and selenium webdriver,” in A-TEST. IEEE, Sep. 2023, pp. 171–174

work page 2023

[8] [7]

First experiments on automated execution of gherkin test specifications with collaborating llm agents,

S. Bergsmann, A. Schmidt, S. Fischer, and R. Ramler, “First experiments on automated execution of gherkin test specifications with collaborating llm agents,” in A- TEST, ser. A-TEST ’24. ACM, Sep. 2024, pp. 12–15

work page 2024