Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Automated Soap Opera Testing Directed by LLMs and Scenario Knowledge: Feasibility, Challenges, and Road Ahead

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Automated soap opera testing driven by LLMs and scenario knowledge is feasible and finds real bugs, but it trails human testers in exploring scenario boundaries and in avoiding false positives.

desk verdict A credible feasibility study that automates soap opera testing with LLM agents; the evaluation overclaims a bit due to author-adjudicated ground truth, but the contribution and transparency warrant peer review. read the letter →

arxiv 2412.08581 v1 pith:NBJR3HNT submitted 2024-12-11 cs.SE

Yanqi Su , Zhenchang Xing , Chong Wang , Chunyang Chen , Xiwei Xu , Qinghua Lu , Liming Zhu This is my paper · ORCID
classification cs.SE
keywords soapoperatestingexploratorylargelanguagemodelsmulti-agentsystemsscenarioknowledgegraphmobileGUIretrieval-augmentedgenerationbugdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether soap opera testing—scenario-based exploratory testing in which a tester follows a complex user story and watches for unexpected behavior—can be automated with large language models. After a formative study of six professional testers, the authors build a multi-agent system in which a Planner, a Player, and a Detector cooperate to execute natural-language test steps against Android apps and flag anomalies in real time. On 30 soap opera tests across Firefox, WordPress, and AntennaPod, the system fully executed 19 tests and produced 68 bugs verified by the authors, among them bugs later fixed or confirmed by the projects' developers. False positives made up roughly 40-50 percent of automated findings, and automated and manual testers shared only two identical bugs. The authors conclude that automation is feasible but not yet at human level, and they lay out a research agenda centered on neural-plus-symbolic reasoning, human-AI co-learning, and connecting test scenarios to code-level root causes.

What carries the argument

The load-bearing machinery is a Scenario Knowledge Graph (SKG) built from each app's bug reports and issues, where normalized step descriptions carry alternative phrasings and link to expected-behavior oracles. Retrieval-augmented generation injects relevant SKG steps and oracles into the prompts of three multimodal, multi-turn agents: Planner generates an actionable sub-step plan for the current GUI screenshot; Player translates each sub-step into a structured UI action (tap, long-tap, double-tap, input, scroll, home, enter, landscape, portrait), targeting locations by numbered labels overlaid on a 100-pixel grid and executing through a command-line device tool; Detector retrieves oracles matching the executed instruction, asks the LLM to add creative-thinking oracles, and compares screenshots before and after execution to report violations as structured bug reports. The grid-labeling scheme and the oracle-retrieval loop are what let natural-language scenarios become executable actions with real-time anomaly checks.

What would settle it

In a fresh replication, replace each app's scenario knowledge graph with an equal-sized set of unrelated text chunks and keep everything else fixed; if full-test execution rates and developer-confirmed bug counts do not fall substantially relative to the full-knowledge run, the paper's central claim that scenario knowledge drives the automation would be contradicted.

Watch

Extended reading notes

Core claim

The central discovery is that scenario knowledge—not just a generic multimodal LLM—is what makes automated soap opera testing work, and that the approach is real but incomplete. The authors construct a Scenario Knowledge Graph (SKG) from public bug reports and issues, with normalized steps and associated expected-behavior oracles, and feed it to three role-specialized LLM agents through retrieval-augmented generation. In their evaluation, the Planner identified the next test step correctly 92.0 percent of the time, the Player parsed steps into UI instructions correctly 84.0 percent of the time and located elements 70.0 percent of the time, and the system fully executed 19 of 30 soap opera tests. The Detector found 68 true-positive bugs across the three apps, but also 52 false positives, with per-app accuracy of 0.516, 0.656, and 0.577. Ablation runs show that removing oracle knowledge or step knowledge from the SKG lowers true-positive counts and accuracy on every app, and that without step knowledge the number of fully executed tests drops from 19 to 11. The paper reads these results as evidence of feasibility plus a clear gap: automated testing over-generates UI-enhancement suggestions and under-explores the input and branch boundaries that human testers reach through creative thinking.

Load-bearing premise

The system assumes that historical bug reports and issue trackers contain enough accurate, up-to-date step instructions and expected behaviors to steer both execution and bug detection; a project lacking such records, or with stale ones, would weaken the automation.

Editorial extensions

If this is right

  • If the results hold, a maintainer can run soap opera tests automatically on Android apps that have a history of bug reports, harvesting genuine bug reports without a human executing each scenario.
  • The ablation data imply that building and curating the scenario knowledge graph is the primary engineering investment; a generic LLM without app-specific steps and oracles loses true-positive detections and leaves more tests unfinished.
  • The false-positive analysis implies that a practical deployment needs a triage step, because roughly one in two automated findings is not a real defect, and many real findings are enhancement suggestions rather than functional bugs.
  • Because only two bugs overlapped between automated and manual runs, the paper's view is that automated soap opera testing finds a different slice of the bug space, not merely a cheaper version of manual testing.
  • The roadmap's human-AI co-learning direction implies an immediate middle-ground use: humans filter automated reports while AI-generated unusual actions seed new manual exploration, a pattern already observed when the authors found 16 additional bugs by pursuing branches the automated run opened.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to add an execution-history check to the Detector: before accepting an oracle-triggered bug, verify that the oracle's own action sequence was actually performed, which would directly address the paper's reported 'unexecuted execution plans' hallucinations.
  • If the low overlap between automated and manual findings holds beyond three apps, then automated soap opera testing is best positioned as a complement to human exploration rather than a replacement, and a workflow that alternates the two should be measured for unique-bug yield.
  • The paper's 70 percent element-location accuracy points to a cheap improvement route: ground the Player in the app's accessibility hierarchy or widget coordinates instead of 100-pixel grid labels, which would also remove screenshot-timing false positives caused by delayed rendering.
  • The authors' observation that oracle text can suggest new branches implies a concrete experiment: convert actionable oracle sentences into executable sub-steps and measure whether true-positive rate rises or false-positive rate falls relative to the current Detector-only use of oracles.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper investigates whether soap opera testing (scenario-based exploratory testing) can be automated. The authors first conduct a formative study with six practitioners, deriving two insights (observant discovery of unexpected behaviors; creative expansion of exploration boundaries) and two automation challenges (test auto-play and real-time bug detection). They then build a multi-agent LLM system with Planner, Player, and Detector agents augmented by a scenario knowledge graph (SKG) built from each app's bug reports and issues. The system is evaluated on Firefox, WordPress, and AntennaPod, using 30 soap opera tests from the formative study. The reported results are 19/30 tests fully executed, 68 author-confirmed bug reports across the three apps with accuracy between 0.516 and 0.656, and an ablation showing that both step and oracle knowledge improve performance. The paper concludes that automated soap opera testing is feasible but still lags manual execution, and outlines a research roadmap.

Significance. The paper's main value is a systematic early feasibility study and a concrete architecture (RAG over SKG plus three LLM agents) that others can build on; the replication package and the detailed failure analysis (four causes of false positives) are useful contributions. If the results are taken at face value, they provide the first evidence that LLM-driven soap opera testing can execute complex NL scenarios and surface developer-confirmed bugs, and the roadmap (neural-symbolic synergy, human-AI co-learning, integration with code-level knowledge) is plausible and grounded in the observed failure modes. However, the central quantitative comparison to manual testing is not yet on equal footing, and the circularity of oracle-based detection needs a concrete overlap analysis before the 'significant gap' conclusion can be accepted.

major comments (3)
  1. [Table 1; Sections 2.2, 4.1.1, 4.2] The headline comparison in Table 1 is between two differently measured quantities. The manual rows count bugs that were reported to project trackers and received a developer status (Section 2.2 lists 5 fixed, 28 confirmed, 4 assigned, 5 wontfix), while the automated rows count bugs that two authors reproduced and adjudicated as valid (Section 4.1.1). Of the 68 automated true positives, only 34 were submitted to the projects, and of those only 23 have a reported developer status (3 fixed, 13 confirmed, 6 assigned, 1 duplicate; Section 4.2); the other 34 automated true positives were never seen by developers. Because the abstract and Section 1 use this table to support the claim that automated testing shows 'a significant gap compared to manual execution,' the claim currently conflates a measurement difference with a capability difference. Please developer-validate all automated detections (or a clearly specified random sample) and re-present Table 1 with matched validation standards, or explicitly reframe the automated rows as author-adjudicated and soften the gap claim.
  2. [Sections 4.2.1 and 4.2.2] Even under author adjudication, the two columns of Table 1 are not comparable because a large share of automated true positives are enhancement suggestions rather than defect reports. Section 4.2.1 reports that automated testing produced 19/32 (Firefox), 15/21 (WordPress), and 11/15 (AntennaPod) enhancements, whereas manual testers found only 3/27, 1/14, and 2/16 enhancements. Section 4.2.2 also classifies 20 of 52 false positives as 'unreasonable or unnecessary improvement suggestions,' which shows that the author-adjudication boundary for what counts as a valid finding is more permissive than the manual bug counts suggest. Please stratify Table 1 by bug type (defect vs. enhancement) and, for the 34 submitted automated bugs, report how many were enhancements and whether the developer responses treated them as valid.
  3. [Sections 3.2.1, 3.5.2, and Table 2] There is a circularity risk in the evaluation: the SKG is built from historical bug reports and issues (Section 3.2.1), and the Detector identifies bugs by retrieving oracles from that same SKG (Section 3.5.2). A detected 'true positive' can therefore be a reproduction of a known, previously reported issue rather than evidence of unexpected-behavior discovery. The ablation in Table 2 shows that oracle knowledge improves detection, but it does not measure how many automated true positives correspond to oracles already present in the SKG. Please add an overlap analysis that maps each detected bug to the historical reports used to construct the SKG (e.g., by keyword or step matching) and reports the number of detections that do not appear in the knowledge base. The paper's own examples of branch-exploration discoveries (e.g., Bug1912905 in Section 4.3) show that genuinely novel detections occur, so this is a correctness-risk concern about the magnitude of the feasibility claim, not an assertion that the approach is vacuous.
minor comments (5)
  1. [Sections 1 and 4.2] Please state the status of the 11 submitted reports that are not covered by the 3 fixed, 13 confirmed, 6 assigned, and 1 duplicate counts (e.g., pending or unanswered).
  2. [Section 7 (Related Work)] There is a typo: 'challendges' should be 'challenges'.
  3. [Section 4.3] Please report the full Likert distribution for sub-step generation and provide the rubric used by the two authors; currently only the extremes (72.0% rated 5, 10.0% rated 1 or 2) are given.
  4. [Section 3.2.2] The chunk size (800 tokens) and embedding dimension (256) are stated as defaults; a short sensitivity analysis would strengthen the claim that the RAG configuration is not responsible for the observed improvements.
  5. [Section 4.1.2] The random sample of 50 Planner-Player execution pairs is not described in enough detail (e.g., stratification by app, test, and step); please include the sampling procedure and the number of disagreements resolved by discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central feasibility claim is supported by external developer feedback and internal ablations, and the SKG oracle knowledge is applied as standard expected-behavior checking rather than being defined as the output.

full rationale

I walked the paper's derivation chain from the formative study through system design to the experiments and found no step in which a reported result is equivalent to its input by construction. The Detector retrieves oracles from the Scenario Knowledge Graph, and those oracles are extracted from historical bug reports' 'EBs' sections; however, the paper uses these oracles as expected-behavior checks against newly observed GUI states, not as the definition of the bugs it claims to find. The LLM prompt explicitly instructs the Detector not to restrict itself to retrieved oracles ('please do not restrict to these oracles'), and the paper reports independently discovered bugs (e.g., Bug1912905) plus developer-confirmed outcomes (3 fixed, 13 confirmed, 6 assigned priority/severity). The self-citations to the authors' prior SYSKG work [32-34] are present, but they are not load-bearing: the SKG construction is described in the paper, and its contribution is tested directly via the ablation in Table 2. The authors' adjudication of automated true positives and the partial submission for developer validation are measurement limitations that the paper transparently acknowledges; they affect the strength of the feasibility claim but do not make the claim circular. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported, and no known result is merely relabeled. Therefore the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the assumption that LLMs can follow role-play prompts and understand GUI screenshots, that the history-derived knowledge graph is representative, and that the author-labeled validation is a reasonable ground truth. The free hyperparameters (grid size, chunk size, embedding dimension) are design choices that influence performance but are not fitted to the target result.

free parameters (3)
  • grid_size = 100px x 100px
    UI screenshot divided into fixed-size elements for element labeling; chosen by hand, affects Player's position output.
  • chunk_size = 800 tokens
    Fixed token chunk size for vectorizing SKG; default file-search setting.
  • embedding_dimension = 256
    Dimensionality of chunk embeddings; default file-search setting.
assumptions (3)
  • domain assumption GPT-4o multimodal LLMs can reliably execute role-play prompts and understand GUI screenshots for element localization.
    The entire system relies on commercial LLM capabilities; failures in understanding directly cause false positives and auto-play failures (Section 3.3-3.5, 4.2.2).
  • domain assumption Historical bug reports and issues are representative of the app's UI operations and expected behaviors.
    SKG is constructed from these sources and used for retrieval; if they are outdated or biased, system performance degrades (Section 3.2.1).
  • domain assumption Two-author consensus is a valid proxy for bug validity when developer confirmation is not available.
    Ground truth for TP/FP is author-assessed for most bugs; only a subset was submitted to projects (Section 4.1.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated Soap Opera Testing Directed by LLMs and Scenario Knowledge: Feasibility, Challenges, and Road Ahead." pith.science (2026). https://pith.science/paper/NBJR3HNT

@misc{pith2026241208581,
  author       = {Pith},
  title        = {Pith review of: Automated Soap Opera Testing Directed by LLMs and Scenario Knowledge: Feasibility, Challenges, and Road Ahead},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NBJR3HNT}},
  note         = {Machine review of arXiv:2412.08581}
}
read the original abstract

Exploratory testing (ET) harnesses tester's knowledge, creativity, and experience to create varying tests that uncover unexpected bugs from the end-user's perspective. Although ET has proven effective in system-level testing of interactive systems, the need for manual execution has hindered large-scale adoption. In this work, we explore the feasibility, challenges and road ahead of automated scenario-based ET (a.k.a soap opera testing). We conduct a formative study, identifying key insights for effective manual soap opera testing and challenges in automating the process. We then develop a multi-agent system leveraging LLMs and a Scenario Knowledge Graph (SKG) to automate soap opera testing. The system consists of three multi-modal agents, Planner, Player, and Detector that collaborate to execute tests and identify potential bugs. Experimental results demonstrate the potential of automated soap opera testing, but there remains a significant gap compared to manual execution, especially under-explored scenario boundaries and incorrectly identified bugs. Based on the observation, we envision road ahead for the future of automated soap opera testing, focusing on three key aspects: the synergy of neural and symbolic approaches, human-AI co-learning, and the integration of soap opera testing with broader software engineering practices. These insights aim to guide and inspire the future research.

Figures

Figures reproduced from arXiv: 2412.08581 by the authors.

Figure 1
Figure 1. Approach Overview of Automated Soap Opera Testing [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Role-Play Prompt for Planner Initialization [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Role-Play Prompt for Player Initialization [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Role-Play Prompt for Detector Initialization [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Prompt for Oracle Retrieval 3.5.2 Oracle Retrieval. We retrieve relevant or￾acle knowledge from SKG to assist in bug detec￾tion as follows. First, we extract steps similar to the executed UI instruction from the SKG using the KG-based RAG method described in Section 3.…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Fixing

    cs.SE 2025-06 conditional novelty 6.0 of 10

    GUIRepair, a cross-modal LLM pipeline that converts issue screenshots into reproduction code and rendered patch screenshots into validation feedback, resolves 157/517 SWE-bench M instances with GPT-4o and 175 with o4-mini.

Reference graph

Works this paper leans on

48 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    Wasif Afzal, Ahmad Nauman Ghazi, Juha Itkonen, Richard Torkar, Anneliese Andrews, and Khurram Bhatti. 2015. An experiment on the effectiveness and efficiency of exploratory testing. Empirical Software Engineering 20 (2015), 844–878

    work page 2015

  2. [2]

    Jonathan Bach. 2000. Session-based test management. Software Testing and Quality Engineering Magazine 2, 6 (2000), 32–37

    work page 2000

  3. [3]

    James Bach. 2004. Exploratory testing. The testing practitioner (2004), 253–265

    work page 2004

  4. [4]

    Carlos Bernal-Cárdenas, Nathan Cooper, Kevin Moran, Oscar Chaparro, Andrian Marcus, and Denys Poshyvanyk

  5. [5]

    Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101

    2006

  6. [6]

    Hans Buwalda. 2004. Soap opera testing. Better Software 6, 2 (2004), 30–37

    work page 2004

  7. [7]

    JD Cem Kaner. 2013. An introduction to scenario testing. Florida Institute of Technology, Melbourne (2013), 1–13

    work page 2013

  8. [8]

    JD Cem Kaner and James Bach. 2006. The nature of exploratory testing. (2006)

    work page 2006

Show all 48 references
  1. [9]

    Mattia Fazzini, Martin Prammer, Marcelo d’Amorim, and Alessandro Orso. 2018. Automatically translating bug reports into test cases for mobile apps. In Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis. 141–152

    2018

  2. [10]

    Sidong Feng, Mulong Xie, and Chunyang Chen. 2023. Efficiency matters: Speeding up automated testing with gui rendering inference. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 906–918

    2023

  3. [11]

    Google. 2024. Android Debug Bridge (adb) - Android Developers. https://developer.android.com/studio/command- line/adb

    2024

  4. [12]

    Anastasiia Grishina. 2022. Enabling automatic repair of source code vulnerabilities using data-driven methods. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings . 275–277

    2022

  5. [13]

    Madeleine Havranek, Carlos Bernal-Cárdenas, Nathan Cooper, Oscar Chaparro, Denys Poshyvnayk, and Kevin Moran

  6. [14]

    Saki Imai. 2022. Is github copilot a substitute for human pair-programming? an empirical study. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings . 319–321

    2022

  7. [15]

    Juha Itkonen and Mika V Mäntylä. 2014. Are test cases needed? Replicated comparison between exploratory and test-case-based software testing. Empirical Software Engineering 19, 2 (2014), 303–342

    2014

  8. [16]

    Juha Itkonen, Mika V Mäntylä, and Casper Lassenius. 2012. The role of the tester’s knowledge in exploratory software testing. IEEE Transactions on Software Engineering 39, 5 (2012), 707–724

    2012

  9. [17]

    Juha Itkonen and Kristian Rautiainen. 2005. Exploratory testing: a multiple case study. In2005 International Symposium on Empirical Software Engineering, 2005. IEEE, 10–pp

    2005

  10. [18]

    Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. 2022. Jigsaw: Large language models meet program synthesis. InProceedings of the 44th International Conference on Software Engineering. 1219–1231

    2022

  11. [19]

    Cem Kaner. 2008. A tutorial in exploratory testing. Tutorial presented at QUEST2008.(A vailable online at: http://www. kaner. com/pdfs/QAIExploring. pdf, accessed: 26 Jan 2014) (2008)

    2008

  12. [20]

    Cem Kaner, Jack Falk, and Hung Q Nguyen. 1999. Testing computer software. John Wiley & Sons

    1999

  13. [21]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in Neural Information Processing S...

    2020

  14. [22]

    Ziqiang Li and Shin Hwei Tan. 2020. Bugine: a bug report recommendation system for Android apps. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Companion Proceedings . 278–279. , Vol. 1, No. 1, Article . Publication date: December 2024. 2...

    2020

  15. [23]

    Rensis Likert. 1932. A technique for the measurement of attitudes. Archives of psychology (1932)

    1932

  16. [24]

    Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2023. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1355ś1367 (2023)

    2023

  17. [25]

    Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Zhilin Tian, Yuekai Huang, Jun Hu, and Qing Wang. 2024. Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model. In Proceedings of the IEEE/ACM 46th International C...

    2024

  18. [26]

    James Lyndsay and Neil Van Eeden. 2003. Adventures in session-based testing. Workroom Productions Ltd. May 27 (2003)

    2003

  19. [27]

    Noor Nashid, Mifta Sintaha, and Ali Mesbah. 2023. Retrieval-based prompt selection for code-related few-shot learning. In Proceedings of the 45th International Conference on Software Engineering (ICSE’23)

    2023

  20. [28]

    Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining zero-shot vulnerability repair with large language models. In2023 IEEE Symposium on Security and Privacy (SP) . IEEE, 2339–2356

    2023

  21. [29]

    Dietmar Pfahl, Huishi Yin, Mika V Mäntylä, and Jürgen Münch. 2014. How is exploratory testing used? A state-of-the- practice survey. In Proceedings of the 8th ACM/IEEE international symposium on empirical software engineering and measurement. 1–10

    2014

  22. [30]

    Advait Sarkar, Andrew D Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence? arXiv preprint arXiv:2208.06213 (2022)

    2022 arXiv

  23. [31]

    Feng Sidong and Chen Chunyang. 2024. Prompting Is All Your Need: Automated Android Bug Replay with Large Language Models. In Proceedings of the 46th International Conference on Software Engineering (ICSE’24)

    2024

  24. [32]

    Yanqi Su, Zheming Han, Zhenchang Xing, Xin Xia, Xiwei Xu, Liming Zhu, and Qinghua Lu. 2022. Constructing a System Knowledge Graph of User Tasks and Failures from Bug Reports to Support Soap Opera Testing. In Proceedings of the 37th IEEE/ACM International Conference on Automate...

    2022

  25. [33]

    Yanqi Su, Zheming Han, Zhenchang Xing, Xiwei Xu, Liming Zhu, and Qinghua Lu. 2023. SoapOperaTG: A Tool for System Knowledge Graph Based Soap Opera Test Generation. In 2023 IEEE/ACM 45th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) . ...

    2023

  26. [34]

    Yanqi Su, Dianshu Liao, Zhenchang Xing, Qing Huang, Mulong Xie, Qinghua Lu, and Xiwei Xu. 2024. Enhancing exploratory testing by large language model and knowledge graph. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering . 1–12

    2024

  27. [35]

    Maryam Taeb, Amanda Swearngin, Eldon Schoop, Ruijia Cheng, Yue Jiang, and Jeffrey Nichols. 2024. Axnav: Replaying accessibility tests from natural language. In Proceedings of the CHI Conference on Human Factors in Computing Systems . 1–16

    2024

  28. [36]

    Shin Hwei Tan and Ziqiang Li. 2020. Collaborative bug finding for android apps. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 1335–1347

    2020

  29. [37]

    Jarle Våga and Ståle Amland. 2002. Managing high-speed web testing. In Software quality and software testing in internet times. Springer, 23–30

    2002

  30. [38]

    Bryan Wang, Gang Li, and Yang Li. 2023. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems . 1–17

    2023

  31. [39]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al . 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  32. [40]

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2023. Empowering llm to use smartphone for intelligent task automation. arXiv e-prints (2023), arXiv–2308

    2023

  33. [41]

    James A Whittaker. 2009. Exploratory software testing: tips, tricks, tours, and techniques to guide test design . Pearson Education

    2009

  34. [42]

    Bill Wood and David James. 2003. Applying session-based testing to medical software. Medical Device and Diagnostic Industry 25, 5 (2003), 90–103

    2003

  35. [43]

    Yiheng Xiong, Mengqian Xu, Ting Su, Jingling Sun, Jue Wang, He Wen, Geguang Pu, Jifeng He, and Zhendong Su

  36. [44]

    Zhizheng Zhang, Xiaoyi Zhang, Wenxuan Xie, and Yan Lu. 2023. Responsible task automation: Empowering large language models as responsible task automators. arXiv preprint arXiv:2306.01242 (2023)

    2023 arXiv

  37. [45]

    Yu Zhao, Tingting Yu, Ting Su, Yang Liu, Wei Zheng, Jingzhi Zhang, and William GJ Halfond. 2019. Recdroid: automatically reproducing android application crashes from bug reports. In2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) . IEEE, 128–139. , Vo...

    2019

  38. [2020]

    In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering

    Translating video recordings of mobile app usages into replayable scenarios. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering . 309–321

  39. [2021]

    In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion)

    V2S: a tool for translating video recordings of mobile app usages into replayable scenarios. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) . IEEE, 65–68

    2021

  40. [2023]

    In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2023)

    An Empirical Study of Functional Bugs in Android Apps. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2023) . 1319–1331. https://doi.org/10.1145/3597926.3598138

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.