Pith. sign in

REVIEW 4 major objections 6 minor 53 references

An LLM-driven browser agent that reconstructs missing setup, executes a high-level plan in a live browser, and checks the final page state against the report reproduces about half of 667 real-world web GUI bug reports.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

ReBug reproduces 49.96% of 667 real web GUI bug reports by reconstructing prerequisites, executing state-aware browser actions, and validating outcomes, beating both browser-agent and script-generation baselines.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid LLM-agent system for web GUI bug reproduction with a real benchmark; headline RSR is a procedural-completion metric on current deployments, not a fully validated reproduction rate. the 4 major comments →

arxiv 2608.03598 v1 pith:2TON36QT submitted 2026-08-04 cs.SE

From Bug Reports to Browser-Executable Procedures: An LLM-Driven Agent for Web GUI Bug Reproduction

Cunming Zhang , Yu Pei , Michail Papadakis This is my paper
classification cs.SE
keywords web GUI bug reproductionLLM agentsbrowser automationnatural-language bug reportsprerequisite reconstructionstate-aware executionmetamorphic oracleempirical study
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ReBug is a self-contained LLM-driven agent that takes a natural-language web bug report and turns it into a working browser procedure, then checks whether the final page state matches what the report says should or should not happen. The paper claims that splitting the work into two stages—first reconstructing missing files, login state, and other prerequisites and writing a high-level plan, then executing step-by-step while grounding every action in the live DOM and screenshots—lets the agent reproduce about half of 667 real-world reports. That matters because manual reproduction is a major bottleneck in fixing and regression-testing web applications. The paper also reports that when 40 successful runs were rerun against restored historical buggy versions, 39 triggered the originally described failure, suggesting the successful procedures are not just completing generic workflows.

Core claim

The paper's central claim is that an agent combining prerequisite reconstruction, state-aware browser execution, and report-derived outcome validation can reproduce web GUI bugs from natural-language reports at scale. In the preparation stage, the agent assesses what the report implicitly requires—files, data, configuration, or entry points—materializes feasible artifacts inside a sandbox, and produces an ordered but flexible high-level plan. In the execution stage, it drives a real browser through tool-mediated actions, keeps a structured memory of the current page state and action history, and terminates with a validation phase in which an LLM judge compares the final URL, visible DOM, and

What carries the argument

The central mechanism is the two-stage prepare-and-execute pipeline with a strict success oracle. The preparation stage materializes missing prerequisites and emits a high-level plan; the execution stage repeatedly captures DOM and screenshot evidence, chooses tool actions grounded in the current browser state, and updates a shared Information Center that records agent state, action results, and browser-state history. Success is defined by the reproduction success rate equation: a run counts as successful only if it terminates with the task marked complete, matches the annotator's expected-state description, and does not match the unexpected-state description. The LLM-based evaluator applies

Load-bearing premise

The headline success rate counts a run as reproducing the bug when the agent finishes its own plan and the final page matches human-written expected and unexpected descriptions on a current, likely-fixed deployment—not when the originally reported failure is actually observed.

What would settle it

The decisive check would be to run all 667 reports through the agent against their corresponding historical buggy versions, not just 40 successful cases, and count how many reproduce the actual reported failure; if that fraction is far below 49.96%, the current-deployment oracle was measuring procedure completion rather than bug reproduction.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Developers could hand a bug report to the agent and receive a repeatable browser trace plus a pass/fail decision, instead of manually reconstructing the reproduction steps.
  • Because 39 of 40 oracle-successful runs triggered the original report-described behavior on restored buggy versions, current-deployment success can serve as a proxy for failure-triggering reproduction on this benchmark.
  • Removing prerequisite reconstruction lowers reproduction success by 6.6 to 11.3 percentage points across projects, so missing files and setup steps are a recoverable rather than hopeless class of incompleteness.
  • The dominant failure mode is grounding intended actions to dynamic UI elements, accounting for 60–74% of failure labels, which identifies the next concrete bottleneck for browser-based reproduction systems.
  • The pipeline's success is stable across several LLM backbones (RSR remains roughly 46–55%), suggesting the agent design and execution structure matter more than the specific model choice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its oracle-based metric probably undercounts true reproduction on reports where the current deployment already contains the reported bug, since matching the unexpected description would fail the success condition by design; restaging historical versions could raise measured success.
  • The 40-case replay, not the 667-case main evaluation, is the direct evidence of failure-triggering reproduction; a full historical replay of all 667 reports would convert the headline 49.96% from procedure completion into verified bug reproduction.
  • The same two-stage structure—reconstruct context, execute state-aware, validate outcomes—could transfer to mobile or desktop GUI reproduction by swapping browser-observable signals for platform-specific hierarchies, crash evidence, and instrumentation.
  • The near-zero performance of one-shot script generation implies that natural-language-to-script translation is insufficient on its own; an interactive loop with live page observations is the load-bearing design choice.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ReBug, a two-stage LLM-driven agent for reproducing web GUI bugs from natural-language bug reports. In the preparation stage, the system assesses and materializes report-specific prerequisites and generates a high-level reproduction plan; in the execution stage, it drives a real browser through tool-mediated actions, maintains structured state summaries and action histories, and terminates with an LLM-based evaluation of the final browser state against expected/unexpected descriptions. The evaluation covers 667 confirmed bug reports from Ghost, Metabase, NocoDB, and n8n, comparing ReBug against Browser-use and one-shot Playwright/Selenium script generation. Headline results: RSR 49.96%, TCR 74.96%, AESR 86.54%; the ablation shows the Context Builder contributes 6.63–11.29 percentage points; a historical replay on 40 successful runs triggers bug-present behavior in 39 cases. The paper also promises an artifact repository with implementation, prompts, schemas, and traces.

Significance. If the evaluation oracle is valid, this is a strong and useful empirical contribution: it is, to my knowledge, the first large-scale study of end-to-end web GUI bug reproduction from raw issue reports with real deployments, and it provides a reusable benchmark with outcome oracles. The comparison against two baselines, the three-model backbone sensitivity analysis, the ablation, and the trace-based failure taxonomy are well designed and reported. The promised artifact repository is a concrete strength. However, the headline metric rests on an oracle that is not independently validated, and the historical replay is limited to successful runs. Thus the magnitude of the claimed 49.96% RSR should be treated as provisional until the authors close that measurement-validity gap.

major comments (4)
  1. [Sections III-C2 and IV-C, Eq. (1)] The RSR definition in Eq. (1) requires done_i (the agent's self-declared termination flag), m_E(i), and not m_U(i), where E_i/U_i are author-written descriptions and the final check is performed by an LLM judge. No inter-annotator agreement is reported for E_i/U_i (Section IV-A2 only states two authors reviewed; no IAA), and no accuracy/reliability study of the LLM judge against human judgments is provided. Consequently, RSR may measure self-consistency rather than bug reproduction. This is load-bearing for the central claim. I ask for (a) IAA on a sample of annotations, (b) a human-judge-vs-LLM-judge agreement study on a stratified sample of final states, and (c) an error analysis of judge divergences.
  2. [Section V-A1, Table IV] The 40 historical replay cases are sampled only from successful ReBug runs. This design cannot estimate the false-positive rate of the current-deployment oracle: it never tests runs that failed the oracle, nor does it test a random sample of the 667 reports. The statement that 'successful procedures often expose the original bug-present behavior' is conditional on the very runs that already passed the current-deployment oracle. To validate the oracle, sample a random set of reports (including oracle-failed runs), replay on restored buggy versions, and report sensitivity/specificity relative to ground-truth bug presence; the manual inspection should be performed blind.
  3. [Sections IV-A2 and IV-E, filters R1–R4] The dataset filters exclude version-locked issues (R3), yet every report is executed on a current deployment where the reported bug may have been fixed or may still be present. The expected/unexpected descriptions are written with respect to that current deployment, not the historical buggy version, and no per-report verification is reported of the bug's status at evaluation time. If a bug is still present, passing the expected-state check does not necessarily demonstrate reproduction; if it is fixed, U_i may be stale. Please add a per-report classification of bug status (e.g., issue resolution, fix commit date) and a sensitivity analysis of RSR by that status.
  4. [Section III-C1 and Eq. (1)] The done_i signal in Eq. (1) is emitted by the agent itself, and the same model family both generates actions and decides completion. The paper does not report any independent verification that done_i corresponds to the plan being genuinely completed. Please report how often done_i is asserted when not all plan steps are marked complete, and consider replacing the self-declared flag with an independent plan-completion check or human verification.
minor comments (6)
  1. [Global formatting] The name 'REBUG' is missing spaces in several places, e.g., 'REBUGseparates', 'REBUGon', 'REBUGachieves', 'REBUGand', 'REBUGto' in the abstract and body. Please use a consistent typographic form.
  2. [Section IV-E vs Table III] The global parameter section sets B_step = 20 executed steps, but Table III reports mean #Actions between 26 and 34. Since Section III-C1 says a step can contain one or more low-level actions, clarify the distinction between steps and actions and define whether the budget limits steps or actions.
  3. [Section IV-C, Eq. (1)] Eq. (1) defines N as total runs, while Section IV-A initially presents D as a set of N bug reports. Please explicitly state that N in Eqs. (1)–(3) is run-level, and clarify how per-report aggregation is performed.
  4. [Section IV-A2] Two authors independently reviewed reports and resolved disagreements, but no agreement rate or number of disagreements is reported. A brief inter-annotator agreement figure would strengthen confidence in the annotation process.
  5. [Section III-C2] The LLM judge is described only functionally. Please provide the evaluator prompt/rubric in the artifact or appendix, and state the sampling temperature and other settings used for the judge, since these affect reproducibility.
  6. [Table II] The 'Average' row is given only for REBUG(Full), GPT-5 mini. Adding average RSR for the baselines would directly support the claim that ReBug outperforms both baselines across datasets.

Circularity Check

0 steps flagged

No significant circularity: RSR is a defined metric with independently authored annotations; oracle-validity concerns are measurement limitations, not circular reductions.

full rationale

The paper does not contain a mathematical derivation chain that reduces to its inputs. RSR (Eq. 1) is an explicitly defined evaluation metric: a run succeeds iff the agent terminates with done_i, the final state matches the annotated expected description E_i, and it does not match the annotated unexpected description U_i. E_i and U_i are authored by two human annotators in Section IV-A2, independently of ReBug's execution traces, so success is not defined as 'the agent matches its own plan.' The done flag is self-reported, but the expected/unexpected matching conditions are externally specified, and the metric is therefore not equivalent to the agent's self-assessment. The evaluation is further anchored by external baselines (Browser-use and direct script generation), and the historical replay subset (Section V-A1) is explicitly presented as complementary evidence, not as an unbiased re-estimate of RSR. Selection of successful runs for replay is a statistical limitation, not a circular step. No self-citation is load-bearing, and no uniqueness theorem or ansatz is imported from the authors' prior work. The construct-validity threat that the oracle may not capture true bug reproduction is acknowledged by the authors in Section VII and is a measurement-validity concern, not a circularity of the paper's reasoning.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

ReBug's reported performance rests on a chain of hand-set budgets, human annotations, and an LLM judge rather than on a parameter-free derivation. No new physical or abstract entities are introduced. The numbers are therefore only as strong as the annotation and evaluation pipeline, which is not independently verified in the paper.

free parameters (4)
  • Execution step budget B_step = 20
    Maximum number of execution steps per report; hand-set and affects RSR because runs exceeding it are terminated.
  • Consecutive failure limit B_fail = 3
    Number of consecutive action failures tolerated before forcing termination; influences whether a run can finish.
  • Per-action timeout T_act = 120s
    Binds individual automation steps and can cause failures on slow pages.
  • Attempts per report K = 3 (early-stop)
    Independent attempts with early-stop protocol; the run-level aggregation in Eq. (1) depends on this choice.
axioms (5)
  • domain assumption The deployed current versions of Ghost, Metabase, NocoDB, and n8n are bug-absent for the selected reports.
    Section IV-A2 evaluates on controlled current deployments; if a selected bug persists in the current version, a correct procedure could match the unexpected description and be counted as failure.
  • domain assumption The human-authored expected/unexpected descriptions are accurate and complete oracles.
    Section IV-A2: two authors annotated the descriptions, but no inter-annotator agreement metric is reported; the RSR definition relies entirely on these descriptions.
  • domain assumption The LLM-based judge reliably evaluates final browser state against the oracle descriptions.
    Section III-C2: the evaluator is an LLM judge comparing URL, DOM summary, and screenshots; no calibration or accuracy measurement is provided.
  • domain assumption The metamorphic oracle (end-state consistency on a current deployment) is a valid proxy for actual bug reproduction.
    Section IV-C, Eq. (1): RSR defines success as procedural completion plus end-state consistency; actual bug presence is only checked on 40 successful runs in the historical replay.
  • domain assumption The agent's own completion flag (done_i) is trustworthy.
    Eq. (1): a run counts as successful only when the agent itself marks the task completed; no analysis of premature or false termination is provided.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of From Bug Reports to Browser-Executable Procedures: An LLM-Driven Agent for Web GUI Bug Reproduction." pith.science (2026). https://pith.science/paper/2TON36QT

@misc{pith2026260803598,
  author       = {Pith},
  title        = {Pith review of: From Bug Reports to Browser-Executable Procedures: An LLM-Driven Agent for Web GUI Bug Reproduction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2TON36QT}},
  note         = {Machine review of arXiv:2608.03598}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reproducing web GUI bugs from natural-language bug reports is critical for software maintenance, but remains difficult because reports often lack prerequisites such as dependencies and input files. Existing bug reproduction techniques mainly target code units or mobile applications and lack end-to-end visual execution and validation for web GUIs. We present ReBug, a context-aware agent system that reconstructs, executes, and validates browser-level reproduction procedures from web GUI bug reports by driving a real browser. ReBug separates reproduction into two stages. In the preparation stage, ReBug reconstructs missing prerequisites from the report and available artifacts, and it produces a high-level reproduction plan. In the execution stage, it performs tool-mediated interactions in the browser, maintains structured summaries of page state and action history, and validates the final state against expectations derived from the report. We evaluate ReBug on 667 real-world bug reports from four open-source web applications. On controlled current deployments, ReBug outperforms both baselines, achieving an average RSR of 49.96%, a mean task completion rate of 74.96%, and a mean action execution success rate of 86.54%. Our results show that explicit context reconstruction and state-aware browser execution effectively support report-derived browser reproduction, while historical replay shows that successful procedures often expose the original bug-present behavior on restored buggy versions.

Figures

Figures reproduced from arXiv: 2608.03598 by Cunming Zhang, Michail Papadakis, Yu Pei.

Figure 1
Figure 1. Figure 1: Overview of REBUG. The workflow begins with the preparation stage, which reconstructs the required context and produces a reproduction plan. It then enters the execution stage, which performs an iterative execution loop in a real browser with continuous state tracking and trace logging, followed by automatic outcome evaluation. snippets, spreadsheet-like data, or other local artifacts con￾sumed by the brow… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 34 canonical work pages

  1. [1]

    Monitoring user interactions for supporting failure reproduction,

    T. Roehm, N. Gurbanova, B. Bruegge, C. Joubert, and W. Maalej, “Monitoring user interactions for supporting failure reproduction,” in 2013 21st International Conference on Program Comprehension (ICPC). IEEE, 2013, pp. 73–82

    work page 2013

  2. [2]

    Interactive record/replay for web application debugging,

    B. Burg, R. Bailey, A. J. Ko, and M. D. Ernst, “Interactive record/replay for web application debugging,” inProceedings of the 26th annual ACM symposium on User interface software and technology, 2013, pp. 473– 484

    work page 2013

  3. [3]

    What makes a good bug report?

    N. Bettenburg, S. Just, A. Schr ¨oter, C. Weiss, R. Premraj, and T. Zim- mermann, “What makes a good bug report?” inProceedings of the 16th ACM SIGSOFT International Symposium on Foundations of software engineering, 2008, pp. 308–318

    2008

  4. [4]

    Assessing the quality of the steps to reproduce in bug reports,

    O. Chaparro, C. Bernal-C ´ardenas, J. Lu, K. Moran, A. Marcus, M. Di Penta, D. Poshyvanyk, and V . Ng, “Assessing the quality of the steps to reproduce in bug reports,” inProceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2019, pp. 86–96

    2019

  5. [5]

    Improved bug reporting and reproduction through non-intrusive gui usage monitoring and automated replaying,

    S. Herbold, J. Grabowski, S. Waack, and U. B ¨unting, “Improved bug reporting and reproduction through non-intrusive gui usage monitoring and automated replaying,” in2011 IEEE fourth international conference on software testing, verification and validation workshops. IEEE, 2011, pp. 232–241

    work page 2011

  6. [6]

    Who should fix this bug?

    J. Anvik, L. Hiew, and G. C. Murphy, “Who should fix this bug?” in 28th International Conference on Software Engineering (ICSE 2006), Shanghai, China, May 20-28, 2006, L. J. Osterweil, H. D. Rombach, and M. L. Soffa, Eds. ACM, 2006, pp. 361–370

    work page 2006

  7. [7]

    Analysis and design of selenium webdriver automation testing framework,

    S. Gojare, R. Joshi, and D. Gaigaware, “Analysis and design of selenium webdriver automation testing framework,”Procedia Computer Science, vol. 50, pp. 341–346, 2015

    work page 2015

  8. [8]

    Pathak,Web Automation Testing Using Playwright: End-to-end, API, accessibility, and visual testing using Playwright

    K. Pathak,Web Automation Testing Using Playwright: End-to-end, API, accessibility, and visual testing using Playwright. BPB Publications, 2024

    work page 2024

  9. [9]

    Erratum: Leverag- ing flexible tree matching to repair broken locators in web automation scripts,

    S. Brisset, R. Rouvoy, L. Seinturier, and R. Pawlak, “Erratum: Leverag- ing flexible tree matching to repair broken locators in web automation scripts,”Information and Software Technology, vol. 144, p. 106754, 2022

    work page 2022

  10. [10]

    Large language models are few- shot testers: Exploring llm-based general bug reproduction,

    S. Kang, J. Yoon, and S. Yoo, “Large language models are few- shot testers: Exploring llm-based general bug reproduction,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 2312–2323

    2023

  11. [11]

    Assertflip: Reproduc- ing bugs via inversion of llm-generated passing tests,

    L. Khatib, N. S. Mathews, and M. Nagappan, “Assertflip: Reproduc- ing bugs via inversion of llm-generated passing tests,”arXiv preprint arXiv:2507.17542, 2025

    arXiv 2025

  12. [12]

    Automated bug reproduction from user reviews for android applications,

    S. Li, J. Guo, M. Fan, J.-G. Lou, Q. Zheng, and T. Liu, “Automated bug reproduction from user reviews for android applications,” inPro- ceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice, 2020, pp. 51–60

    work page 2020

  13. [13]

    Recdroid+: Automated end-to-end crash reproduction from bug reports for android apps,

    Y . Zhao, T. Su, Y . Liu, W. Zheng, X. Wu, R. Kavuluru, W. G. Halfond, and T. Yu, “Recdroid+: Automated end-to-end crash reproduction from bug reports for android apps,”ACM Transactions on Software Engineer- ing and Methodology (TOSEM), vol. 31, no. 3, pp. 1–33, 2022

    work page 2022

  14. [14]

    Prompting is all you need: Automated android bug replay with large language models,

    S. Feng and C. Chen, “Prompting is all you need: Automated android bug replay with large language models,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–13

    work page 2024

  15. [15]

    Feedback-driven automated whole bug report reproduction for android apps,

    D. Wang, Y . Zhao, S. Feng, Z. Zhang, W. G. Halfond, C. Chen, X. Sun, J. Shi, and T. Yu, “Feedback-driven automated whole bug report reproduction for android apps,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1048–1060

    2024

  16. [16]

    Application monitoring for bug reproduction in web-based applications,

    D. Wang, M. Galster, and M. Morales-Trujillo, “Application monitoring for bug reproduction in web-based applications,”Journal of Systems and Software, vol. 207, p. 111834, 2024

    work page 2024

  17. [17]

    Standing on the shoulders of giants: Bug-aware automated gui testing via retrieval augmentation,

    M. Chen, Z. Liu, C. Chen, J. Wang, B. Wu, J. Hu, and Q. Wang, “Standing on the shoulders of giants: Bug-aware automated gui testing via retrieval augmentation,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 825–846, 2025

    work page 2025

  18. [18]

    Mind2web: Towards a generalist agent for the web,

    X. Deng, Y . Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y . Su, “Mind2web: Towards a generalist agent for the web,”Advances in Neural Information Processing Systems, vol. 36, pp. 28 091–28 114, 2023

    2023

  19. [19]

    Webarena: A realistic web environment for building autonomous agents,

    S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y . Bisk, D. Friedet al., “Webarena: A realistic web environment for building autonomous agents,”arXiv preprint arXiv:2307.13854, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Gpt-4v (ision) is a generalist web agent, if grounded,

    B. Zheng, B. Gou, J. Kil, H. Sun, and Y . Su, “Gpt-4v (ision) is a generalist web agent, if grounded,”arXiv preprint arXiv:2401.01614, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” inThe eleventh international conference on learning representations, 2022

    2022

  22. [22]

    Recent trends and challenges in using nlp techniques in software debugging: A systematic literature review,

    L. N. Lota, T. S. Zaman, M. M. Azwad, L. Farah, A. Chowdhury, Z. Anjum, C. Islam, and A. R. M. Kamal, “Recent trends and challenges in using nlp techniques in software debugging: A systematic literature review,”Available at SSRN 5060080

    work page

  23. [23]

    Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports,

    J. Zhou, H. Zhang, and D. Lo, “Where should the bugs be fixed? more accurate information retrieval-based bug localization based on bug reports,” in2012 34th International conference on software engineering (ICSE). IEEE, 2012, pp. 14–24

    work page 2012

  24. [24]

    Detecting missing information in bug descrip- tions,

    O. Chaparro, J. Lu, F. Zampetti, L. Moreno, M. Di Penta, A. Marcus, G. Bavota, and V . Ng, “Detecting missing information in bug descrip- tions,” inProceedings of the 2017 11th joint meeting on foundations of software engineering, 2017, pp. 396–407

    2017

  25. [25]

    What’s in a bug report?

    S. Davies and M. Roper, “What’s in a bug report?” inProceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2014, pp. 1–10

    2014

  26. [26]

    Aegis: An agent-based framework for general bug reproduction from issue descriptions,

    X. Wang, P. Gao, X. Meng, C. Peng, R. Hu, Y . Lin, and C. Gao, “Aegis: An agent-based framework for general bug reproduction from issue descriptions,”arXiv preprint arXiv:2411.18015, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Automated system testing using visual gui testing tools: A comparative study in industry,

    E. Borjesson and R. Feldt, “Automated system testing using visual gui testing tools: A comparative study in industry,” in2012 IEEE Fifth International Conference on Software Testing, Verification and Validation. IEEE, 2012, pp. 350–359

    work page 2012

  28. [28]

    Automated testing of gui applications: models, tools, and controlling flakiness,

    A. M. Memon and M. B. Cohen, “Automated testing of gui applications: models, tools, and controlling flakiness,” in2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 1479– 1480

    work page 2013

  29. [29]

    Graphical user interface (gui) testing: Systematic mapping and repository,

    I. Banerjee, B. Nguyen, V . Garousi, and A. Memon, “Graphical user interface (gui) testing: Systematic mapping and repository,”Information and Software Technology, vol. 55, no. 10, pp. 1679–1694, 2013

    work page 2013

  30. [30]

    Finding bugs in web applications using dynamic test generation and explicit-state model checking,

    S. Artzi, A. Kiezun, J. Dolby, F. Tip, D. Dig, A. Paradkar, and M. D. Ernst, “Finding bugs in web applications using dynamic test generation and explicit-state model checking,”IEEE Transactions on Software Engineering, vol. 36, no. 4, pp. 474–494, 2010

    work page 2010

  31. [31]

    A framework for automated testing of javascript web applications,

    S. Artzi, J. Dolby, S. H. Jensen, A. Møller, and F. Tip, “A framework for automated testing of javascript web applications,” inProceedings of the 33rd international conference on software engineering, 2011, pp. 571–580

    work page 2011

  32. [32]

    Model-based vul- nerability testing for web applications,

    F. Lebeau, B. Legeard, F. Peureux, and A. Vernotte, “Model-based vul- nerability testing for web applications,” in2013 IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops. IEEE, 2013, pp. 445–452

    work page 2013

  33. [33]

    Model-based testing leveraged for automated web tests,

    G. R. Mattiello and A. T. Endo, “Model-based testing leveraged for automated web tests,”Software Quality Journal, vol. 30, no. 3, pp. 621– 649, 2022

    work page 2022

  34. [34]

    Automating functional tests using sele- nium,

    A. Holmes and M. Kellogg, “Automating functional tests using sele- nium,” inAGILE 2006 (AGILE’06). IEEE, 2006, pp. 6–pp

    work page 2006

  35. [35]

    Selenium framework for web automation testing: A systematic literature review,

    H. A. Thooriqoh, T. N. Annisa, and U. L. Yuhana, “Selenium framework for web automation testing: A systematic literature review,”JUTI: Jurnal Ilmiah Teknologi Informasi, pp. 65–76, 2021

    work page 2021

  36. [36]

    Exploring browser automation: A comparative study of selenium, cypress, puppeteer, and playwright,

    B. Garc ´ıa, J. M. del Alamo, M. Leotta, and F. Ricca, “Exploring browser automation: A comparative study of selenium, cypress, puppeteer, and playwright,” inInternational Conference on the Quality of Information and Communications Technology. Springer, 2024, pp. 142–149

    work page 2024

  37. [37]

    Automatically reproducing android bug reports using natural language processing and reinforcement learning,

    Z. Zhang, R. Winn, Y . Zhao, T. Yu, and W. G. Halfond, “Automatically reproducing android bug reports using natural language processing and reinforcement learning,” inProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2023, pp. 411–422

    work page 2023

  38. [38]

    An empirical investigation into the reproduction of bug reports for android apps,

    J. Johnson, J. Mahmud, T. Wendland, K. Moran, J. Rubin, and M. Fazz- ini, “An empirical investigation into the reproduction of bug reports for android apps,” in2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2022, pp. 321– 322

    work page 2022

  39. [39]

    Considering browser interaction in web application testing,

    G. A. Di Lucca and M. Di Penta, “Considering browser interaction in web application testing,” inFifth IEEE International Workshop on Web Site Evolution, 2003. Theme: Architecture. Proceedings.IEEE, 2003, pp. 74–81

    work page 2003

  40. [40]

    Leveraging user interactions for in-depth testing of web applications,

    S. McAllister, E. Kirda, and C. Kruegel, “Leveraging user interactions for in-depth testing of web applications,” inInternational Workshop on Recent Advances in Intrusion Detection. Springer, 2008, pp. 191–210

    work page 2008

  41. [41]

    Wevquery: Testing hypotheses about web interaction patterns,

    A. Apaolaza and M. Vigo, “Wevquery: Testing hypotheses about web interaction patterns,”Proceedings of the ACM on Human-Computer Interaction, vol. 1, no. EICS, pp. 1–17, 2017

    work page 2017

  42. [42]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 572–576

    work page 2024

  43. [43]

    An empirical evaluation of using large language models for automated unit test generation,

    M. Sch ¨afer, S. Nadi, A. Eghbali, and F. Tip, “An empirical evaluation of using large language models for automated unit test generation,”IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2023

    2023

  44. [44]

    Llm for test script generation and migration: Challenges, capabilities, and opportunities,

    S. Yu, C. Fang, Y . Ling, C. Wu, and Z. Chen, “Llm for test script generation and migration: Challenges, capabilities, and opportunities,” in2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS). IEEE, 2023, pp. 206–217

    work page 2023

  45. [45]

    Inferfix: End-to-end program repair with llms,

    M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy, “Inferfix: End-to-end program repair with llms,” in Proceedings of the 31st ACM joint european software engineering conference and symposium on the foundations of software engineering, 2023, pp. 1646–1656

    work page 2023

  46. [46]

    Can we enhance bug report quality using llms?: An empirical study of llm-based bug report generation,

    J. Acharya and G. Ginde, “Can we enhance bug report quality using llms?: An empirical study of llm-based bug report generation,” in Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, 2025, pp. 994–1003

    2025

  47. [47]

    Guardian: A runtime framework for llm-based ui exploration,

    D. Ran, H. Wang, Z. Song, M. Wu, Y . Cao, Y . Zhang, W. Yang, and T. Xie, “Guardian: A runtime framework for llm-based ui exploration,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 958–970

    work page 2024

  48. [48]

    Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions,

    Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering, 2024, pp. 1–13

    work page 2024

  49. [49]

    Towards generating ui design feedback with llms,

    P. Duan, J. Warner, and B. Hartmann, “Towards generating ui design feedback with llms,” inAdjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023, pp. 1–3

    2023

  50. [50]

    Issue2test: Generating reproducing test cases from issue reports,

    N. Nashid, I. Bouzenia, M. Pradel, and A. Mesbah, “Issue2test: Generating reproducing test cases from issue reports,”arXiv preprint arXiv:2503.16320, 2025

    arXiv 2025

  51. [51]

    Agentoccam: A simple yet strong baseline for llm-based web agents,

    K. Yang, Y . Liu, S. Chaudhary, R. Fakoor, P. Chaudhari, G. Karypis, and H. Rangwala, “Agentoccam: A simple yet strong baseline for llm-based web agents,”arXiv preprint arXiv:2410.13825, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Laser: Llm agent with state-space exploration for web navigation,

    K. Ma, H. Zhang, H. Wang, X. Pan, W. Yu, and D. Yu, “Laser: Llm agent with state-space exploration for web navigation,”arXiv preprint arXiv:2309.08172, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks,

    J. Y . Koh, R. Lo, L. Jang, V . Duvvur, M. Lim, P.-Y . Huang, G. Neubig, S. Zhou, R. Salakhutdinov, and D. Fried, “Visualwebarena: Evaluating multimodal agents on realistic visual web tasks,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 881–905

    2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.