{"id":"a9834864-418a-41fe-b42c-14f02fc076c9","arxiv_id":"2504.19335","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper outlines a proposed empirical study to measure whether integrating UI testing into GitHub Actions CI/CD workflows affects development activity and pipeline performance.","lead":"This paper is a study plan, not a completed study. It proposes to mine GitHub repositories to see whether integrating UI testing frameworks like Selenium, Playwright, and Cypress into CI/CD workflows changes developer activity or pipeline performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Even with perfect keyword detection, RQ1/RQ2 remain unanswerable: commits/PRs/issues are assigned to workflow runs by timestamp and message heuristics, and the before/after design has no control for confounding.","rationale":"The reader correctly identifies keyword-based detector validity as a weak assumption, and the paper itself acknowledges it. However, that specific threat is explicitly addressed by manual review and the requirement of UI test files, so it is not the deepest dependency. The more serious problem is that even a perfect detector does not make the outcome metrics interpretable as measures of impact. The design's attribution of commits, pull requests, and issues to workflow runs is heuristic, and the before/after time-series comparison lacks controls for concurrent confounds. Because the preprint reports no data and no tested hypotheses, the UNVERDICTED verdict is appropriate. My concern reinforces that status rather than changing it: the proposed study, as designed, cannot yet answer its research questions, and the authors should validate the attribution and causal identification strategy before execution. This is a partial agreement because the reader focused on sampling validity while I focus on internal validity and causal inference.","tokens_in":10257,"tokens_out":4738,"duration_ms":54510,"concrete_test":"Pilot the attribution and causal pipeline on 30–50 manually labeled repositories. First, for a one-month window, manually label every commit, pull request, and issue as UI-test-related or not, then compare with the proposed timestamp-plus-message heuristic; require at least 80% precision and recall, and report the confusion matrix. Second, in the interrupted time-series analysis, add a negative control outcome that should not be affected by UI testing, such as dependency-update commits or bot-generated closed issues; if the control outcome also shows a discontinuity at the keyword-introduction date, the estimated effect is spurious. The authors should run this pilot and report thresholds before executing the full study.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the planned pipeline can reveal whether UI testing in CI/CD impacts development activity and CI/CD workflows. The keyword false-positive threat identified in Section IV-A is real but partially mitigated by the planned manual review and requirement that corresponding UI test files be explicitly referenced (Section III-A, criterion 7). A more load-bearing, unaddressed threat is attribution and confounding. In Section III-C.1b, RQ1 metrics are split into 'UI test-related' versus 'non-UI test-related' workflow runs, yet the authors concede there is 'no direct way to fetch commits, pull requests or issues influenced by a specific workflow run.' Their workaround is to take the triggering commit/PR and filter by timestamp and message text. That heuristic conflates 'a commit that happened to trigger a workflow containing a UI-test keyword' with 'development activity caused by UI testing.' Pull requests and issues created in the same time window or mentioning UI-test terms cannot be causally attributed to the workflow. Similarly, the interrupted time series in Section III-C.2 uses the first commit that adds a keyword to a GitHub Actions file as an intervention marker. But actual UI-test usage is an ongoing, uneven practice, not a one-time intervention, and any before/after difference in repo-level metrics is confounded by simultaneous changes in project direction, contributor composition, and tooling. Since the sample is restricted to repositories already using UI testing, with no control group, the design identifies correlation at best, not the 'impact' stated in the objective.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes an empirical study plan to investigate whether integrating UI testing frameworks (Selenium, Cypress, Playwright) into GitHub Actions CI/CD workflows affects open-source development activity and CI/CD processes. The authors describe a repository selection strategy based on keyword search of workflow files, a set of filtering criteria, plans to collect development and workflow metrics, and a two-part analysis plan (comparative statistics and interrupted time series). No data collection, analysis, or results are presented; the paper is explicitly a study protocol.","tokens_in":10535,"tokens_out":2905,"duration_ms":31975,"significance":"If executed with a valid design, a study of UI testing in CI/CD workflows on GitHub could provide useful empirical evidence for a topic with limited prior work. The manuscript's strengths include a clear research question, an explicit filter criteria list, a plan to publish data and scripts, and an acknowledgment of several validity threats. However, as submitted, the paper makes no empirical contribution, and the proposed design contains load-bearing flaws that likely prevent it from answering the stated research questions. The significance is therefore conditional on a substantially revised protocol, not on the current content.","major_comments":[{"comment":"The attribution of development activity to UI-test-related workflow runs is not valid as described. The authors state that there is 'no direct way to fetch commits, pull requests or issues influenced by a specific workflow run' and propose to filter by timestamp and message text. This heuristic conflates temporal proximity with causation: a commit that triggers a workflow containing a UI-test keyword is not necessarily caused by UI testing, and issues/PRs mentioning UI-test terms may be unrelated to the workflow run. Consequently, the planned comparative analysis between 'UI test-related' and 'non-UI test-related' metrics cannot support the causal language of RQ1 ('affect development processes'). This is a central threat to the validity of the main research question.","section":"§III-C.1b"},{"comment":"The interrupted time series design uses the first commit introducing a UI-test keyword as the intervention marker, but UI testing integration is an ongoing practice rather than a one-time event: repositories add, remove, and update UI tests over time, and workflow files change frequently. Moreover, the design lacks a control group or any adjustment for confounding factors such as project popularity, contributor turnover, or simultaneous changes in other CI/CD tooling. Without such controls, any observed before/after differences in development or workflow metrics cannot be attributed to UI testing integration. The authors do not mention difference-in-differences, matched controls, or other identification strategies that would address this issue.","section":"§III-C.2"},{"comment":"The keyword-based detector for identifying UI-testing repositories is foundational to the entire sample, and the acknowledged false-positive threat (comments, web scraping, non-testing automation) is only partially mitigated by the planned manual review. Criterion 7 requires 'corresponding UI test files explicitly specified for execution in the workflow files,' but the manuscript does not specify how this will be operationalized, how the manual review will be conducted to avoid researcher bias, or how inter-rater reliability will be measured. A preliminary manual check on a subset of the same repositories is not a validation protocol; the authors should pre-register a detection rule and a validation threshold before proceeding.","section":"§III-A and §IV-A"}],"minor_comments":[{"comment":"The sentence 'to our knowledge their is no prior work' contains a typo ('their is' should be 'there is').","section":"§II-C"},{"comment":"The phrase 'In necessary' appears near the end of the repository selection section; it should be 'If necessary.'","section":"§III-A"},{"comment":"The path expression '.github.workflows/.*.yml' should use the correct syntax '.github/workflows/*.yml' to denote GitHub Actions workflow files.","section":"§III-A"},{"comment":"The list of keywords is introduced with inconsistent quotation marks ('selenium', 'cypress', or playwright\"), which makes the text slightly confusing.","section":"§III-A"},{"comment":"The metric types 'integer' and 'boolean' are used as data types, but some columns (e.g., 'Pull Request Duration datetime') mix units with types; this should be clarified.","section":"Table II"}],"recommendation":"reject","confidential_remarks":"The manuscript is a study protocol rather than a completed empirical study. Given the venue's journal format, the lack of any data or results is already a significant concern. More importantly, the proposed design has fundamental threats to causal validity (attribution heuristic in §III-C.1b and uncontrolled interrupted time series in §III-C.2) that are not addressed by the acknowledged threats to validity. Even if the authors were to execute the study as planned, the results would not support the research questions. The appropriate path would be to redesign the study with a control group and a more rigorous definition of UI-test-related activity, and only then resubmit with actual results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a research proposal, not an empirical paper, and that is the key thing to know before reading it. There are no data, no results, and no model. What it does well: it identifies a real gap in the literature, focuses on three popular UI testing frameworks and GitHub Actions, and lays out a concrete, well-organized pipeline for repository mining. The related work is adequately cited, and the authors explicitly acknowledge a real threat with keyword-based detection (false positives from comments or scraping), mitigation via manual review and requiring explicit UI test files. That is more self-aware than many mining studies. That is the extent of what is new: a proposed application of established methods to a new domain, with no findings.\n\nNow the soft spots, and they are substantive. The stress-test note is right: even with perfect keyword detection, the attribution logic is too weak to answer RQ1. The paper admits there is no direct way to fetch commits, PRs, or issues influenced by a specific workflow run, then proposes to infer relevance from the triggering commit, timestamps, and message text. That conflates “happened near a UI-test workflow run” with “caused by UI testing.” The interrupted time series has the same problem. The intervention marker is the first commit that adds a UI-testing keyword to a workflow file, but adoption is usually not a discrete one-time event, and there is no control group. Simultaneous project changes and contributor composition shifts are just not addressed. The paper acknowledges some generalizability and construct validity concerns, but the missing-control-group issue is never really discussed.\n\nI think the paper is a solid skeleton for a registered report or a pre-registration, and the authors would benefit from feedback before collecting data. But as a full empirical submission, it would need major redesign: add a control group of repositories without UI testing, use matching or difference-in-differences, and soften the language from “impact” to “association.” The claim that there is “no prior work exploring the impact of UI testing on CI/CD in open-source software” is also too strong given the Lu et al. case study, though I grant that case is not open-source.\n\nWho gets value from this: anyone studying empirical software engineering, GitHub Actions, or UI testing adoption. It is more useful as a cautionary example of causal inference in mining studies than as a source of findings. I would not cite it in my own work next year, but I would read it to see how the authors revise. My recommendation: treat it as a pre-registration proposal and referee it on design merits, with the expectation of major revision.","headline":"A clear, honest study plan for an under-explored topic, but the proposed designs cannot back the causal wording in the research questions.","tokens_in":11018,"tokens_out":2345,"would_cite":false,"duration_ms":26430,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a GitHub-wide keyword search of GitHub Actions workflow files can reveal whether adding UI testing to CI/CD changes open-source development, and it proposes an empirical study with comparative and interrupted…","keywords":["CI/CD","UI testing","GitHub Actions","Selenium","Cypress","Playwright","open-source software","empirical study"],"falsifier":"Manually inspect a random sample of the candidate workflow files the keyword search returns and check whether each one references an actual UI test file or command; if a substantial share of matches are comments, installs without tests, or scraping scripts, the repository sample and every downstream comparison would not measure what the study claims.","tokens_in":10083,"feed_emoji":"🧪","tokens_out":5408,"duration_ms":48776,"temperature":0.7,"pith_summary":"This paper proposes an empirical study to find out whether integrating UI testing into CI/CD pipelines changes how open-source projects develop. It argues that repositories using Selenium, Cypress, or Playwright in GitHub Actions workflow files can be detected by searching for those keywords in .github/workflows/*.yml and .yaml files, and that comparing UI-test workflows against non-UI workflows before and after the keyword first appears will reveal impacts on development activity and CI/CD health. The authors care because UI testing is widely considered difficult to automate and practitioners report the effort is not worth the benefit, yet no prior work has measured the impact of UI testing on CI/CD in open-source software. The paper is a study plan; it states no results, but it lays out testable research questions and a concrete pipeline for answering them.","feed_headline":"A GitHub study will test UI testing's impact on CI/CD pipelines","feed_subtitle":"They will scan GitHub Actions workflows for the tools and compare metrics before and after they appear.","key_machinery":"The load-bearing mechanism is keyword-based detection: the GitHub REST API code search looks for the strings 'selenium', 'cypress', or 'playwright' inside GitHub Actions workflow files, supplemented by Sourcegraph to bypass the API's 1,000-record limit. Repository filtering then keeps only active, licensed, non-forked projects with enough workflow runs and at least one UI-test and one non-UI workflow, and the analytic core is an interrupted time series model using the first commit that introduced a UI-testing keyword as the intervention marker. This machinery carries the argument because the validity of both research questions rests on whether a workflow file containing one of these strings is a genuine UI-testing workflow.","core_discovery":"In the paper's own terms, the central claim is empirical and still unverified: open-source repositories that run UI testing inside GitHub Actions workflows experience measurable differences in development activity and CI/CD workflow behavior compared with their own non-UI workflows. The authors' planned evidence is a dataset of repositories whose GitHub Actions configuration files mention Selenium, Cypress, or Playwright, filtered to active, licensed, non-forked projects with enough workflow runs, then analyzed with descriptive and inferential statistics (paired t-test or Wilcoxon signed-rank) and interrupted time series around the commit that first introduced a UI-testing keyword. If the study succeeds, it would provide the first large-scale evidence on the cost and benefit of UI testing in CI/CD for open-source software.","pith_inferences":["A natural next step the paper does not pursue is to validate the keyword detector's precision and recall on a labeled random sample before running any impact analysis; that validation alone would be a publishable measurement result.","The same GitHub Actions corpus could also be used to ask whether UI tests are more likely to be removed or disabled after flaky runs, a maintenance pattern the paper's metrics would only indirectly capture.","Because the paper selects only projects that already have both UI and non-UI workflows, it will not say anything about projects that tried UI testing and deleted it; a separate survival-style analysis would be needed."],"forward_implications":["If the detection pipeline works, the study will produce the first large-scale census of open-source repositories using Selenium, Cypress, or Playwright inside GitHub Actions.","If UI-testing workflows differ on failure rate and execution time, teams will have concrete evidence about the CI/CD cost of UI testing.","If development activity shifts after the introducing commit, maintainers will learn whether UI testing tends to accompany busier or quieter projects.","The interrupted time series design can separate a step-change at UI-test adoption from gradual project trends, giving a stronger basis for causal language.","A public dataset and scripts would let other researchers extend the same pipeline to other frameworks and CI platforms."],"supporting_citations":[{"why":"Closest prior result the study extends: automated UI testing combined with CI improved productivity and quality in a smart-grid system.","marker":"[45]"},{"why":"Provides the evidence that testing in CI increases failure detection, the empirical base for RQ2 on CI/CD impact.","marker":"[43]"},{"why":"Shows which testing types CI/CD projects adopt and notes UI testing is largely absent, motivating the gap.","marker":"[3]"},{"why":"Grounds the choice of GitHub Actions by describing how developers use it to automate workflows.","marker":"[24]"},{"why":"Supplies the known pitfalls of mining GitHub repositories that shape the filtering criteria and validity threats.","marker":"[48]"},{"why":"Supplies the interrupted time series method used to estimate effects before and after UI-test introduction.","marker":"[49]"},{"why":"Adds Sourcegraph as a supplementary code search source because the GitHub API limits results to 1,000 records.","marker":"[47]"},{"why":"Documents rising and multi-tool CI/CD adoption in open source, the context for why UI-test integration matters.","marker":"[23]"}],"fun_headline_variants":["GitHub study to test UI testing's impact on CI/CD","Will UI testing in CI/CD change development? New study","Exploring UI testing in CI/CD: A GitHub study","UI testing in CI/CD: New GitHub research to measure effects","Selenium, Playwright, Cypress in CI/CD: Impact under study"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole study depends on the assumption that a GitHub Actions workflow file containing the word 'selenium', 'cypress', or 'playwright' actually runs UI tests, rather than mentioning the tool in a comment or using it for unrelated web automation.","fun_headline_variants_meta":{"raw":{"variants":["GitHub study to test UI testing's impact on CI/CD","Will UI testing in CI/CD change development? New study","Exploring UI testing in CI/CD: A GitHub study","UI testing in CI/CD: New GitHub research to measure effects","Selenium, Playwright, Cypress in CI/CD: Impact under study"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000247,"raw_usage":{"total_tokens":1513,"prompt_tokens":882,"completion_tokens":631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":498,"tokens_out":631,"duration_ms":6663,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:54:29.289085+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually inspect a random sample of the candidate workflow files the keyword search returns and check whether each one references an actual UI test file or command; if a substantial share of matches are comments, installs without tests, or scraping scripts, the repository sample and every downstream comparison would not measure what the study claims.","supporting_citations":[{"cited_title":"Implementation of continuous integration and automated testing in software development of smart grid scheduling support system,","cited_arxiv_id":null,"evidence_quote":"Closest prior result the study extends: automated UI testing combined with CI improved productivity and quality in a smart-grid system."},{"cited_title":"Oops, my tests broke the build: An explorative analysis of travis ci with github,","cited_arxiv_id":null,"evidence_quote":"Provides the evidence that testing in CI increases failure detection, the empirical base for RQ2 on CI/CD impact."},{"cited_title":"Synthesizing continuous deployment practices used in software development,","cited_arxiv_id":null,"evidence_quote":"Shows which testing types CI/CD projects adopt and notes UI testing is largely absent, motivating the gap."},{"cited_title":"How do software developers use github actions to automate their workflows?,","cited_arxiv_id":null,"evidence_quote":"Grounds the choice of GitHub Actions by describing how developers use it to automate workflows."},{"cited_title":"The promises and perils of mining github,","cited_arxiv_id":null,"evidence_quote":"Supplies the known pitfalls of mining GitHub repositories that shape the filtering criteria and validity threats."},{"cited_title":"McDowall, R","cited_arxiv_id":null,"evidence_quote":"Supplies the interrupted time series method used to estimate effects before and after UI-test introduction."},{"cited_title":"Sourcegraph: Universal code search and intelligence,","cited_arxiv_id":null,"evidence_quote":"Adds Sourcegraph as a supplementary code search source because the GitHub API limits results to 1,000 records."},{"cited_title":"Chronicles of CI/CD: A Deep Dive into its Usage Over Time","cited_arxiv_id":"2402.17588","evidence_quote":"Documents rising and multi-tool CI/CD adoption in open source, the context for why UI-test integration matters."}],"review_version":1}