{"id":"cf74a8e6-b0a8-42a3-983c-325ec4af94a8","arxiv_id":"2504.16777","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Most flaky tests in a 24-project Java dataset co-occur in failure clusters, and machine learning on static test similarity can partially predict which flaky tests cluster.","lead":"Flaky tests, which pass or fail without code changes, often fail together in clusters that share a single root cause. The paper measures this 'systemic flakiness' across 24 Java projects and tests whether static code similarity can predict the clusters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RQ1's 75% prevalence is not tested against a null model of independent failures; co-occurrence may be a clustering artifact or shared environment event, so the shared-fixable-root-cause claim is not yet established.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing premise: co-occurrence of failures in the same test suite runs is taken as evidence of a shared, fixable root cause. My stress-test sharpens this into a concrete, testable gap: the clustering pipeline lacks any null-model comparison. Since the threshold is selected per project to maximize silhouette on the observed data, and failure counts are often small, non-singleton clusters can arise under independent failures. The paper's manual inspection of stack traces is useful qualitative evidence but is not a statistical baseline; it cannot rule out the possibility that a substantial fraction of the 606 clustered tests would cluster under a null model. I therefore agree with the reader's conditional verdict. I do not recommend changing the verdict because the dataset and methodology are transparent, the replication package is available, and the proposed permutation test can settle the issue. If the null test shows the observed clustering is significantly above chance, the core prevalence claim is substantially strengthened; if not, the central claim would need to be reframed. The RQ2 prediction weakness identified by the reader is real but secondary, since it does not affect the existence of the clusters, only the lightweight prediction of them.","tokens_in":18432,"tokens_out":5432,"duration_ms":53840,"concrete_test":"Run a permutation null-model test on the replication-package data: for each project, preserve each flaky test's failure count but randomly reassign its failing run IDs independently, then rerun the exact RQ1 pipeline (agglomerative clustering with the same distance metric and threshold selection via maximum silhouette >= 0.6, ignoring singleton clusters) and record the percentage of flaky tests in non-singleton clusters. Repeat at least 1000 times to form a null distribution. If the observed 75% (or per-project rates) is not above the 95th percentile of the null, the prevalence claim is not evidence of shared root causes; if it is far above, the co-occurrence result survives this threat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is that 75% of flaky tests belong to clusters whose co-occurring failures share a fixable root cause. The only quantitative evidence is agglomerative clustering of Jaccard distances between failing-run sets, with per-project thresholds chosen to maximize silhouette and a requirement that mean silhouette >= 0.6 (Section 2.2). Nothing in this procedure compares the observed cluster structure with what would arise if each flaky test's failing runs were independent of every other test. With 810 flaky tests across 10,000 runs and typically low per-test failure counts, hierarchical clustering can produce non-singleton, high-silhouette clusters by chance, especially because the threshold is optimized on the same data. The paper's Section 2.5 construct-validity note concedes 'random chance' as a possible contributor, and the only check is a manual stack-trace inspection (RQ3) that cannot quantify how much co-occurrence would remain under independence. This matters because the practical payoff, 'fixing one root cause repairs multiple flaky tests', requires that clusters correspond to a common, addressable cause; a shared external outage (the dominant RQ3 theme) is a common cause but may not be a single code-level fix. If the 75% figure is near the null expectation, the paper's foundational claim is unsupported even though the clusters themselves are real.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the concept of 'systemic flakiness': flaky tests whose failures co-occur in the same test-suite runs and are assumed to share a root cause. Using the existing FlakeFlagger dataset of 10,000 runs across 24 Java projects (810 flaky tests), the authors apply agglomerative clustering to Jaccard distances between failing-run sets, report that 75% of flaky tests fall into 45 non-singleton clusters, train tree-based models to predict pairwise Jaccard distances and cluster membership from static test-code distance measures, and manually inspect stack traces to attribute cluster causes. The central claims are that systemic flakiness is widespread, that it can be predicted without full reruns, and that developers can fix multiple flaky tests by addressing shared root causes.","tokens_in":18673,"tokens_out":6212,"duration_ms":57802,"significance":"If the prevalence claim withstands scrutiny, the paper identifies a genuinely under-explored phenomenon with practical implications for flaky-test repair and for simulating flakiness in software-engineering experiments. Strengths include the use of a well-established, expensive 10,000-run dataset; the public replication package; the mixed-method design combining clustering, prediction, and manual inspection; and the negotiated-agreement protocol for qualitative analysis. The manual inspection gives credible evidence that some clusters share environmental causes, such as networking failures. However, the RQ1 prevalence figure is not tested against a null model of independent failures, and RQ2 lacks the baselines needed to interpret its predictive results; both issues are load-bearing for the paper's headline conclusions rather than cosmetic.","major_comments":[{"comment":"The headline 75% prevalence figure is not tested against a null model in which each flaky test's failing runs are independent of every other test. The pipeline selects, per project, the distance threshold that maximizes mean silhouette and then discards projects whose best silhouette is below 0.6; with 810 flaky tests spread over 10,000 runs and low per-test failure counts, non-singleton high-silhouette clusters can arise by chance. I request a permutation baseline that preserves each flaky test's failure count and, ideally, each run's failure count, reruns the same clustering pipeline, and reports the distribution of the percentage of flaky tests in clusters. Without such a baseline, the statement that 75% of flaky tests 'belong to a cluster' cannot support the conclusion that systemic flakiness is widespread, because some clustering is expected under independence.","section":"Section 2.2, Table 1"},{"comment":"The inference from co-occurrence to a shared fixable root cause is load-bearing for the practical payoff, but the evidence supports a weaker conclusion. The construct-validity section explicitly lists random chance as a possible contributor, and RQ3's dominant themes are networking (25/45 clusters) and external dependency (14/45) — common environmental conditions rather than code-level defects that a single developer action repairs. Table 5 reports Unknown for 14 clusters and lists mitigations such as Avoid Networking and Better Error Checking, which are not single-fix repairs. The authors should either provide evidence that a meaningful fraction of clusters trace to a specific fixable defect in the project's own code, or reframe the contribution as characterizing co-occurrence and its environmental causes rather than as 'fix one root cause, repair many tests.'","section":"Section 2.5, Tables 4 and 5"},{"comment":"RQ2's evaluation lacks the baselines needed to interpret the reported R2 and MCC values. The regression target is a pairwise Jaccard distance, which is strongly related by construction to the name- and code-based distance features, several of which are themselves Jaccard distances over static tokens; the reported R2 is relative only to a constant-mean predictor. The classification labels come from the same silhouette-optimized clustering, so the experiments evaluate in-sample predictability of the clustering output rather than prediction of independently validated systemic flakiness. I ask the authors to add at least (i) a baseline using only the hierarchy distance or a single best feature, (ii) a permutation or random-feature baseline, and (iii) ideally a cross-project evaluation, and to temper the RQ2 conclusion accordingly.","section":"Section 2.3, Table 2"},{"comment":"The clustering procedure is incompletely specified. The text does not state the linkage criterion used by SciPy's agglomerative clustering (for example, single, complete, average, or Ward) or the range and step size of the distance thresholds searched when maximizing mean silhouette. These choices materially change cluster membership and therefore the 606/810 count; they need to be reported either in this section or in the replication package for the analysis to be reproducible and for the threshold sensitivity of the headline result to be assessed.","section":"Section 2.2"}],"minor_comments":[{"comment":"The sentence 'We call this phenomenonsystemic flakiness' is missing a space; it should read 'We call this phenomenon systemic flakiness.'","section":"Abstract"},{"comment":"The text 'it is does not require us to prespecify the number of clusters' contains a duplicated verb; it should read 'it does not require us to prespecify.'","section":"Section 2.2"},{"comment":"The text 'a value of 0 indicates they they differ in their first component' has a doubled 'they'; one occurrence should be removed.","section":"Section 2.3"},{"comment":"The caption reads '21 static test case distances measures'; this should be '21 static test case distance measures.'","section":"Table 3 caption"},{"comment":"The number of stack traces sampled per cluster is not reported, and the Levenshtein-diversity sampling procedure is described only briefly; adding these details would improve the reproducibility of RQ3.","section":"Section 2.4"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the missing null model for RQ1. If the authors can show that 606/810 exceeds an appropriate permutation baseline and add simple baselines to RQ2, the paper could become a solid empirical contribution. I would not require new data collection, but the authors may need to reframe the practical 'fix one root cause' claim to match the environmental causes that their own RQ3 analysis identifies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe bottom line: this is a serious empirical paper, and the reader's conditional verdict is about right. The genuinely new thing is the systemic perspective on flaky tests – that flaky failures co-occur in clusters and that the clusters often trace to a shared external cause. The authors define this cleanly, use an established 10,000-run dataset, and back the clustering with a manual review of all 45 clusters, which is real work. The replication package is cited, and the methodology is transparent enough to reproduce.\n\nThe main soft spot is the one the stress-test flags: the 75% prevalence figure is not compared to a null model. Agglomerative clustering on sparse binary vectors can produce high-silhouette clusters by chance, and the per-project threshold is chosen on the same data. So 75% should be read as an upper bound, not a precisely measured rate. That said, this is not a fatal flaw – the manual inspection found plausible shared causes (networking, external dependencies) for many clusters, which is evidence the clusters are not pure artifacts. The deeper question is whether those causes are 'fixable by one developer action' in the sense the abstract implies. A networking outage is a common cause, but the repair may be environmental rather than a code fix. The paper's own repair-mitigation themes (look before you leap, mocking) are honest about this.\n\nRQ2 is the weakest section. The models are trained to predict labels derived from the exact same 10,000 runs, with no simple baselines (e.g., predict the mean distance, or always predict the majority cluster). An R2 of 0.74 sounds good, but without a baseline it is hard to know how much is signal. The SHAP analysis is also only about the model, not the phenomenon. This belongs in the paper, but as a preliminary feasibility result rather than an established lightweight method.\n\nThe authors are honest about the co-occurrence-to-root-cause assumption in Section 2.5, and they acknowledge random chance as a factor. That is good practice. The citation pattern looks appropriate – they distinguish prior clustering work (Golagha et al., An et al.) and the order-dependent flaky test literature.\n\nWho should read this: flaky test researchers, people building flakiness simulations, and CI tooling folks. The descriptive finding changes how we think about flaky test repair – even if the exact 75% needs calibration.\n\nRecommendation: send to peer review. The core empirical contribution is new, the dataset is heavily used, and the limitations are acknowledged. Revisions should ask for a null model comparison and simple baselines in RQ2. This is exactly the kind of paper that benefits from a serious referee.","headline":"Solid novel empirical study of flaky-test co-occurrence; the 75% prevalence is an upper bound, but the phenomenon is real and deserves serious peer review.","tokens_in":19240,"tokens_out":2896,"would_cite":true,"duration_ms":25342,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Flaky tests frequently fail in clusters that share root causes, so one repair can fix many tests at once.","keywords":["flaky tests","systemic flakiness","failure co-occurrence","agglomerative clustering","Jaccard distance","machine learning prediction","root cause analysis","continuous integration"],"falsifier":"For each of the 45 clusters, count the member tests whose failure stack traces end in the same root exception; if most clusters lack a single shared root exception, the claim that co-occurrence implies a shared fixable cause fails. A stronger test would stub out the suspected network path or external dependency for a cluster and check whether all member tests stop failing together.","tokens_in":18195,"feed_emoji":"🧪","tokens_out":9991,"duration_ms":83555,"temperature":0.7,"pith_summary":"This paper sets out to show that flaky tests, tests that pass or fail unpredictably without code changes, are not isolated problems. By clustering 810 flaky tests from 22 Java projects according to the test-suite runs in which they failed, the authors find that 75% of them belong to 45 clusters, with an average of 13.5 tests per cluster spanning multiple test classes. They call this systemic flakiness and argue that it changes the economics of flaky-test repair: instead of debugging one test at a time, a developer can address a shared root cause and repair many tests at once. The paper also shows that machine-learning models using static code and naming distances can predict which flaky tests belong together, suggesting that thousands of reruns are not always needed to find the clusters.","feed_headline":"75% of flaky tests fail in clusters with shared causes","feed_subtitle":"One root-cause fix can repair an average of 13.5 flaky tests at once, cutting the cost of flaky-test debugging.","key_machinery":"The central object is the Jaccard distance over failure sets. For flaky tests $A$ and $B$, let $A$ and $B$ also denote the sets of test-suite run IDs in which each test fails; then $J(A,B) = 1 - |A \\cap B| / |A \\cup B|$, where 0 means the two tests always fail in the same runs and 1 means they never co-occur. Agglomerative clustering over these distances, with the clustering cutoff chosen by maximizing mean silhouette score subject to a minimum of 0.6, converts raw co-occurrence into named clusters. Two supporting mechanisms carry the rest: a prediction pipeline in which character- and set-based string distances on test names and tokenized source, plus a hierarchy distance on package/class paths, feed tree-ensemble models that predict $J(A,B)$ and cluster membership; and a manual inspection protocol in which sampled stack traces and error messages are reviewed by multiple inspectors and resolved through negotiated agreement to assign cause themes to clusters. Together these carry the argument from \"these tests fail in the same runs\" to \"these tests fail for a shared, repairable reason.\"","core_discovery":"The paper's central discovery is that flaky-test failures co-occur in structured clusters, not as isolated events. On a dataset of 10,000 test-suite runs for each of 24 Java projects, of which 22 contain at least one flaky test and 810 flaky tests in total, the authors compute the Jaccard distance between every pair of flaky tests' sets of failing run IDs and cluster the tests agglomeratively, choosing the distance threshold that maximizes the mean silhouette score. This produces 45 non-singleton clusters in 10 projects; 606 of the 810 flaky tests (75%) fall into a cluster, the mean cluster size is 13.5 tests, and clusters span 2.9 test classes on average. The paper argues that these co-occurrence clusters correspond to shared root causes, and manual inspection of stack traces identifies intermittent networking issues and instabilities in external dependencies as the dominant causes. It then shows that an extra-trees model trained only on static distance measures between test names and tokenized source code predicts pairwise Jaccard distances with mean $R^2 = 0.74$ and cluster membership with mean Matthews correlation coefficient 0.74, so much of the cluster structure is recoverable without exhaustive reruns. The discovery, if correct, overturns the default assumption that flaky failures are isolated and makes systemic flakiness a first-class target for repair and tooling.","pith_inferences":["Editorial extension: the dataset rebooted the machine between test-suite runs to isolate runs; a CI pipeline that reuses processes, caches, and network connections may show weaker or different clustering, so the 75% prevalence should be re-estimated in production-like settings.","Editorial extension: if the hierarchy distance is the strongest predictor, then tests in the same package or class subtree tend to share failure conditions; a cheap, proactive risk signal would flag same-hierarchy tests that touch the network or external services before any rerun is done.","Editorial extension: the cluster-then-fix view suggests a measurable triage workflow: group newly failing tests by co-occurrence in a sliding window of CI runs, fix the cluster's shared cause, and record how many tests are repaired per action; that yield can be compared against per-test debugging.","Editorial extension: because the dominant causes are environmental, effective fixes may be infrastructure changes rather than test-code changes; adding a precondition that checks a server or directory before a test runs is a concrete mitigation that could be tested A/B in a repository."],"forward_implications":["One repair action aimed at a cluster's shared cause can fix, on average, 13.5 flaky tests at once, directly attacking the measured cost of flaky-test repair.","Impact studies of flaky tests on fault localization, mutation testing, and automated program repair should be repeated with co-occurring clusters; the paper argues that ignoring them misrepresents the real effect.","Previous root-cause taxonomies that rank concurrency and asynchronous issues highest for individual flaky tests may be skewed, because the dominant cluster-level causes are networking and external-dependency instability.","Static analysis alone, using test names, code tokens, and package/class hierarchy, can predict a substantial share of the cluster structure, so teams can triage systemic flakiness without 10,000 reruns.","Because clusters span about 2.9 test classes on average, developers should treat test-class decoupling and isolation from environmental variability as protective measures."],"supporting_citations":[{"why":"Supplies the dataset of 10,000 test-suite runs per project and the 810 flaky tests that the clustering analysis is built on.","marker":"[8]"},{"why":"Provides the industrial cost estimate, 1.28% of developer time and $2,250 per month, that motivates repairing multiple flaky tests at once.","marker":"[36]"},{"why":"Represents the developer-perspective cause study whose isolated-test framing the paper challenges and re-examines.","marker":"[16]"},{"why":"Simulated flaky-test impact on software engineering techniques without co-occurring failures; the paper argues this misrepresents real flakiness.","marker":"[14]"},{"why":"Simulation of flaky-test effect on fault localization that likewise ignores clustered co-failures.","marker":"[55]"},{"why":"Provides the static string-distance measures that the prediction models use as features.","marker":"[17]"},{"why":"Survey of diversity-based testing metrics that motivates applying static test-case distance measures to flakiness prediction.","marker":"[18]"},{"why":"Silhouette score method used to choose the clustering distance threshold.","marker":"[50]"}],"fun_headline_variants":["Flaky test failures cluster: 75% share root causes","Fix shared root causes to fix 13.5 flaky tests at once","Flaky failures aren't independent: 75% cluster with shared causes","ML predicts flaky-test clusters without 10,000 reruns"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that two flaky tests failing in the same test-suite run share one repairable root cause; the paper itself acknowledges that random chance, a common external outage, or an artifact of the run environment could also produce co-occurrence.","fun_headline_variants_meta":{"raw":{"variants":["Flaky test failures cluster: 75% share root causes","Fix shared root causes to fix 13.5 flaky tests at once","Flaky failures aren't independent: 75% cluster with shared causes","ML predicts flaky-test clusters without 10,000 reruns"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001306,"raw_usage":{"total_tokens":5421,"prompt_tokens":1139,"completion_tokens":4282,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":755,"completion_tokens_details":{"reasoning_tokens":4205}},"tokens_in":755,"tokens_out":4282,"duration_ms":27968,"temperature":1.0,"reasoning_tokens":4205,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:55:22.179132+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For each of the 45 clusters, count the member tests whose failure stack traces end in the same root exception; if most clusters lack a single shared root exception, the claim that co-occurrence implies a shared fixable cause fails. A stronger test would stub out the suspected network path or external dependency for a cluster and check whether all member tests stop failing together.","supporting_citations":[{"cited_title":"Alshammari, C","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of 10,000 test-suite runs per project and the 810 flaky tests that the clustering analysis is built on."},{"cited_title":"Leinen, D","cited_arxiv_id":null,"evidence_quote":"Provides the industrial cost estimate, 1.28% of developer time and $2,250 per month, that motivates repairing multiple flaky tests at once."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Represents the developer-perspective cause study whose isolated-test framing the paper challenges and re-examines."},{"cited_title":"Cordy, R","cited_arxiv_id":null,"evidence_quote":"Simulated flaky-test impact on software engineering techniques without co-occurring failures; the paper argues this misrepresents real flakiness."},{"cited_title":"Vancsics, T","cited_arxiv_id":null,"evidence_quote":"Simulation of flaky-test effect on fault localization that likewise ignores clustered co-failures."},{"cited_title":"Elgendy, R","cited_arxiv_id":null,"evidence_quote":"Provides the static string-distance measures that the prediction models use as features."},{"cited_title":"Elgendy, R","cited_arxiv_id":null,"evidence_quote":"Survey of diversity-based testing metrics that motivates applying static test-case distance measures to flakiness prediction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Silhouette score method used to choose the clustering distance threshold."}],"review_version":1}