Pith. sign in

REVIEW 2 major objections 6 minor 56 references

Code-based flaky-test detectors collapse to majority baselines once labeling shortcuts and project overlap are removed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-13 03:42 UTC pith:SKK2DPKJ

load-bearing objection Controlled counterfactual shows published code-based flaky-test F1 scores rest on labeling shortcuts and CV protocol, not a generalizable signal in test code; collapse to baseline is real and well-isolated. the 2 major comments →

arxiv 2607.09345 v1 pith:SKK2DPKJ submitted 2026-07-10 cs.SE

How Far Are We from Detecting Flaky Tests? On the Limits of Code-Based Detection

\"Omer Oktay G\"ultekin , Alexander Berndt , Jonathan Bell , Thomas Bach , Sebastian Baltes This is my paper
classification cs.SE
keywords flaky testscode-based detectionbenchmark shortcutsproject-disjoint evaluationCI log miningfailure-level classificationCodeBERT
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that asking whether a test is flaky from its source code alone is often the wrong problem: flakiness typically arises at runtime from interaction with the system and environment, not from static syntax. Reproducing three published code-model detectors, the authors show that high F1 scores on the popular IDoFT benchmark rest on two artifacts—a fix-commit labeling shortcut that pairs nearly identical flaky and fixed versions, and evaluation protocols that leak information across folds or projects. They rebuild only the non-flaky class of that benchmark from 500 repeated executions (C-IDoFT) while keeping developer-confirmed flaky tests; under project-disjoint evaluation the same models fall to constant majority baselines. On a curated subset of flakiness types that are visible in code they still succeed, and a CI-mined set of 86 end-to-end flaky tests shows that test code plus logs explain only 42% of causes. The practical upshot is a reframing: classify whether an observed failure is flaky, or estimate how likely a test is to fail under its actual execution environment.

Core claim

Once the fix-commit labeling shortcut is removed and projects are held out, CodeBERT reimplementations of published code-based detectors do not meaningfully exceed constant majority baselines on developer-confirmed flaky tests with rerun-confirmed non-flaky labels; the previously reported high F1 scores rested on that labeling shortcut and the cross-validation protocol, not on a generalizable signal in the test code.

What carries the argument

C-IDoFT, a controlled counterfactual that keeps developer-confirmed IDoFT flaky tests and rebuilds only the non-flaky class from 500 module-level reruns, evaluated under project-disjoint folds against majority-class baselines.

Load-bearing premise

That 500 repeated runs on a single CI configuration are enough to label a test non-flaky for the counterfactual; rare environment-dependent failures can still be mislabeled.

What would settle it

A project-disjoint evaluation on a larger set of developer-confirmed flaky tests whose non-flaky counterparts survive deep multi-environment reruns in which a code-only model clearly and stably exceeds the always-majority baseline on flaky-class F1.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Detector papers should report flaky-class precision/recall/F1 under project-disjoint splits and against majority baselines; matching the baseline is not detection.
  • Non-flaky labels should come from repeated execution or CI history, not from later fixed versions of the same tests.
  • Benchmarks restricted to code-visible flakiness types should be reported as such, not as general flaky-test detection.
  • Progress is more likely by classifying individual observed failures from logs and runtime evidence, or by estimating environment-conditioned failure probability.
  • Static code models remain useful only for the narrow subset of flakiness that is lexically obvious in the test method.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same critique—task ill-posedness plus labeling shortcuts—likely applies to other software-engineering ML tasks that pair near-duplicate fixed/unfixed code fragments.
  • Industrial CI systems already hold the decisive signal (pass/fail histories and logs); the bottleneck is labeling genuine versus flaky failures at scale rather than inventing better static models.
  • Language-model agents that actively perturb the environment (latency, clock, resource pressure) to test a hypothesized cause would close the loop that pure static detection lacks.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper argues that code-based flaky-test detection is often ill-posed because flakiness is not a static property of test source code. It reproduces three published LLM/CodeBERT detectors (Flakify, FlakyQ, FlakyXBert), shows that reported high F1 scores on IDoFT rest on data leakage, fix-commit non-flaky labeling, and non-project-disjoint evaluation, and constructs C-IDoFT as a controlled counterfactual that keeps developer-confirmed IDoFT flaky tests while rebuilding non-flaky labels from 500 reruns. Under project-disjoint evaluation on C-IDoFT, CodeBERT reimplementations collapse to constant majority baselines (flaky-class F1 0.035/0.070 vs always-flaky 0.054), while the same models recover nearly all flaky tests on FlakeBench. A second corpus, FlakeCI (86 same-commit flaky E2E tests), finds that test code plus CI logs attribute a cause for only 42% of cases. The authors reframe the task toward failure-level classification and environment-conditioned failure probability, and release datasets and a replication package.

Significance. If the results hold, the paper substantially revises how the SE community should evaluate and pursue flaky-test prediction: much of the recent LLM-based progress on standard benchmarks is an artifact of labeling shortcuts and evaluation protocol rather than a generalizable code signal. Strengths include a carefully designed counterfactual (C-IDoFT holds the flaky population fixed while changing only non-flaky construction), an explicit counterfactual square separating labeling from protocol (Table VII), project-disjoint evaluation with constant baselines, reproduction of three published detectors including leakage fixes, and a released replication package with curation pipelines. The FlakeBench contrast and FlakeCI diagnosis study usefully bound when code is and is not informative. This is a high-value critical empirical contribution for flaky-test research and for SE-ML evaluation practice more broadly.

major comments (2)
  1. [§IV-B, Table V] Table V and §IV-B: C-IDoFT project-disjoint results rest on only four folds, and the flaky-class F1 standard deviations (0.035±0.058; 0.070±0.078) are comparable to or larger than the means. The direction of collapse relative to the always-flaky baseline (0.054) and the CV-vs-Disjoint contrast on identical data are still clear, but the paper should either increase the number of project-disjoint folds (or use leave-one-project-out where feasible) or state the claim more carefully as a consistent near-baseline collapse rather than a precisely estimated F1. This is the main quantitative support for the central negative result and deserves tighter uncertainty reporting.
  2. [§IV-C, Table VI] §IV-C and Table VI: The 42%/58% attribution split on FlakeCI is used to support the reframe away from static code. Root-cause labels were assigned by a single author over 86 tests, with category concentration (15 of 23 Network cases from two projects). The threats section correctly treats 42% as a lower bound, but if this split remains a headline result, the manuscript should add at least a second independent coder on a substantial subset and report agreement (or restrict the claim more tightly to “in our manual analysis, evidence was insufficient without speculation”). This is secondary to the C-IDoFT collapse but load-bearing for the diagnosis/reframe half of the contribution.
minor comments (6)
  1. [§II-B, Footnote 1, Table I] Footnote 1 and Table I: The IDoFT label-count discrepancies (3813 vs 3862; 3195/618 vs 3226/587) are carefully documented but still easy to miss. A short explicit “label inventory” table in the appendix would help readers reconcile paper, package, and your filtered subsets.
  2. [§III] §III RQ1 metrics: The always-flaky flaky-class F1 formula F1_f = 2p/(1+p) is useful; stating it once in a numbered equation would make later baseline comparisons easier to audit.
  3. [§V, Table VII] Table VII is excellent for the counterfactual square; consider also reporting flaky-class F1 (not only weighted F1) in every cell so readers do not have to cross-reference Tables II–V.
  4. [§IV-A, Table III] §IV-A FlakyXBert: The four notebooks shipping another project’s outputs are a serious artifact issue; the corrected 0.83 figure is clear, but a one-line note on whether authors updated the public package after notification would help reproducibility.
  5. [§IV-C] Collection windows “between January and April 2026” (§IV-C) will look odd relative to a 2026 arXiv stamp depending on final publication date; confirm consistency of dates in the camera-ready text.
  6. [§IV-B] Minor prose: “C-IDoFT is a controlled counterfactual, not a benchmark for reuse” is stated well in the abstract and §II-D; repeating that caveat once more near Table V would reduce the risk of others adopting it as a leaderboard set.

Circularity Check

0 steps flagged

No significant circularity: empirical reproduction and controlled counterfactual, not a derivation that re-uses fitted quantities as predictions.

full rationale

The paper is a critical empirical study of code-based flaky-test detectors. Its central claim (that published high F1 scores collapse to constant majority baselines once the fix-commit labeling shortcut is removed and projects are held out) is established by re-running published pipelines, constructing C-IDoFT as an explicit control that keeps the same developer-confirmed flaky tests while rebuilding only the non-flaky class from 500 reruns, and evaluating under project-disjoint folds against always-flaky / always-non-flaky baselines (Tables II–V, VII). Performance numbers are measured outputs, not inputs renamed as predictions. Self-citations to the authors’ prior industrial and LLM studies appear as background or related work and are not load-bearing for the collapse result; that result is produced by the new experiments. FlakeCI is a separately mined diagnostic corpus, not a fitted parameter. No self-definitional loop, fitted-input-as-prediction, uniqueness theorem imported from the authors, or ansatz smuggled via citation is present. The derivation chain is therefore self-contained against external benchmarks and ordinary scientific practice.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The paper is empirical; its load-bearing premises are methodological choices about labeling oracles, evaluation splits, and what counts as a diagnosable cause, not free physical parameters or new particles. The free parameters that do appear are standard ML hyper-parameters and the 500-run budget. Invented entities are the two curated datasets, which are operational constructs with public artifacts rather than theoretical posits.

free parameters (3)
  • rerun budget n=500 = 500
    Chosen to balance cost and statistical power (1% failure probability yields 99.3% detection chance). Directly determines which tests are labeled non-flaky in C-IDoFT.
  • CodeBERT fine-tuning hyper-parameters = lr=1e-5, bs=4, epochs<=20, seed=42
    Learning rate 1e-5, batch size 4, max 20 epochs, early-stopping patience 5, seed 42, 1024-token cap. Standard but affect absolute F1 numbers.
  • project-disjoint fold count = 4
    Four folds used for C-IDoFT / FlakeBench project-disjoint evaluation; small enough that fold variance is large relative to the tiny flaky-class F1.
axioms (5)
  • domain assumption A test that never fails in 500 re-executions on one CI configuration may be treated as non-flaky for the counterfactual.
    Section III RQ2 construction; acknowledged threat that rarer or environment-specific flakiness is missed.
  • domain assumption Developer-confirmed IDoFT statuses (Accepted / DeveloperFixed / InspiredAFix) are high-precision positive flaky labels.
    Used to restrict C-IDoFT's flaky class; if many of these are not truly flaky the collapse result would be misinterpreted.
  • domain assumption Project-disjoint evaluation is the appropriate proxy for practical cross-project generalization of code-based detectors.
    Section III and threats; within-project temporal drift is noted as a possible alternative but not measured.
  • domain assumption Same-workflow-and-job pass/fail pairs on the same commit indicate non-determinism rather than deliberate configuration differences.
    FlakeCI construction (Section III RQ3); residual risk of intentional config variation is acknowledged.
  • standard math Standard classification metrics (flaky-class F1, weighted F1) and constant majority baselines are the right yardsticks for detection ability.
    Used throughout RQ1/RQ2; the paper correctly prefers flaky-class metrics on imbalanced data.
invented entities (2)
  • C-IDoFT dataset independent evidence
    purpose: Controlled counterfactual that keeps developer-confirmed IDoFT flaky tests and rebuilds non-flaky labels from 500 re-runs, removing the fix-commit shortcut.
    Operational construct, not a theoretical posit; released as a control rather than a recommended reuse benchmark.
  • FlakeCI dataset independent evidence
    purpose: Corpus of 86 same-commit flaky E2E failures mined from GitHub Actions Cypress/Playwright logs, with manual cause categories.
    New empirical resource supporting the claim that most causes need execution evidence beyond code+log.

pith-pipeline@v1.1.0-grok45 · 28999 in / 3507 out tokens · 29631 ms · 2026-07-13T03:42:59.360768+00:00 · methodology

0 comments
read the original abstract

Flaky tests pass and fail on the same code version, weakening the signal of test results and disrupting continuous integration (CI) pipelines. Code-based flakiness detectors report strong benchmark results, yet their use in practice remains limited. We argue that the field is studying the wrong problem: Flakiness is not a static property of test code, which often lacks the information needed to decide whether a test is flaky. Analyzing three code-based detectors operating on test code, we found that widely used benchmarks contain shortcuts that inflate reported F1 scores and that evaluation protocols overstate generalizability. To control for these shortcuts, we curated two datasets. The first, C-IDoFT (54,468 unit tests from 57 GitHub projects), keeps a developer-confirmed subset of IDoFT's flaky tests and rebuilds only the non-flaky class from repeated executions instead of fixed versions of flaky tests. C-IDoFT is a controlled counterfactual, not a benchmark for reuse. Our CodeBERT reimplementations of two published detectors scored far above its constant baselines under the published cross-validation protocol but no better than them once projects were separated. The high scores rested on the labeling shortcut and the evaluation protocol, not on the test code. On FlakeBench, a benchmark restricted to flakiness types typically recognizable from test code, and the same project-disjoint protocol, the models identified nearly all flaky tests. The second dataset, mined from CI logs, contains 86 flaky end-to-end tests that passed and failed on the same commit. The test code and CI log yielded a cause for 42% of them; the other 58% required further execution evidence. Rather than abandoning flakiness prediction, we reframe it around whether an observed failure is flaky and how likely a test is to fail given its execution environment. Our datasets and CI-mining method support this direction.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 1 linked inside Pith

  1. [1]

    Cost of flaky tests in con- tinuous integration: An industrial case study,

    F. Leinen, D. Elsner, A. Pretschner, A. Stahlbauer, M. Sailer, and E. J ¨urgens, “Cost of flaky tests in con- tinuous integration: An industrial case study,” inProc. ICST. IEEE, 2024, pp. 329–340

    2024

  2. [2]

    Understanding flaky tests: the developer’s perspective,

    M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli, “Understanding flaky tests: the developer’s perspective,” inProc. ESEC/FSE. ACM, 2019, pp. 830–840

    2019

  3. [3]

    Presubmit rescue: Auto- matically ignoring flakytest executions,

    M. Hoang and A. Berding, “Presubmit rescue: Auto- matically ignoring flakytest executions,” inProc. FTW. ACM, 2024, pp. 1–2

    2024

  4. [4]

    Software testing research challenges: An industrial perspective,

    N. Alshahwan, M. Harman, and A. Marginean, “Software testing research challenges: An industrial perspective,” in Proc. ICST. IEEE, 2023, pp. 1–10

    2023

  5. [5]

    Taming google- scale continuous testing,

    A. M. Memon, Z. Gao, B. N. Nguyen, S. Dhanda, E. Nickell, R. Siemborski, and J. Micco, “Taming google- scale continuous testing,” inProc. ICSE-SEIP. IEEE Computer Society, 2017, pp. 233–242

    2017

  6. [6]

    Do test and environ- mental complexity increase flakiness? an empirical study of SAP HANA,

    A. Berndt, T. Bach, and S. Baltes, “Do test and environ- mental complexity increase flakiness? an empirical study of SAP HANA,” inProc. ESEM. ACM, 2024, pp. 572– 581

    2024

  7. [7]

    Modeling and ranking flaky tests at apple,

    E. Kowalczyk, K. Nair, Z. Gao, L. Silberstein, T. Long, and A. M. Memon, “Modeling and ranking flaky tests at apple,” inProc. ICSE-SEIP. ACM, 2020, pp. 110–119

    2020

  8. [8]

    Empirically detecting false test alarms using association rules,

    K. Herzig and N. Nagappan, “Empirically detecting false test alarms using association rules,” inProc. ICSE. IEEE Computer Society, 2015, pp. 39–48

    2015

  9. [9]

    Toward static test flakiness prediction: a feasibility study,

    V . Pontillo, F. Palomba, and F. Ferrucci, “Toward static test flakiness prediction: a feasibility study,” inProc. MaLTeSQuE@ESEC/FSE. ACM, 2021, pp. 19–24

    2021

  10. [10]

    Deepflaky: Deep hybrid representation learning for flaky test prediction,

    J. Cai, Y . Lei, Z. Li, C. Liu, H. Xie, and Z. Wu, “Deepflaky: Deep hybrid representation learning for flaky test prediction,”Inf. Softw. Technol., vol. 193, p. 108070, 2026

    2026

  11. [11]

    What is the vocabulary of flaky tests?

    G. Pinto, B. Miranda, S. Dissanayake, M. d’Amorim, C. Treude, and A. Bertolino, “What is the vocabulary of flaky tests?” inProc. MSR. Seoul, Republic of Korea: ACM, 2020, pp. 492–502

    2020

  12. [12]

    A replication study on the usability of code vocabulary in predicting flaky tests,

    G. Haben, S. Habchi, M. Papadakis, M. Cordy, and Y . L. Traon, “A replication study on the usability of code vocabulary in predicting flaky tests,” inProc. MSR. Madrid, Spain: IEEE, 2021, pp. 219–229

    2021

  13. [13]

    The vocabulary of flaky tests in the context of SAP HANA,

    A. Berndt, Z. Nochta, and T. Bach, “The vocabulary of flaky tests in the context of SAP HANA,” inProc. ESEM. IEEE, 2023, pp. 1–9

    2023

  14. [14]

    A preliminary study on the vocabulary of flaky tests in swift,

    J. Medeiros and B. Miranda, “A preliminary study on the vocabulary of flaky tests in swift,” inProc. FTW. ACM, 2026, pp. 11–12

    2026

  15. [15]

    Flakeflagger: Predicting flakiness without rerunning tests,

    A. Alshammari, C. Morris, M. Hilton, and J. Bell, “Flakeflagger: Predicting flakiness without rerunning tests,” inProc. ICSE. Madrid, Spain: IEEE, 2021, pp. 1572–1584

    2021

  16. [16]

    Flakify: A black-box, language model-based predictor for flaky tests,

    S. Fatima, T. A. Ghaleb, and L. C. Briand, “Flakify: A black-box, language model-based predictor for flaky tests,”IEEE Trans. Software Eng., vol. 49, no. 4, pp. 1912–1927, 2023

    1912

  17. [17]

    Understanding and improving flaky test classification,

    S. Rahman, S. Dutta, and A. Shi, “Understanding and improving flaky test classification,”Proc. ACM Program. Lang., vol. 9, no. OOPSLA2, pp. 1345–1371, 2025

    2025

  18. [18]

    An analysis of LLM fine- tuning and few-shot learning for flaky test detection and classification,

    R. More and J. S. Bradbury, “An analysis of LLM fine- tuning and few-shot learning for flaky test detection and classification,” inProc. ICST. IEEE, 2025, pp. 349–359

    2025

  19. [19]

    Can we classify flaky tests using only test code? an LLM-based empirical study,

    A. Berndt, V . Bekmyradov, R. Gemulla, M. Kessel, T. Bach, and S. Baltes, “Can we classify flaky tests using only test code? an LLM-based empirical study,” inProc. SANER, RENE Track. Limassol, Cyprus: IEEE, 2026

    2026

  20. [20]

    The importance of accounting for execution failures when predicting test flakiness,

    G. Haben, S. Habchi, J. Micco, M. Harman, M. Pa- padakis, M. Cordy, and Y . L. Traon, “The importance of accounting for execution failures when predicting test flakiness,” inProc. ASE. ACM, 2024, pp. 1979–1989

    2024

  21. [21]

    Quan- tizing large-language models for predicting flaky tests,

    S. Rahman, A. Baz, S. Misailovic, and A. Shi, “Quan- tizing large-language models for predicting flaky tests,” inProc. ICST. Toronto, ON, Canada: IEEE, 2024, pp. 93–104

    2024

  22. [22]

    How Far Are We from Detecting Flaky Tests? On the Limits of Code-Based Detection (Research Arti- fact),

    ¨O. O. G¨ultekin, A. Berndt, J. Bell, T. Bach, and S. Baltes, “How Far Are We from Detecting Flaky Tests? On the Limits of Code-Based Detection (Research Arti- fact),” Zenodo, Jul. 2026, https://doi.org/10.5281/zenodo. 21102431

    doi:10.5281/zenodo 2026

  23. [23]

    A survey of flaky tests,

    O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, “A survey of flaky tests,”ACM Trans. Softw. Eng. Methodol., vol. 31, no. 1, pp. 17:1–17:74, 2022

    2022

  24. [24]

    Know you neighbor: Fast static prediction of test flaki- ness,

    R. Verdecchia, E. Cruciani, B. Miranda, and A. Bertolino, “Know you neighbor: Fast static prediction of test flaki- ness,”IEEE Access, vol. 9, pp. 76 119–76 134, 2021

    2021

  25. [25]

    What is the vocabulary of flaky tests? an extended replication,

    B. H. P. Camara, M. A. G. Silva, A. T. Endo, and S. R. Vergilio, “What is the vocabulary of flaky tests? an extended replication,” inProc. ICPC. IEEE, 2021, pp. 444–454

    2021

  26. [26]

    Flakyfix: Using large language models for predicting flaky test fix categories and test code repair,

    S. Fatima, H. Hemmati, and L. C. Briand, “Flakyfix: Using large language models for predicting flaky test fix categories and test code repair,”IEEE Trans. Software Eng., vol. 50, no. 12, pp. 3146–3171, 2024

    2024

  27. [27]

    Deflaker: automatically detecting flaky tests,

    J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “Deflaker: automatically detecting flaky tests,” inProc. ICSE. ACM, 2018, pp. 433–444

    2018

  28. [28]

    International Dataset of Flaky Tests (IDoFT),

    W. Lam, “International Dataset of Flaky Tests (IDoFT),” 2020, http://mir.cs.illinois.edu/flakytests

    2020

  29. [29]

    iDFlakies: A framework for detecting and partially classifying flaky tests,

    W. Lam, R. Oei, A. Shi, D. Marinov, and T. Xie, “iDFlakies: A framework for detecting and partially classifying flaky tests,” inICST 2019: 12th IEEE Interna- tional Conference on Software Testing, Verification and Validation, Xi’an, China, April 2019, pp. 312–322

    2019

  30. [30]

    FlakyCat: Predicting flaky tests categories using few-shot learning,

    A. Akli, G. Haben, S. Habchi, M. Papadakis, and Y . L. Traon, “FlakyCat: Predicting flaky tests categories using few-shot learning,” inProc. AST. IEEE, 2023, pp. 140– 151

    2023

  31. [31]

    An empirical analysis of flaky tests,

    Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” inProc. FSE. ACM, 2014, pp. 643–653

    2014

  32. [32]

    Code- bert: A pre-trained model for programming and natural languages,

    Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Code- bert: A pre-trained model for programming and natural languages,” inFindings of the Association for Compu- tational Linguistics: EMNLP 2020, Online Event, 16- 20 November 2020, ser. Findings of ACL, vol. EMNLP

    2020

  33. [33]

    1536–1547

    Association for Computational Linguistics, 2020, pp. 1536–1547

    2020

  34. [34]

    Artifact Review and Badging — Current,

    Association for Computing Machinery, “Artifact Review and Badging — Current,” https://www.acm.org/publications/policies/ artifact-review-and-badging-current, 2020, accessed: May 2026

    2020

  35. [35]

    Do automatic test generation tools generate flaky tests?

    M. Gruber, M. F. Roslan, O. Parry, F. Scharnb ¨ock, P. McMinn, and G. Fraser, “Do automatic test generation tools generate flaky tests?” inProc. ICSE. ACM, 2024, pp. 47:1–47:12

    2024

  36. [36]

    On the flakiness of llm-generated tests for industrial and open-source database management systems,

    A. Berndt, T. Bach, R. Gemulla, M. Kessel, and S. Baltes, “On the flakiness of llm-generated tests for industrial and open-source database management systems,”CoRR, vol. abs/2601.08998, 2026

    arXiv 2026

  37. [37]

    Research on test flakiness: from unit to system testing,

    K. Ngo, V . Nguyen, and T. N. Nguyen, “Research on test flakiness: from unit to system testing,” inProc. ASE. ACM, 2022, pp. 218:1–218:4

    2022

  38. [38]

    Taming timeout flakiness: An empirical study of SAP HANA,

    A. Berndt, S. Baltes, and T. Bach, “Taming timeout flakiness: An empirical study of SAP HANA,” inProc. ICSE-SEIP. ACM, 2024, pp. 69–80

    2024

  39. [39]

    An empirical study of flaky tests in javascript,

    N. Hashemi, A. Tahir, and S. Rasheed, “An empirical study of flaky tests in javascript,” inProc. ICSME. IEEE, 2022, pp. 24–34

    2022

  40. [40]

    An empirical analysis of ui-based flaky tests,

    A. Romano, Z. Song, S. Grandhi, W. Yang, and W. Wang, “An empirical analysis of ui-based flaky tests,” inProc. ICSE. IEEE, 2021, pp. 1585–1597

    2021

  41. [41]

    Empirical study of restarted and flaky builds on travis CI,

    T. Durieux, C. Le Goues, M. Hilton, and R. Abreu, “Empirical study of restarted and flaky builds on travis CI,” inProc. MSR. ACM, 2020, pp. 254–264

    2020

  42. [42]

    Silent data corruptions at scale,

    H. D. Dixit, S. Pendharkar, M. Beadon, C. Mason, T. Chakravarthy, B. Muthiah, and S. Sankar, “Silent data corruptions at scale,”CoRR, vol. abs/2102.11245, 2021

    Pith/arXiv arXiv 2021

  43. [43]

    The relationship between precision-recall and ROC curves,

    J. Davis and M. Goadrich, “The relationship between precision-recall and ROC curves,” inProc. ICML. Pitts- burgh, PA, USA: ACM, 2006, pp. 233–240

    2006

  44. [44]

    A container- based infrastructure for fuzzy-driven root causing of flaky tests,

    V . Terragni, P. Salza, and F. Ferrucci, “A container- based infrastructure for fuzzy-driven root causing of flaky tests,” inProc. ICSE-NIER. ACM, 2020, pp. 69–72

    2020

  45. [45]

    Research Design and Issues of Validity,

    M. B. Brewer and W. D. Crano, “Research Design and Issues of Validity,”Handbook of research methods in social and personality psychology, pp. 11–26, 2014

    2014

  46. [46]

    The effects of computational resources on flaky tests,

    D. Silva, M. Gruber, S. Gokhale, E. Arteca, A. Turcotte, M. d’Amorim, W. Lam, S. Winter, and J. Bell, “The effects of computational resources on flaky tests,”IEEE Trans. Software Eng., vol. 50, no. 12, pp. 3104–3121, 2024

    2024

  47. [47]

    Sampling in software engineer- ing research: a critical review and guidelines,

    S. Baltes and P. Ralph, “Sampling in software engineer- ing research: a critical review and guidelines,”Empir. Softw. Eng., vol. 27, no. 4, p. 94, 2022

    2022

  48. [48]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. C. Ohlsson, B. Reg- nell, and A. Wessl ´en,Experimentation in Software Engi- neering, 2nd ed. Berlin, Heidelberg: Springer, 2024

    2024

  49. [49]

    Construct validity in software engineering research and software metrics,

    P. Ralph and E. D. Tempero, “Construct validity in software engineering research and software metrics,” in Proc. EASE. ACM, 2018, pp. 13–23

    2018

  50. [50]

    Deep learning based vulnerability detection: Are we there yet?

    S. Chakraborty, R. Krishna, Y . Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet?” IEEE Trans. Software Eng., vol. 48, no. 9, pp. 3280– 3296, 2022

    2022

  51. [51]

    Root causing flaky tests in a large-scale industrial setting,

    W. Lam, P. Godefroid, S. Nath, A. Santhiar, and S. Thum- malapenta, “Root causing flaky tests in a large-scale industrial setting,” inProc. ISSTA. ACM, 2019, pp. 101–111

    2019

  52. [52]

    A survey on how test flakiness affects developers and what support they need to address it,

    M. Gruber and G. Fraser, “A survey on how test flakiness affects developers and what support they need to address it,” inProc. ICST. IEEE, 2022, pp. 82–92

    2022

  53. [53]

    230,439 test failures later: An empirical evaluation of flaky failure classifiers,

    A. Alshammari, P. Ammann, M. Hilton, and J. Bell, “230,439 test failures later: An empirical evaluation of flaky failure classifiers,” inProc. ICST. Toronto, ON, Canada: IEEE, 2024, pp. 257–268

    2024

  54. [54]

    Just-in- time flaky test detection via abstracted failure symptom matching,

    G. An, J. Yoon, T. Bach, J. Hong, and S. Yoo, “Just-in- time flaky test detection via abstracted failure symptom matching,” inProc. ICSME. IEEE, 2024, pp. 741–752

    2024

  55. [55]

    When life gives you oranges: detecting and diagnosing intermittent job failures at mozilla,

    J. Lampel, S. Just, S. Apel, and A. Zeller, “When life gives you oranges: detecting and diagnosing intermittent job failures at mozilla,” inProc. ESEC/FSE. ACM, 2021, pp. 1381–1392

    2021

  56. [56]

    On the diagnosis of flaky job failures: Understanding and prior- itizing failure categories,

    H. A ¨ıdasso, F. Bordeleau, and A. Tizghadam, “On the diagnosis of flaky job failures: Understanding and prior- itizing failure categories,” inProc. ICSE-SEIP. IEEE, 2025, pp. 192–202

    2025