Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript

Amjed Tahir; August Shi; Negar Hashemi; Rachel Blagojevic; Shawn Rasheed

REVIEW 4 major objections 3 minor 65 references

Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript

T0 review · 4 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Systematic reordering in 81 Jest projects finds 55 order-dependent flaky tests, most caused by shared mocking state.

desk verdict First systematic Jest order-dependence study with a plausible new cause category, but the missing transformed-original-order control and a likely hoisting artifact in the headline example undermine the shared-mocking-state count. read the letter →

arxiv 2501.12680 v1 pith:Q4EEXFM7 submitted 2025-01-22 cs.SE

Negar Hashemi , Amjed Tahir , Shawn Rasheed , August Shi , Rachel Blagojevic This is my paper · ORCID

classification cs.SE

keywords flakytestsorder-dependentJestJavaScripttestingsharedmockingstatefilestestreorderingpollution

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Flaky tests are tests whose outcomes vary between runs, and one known trigger is the order in which tests execute. This paper asks whether JavaScript projects using Jest, the most widely used JavaScript testing framework, contain hidden order-dependent tests and what causes them. To find out, the authors built a tool that reorders tests, describe blocks, and whole test suites into random orders and reruns each order, then manually analysed every failure. Across 81 projects they found 55 order-dependent tests: 52 between individual tests, 3 between describe blocks, and none between test suites. The causes were shared files (13) and shared mocking state (42), with mocking-state persistence a cause that earlier flaky-test studies in Java and Python had not reported.

What carries the argument

The load-bearing mechanism is JS-TOD, the paper's detector, built on four pieces: Jest's -listTests option to enumerate suites, a custom TestSequencer subclass that enforces a supplied suite order, Babel's abstract syntax tree to lift individual tests and describe blocks out of a suite into new files, and Algorithm 1, which generates up to 10 unique random orders per level (or all permutations if fewer exist). Each reordered file is rerun 10 times; a failure that recurs in every rerun of the reordered file, while passing in the default order, is classified as order-dependent. Manual inspection then classifies each failure as stemming from shared files or shared mocking state.

What would settle it

Rerun the reported 55 tests in both their failing and passing orders with the transformation removed, using Jest's own randomize option on the original files; if a reported test fails in both orders, or passes in the order that supposedly fails, its order-dependence classification is wrong. Alternatively, add a global beforeEach with jest.clearAllMocks() to jest-webextension-mock and check whether all 35 reported failures disappear.

Watch

Extended reading notes

Core claim

The paper's central claim is that Jest projects do contain real order-dependent flakiness, and that the dominant mechanism is shared mocking state rather than the shared-file or static-state mechanisms documented for Java and Python. A test is counted as order-dependent when it passes in the project's default order, fails in every rerun of a reordered file, and the failure disappears when the original order is restored. Of the 55 detected tests, 52 appear when the order of tests inside a suite changes, and 3 appear when the order of describe blocks changes; no order dependence was found between test suites in the 49 projects analysed at that level. Manual inspection attributes 13 tests to shared files and 42 to mocking state that persists across tests through Jest's module and mock registry, a category the authors say previous order-dependence studies did not report.

Load-bearing premise

The whole classification rests on the assumption that Babel's AST-based rewriting of a test suite into reordered files leaves the tests' behavior unchanged, so a new failure reflects order rather than transformation; the authors themselves report nine projects where Babel missed code and required manual repair.

Editorial extensions

If this is right

Developers using Jest can no longer assume that running tests in file order guarantees deterministic outcomes; hidden order dependence can surface whenever Jest reorders suites or a developer uses the randomize option.
Resetting shared state, for example jest.clearAllMocks() in a beforeEach hook, should fix a large share of order-dependent tests: the paper reports clearing mock state fixed 34 of the 39 test-level shared-mocking-state failures.
Order dependence can also hide between describe blocks, so tools and best practices should treat describe blocks as a reorderable unit, not just individual tests.
Because no order-dependent test was found at the test-suite level across 49 projects, the practical risk in Jest projects is concentrated inside suites rather than across suite files.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the paper's central claim holds, its count of 55 is a lower bound: with only 10 random orders per level, other order-dependent tests may require rarer orderings, and more aggressive or pairwise reordering would likely find more.
The new shared-mocking-state cause suggests a cheap preventive check: statically flag any test that calls jest.mock without a matching reset in beforeEach or afterEach, since such mocks persist across tests.
If shared mocking state is as prevalent in other JavaScript frameworks as it is in Jest, current flaky-test taxonomies, built mostly from Java and Python, may need a new category for framework-level mock registries.
The methodology could be extended to other Jest-like frameworks by replacing the AST extraction and sequencer hooks, but the describe-block level is Jest-specific and would not transfer directly.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things. First, this paper does real work: it actively reorders Jest tests, describe blocks, and suites across 81 popular GitHub projects, finds 55 order-dependent tests, and proposes a new cause category—shared mocking state—that the cited Java/Python literature does not report. Second, the methodology has a load-bearing gap: JS-TOD rewrites test files via Babel but never runs a transformed file in the original order as a control. Section V-B1 admits nine projects where the rewrite dropped code and had to be manually fixed. Without a control, some of the 55 reported failures could be transformation artifacts rather than order dependence. The new-cause claim also has a concrete problem. The native-testing-library example in Listing 6 says that moving a test containing a local jest.mock('react-test-renderer', ...) earlier in the file causes that mock to pollute other tests. But Jest hoists jest.mock calls to the top of the module scope, so that mock should already be active in the original order. The described mechanism is inconsistent with Jest semantics, and some of the 42 shared-mocking-state cases may be artifacts of physical source reordering. The other large cluster—35 call-count assertions in jest-webextension-mock—is much cleaner and almost certainly genuine order dependence. What the paper does well: the manual root-cause analysis is careful (two independent inspectors, a third for disagreements), all 55 tests failed consistently across reruns of the same order, and the tool and replication package are public. The three-level reordering—individual tests, describe blocks, and suites—is a real extension beyond iDFlakies and FlaPy, even if the core idea is ported. The authors are upfront about threats: only 10 reorders, no recall measurement, manual project selection. They also report, rather than hide, the transformation failures; that honesty is creditworthy. The missing control baseline is the main issue, and it directly affects the headline count. The 'not reported previously' claim lacks a systematic literature comparison. The prevalence numbers are lower bounds given the sampling and the all-passing-default-order filter. None of this refutes the existence of order-dependent flakiness in Jest—the paper has that—but the shared-mocking-state category needs re-verification with a proper original-order control of the transformed files, and the Listing 6 explanation should be reconciled with Jest's hoisting behavior. I would bring this to a reading group and likely cite the paper. A serious editor should send it to peer review: it needs revision, not rejection. The referee should push for the control experiment and a re-classification of the shared-mocking-state examples.

Referee Report

4 major / 3 minor

Summary. The paper presents JS-TOD, a tool that randomizes Jest tests at the level of individual tests, describe blocks, and test suites, and reruns the reordered files multiple times to detect order-dependent flaky tests. The authors evaluate the approach on 81 GitHub projects and report 55 order-dependent tests across 10 projects, attributing them to shared files (13 tests) and shared mocking state (42 tests). They claim that shared mocking state is a previously unreported cause of order-dependent flakiness. The study includes a manual root-cause analysis with independent author verification and a public replication package.

Significance. If the findings hold, the paper provides the first systematic evidence of order-dependent flakiness in Jest projects at scale, and its proposed new cause category (shared mocking state) would be a useful addition to the flaky-test taxonomy. The strengths are the careful two-author manual analysis with conflict resolution, the consistent failure of all 55 tests across reruns of the same order, and the availability of a replication package. The main weakness is the lack of a control for transformation equivalence, which is load-bearing for the central claim that the observed failures are caused by test order rather than by the AST rewriting itself.

major comments (4)

[Section III-B, Algorithm 2] The paper does not report a control in which the AST-transformed test file is run with the tests in their original order. Since the classification of a test as order-dependent depends on failures appearing only after reordering, and since the authors themselves report nine projects where the transformation dropped code (Section V-B1), the absence of this control leaves open the possibility that some of the 55 reported failures are transformation artifacts rather than order dependence.
[Section III-B, Algorithm 2] The code-generation step is underspecified: it is unclear whether top-level statements outside test/describe nodes (imports, requires, beforeEach/afterEach hooks, and top-level jest.mock calls) are preserved in the generated files. The native-testing-library example in Listing 6 includes a top-level jest.mock and beforeEach/afterEach; if those are not included in the new test files, the observed inter-test effects would not correspond to the original test suite's semantics. The paper should state exactly which AST nodes are copied into each generated file and demonstrate that the transformed original-order file is equivalent.
[Section V-B1 and abstract] The assertion that shared mocking state is a cause "not reported previously" is not supported by a comparison with prior classification taxonomies. In Java, for example, mocking frameworks maintain global static state, and prior taxonomies (e.g., Luo et al. [11]) subsume such state under shared static state. The paper should either differentiate shared mocking state from existing categories with concrete evidence from prior work or soften the novelty claim.
[Section V-A] The statement "All 55 tests fail across all 10 reruns of the same order, confirming that they indeed are order-dependent tests" overstates the evidence. Consistent failure across reruns of the same reordered file only confirms deterministic behavior of that transformed file; it does not by itself confirm order dependence without a comparison against the transformed original-order file. This wording should be revised to reflect the need for the control.

minor comments (3)

[Table III] The footnote says shaded rows are projects with failing tests but not order-dependent flaky tests; for clarity, consider explicitly distinguishing these non-OD failures from the OD rows in the table header or caption.
[Table II] The "Total" row sums the three reordering levels, but the levels overlap in projects; reporting unique project counts in addition to the sums would avoid conflation.
[References] Reference [39] contains a typo: "Ssoftware" should be "Software".

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical detection result grounded in external GitHub projects, with no fitted input or load-bearing self-citation.

full rationale

The paper's central claim is an empirical measurement: JS-TOD reorders test suites, tests, and describe blocks in 81 external GitHub projects, and the 55 reported failures were manually inspected and classified. No parameter is fitted to the outcome, no prediction is derived from a definition that already contains the result, and no uniqueness theorem or prior result by these authors is invoked to force the conclusion. The self-citation to Hashemi et al. [13] is background only, used to note that prior issue-tracker studies found few order-dependent tests in JavaScript; the claimed new cause (shared mocking state) is grounded in the present study's concrete examples (Listings 5, 6, and 9) and in two-author-plus-adjudicator manual classification, not in the cited paper. The transformation limitation is real and disclosed: Section V-B1 reports nine projects where failures occurred because Babel did not parse parts of the original test suites and required manual fixes, and the paper does not report a transformed original-order control to rule out transformation artifacts; this is a validity threat to specific classifications, not a circularity, because the detection rule 'a test that fails on every rerun of a reordered file is order-dependent' is not equivalent by construction to the cause categories, which are assigned by external code inspection. The potential mock-hoisting concern raised by the reviewer is likewise a correctness risk for the shared-mocking-state category, not a circular reduction. No step in the derivation chain reduces to its own inputs, so the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The ledger captures experimental design choices and domain assumptions that the central claim depends on. There are no fitted model parameters and no newly postulated physical or conceptual entities, but the representative sample, the AST preservation assumption, and the rerun-count assumption are load-bearing for the reported prevalence numbers.

free parameters (2)

number_of_reorders = 10
Chosen by hand as an experimental design constant. The paper claims in the threats-to-validity section that increasing to 20 or 30 reruns did not change results, but no supporting data are shown.
number_of_reruns = 10
Chosen by hand as an experimental design constant. The paper justifies this by pilot runs and experience, but the supporting comparison is not presented.

assumptions (4)

domain assumption Projects selected from the 700 most-starred GitHub repositories using Jest are representative of JavaScript test suites.
Section IV-A builds the dataset from the 700 most-starred repositories, then filters to projects with pass-by-default tests. This affects prevalence estimates and may exclude projects with existing failures.
domain assumption A test that fails in all 10 reruns of one reordered run, but passes in the default order, is order-dependent.
Section III defines this detection criterion. It assumes no non-deterministic flakiness or infrastructure issue is responsible, though the authors later attribute some failures to test code structure and resource issues.
domain assumption Babel AST extraction and code generation preserve the semantics of the original test files.
Section III-B replaces test suites with rewritten files. Section V-B1 documents parsing misses and manual fixes, so this assumption is load-bearing and partially violated.
domain assumption Ten reruns are sufficient to distinguish order-dependence from other forms of non-deterministic flakiness.
The threats-to-validity section claims equivalence with 20 or 30 reruns, but no data or statistical argument is provided to support this claim.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript." pith.science (2026). https://pith.science/paper/Q4EEXFM7

@misc{pith2026250112680,
  author       = {Pith},
  title        = {Pith review of: Detecting and Evaluating Order-Dependent Flaky Tests in JavaScript},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q4EEXFM7}},
  note         = {Machine review of arXiv:2501.12680}
}

read the original abstract

Flaky tests pose a significant issue for software testing. A test with a non-deterministic outcome may undermine the reliability of the testing process, making tests untrustworthy. Previous research has identified test order dependency as one of the most prevalent causes of flakiness, particularly in Java and Python. However, little is known about test order dependency in JavaScript tests. This paper aims to investigate test order dependency in JavaScript projects that use Jest, a widely used JavaScript testing framework. We implemented a systematic approach to randomise tests, test suites and describe blocks and produced 10 unique test reorders for each level. We reran each order 10 times (100 reruns for each test suite/project) and recorded any changes in test outcomes. We then manually analysed each case that showed flaky outcomes to determine the cause of flakiness. We examined our detection approach on a dataset of 81 projects obtained from GitHub. Our results revealed 55 order-dependent tests across 10 projects. Most order-dependent tests (52) occurred between tests, while the remaining three occurred between describe blocks. Those order-dependent tests are caused by either shared files (13) or shared mocking state (42) between tests. While sharing files is a known cause of order-dependent tests in other languages, our results underline a new cause (shared mocking state) that was not reported previously

Figures

Figures reproduced from arXiv: 2501.12680 by the authors.

**Figure 2.** An overview of the data collection and analysis process [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 64 canonical work pages

[11]

An empirical analysis of flaky tests,

Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” in Proceedings of the ACM International Symposium on Foundations of Software Engineering (FSE) , 2014

work page 2014
[1]

Eradicating non-determinism in tests,

M. Fowler, “Eradicating non-determinism in tests,” https://martinfowler .com/articles/nonDeterminism.html, (Accessed on 10/02/2024)

work page 2024
[2]

How to fix flaky tests,

A. Sandhu, “How to fix flaky tests,” 2015. [Online]. Available: https://tech.justeattakeaway.com/2015/03/30/how-to-fix-flaky-tests/

work page 2015
[3]

Test flakiness – methods for identifying and dealing with flaky tests,

J. Palmer, “Test flakiness – methods for identifying and dealing with flaky tests,” 2019. [Online]. Available: https://engineering.atspotify.com/ 2019/11/18/test-flakiness-methods-for-identifying-and-dealing-with-fla ky-tests/

work page 2019
[4]

Predictive test selection,

M. Machalica, A. Samylkin, M. Porth, and S. Chandra, “Predictive test selection,” in Proceedings of the International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) , 2019

work page 2019
[5]

Dependent- test-aware regression testing techniques,

W. Lam, A. Shi, R. Oei, S. Zhang, M. D. Ernst, and T. Xie, “Dependent- test-aware regression testing techniques,” in Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA), 2020

work page 2020
[6]

An empirical study of flaky tests in android apps,

S. Thorve, C. Sreshtha, and N. Meng, “An empirical study of flaky tests in android apps,” in Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME) , 2018

work page 2018
[7]

Google testing blog: Flaky tests at google and how we mitigate them,

J. Micco, “Google testing blog: Flaky tests at google and how we mitigate them,” https://testing.googleblog.com/2016/05/flaky-tests-at-google-and -how-we.html, (Accessed on 25/03/2024)

work page 2016

Show all 65 references

[8]

Probabilistic flakiness: How do you test your tests? - engineering at Meta,

M. Machalica, W. Chmiel, S. Swierc, and R. Sakevych, “Probabilistic flakiness: How do you test your tests? - engineering at Meta,” https:// engineering.fb.com/2020/12/10/developer-tools/probabilistic-flakiness/, 2020, (Accessed on 12/03/2024)

2020
[9]

The art of testing less without sacrificing quality,

K. Herzig, M. Greiler, J. Czerwonka, and B. Murphy, “The art of testing less without sacrificing quality,” in Proceedings of the IEEE/ACM 37th IEEE International Conference on Software Engineering , 2015

2015
[10]

Reducing flaky builds by 18x - the GitHub blog,

J. Raine, “Reducing flaky builds by 18x - the GitHub blog,” https://gith ub.blog/engineering/reducing-flaky-builds-by-18x/, 2021, (Accessed on 31/03/2024)

2021
[12]

An empirical study of flaky tests in python,

M. Gruber, S. Lukasczyk, F. Kroiß, and G. Fraser, “An empirical study of flaky tests in python,” in Proceedings of the IEEE Conference on Software Testing, Verification and Validation (ICST) , 2021

2021
[13]

An empirical study of flaky tests in javascript,

N. Hashemi, A. Tahir, and S. Rasheed, “An empirical study of flaky tests in javascript,” in Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME) , 2022

2022
[14]

Understanding flaky tests: The developer’s perspective,

M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli, “Understanding flaky tests: The developer’s perspective,” in Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) , 2019

2019
[15]

Finding flaky tests in javascript applications using stress and test suite reordering,

G. A. Yost, “Finding flaky tests in javascript applications using stress and test suite reordering,” Master’s thesis, The University of Texas at Austin, 2023

2023
[16]

JavaScript unit testing frameworks in 2024: A comparison · Raygun blog,

M. Taleb, “JavaScript unit testing frameworks in 2024: A comparison · Raygun blog,” https://raygun.com/blog/javascript-unit-testing-framework s/, 2023, (Accessed on 09/02/2024)

2024
[17]

Issue #4386: Flakiness in tests with Jest,

J. Community, “Issue #4386: Flakiness in tests with Jest,” https://github .com/jestjs/jest/issues/4386, 2023, accessed: 2024-09-11

2023
[18]

Empirically revisiting the test independence assumption,

S. Zhang, D. Jalali, J. Wuttke, K. Mu s ¸lu, W. Lam, M. D. Ernst, and D. Notkin, “Empirically revisiting the test independence assumption,” in Proceedings of the 2014 International Symposium on Software Testing and Analysis, 2014, pp. 385–396

2014
[19]

iFixFlakies: A framework for automatically fixing order-dependent flaky tests,

A. Shi, W. Lam, R. Oei, T. Xie, and D. Marinov, “iFixFlakies: A framework for automatically fixing order-dependent flaky tests,” in Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/F...

2019
[20]

react-testing-library,

“react-testing-library,” https://github.com/testing-library/react-testing -library/blob/main/src/ tests /auto-cleanup-skip.js, (Accessed on 10/02/2024)

2024
[21]

State of JavaScript 2023: Testing,

“State of JavaScript 2023: Testing,” https://2023.stateofjs.com/en-US/libr aries/testing/, (Accessed on 09/02/2024)

2023
[22]

Globals · Jest,

“Globals · Jest,” https://jestjs.io/docs/api#describename-fn, (Accessed on 10/02/2024)

2024
[23]

Option to run Jest in band and sort tests alphabetically,

“Option to run Jest in band and sort tests alphabetically,” https://github .com/jestjs/jest/issues/4032#issuecomment-424427942, 2018, (Accessed on 10/02/2024)

2018
[24]

Jest test schedular,

“Jest test schedular,” https://github.com/jestjs/jest/blob/1a487c1803124c 594bdd86ee24b5949025660bc3/packages/jest-cli/src/test scheduler.js# L88-L96, (Accessed on 10/02/2024)

2024
[25]

Setup and teardown · Jest,

“Setup and teardown · Jest,” https://jestjs.io/docs/setup-teardown#orde r-of-execution, (Accessed on 10/02/2024)

2024
[26]

Jest cli options · jest,

“Jest cli options · jest,” https://jestjs.io/docs/cli, (Accessed on 10/02/2024)

2024
[27]

Jest testsequencer,

“Jest testsequencer,” https://jestjs.io/docs/configuration#testsequencer-str ing, (Accessed on 10/02/2024)

2024
[28]

Babel · babel,

“Babel · babel,” https://babel.dev/, (Accessed on 10/01/2024)

2024
[29]

REST API endpoints for search - GitHub docs,

“REST API endpoints for search - GitHub docs,” https://docs.github.co m/en/rest/search/search, (Accessed on 10/01/2024)

2024
[30]

We are family: analyzing communi- cation in GitHub software repositories and their forks,

S. Brisson, E. Noei, and K. Lyons, “We are family: analyzing communi- cation in GitHub software repositories and their forks,” in Proceedings of the IEEE 27th International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2020, pp. 59–69

2020
[31]

Jest v20.0.0,

“Jest v20.0.0,” https://github.com/jestjs/jest/releases/tag/v20.0.0, (Accessed on 10/02/2024)

2024
[32]

Detecting and evaluating order-dependent flaky tests in JavaScript - replication package,

Anonymous, “Detecting and evaluating order-dependent flaky tests in JavaScript - replication package,” 2024. [Online]. Available: https://doi.org/10.5281/zenodo.13852085

2024 doi
[33]

Jasmine-matchers,

“Jasmine-matchers,” https://github.com/JamieMason/Jasmine-Matchers, (Accessed on 10/02/2024)

2024
[34]

timkindberg/jest-when: Jest support for mock argument-matched return values

“timkindberg/jest-when: Jest support for mock argument-matched return values.” https://github.com/timkindberg/jest-when, (Accessed on 10/01/2024)

2024
[35]

tj/commander.js: node.js command-line interfaces made easy,

“tj/commander.js: node.js command-line interfaces made easy,” https: //github.com/tj/commander.js, (Accessed on 10/01/2024)

2024
[36]

rbardini/jest-it-up: Automatically bump up global jest thresholds when- ever coverage goes above them,

“rbardini/jest-it-up: Automatically bump up global jest thresholds when- ever coverage goes above them,” https://github.com/rbardini/jest-it-up, (Accessed on 10/01/2024)

2024
[37]

fetch-mock-jest,

“fetch-mock-jest,” https://github.com/wheresrhys/fetch-mock-jest/releas es/tag/v1.5.1, (Accessed on 10/02/2024)

2024
[38]

iPFlakies: A framework for detecting and fixing Python order-dependent flaky tests,

R. Wang, Y . Chen, and W. Lam, “iPFlakies: A framework for detecting and fixing Python order-dependent flaky tests,” in Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, 2022

2022
[39]

iDFlakies: A framework for detecting and partially classifying flaky tests,

W. Lam, R. Oei, A. Shi, D. Marinov, and T. Xie, “iDFlakies: A framework for detecting and partially classifying flaky tests,” in Proceedings of the IEEE Conference on Ssoftware Testing, Validation and Verification (ICST), 2019

2019
[40]

Unit test virtualization with VMVM,

J. Bell and G. Kaiser, “Unit test virtualization with VMVM,” in Pro- ceedings of the 36th International Conference on Software Engineering , 2014, pp. 550–561

2014
[41]

Building a JavaScript testing framework: Run all the tests in parallel,

“Building a JavaScript testing framework: Run all the tests in parallel,” https://cpojer.net/posts/building-a-javascript-testing-framework#run-all -the-tests-in-parallel, (Accessed on 10/01/2024)

2024
[42]

Reliable testing: detecting state-polluting tests to prevent test dependency,

A. Gyori, A. Shi, F. Hariri, and D. Marinov, “Reliable testing: detecting state-polluting tests to prevent test dependency,” in Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), 2015

2015
[43]

FlaPy: mining flaky python tests at scale,

M. Gruber and G. Fraser, “FlaPy: mining flaky python tests at scale,” in Proceedings of the IEEE/ACM International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) , 2023

2023
[44]

The Jest object · Jest,

“The Jest object · Jest,” https://jestjs.io/docs/jest-object#jestmockmodule name-factory-options, (Accessed on 10/01/2024)

2024
[45]

Test flakiness’ causes, detection, impact and responses: A multivocal review,

A. Tahir, S. Rasheed, J. Dietrich, N. Hashemi, and L. Zhang, “Test flakiness’ causes, detection, impact and responses: A multivocal review,” Journal of Systems and Software , vol. 206, p. 111837, 2023

2023
[46]

Practical test dependency detection,

A. Gambi, J. Bell, and A. Zeller, “Practical test dependency detection,” in Proceedings of IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 2018

2018
[47]

FlakeSync: Automatically repairing async flaky tests,

S. Rahman and A. Shi, “FlakeSync: Automatically repairing async flaky tests,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , 2024

2024
[48]

Automatically re- producing timing-dependent flaky-test failures,

S. Rahman, A. Massey, W. Lam, A. Shi, and J. Bell, “Automatically re- producing timing-dependent flaky-test failures,” in 2024 IEEE Conference on Software Testing, Verification and Validation (ICST) , 2024

2024
[49]

Flaky test detection in android via event order exploration,

Z. Dong, A. Tiwari, X. L. Yu, and A. Roychoudhury, “Flaky test detection in android via event order exploration,” in Proceedings of the ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) , 2021

2021
[50]

Detecting assumptions on deterministic implementations of non-deterministic specifications,

A. Shi, A. Gyori, O. Legunsen, and D. Marinov, “Detecting assumptions on deterministic implementations of non-deterministic specifications,” in 2016 IEEE International Conference on Software Testing, Verification and Validation (ICST), 2016

2016
[51]

Detecting flaky tests in probabilistic and machine learning applications,

S. Dutta, A. Shi, R. Choudhary, Z. Zhang, A. Jain, and S. Misailovic, “Detecting flaky tests in probabilistic and machine learning applications,” in Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA) , 2020

2020
[52]

FLEX: fixing flaky tests in machine learning projects by updating assertion bounds,

S. Dutta, A. Shi, and S. Misailovic, “FLEX: fixing flaky tests in machine learning projects by updating assertion bounds,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2021

2021
[53]

On the effect of instrumentation on test flakiness,

S. Rasheed, J. Dietrich, and A. Tahir, “On the effect of instrumentation on test flakiness,” in Proceedings of IEEE/ACM International Conference on Automation of Software Test (AST) , 2023

2023
[54]

Flaky test sanitisation via on- the-fly assumption inference for tests with network dependencies,

J. Dietrich, S. Rasheed, and A. Tahir, “Flaky test sanitisation via on- the-fly assumption inference for tests with network dependencies,” in Proceedings of the IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) , 2022

2022
[55]

Probabilistic and systematic coverage of consecutive test-method pairs for detecting order- dependent flaky tests,

A. Wei, P. Yi, T. Xie, D. Marinov, and W. Lam, “Probabilistic and systematic coverage of consecutive test-method pairs for detecting order- dependent flaky tests,” in Tools and Algorithms for the Construction and Analysis of Systems: 27th International Conference (TACAS) , 2021

2021
[56]

Systematically producing test orders to detect order-dependent flaky tests,

C. Li, M. M. Khosravi, W. Lam, and A. Shi, “Systematically producing test orders to detect order-dependent flaky tests,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2023

2023
[57]

Repairing order-dependent flaky tests via test generation,

C. Li, C. Zhu, W. Wang, and A. Shi, “Repairing order-dependent flaky tests via test generation,” in Proceedings of the International Conference on Software Engineering (ICSE) , 2022

2022
[58]

Evolution-aware detection of order-dependent flaky tests,

C. Li and A. Shi, “Evolution-aware detection of order-dependent flaky tests,” in Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) , 2022

2022
[59]

Evaluating features for machine learning detection of order-and non-order-dependent flaky tests,

O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, “Evaluating features for machine learning detection of order-and non-order-dependent flaky tests,” in Proceedings of the IEEE Conference on Software Testing, Verification and Validation (ICST), 2022

2022
[60]

Test flakiness across programming languages,

K. Costa, R. Ferreira, G. Pinto, M. d’Amorim, and B. Miranda, “Test flakiness across programming languages,” IEEE Transactions on Software Engineering, 2022

2022
[61]

FlakyFix: Using large language models for predicting flaky test fix categories and test code repair,

S. Fatima, H. Hemmati, and L. Briand, “FlakyFix: Using large language models for predicting flaky test fix categories and test code repair,” arXiv preprint arXiv:2307.00012, 2024

2024 arXiv
[62]

A survey of flaky tests,

O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, “A survey of flaky tests,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 1, 2021

2021
[63]

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap,

B. Zolfaghari, R. M. Parizi, G. Srivastava, and Y . Hailemariam, “Root causing, detecting, and fixing flaky tests: State of the art and future roadmap,” Software: Practice and Experience , vol. 51, no. 5, 2021

2021
[64]

Towards a bayesian network model for predicting flaky automated tests,

T. M. King, D. Santiago, J. Phillips, and P. J. Clarke, “Towards a bayesian network model for predicting flaky automated tests,” in Proceedings of the IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) , 2018

2018
[65]

NodeRacer: Event race detection for node.js applications,

A. T. Endo and A. Møller, “NodeRacer: Event race detection for node.js applications,” in Proceedings of the IEEE International Conference on Software Testing, Validation and Verification (ICST) , 2020

2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [11]

An empirical analysis of flaky tests,

Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” in Proceedings of the ACM International Symposium on Foundations of Software Engineering (FSE) , 2014

work page 2014

[2] [1]

Eradicating non-determinism in tests,

M. Fowler, “Eradicating non-determinism in tests,” https://martinfowler .com/articles/nonDeterminism.html, (Accessed on 10/02/2024)

work page 2024

[3] [2]

How to fix flaky tests,

A. Sandhu, “How to fix flaky tests,” 2015. [Online]. Available: https://tech.justeattakeaway.com/2015/03/30/how-to-fix-flaky-tests/

work page 2015

[4] [3]

Test flakiness – methods for identifying and dealing with flaky tests,

J. Palmer, “Test flakiness – methods for identifying and dealing with flaky tests,” 2019. [Online]. Available: https://engineering.atspotify.com/ 2019/11/18/test-flakiness-methods-for-identifying-and-dealing-with-fla ky-tests/

work page 2019

[5] [4]

Predictive test selection,

M. Machalica, A. Samylkin, M. Porth, and S. Chandra, “Predictive test selection,” in Proceedings of the International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) , 2019

work page 2019

[6] [5]

Dependent- test-aware regression testing techniques,

W. Lam, A. Shi, R. Oei, S. Zhang, M. D. Ernst, and T. Xie, “Dependent- test-aware regression testing techniques,” in Proceedings of the ACM International Symposium on Software Testing and Analysis (ISSTA), 2020

work page 2020

[7] [6]

An empirical study of flaky tests in android apps,

S. Thorve, C. Sreshtha, and N. Meng, “An empirical study of flaky tests in android apps,” in Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME) , 2018

work page 2018

[8] [7]

Google testing blog: Flaky tests at google and how we mitigate them,

J. Micco, “Google testing blog: Flaky tests at google and how we mitigate them,” https://testing.googleblog.com/2016/05/flaky-tests-at-google-and -how-we.html, (Accessed on 25/03/2024)

work page 2016