Pith. sign in

REVIEW 4 major objections 3 minor 1 cited by

Detecting Flakiness in Quantum Software: A Dynamic Testing Approach

T0 review · 4 major / 3 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By rerunning the Qiskit Terra test suite 10,000 times per release across 23 versions, this paper shows that quantum software harbors hundreds of flaky tests—most failing so rarely that standard CI reruns cannot catch them.

desk verdict First large dynamic rerun dataset for quantum flakiness is a real contribution, but the detection floor and an abstract/body numbers mismatch keep the headline claims from being taken at face value. read the letter →

arxiv 2512.18088 v3 pith:AVUBGHR4 submitted 2025-12-19 cs.SE

Dongchan Kim , Hamidreza Khoramrokh , Lei Zhang , Andriy Miranskyy This is my paper · ORCID
classification cs.SE
keywords flakytestsquantumsoftwaredynamictestingQiskitTerraempiricalfailureprobabilityWilsonconfidenceintervaltestrerunbudgetnondeterminism
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to establish, for the first time on a large scale, how flaky tests behave in quantum software. It executes the Qiskit Terra test suite 10,000 times per release across 23 releases, labeling any test that both passes and fails as flaky. The result is 290 unique flaky tests out of 27,026 distinct test cases, with most failing exactly once in every 10,000 runs. Because such rare failures require tens of thousands of reruns to detect with statistical confidence, typical continuous-integration budgets will miss most quantum flakiness. This matters because it provides the first dynamic ground-truth dataset for quantum flaky tests and exposes a structural detection problem.

What carries the argument

The central object is the per-test execution record: 10,000 runs of each test in each release, summarized as passed_attempts and failed_attempts. A test is flaky if both counts are positive. The empirical failure probability p_hat = failed_attempts / 10,000 anchors the analysis, and the Wilson confidence interval's lower bound L is used to compute a conservative rerun budget n = ceil( ln(1 - q) / ln(1 - L) ) for detecting at least one failure with probability q. This machinery converts raw rerun data into a statistical statement about how many executions a test needs before flakiness becomes visible.

What would settle it

Run a test whose true failure probability is known to be 5×10^-5 (for example, by injecting a rare random failure) for 10,000 executions; the paper's definition would almost certainly classify it as stable. If such tests exist in Qiskit, the reported prevalence of 290 is an underestimate. Alternatively, rerun a sample of tests the paper classifies as stable for 100,000 executions; any that reveal a pass and a fail would falsify the completeness of the 10,000-run detection floor.

Watch

Extended reading notes

Core claim

The paper's central discovery is that flakiness in Qiskit Terra is real, episodic, and often extremely sparse in failure probability. Across 23 releases and 10,000 executions per test, 290 of 27,026 distinct tests showed at least one pass and at least one fail, for an overall flakiness rate of 0–0.40% per release. Most flaky tests (192) failed exactly once in 10,000 runs, giving an empirical failure probability of 10^-4. Using a Wilson confidence interval, the paper derives a conservative rerun budget: detecting such a test with 95% probability requires about 169,000 executions, far beyond typical CI rerun budgets. Flakiness is concentrated in the transpiler and quantum_info subcomponents, a

Load-bearing premise

A test is classified as flaky only if it shows at least one pass and at least one fail in 10,000 runs, so tests with true failure probabilities below roughly 10^-4 are effectively invisible to this study.

Editorial extensions

If this is right

  • Flakiness in quantum software is rare overall but not negligible: 290 unique flaky tests exist in the studied version range, concentrated in a few subcomponents.
  • Most flaky tests fail with empirical probability near 10^-4, so typical CI rerun budgets of tens or hundreds of runs will almost always miss them; reliable detection requires tens of thousands of runs.
  • The transpiler and quantum_info subcomponents account for the majority of flaky tests, suggesting that targeted stabilization of these components could remove most observed flakiness.
  • The released per-test execution dataset provides dynamic ground truth that can support future detection, prediction, and root-cause studies of quantum flaky tests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's detection floor—one failure in 10,000 runs—implies the true flakiness count is likely an underestimate; tests with failure probabilities below 10^-4 would be misclassified as stable, so a higher-budget follow-up on a stratified sample could quantify the undercount.
  • The concentration of flakiness in the transpiler suggests a testable mechanism: nondeterminism in transpiler passes (e.g., noise-aware layout or routing) might be reproducible by varying random seeds; if fixing seeds eliminates flakiness, that would confirm the suspected cause.
  • The rerun-budget formula can be inverted: for a given CI budget, teams can compute the smallest failure probability detectable at a chosen confidence, letting them choose budgets based on acceptable miss rates rather than fixed rerun counts.
  • The three temporal patterns (rarely, persistently, intermittently flaky) could be used to prioritize fixes: persistently flaky tests affect many releases and likely indicate stable, systematic nondeterminism, whereas rarely flaky tests may be noise from environment or dependency drift.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. This paper reports the first large-scale dynamic study of flaky tests in Qiskit Terra by executing the test suite 10,000 times per release across 23 releases. It defines a test as flaky within a release if it passes at least once and fails at least once, then analyzes prevalence, recurrence across releases, empirical failure probabilities, Wilson-based rerun budgets, and subcomponent distribution. The authors release the per-test execution dataset. The central empirical claim is that quantum test flakiness is rare but difficult to detect: reported rates are low (0–0.40%), yet many flaky tests have very small failure probabilities around 10^-4, requiring tens of thousands of reruns for reliable detection.

Significance. If the reported numbers are correct, this is a valuable contribution: a 70-CPU-year dynamic rerun study of a major quantum software library, with a public dataset that can ground future static and ML-based flakiness work. The use of standard binomial/Wilson statistics for detectability is appropriate, and the study addresses a genuine gap left by prior incident-report-based and static studies. However, the manuscript's credibility is currently undermined by internal inconsistencies in headline numbers and by an unquantified detection floor that conditions all prevalence and recurrence conclusions.

major comments (4)
  1. [Abstract vs. §4 / Table 2] The abstract and the body report incompatible headline numbers. The abstract states 62 unique flaky tests, median p̂ = 9×10^-4, and 52/62 (83.87%) recurring across releases; the body (Section 4, Table 2, Table 3) reports 820 flaky-test occurrences, 290 unique flaky tests, median p̂ = 0.0033, and 98/290 recurring. These cannot both describe the same dataset. The abstract must be reconciled with the body before any prevalence, recurrence, or rarity claim can be evaluated.
  2. [§3.3 / RQ2 / RQ6] The flakiness definition passed_attempts>0 and failed_attempts>0 over N=10,000 creates a hard detection floor. A test with true failure probability p is detected with probability 1-(1-p)^N: 0.63 at p=1e-4 and 0.39 at p=5e-5. Thus 820 occurrences and 290 unique tests are lower bounds, and the undercount is not quantified in the threats-to-validity section or elsewhere. This also confounds RQ2/RQ6: a test with constant p≈1e-4 will appear flaky in a random ~63% of releases, so the 'intermittently flaky' profile can be a statistical artifact rather than a code-level state. The authors should report N-dependent detection probabilities or restrict recurrence claims to tests with well-constrained p, and explicitly state that all prevalence figures are floor estimates.
  3. [§4.3.2, Eq. (2)] The rerun-budget analysis only computes the number of executions needed to observe at least one failure, but the operational definition of flakiness requires both at least one pass and at least one fail. For high-p tests (several have p̂=0.9999), the pass side is the limiting event and is unanalyzed. Detecting flakiness with probability q requires n ≥ max(ln(1-q)/ln(1-p), ln(1-q)/ln(p)). The paper should present both sides of the budget, or explicitly restrict RQ3 to the low-p regime and justify that restriction.
  4. [Table 2] The column 'Share of Unique Flaky Tests (%)' is defined as per-release flaky count divided by 290 unique tests. Because the same test can be flaky in multiple releases, the column sums to 820/290 = 282.76%, not 100% as printed in the Total row. Either the definition, the denominator (e.g., 820 occurrences), or the table must be corrected; as is, the table misstates the relationship between release-level and unique-level flakiness.
minor comments (3)
  1. [§1] Typo/incomplete sentence: 'Static machine-learning approaches depends critically...' should be 'Static and machine-learning approaches depend critically...'.
  2. [§2.1] 'Flaky tests have been widely extensively' should be 'widely studied' or similar; also 'flakiness patterns' is misspelled 'patters' in §4.7.2.
  3. [Figures 7–9] The placeholder 10^-6 for zero observed failures is used without a note in the main text. Consider explaining it in the figure captions or text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the empirical measurements and standard binomial/Wilson statistics stand on their own; self-citations are contextual only.

full rationale

The paper's central claims are direct empirical measurements: repeatedly executing Qiskit Terra tests, counting pass/fail outcomes, and reporting flaky-test counts, failure probabilities, and recurrence patterns. The flakiness definition (passed_attempts>0 and failed_attempts>0) is an operational labeling rule, not a conclusion derived from the data. The probabilistic analysis in RQ3 uses standard, externally established binomial and Wilson-interval statistics; the rerun-budget formula n >= ln(1-q)/ln(1-p) is a textbook consequence, not a fitted parameter disguised as a prediction. No fitted input is renamed as a prediction. Self-citations (Zhang et al., Zhang and Miranskyy, Sivaloganathan et al., Kaur et al.) appear only in related work and motivation, and the paper's core contribution does not reduce to any of those citations. The acknowledged detection floor at N=10,000 is a real measurement limitation and selection effect, and the paper's own threat-to-validity section notes under-detection of order- and timing-dependent flakiness; however, this is a limitation on completeness, not circular reasoning. There is an internal numerical inconsistency between the abstract (62 unique flaky tests, median p=9e-4) and the body (290 unique flaky tests, median p=0.0033), but that is a reporting/consistency issue, not a circular derivation. Overall, the derivation chain is self-contained against external statistical facts and direct observation, so no circular step is present.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities or fitted parameters. Its load-bearing assumptions are the flakiness detection threshold, the i.i.d. binomial model for rerun budgets, and the standard Wilson interval. No invented physical or mathematical entities.

assumptions (3)
  • domain assumption A test is flaky iff it has at least one pass and at least one fail across 10,000 executions.
    Section 3.3 defines flakiness this way; it sets the detection floor and biases all prevalence estimates.
  • domain assumption Per-run outcomes are independent and identically distributed Bernoulli trials.
    Section 4.3.2 uses the binomial model and Wilson intervals; Section 4.7.1 acknowledges HPC drift violates this to some degree.
  • standard math Wilson confidence interval for a binomial proportion is valid and appropriate.
    Section 4.3.2 relies on Wilson (1927) and Brown et al. (2001); this is standard statistics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting Flakiness in Quantum Software: A Dynamic Testing Approach." pith.science (2026). https://pith.science/paper/AVUBGHR4

@misc{pith2026251218088,
  author       = {Pith},
  title        = {Pith review of: Detecting Flakiness in Quantum Software: A Dynamic Testing Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AVUBGHR4}},
  note         = {Machine review of arXiv:2512.18088}
}
abstract

Flaky tests, tests that pass or fail nondeterministically without changes to code or environment, pose a serious threat to software reliability. While classical software engineering has developed a rich body of techniques to study flakiness, corresponding evidence for quantum software remains limited. Prior work relies mainly on static analysis or small sets of manually reported incidents, leaving open questions about their prevalence, characteristics, and detectability. This paper presents the first large-scale dynamic characterization of flaky tests in quantum software, focusing on the Qiskit Terra core library. We executed the Qiskit Terra test suite 10,000 times across 23 releases in controlled environments. For each release, we measured test-outcome variability, identified flaky tests, estimated empirical failure probabilities, analyzed recurrence across versions, used Wilson confidence intervals to quantify rerun budgets for reliable detection, and mapped flaky tests to Terra subcomponents. Across 27,026 fully qualified test identifiers, we identified 62 unique flaky tests. Although overall flakiness rates were low (0-0.17%), recurrence was substantial: 52 of 62 flaky tests (83.87%) reappeared in multiple releases, while only 10 tests (16.13%) were confined to a single release. Empirical failure probabilities spanned several orders of magnitude, with a median of $9 \times 10^{-4}$ and 34 tests (54.84%) at or below $10^{-3}$, implying that thousands to tens of thousands of executions may be required for confident detection. These results show that quantum test flakiness is rare but difficult to detect under typical continuous integration budgets. To support future research, we release a public dataset of per-test execution outcomes.

Figures

Figures reproduced from arXiv: 2512.18088 by the authors.

Figure 1
Figure 1. Distribution of empirical failure probabilities [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. Wilson confidence intervals for the true failure probability [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Conservative rerun budget for flakiness detection using Wilson lower confidence [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Distribution of failure frequencies for flaky tests across [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Total number of distinct flaky tests by terra subcomponent, aggregated across all releases. Subcomponents are sorted in descending order. transpiler quantum_info compiler synthesis circuit qpy pulse scheduler visualization providers test_user_config basicaer opflow pri…
Figure 6
Figure 6. Figure 6: Distribution of flaky tests across terra subcomponents over all analyzed re￾leases. For each subcomponent, the boxplot summarizes the per-release count of distinct flaky test cases, and subcomponents are ordered as in [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Rarely Flaky pattern. The temporal pattern in the right panel does not imply [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Persistently Flaky pattern. The gray background marks releases in which the [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Intermittently Flaky pattern. These examples show test cases that intermittently [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Longest consecutive flaky-release streak per test across [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Shadow-Based Noise Fingerprinting of Simulated Quantum Noise Models

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Physics-informed classical-shadow features from fixed 3-qubit probes let a random forest identify ten simulated quantum noise channels at 0.84 test accuracy.

Reference graph

Works this paper leans on

16 extracted references · 2 canonical work pages · cited by 1 Pith paper

  1. [7]

    arXiv preprint arXiv:2503.05240

    Mining q&a platforms for empirical evidence on quantum software pro- gramming. arXiv preprint arXiv:2503.05240 . Kurtzer, G.M., Sochat, V., Bauer, M.W.,

    arXiv

  2. [11]

    What is the vocabulary of flaky tests?, in: Proceed- ings of the 17th International Conference on Mining Software Repositories (MSR ’20), ACM/IEEE. pp. 492–502. doi:10.1145/3379597.3387482. Qiskit Development Community, 2025a. Configure qiskit lo- cally. URL:https://quantum.cloud.ibm.com/docs/en/guides/ configure-qiskit-local. [Online; accessed 2025-11-07...

    arXiv 2025

  3. [12]

    arXiv preprint arXiv:2410.23578

    Automat- ing quantum software maintenance: Flakiness detection and root cause analysis. arXiv preprint arXiv:2410.23578 . Tox Development Community, . tox-dev/tox: Command line driven CI fron- tend and development task automation tool. URL:https://github.com/ tox-dev/tox. Verdecchia, R., Cruciani, E., Miranda, B., Bertolino, A.,

    arXiv

  4. [14]

    Automated flakiness detection in quantum software bug reports, in: 2024 IEEE International Conference on Quantum Computing and Engineering (QCE), IEEE. pp. 179–181. Zhang, L., Radnejad, M., Miranskyy, A.,

    2024

  5. [16]

    De-flake your tests: Automatically locating root causes of flaky tests in code at Google, in: 2020 IEEE International 26 Conference on Software Maintenance and Evolution (ICSME), IEEE. pp. 736–745. 27 Appendix A. Module taxonomy Below are descriptions of the main subcomponents of Qiskitterra. transpilerTests passes, preset pass managers, layout, routing, ...

    2020

  6. [187]

    Bell, J., Legunsen, O., Hilton, M., Eloussi, L., Yung, T., Marinov, D.,

    doi:10.1109/ICSE-COMPANION52605.2021.00081. Bell, J., Legunsen, O., Hilton, M., Eloussi, L., Yung, T., Marinov, D.,

    arXiv 2021

  7. [2014]

    The rust language, in: Proceedings of the 2014 ACM SIGAda annual conference on High integrity language technol- ogy, pp. 103–104. Memon, A., Gao, Z., Nguyen, B., Dhanda, S., Nickell, E., Siemborski, R., Micco, J.,

    2014

  8. [2015]

    Science of computer programming 101, 136–152

    Six strategies for generalizing software engineering theories. Science of computer programming 101, 136–152. doi:10.1016/J.SCICO.2014.11.013. Wilson, E.B.,

    doi:10.1016/j.scico.2014.11.013 2014

Show all 16 references
  1. [2017]

    Taming google-scale continuous testing, in: Proceedings of the 2017 IEEE/ACM 39th International Conference on Software Engi- neering: Software Engineering in Practice Track (ICSE-SEIP), IEEE. pp. 233–242. 23 Micco, J.,

    2017

  2. [2018]

    Deflaker: Automatically detecting flaky tests, in: Proceedings of the 40th International Conference on Software Engineering (ICSE ’18), ACM. pp. 433–444. doi:10.1145/3180155.3180164. Brown, L.D., Cai, T.T., DasGupta, A.,

  3. [2020]

    1471–1482

    A study on the lifecycle of flaky tests, in: Proceedings of the ACM/IEEE 42nd Interna- tional Conference on Software Engineering, pp. 1471–1482. Lam, W., Oei, R., Shi, A., Marinov, D., Xie, T., 2019b. idflakies: A frame- work for detecting and partially classifying flaky tests...

    2019 doi

  4. [2021]

    An empirical study of flaky tests in Python, in: 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), IEEE. pp. 148–158. Javadi-Abhari, A., Treinish, M., Krsulich, K., Wood, C.J., Lishman, J., Gacon, J., Martiel, S., Nation, P.D., Bishop, L.S., Cr...

    2021

  5. [2022]

    Asurveyonhowtestflakinessaffectsdevelopers and what support they need to address it, in: Proceedings of 2022 IEEE ConferenceonSoftwareTesting, VerificationandValidation(ICST),IEEE. pp. 82–92. Gruber, M., Lukasczyk, S., Kroiß, F., Fraser, G.,

    2022

  6. [2023]

    doi:10.1109/ESEM56168.2023.10304850

    Identifying flakiness in quantum programs, in: Proceedings of the 17th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM 2023), ACM/IEEE. doi:10.1109/ESEM56168.2023.10304850. Ziftci, C., Cavalcanti, D.,

    2023

  7. [2024]

    doi:10.48550/arXiv.2405.08810,arXiv:2405.08810

    Quantum computing with Qiskit. doi:10.48550/arXiv.2405.08810,arXiv:2405.08810. 22 Kaur, K., Kim, D., Jamshidi, A., Zhang, L.,

    doi arXiv

  8. [2025]

    doi:10.48550/arXiv.2502.04471

    Identifying flaky tests in quantum code: A machine learning approach, in: Proceedings of the 8th Workshop on Validation, Analysis and Evolution of Software Tests (VST 2025). doi:10.48550/arXiv.2502.04471. Khan, A.A., Ye, B., Akbar, M.A., Khan, J.A., Mougouei, D., Ma, X.,

    2025 doi arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.