Quantum Testing in the Wild: A Case Study with Qiskit Algorithms

Erico Augusto da Silva; Higor Amario de Souza; Marcos Lordello Chaim; Neilson Carlos Leite Ramalho

REVIEW 4 major objections 4 minor 24 references

Quantum Testing in the Wild: A Case Study with Qiskit Algorithms

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Classical testing methods still dominate how quantum algorithms are checked, a study of Qiskit Algorithms finds.

desk verdict Useful first taxonomy of testing patterns in a real quantum library, but the headline counts are internally inconsistent and the classification is subjective enough that percentages should be treated as approximate. read the letter →

arxiv 2501.06443 v1 pith:4MHTTXXW submitted 2025-01-11 cs.SE quant-ph

Neilson Carlos Leite Ramalho , Erico Augusto da Silva , Higor Amario de Souza , Marcos Lordello Chaim This is my paper · ORCID

classification cs.SEquant-ph

keywords quantumsoftwaretestingtestpatternsQiskitAlgorithmsblack-boxgray-boxdata-drivenempiricalstudy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This empirical study examines how practitioners actually test quantum algorithms by manually classifying all 309 quantum-related test methods in the Qiskit Algorithms library. The authors find that developers rely on classical software testing techniques: fixed random seeds, deterministic oracles, precise and approximate assertions, data-driven testing, functional (black-box) testing, gray-box checks of intermediate routines, and unitary-matrix equivalence checking. They report that 54.69% of the analyzed tests target classical infrastructure, while the quantum-related tests split between black-box (20.39%) and gray-box (24.92%) approaches. The central claim is that recent advances from quantum-software testing research, such as mutation testing, have not yet been adopted by practitioners. The result quantifies the gap between academic proposals and real-world testing practice in a fast-moving field.

What carries the argument

The central object is the classification scheme itself: each of the 309 test methods is hand-labeled as classical, black-box, or gray-box based on whether it exercises quantum-related concepts and whether the test data are designed with internal knowledge of the algorithm. This scheme produces the seven observed patterns, which are the paper's unit of analysis. The classification is load-bearing because every percentage and pattern count follows from it; the authors note that manual judgment is the main threat to its validity.

What would settle it

Take the same 309 test methods and have an independent team classify them with a written rubric; if the reported proportions (54.69% classical, 20.39% black-box, 24.92% gray-box) shift substantially, or if any of the seven patterns disappears, the central claim is not stable. Alternatively, if the current Qiskit Algorithms repository now contains mutation tests, metamorphic tests, or hardware-recording tests, the non-adoption claim is already outdated.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the quantum parts of Qiskit Algorithms are tested almost entirely with classical testing techniques, organized into seven recurring patterns. Those patterns are: setting fixed seeds to control randomness; using deterministic oracles with known-correct answers; combining precise and approximate assertions (e.g., assertEqual and assertAlmostEqual); applying data-driven testing with parameterized inputs; testing the algorithm as a black box by comparing final outputs to precomputed values; gray-box testing of intermediate results such as gradient computations and callbacks; and checking quantum-circuit equivalence by comparing unitary matrices. The authors count 63 black-box tests, 77 gray-box tests, and 169 classical tests out of 309. They find no use of research-community techniques such as mutation testing, and they observe that all tests run on simulators with no transpilation-related checks.

Load-bearing premise

The load-bearing premise is that the authors' manual classification of each test method as classical, black-box, or gray-box is accurate and consistent; the paper itself acknowledges that this depends on human judgment and could introduce errors.

Editorial extensions

If this is right

If the finding holds, practitioners building quantum libraries are not yet reaping the benefits of mutation testing, metamorphic testing, or other research-grade techniques.
The prevalence of fixed-seed and deterministic-oracle tests suggests that reproducibility is handled by controlling randomness rather than by statistical assertions over multiple runs.
Because no tests cover transpilation, circuit optimizations could silently change behavior without being caught by the current suite.
The gray-box tests focus on intermediate routines and callbacks, indicating that the hybrid classical-quantum interface is the most heavily probed area.
The dataset and scripts are public, allowing other researchers to replicate or extend the classification to other quantum frameworks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension is to run the same classification on other quantum libraries; if the same seven patterns appear, the result generalizes beyond Qiskit's conventions, and if not, the gap between research and practice may be framework-specific.
The absence of mutation testing may reflect a mismatch: mutation operators target circuits, whereas Qiskit Algorithms artifacts are parameterized circuit scaffolds where bugs live in classical orchestration; a testable prediction is that circuit-level mutations would not kill many of the existing tests.
The paper's percentages are a snapshot in time; as quantum hardware access matures, real-device tests with recording or mocking strategies could become an eighth pattern.
Since the authors classify callbacks as gray-box, one could turn those callback checks into metamorphic relations, for instance asserting that optimization trajectories are invariant under reordering of input terms.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The short version: this is the first paper I know of that goes through a real quantum-program test suite and names the testing patterns practitioners actually use. That alone makes it worth a serious look. The seven-pattern taxonomy (fixed seeds, deterministic oracles, precise/approximate assertions, DDT, functional, gray-box intermediate checks, equivalence checking) is clearly explained with code examples, and the authors put their classification data and scripts on GitHub. They also flag their own limitations, which I appreciate.

The central empirical claim — classical techniques dominate quantum-related tests, and none of the recent research techniques appear — is plausible and mostly supported by the examples. But the paper has a real internal inconsistency: Section IV.4 says 80 gray-box and 166 classical tests, while Table II says 77 and 169. Both sum to 309, but they cannot both be right. Since the whole point is the distribution of techniques, this needs to be fixed before publication. The authors should also say which number is correct and double-check the percentages.

The bigger soft spot is the classification method. The authors did have two people classify independently and then reach consensus, which is better than nothing, but they report no agreement measure and no rule for edge cases like hybrid tests. The paper itself admits the process 'depends on human judgment.' With 309 test methods, a handful of misclassifications could shift the pattern counts, though I doubt it would flip the main result. The taxonomy is reasonable as an organizational scheme, but the prevalence percentages should be treated as approximate.

Generalization is another limit: this is one library, Qiskit Algorithms. The abstract says 'practitioners,' but really we're seeing how one open-source project's developers test quantum code. The authors acknowledge this in the threats to validity, so I won't overweight it.

Who should read this: anyone working on quantum software testing, especially researchers trying to understand the research-practice gap. It's a solid empirical data point, not a definitive survey. My advice: send it to peer review, but require the authors to resolve the numerical inconsistency and add more detail on the classification protocol. The central finding is probably right, and the paper deserves a proper venue where those issues can be nailed down.

Referee Report

4 major / 4 minor

Summary. The paper reports a manual, observational study of the testing techniques used in the Qiskit Algorithms library. The authors inspected 309 test methods, classified each as classical, black-box, or gray-box, and identified seven recurring testing patterns: fixed seeds, deterministic oracles, precise and approximate assertions, data-driven testing, functional testing, gray-box testing of intermediate parts, and equivalence checking of quantum circuits. They conclude that practitioners predominantly rely on classical testing techniques for quantum-related elements and that recent research techniques for quantum program testing have not yet been adopted. The study dataset and preprocessing scripts are made available on GitHub.

Significance. If the results are reliable, the paper provides useful empirical evidence about the gap between quantum software testing research and industrial practice, and it is one of the few studies that examines testing patterns in a real, widely used quantum programming library. The open dataset, the independent classification by two authors with consensus resolution, and the concrete code examples are strengths that make the study partially checkable. However, the central prevalence claim depends on a subjective manual classification, and the paper contains a direct numerical inconsistency in the main counts, so the results need strengthening before they can be taken as a solid empirical baseline.

major comments (4)

[IV.4 and Table II] The counts for gray-box and classical tests are inconsistent between the text and Table II. Section IV.4 states that, of the 309 analyzed tests, 63 are black-box, 80 are gray-box, and 166 are classical, which sums to 309. Table II reports 63 black-box, 77 gray-box, and 169 classical, also summing to 309, and the reported percentages (20.39%, 24.92%, 54.69%) correspond to the Table II numbers, not to the text numbers. Because the prevalence percentages and the seven identified patterns are directly derived from this classification, the discrepancy must be resolved and the correct numbers reported.
[III.1 and VI] The classification methodology is not sufficiently operationalized for the central claim to be independently reproducible. The paper acknowledges in Section VI that the manual analysis depends on human judgment, but it does not report inter-rater agreement measures, a coding scheme, or an explicit decision rule for edge cases such as tests that exercise both black-box and gray-box aspects. The criterion stated in Section IV.4 for gray-box versus black-box classification ('need to understand the algorithm's internal functioning to generate appropriate test data') still requires judgment in application. I ask for a more precise operationalization, the classification artifact, and agreement statistics so that the counts and the resulting seven patterns can be checked.
[V] The negative claim that 'no evidence' of recent QP testing techniques was found is hard to evaluate without an explicit checklist. The paper cites Refs. [2,4,5,6,7] as the source of the techniques considered, but it does not specify which concrete techniques were searched for, nor the identification procedure used to determine absence. Please provide the technique list and the search/coding protocol; otherwise the conclusion that advanced techniques 'have yet to achieve wide adoption' is not falsifiable from the paper alone.
[II and III.1] The paper should clarify the relationship between the 587 tests mentioned in Section II and the 309 analyzed tests in Section IV.4. The text in Section III.1 says that purely classical tests were removed, but Table II then reports 169 classical tests among the 309. Please state exactly how the 309 tests were selected, how the 587 total relates to them, and which denominator is used for all prevalence percentages.

minor comments (4)

[IV] In the enumeration of the seven patterns, the fourth item is introduced as 'DDT' without the number '(4)', so the list reads '(1)...(3)... DDT; (5)...' and should be renumbered.
[IV.2 and Listing 2] The text says the assertion tolerance is '10^5' (or '105'), while the code uses places=5, which presumably means a tolerance of 10^{-5}; please correct this typo.
[Table I] Table I reports the top five assertions with percentages that sum to 81.58%, but it does not state the total number of assertions or the number of distinct assertion types. Please add the denominator and clarify what the remaining percentage corresponds to.
[II] The relationship between the 587 tests in the suite and the 309 analyzed tests should be stated explicitly; currently the reader must infer that some tests were excluded without a clear description of the exclusion criterion.

Circularity Check

0 steps flagged · score 0.0 of 10

Observational study; no derivation chain reduces to its inputs, so no circularity.

full rationale

This paper is an empirical, observational study rather than a derivation. The central claims—that seven testing patterns appear in Qiskit Algorithms and that classical techniques prevail—are supported by manually classifying 309 test methods and counting assertion occurrences; there is no fitted parameter that is later presented as a prediction, and no equation defines one quantity in terms of another. The authors cite their own prior survey [2] as one source for the set of quantum testing techniques checked for, alongside independent references [6,7], but the presence/absence conclusion is obtained by inspecting the test suite and is independently checkable from the released dataset, so the self-citation is not load-bearing. The acknowledged subjectivity of the manual classification, and the internal inconsistency between Section IV.4's counts (80 gray-box, 166 classical) and Table II's counts (77 gray-box, 169 classical), are threats to accuracy and reproducibility, not circularity. No circular step can be exhibited with a quote that reduces an output to an input, so the paper should receive a non-circular score of 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The study introduces no free parameters or invented entities. Its conclusions rest on domain assumptions: Qiskit Algorithms represents practitioner testing (external validity, Section VI), manual classification is accurate (internal validity, Section VI), and the absence of a technique in this codebase implies non-adoption.

assumptions (3)

domain assumption Qiskit Algorithms is representative of practitioner quantum software testing practices.
The paper generalizes from this single library to practitioner behavior; external validity is acknowledged as a threat in Section VI.
domain assumption Manual classification of test methods into classical/quantum-related categories is accurate.
Section VI lists human judgment as an internal validity threat.
domain assumption Absence of a technique in the examined test suite implies non-adoption by practitioners.
The conclusion that recent techniques have not achieved wide adoption rests on not finding them in this codebase.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Quantum Testing in the Wild: A Case Study with Qiskit Algorithms." pith.science (2026). https://pith.science/paper/4MHTTXXW

@misc{pith2026250106443,
  author       = {Pith},
  title        = {Pith review of: Quantum Testing in the Wild: A Case Study with Qiskit Algorithms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4MHTTXXW}},
  note         = {Machine review of arXiv:2501.06443}
}

read the original abstract

Although classical computing has excelled in a wide range of applications, there remain problems that push the limits of its capabilities, especially in fields like cryptography, optimization, and materials science. Quantum computing introduces a new computational paradigm, based on principles of superposition and entanglement to explore solutions beyond the capabilities of classical computation. With the increasing interest in the field, there are challenges and opportunities for academics and practitioners in terms of software engineering practices, particularly in testing quantum programs. This paper presents an empirical study of testing patterns in quantum algorithms. We analyzed all the tests handling quantum aspects of the implementations in the Qiskit Algorithms library and identified seven distinct patterns that make use of (1) fixed seeds for algorithms based on random elements; (2) deterministic oracles; (3) precise and approximate assertions; (4) Data-Driven Testing (DDT); (5) functional testing; (6) testing for intermediate parts of the algorithms being tested; and (7) equivalence checking for quantum circuits. Our results show a prevalence of classical testing techniques to test the quantum-related elements of the library, while recent advances from the research community have yet to achieve wide adoption among practitioners.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 15 canonical work pages

[1]

J. D. Hidary, Quantum Computing: An Applied Ap- proach. Springer, 2019

work page 2019
[2]

Testing and debugging quantum programs: The road to 2030,

N. C. L. Ramalho, H. A. de Souza, and M. L. Chaim, “Testing and debugging quantum programs: The road to 2030,” 2024. [Online]. Available: https://arxiv.org/abs / 2405.09178

arXiv 2024
[3]

Quantum computing with qiskit,

A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P . D. Nation, L. S. Bishop, A. W . Cross, B. R. Johnson, and J. M. Gambetta, “Quantum computing with qiskit,” 2024. [Online]. Available: https://arxiv.org/abs/2405.08810

arXiv 2024
[4]

Quantum software testing: State of the art,

A. Garc´ ıa de la Barrera, I. Garc´ ıa-Rodr´ ıguez de Guzm´an, M. Polo, and M. Piattini, “Quantum software testing: State of the art,” Journal of Software: Evolution and Process, vol. 35, no. 4, p. e2419, 2021

work page 2021
[5]

Quantum software engineering: Landscapes and horizons,

J. Zhao, “Quantum software engineering: Landscapes and horizons,” 2020. [Online]. Available: https://arxiv. org/abs/2007.07047

arXiv 2020
[6]

V eriﬁcation and validation of quantum soft- ware,

D. Fortunato, L. Jim´ enez-Navajas, J. Campos, and R. Abreu, “V eriﬁcation and validation of quantum soft- ware,” in Quantum Software: Aspects of Theory and System Design , I. Exman, R. P´ erez-Castillo, M. Piattini, and M. Felderer, Eds. Springer, Cham, 2024, pp. 93– 123

work page 2024
[7]

A survey on testing and analysis of quantum software,

M. Paltenghi and M. Pradel, “A survey on testing and analysis of quantum software,” 2024. [Online]. Available: https://arxiv.org/abs/2410.00650

arXiv 2024
[8]

[Online]

Python Software Foundation, unittest – Unit testing framework, 2024, accessed: 2024-10-24. [Online]. Available: https://docs.python.org/3/library/unittest.html# unittest.TestCase.assertAlmostEqual

work page 2024

Show all 24 references

[9]

Ammann and J

P . Ammann and J. Offutt, Introduction to Software T est- ing, 2nd ed. Cambridge University Press, 2016

2016
[10]

A comparative study of white box, black box and grey box testing techniques,

M. E. Khan and F. Khan, “A comparative study of white box, black box and grey box testing techniques,” International Journal of Advanced Computer Science and Applications, vol. 3, no. 6, 2012

2012
[11]

Identifying ﬂakiness in quantum programs,

L. Zhang, M. Radnejad, and A. Miranskyy, “Identifying ﬂakiness in quantum programs,” in Proceedings of the 2023 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement , ser. ESEM 2023. Los Alamitos, CA, USA: IEEE Computer Society, 2023, pp. 1–7. [On...

2023
[12]

Unit tests for azure quantum python sdk,

Microsoft, “Unit tests for azure quantum python sdk,” https://github.com/microsoft/azure-quantum-python/ blob/main/azure-quantum/tests/README.md, 2024, accessed: 2024-10-21

2024
[13]

[Online]

Kevin Chan, Dustin Lacewell, and contributors, VCR.py: Automatically Mock Y our HTTP Interactions to Simplify and Speed Up T esting , VCR.py Community, 2023, version 5.0.0. [Online]. Available: https://vcrpy. readthedocs.io/en/latest/

2023
[14]

An empirical study into the effects of transpilation on quantum circuit smells,

M. D. Stefano, D. D. Nucci, F. Palomba et al. , “An empirical study into the effects of transpilation on quantum circuit smells,” Empirical Software Engineering, vol. 29, no. 61, 2024. [Online]. Available: https://doi.org/10.1007/s10664-024-10461-9

2024 doi
[15]

Muskit: A mutation analysis tool for quantum software testing,

E. Mendiluze, S. Ali, P . Arcaini, and T. Y ue, “Muskit: A mutation analysis tool for quantum software testing,” in Proceedings of the 2021 36th IEEE/ACM International Conference on Automated Software Engineering , ser. ASE’21, 2021, pp. 1266–1270

2021
[16]

Qmutpy: A mutation testing tool for quantum algorithms and applications in qiskit,

D. Fortunato, J. Campos, and R. Abreu, “Qmutpy: A mutation testing tool for quantum algorithms and applications in qiskit,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software T esting and Analysis , ser. ISSTA 2022. New Y ork, NY , USA: ACM, 2022, pp. 7...

2022
[17]

QCRMut: Quantum circuit random mutant generator tool,

S. G. Gil, L. L. D´ ıaz, and J. I. R. Jarabo, “QCRMut: Quantum circuit random mutant generator tool,” 2024. [Online]. Available: https://arxiv.org/abs/2410.01415

2024 arXiv
[18]

Identifying bug pat- terns in quantum programs,

P . Zhao, J. Zhao, and L. Ma, “Identifying bug pat- terns in quantum programs,” in Proceedings of the 2021 IEEE/ACM 2nd International W orkshop on Quantum Software Engineering , ser. Q-SE 2021, 2021, pp. 16–21

2021
[19]

A com- prehensive study of bug ﬁxes in quantum programs,

J. Luo, P . Zhao, Z. Miao, S. Lan, and J. Zhao, “A com- prehensive study of bug ﬁxes in quantum programs,” in Proceedings of the 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering , ser. SANER 2022, 2022, pp. 1239–1246

2022
[20]

Bugs in quantum computing platforms: An empirical study,

M. Paltenghi and M. Pradel, “Bugs in quantum computing platforms: An empirical study,” Proceedings of the ACM on Programming Languages , vol. 6, no. OOPSLA1, apr 2022. [Online]. Available: https://doi. org/10.1145/3527330

2022 doi
[21]

An empirical study of bugs in quantum machine learning frameworks,

P . Zhao, X. Wu, J. Luo, Z. Li, and J. Zhao, “An empirical study of bugs in quantum machine learning frameworks,” in Proceedings of the 2023 IEEE International Confer- ence on Quantum Software , ser. QSW 2023, 2023, pp. 68–75

2023
[22]

Qchecker: De- tecting bugs in quantum programs via static analysis,

P . Zhao, X. Wu, Z. Li, and J. Zhao, “Qchecker: De- tecting bugs in quantum programs via static analysis,” in Proceedings of the 2023 IEEE/ACM 4th International W orkshop on Quantum Software Engineering , ser. Q-SE 2023, 2023, pp. 50–57

2023
[23]

Analyzing quantum programs with lintq: A static analysis framework for qiskit,

M. Paltenghi and M. Pradel, “Analyzing quantum programs with lintq: A static analysis framework for qiskit,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 2144–2166, 2024. [Online]. Available: https://doi.org/10.1145/3660802

2024 doi
[24]

The smelly eight: An empirical study on the prevalence of code smells in quantum computing,

Q. Chen, R. Cˆ amara, J. Campos, A. Souto, and I. Ahmed, “The smelly eight: An empirical study on the prevalence of code smells in quantum computing,” in Proceedings of the 2023 IEEE/ACM 45th International Conference on Software Engineering , ser. ICSE’23, 2023, pp. 358–370

2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [1]

J. D. Hidary, Quantum Computing: An Applied Ap- proach. Springer, 2019

work page 2019

[2] [2]

Testing and debugging quantum programs: The road to 2030,

N. C. L. Ramalho, H. A. de Souza, and M. L. Chaim, “Testing and debugging quantum programs: The road to 2030,” 2024. [Online]. Available: https://arxiv.org/abs / 2405.09178

arXiv 2024

[3] [3]

Quantum computing with qiskit,

A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P . D. Nation, L. S. Bishop, A. W . Cross, B. R. Johnson, and J. M. Gambetta, “Quantum computing with qiskit,” 2024. [Online]. Available: https://arxiv.org/abs/2405.08810

arXiv 2024

[4] [4]

Quantum software testing: State of the art,

A. Garc´ ıa de la Barrera, I. Garc´ ıa-Rodr´ ıguez de Guzm´an, M. Polo, and M. Piattini, “Quantum software testing: State of the art,” Journal of Software: Evolution and Process, vol. 35, no. 4, p. e2419, 2021

work page 2021

[5] [5]

Quantum software engineering: Landscapes and horizons,

J. Zhao, “Quantum software engineering: Landscapes and horizons,” 2020. [Online]. Available: https://arxiv. org/abs/2007.07047

arXiv 2020

[6] [6]

V eriﬁcation and validation of quantum soft- ware,

D. Fortunato, L. Jim´ enez-Navajas, J. Campos, and R. Abreu, “V eriﬁcation and validation of quantum soft- ware,” in Quantum Software: Aspects of Theory and System Design , I. Exman, R. P´ erez-Castillo, M. Piattini, and M. Felderer, Eds. Springer, Cham, 2024, pp. 93– 123

work page 2024

[7] [7]

A survey on testing and analysis of quantum software,

M. Paltenghi and M. Pradel, “A survey on testing and analysis of quantum software,” 2024. [Online]. Available: https://arxiv.org/abs/2410.00650

arXiv 2024

[8] [8]

[Online]

Python Software Foundation, unittest – Unit testing framework, 2024, accessed: 2024-10-24. [Online]. Available: https://docs.python.org/3/library/unittest.html# unittest.TestCase.assertAlmostEqual

work page 2024