Pith. sign in

REVIEW 5 major objections 7 minor 23 references

Evaluating Mutation-based Fault Localization for Quantum Programs

T0 review · 5 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mutation-based fault localization works well on artificial quantum bugs but poorly on real-world ones.

desk verdict First MBFL-for-quantum study with a real finding, but the artificial benchmark's construction inflates the headline gap. read the letter →

arxiv 2505.09059 v1 pith:4NDTS7T6 submitted 2025-05-14 cs.SE

Yuta Ishimoto , Masanari Kondo , Naoyasu Ubayashi , Yasutaka Kamei , Ryota Katsube , Naoto Sato , Hideto Ogawa This is my paper · ORCID
classification cs.SE
keywords quantumprogramfaultlocalizationmutation-basedmutationtestingoperationsQiskitEXAMscorespectrum-basedreal-worldbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper applies mutation-based fault localization (MBFL) to quantum programs written in Qiskit and tests it on 23 real-world bugs and 305 artificially injected bugs. It finds that artificial bugs are much easier: the median EXAM score (percentage of code examined before hitting the faulty statement, lower is better) is 1.2% in the worst case for artificial bugs versus 19.4% for real-world bugs. The gap matters because any evaluation built only on injected faults will overstate how well MBFL would work in practice. The paper also shows that quantum-specific mutation operations, not classical ones, produce nearly all the behavior-changing mutants behind successful localization on real bugs.

What carries the argument

The load-bearing mechanism is the MUSE-style suspiciousness formula, which scores each statement $s$ by the fraction of failed tests of the original program that are flipped to passing in mutants of $s$: $S(s) = \frac{1}{|mut(s)|}\sum_{m\in mut(s)} |f_P(s)\cap p_m|/|f_P(s)|$. Localization quality is then read off the EXAM score, defined as the percentage of ranked statements a developer must inspect before reaching the first faulty statement, with best- and worst-case tie handling. Mutants come from QMutPy, configured with 20 classical and 3 usable quantum mutation operations (gate deletion, gate replacement, and measurement deletion); the study measures how many of these mutants are behavior-changing, since only those can change the suspiciousness scores.

What would settle it

Run the same MBFL pipeline on all 42 Bugs4Q bugs with dependency versions pinned to the benchmark's original setup, and compare the median worst-case EXAM score. If it falls well below 19.4%, the reported real-versus-artificial gap is an artifact of the 23-bug reproducible subset; if adding a missing-initialization mutation operation drops the median into single digits, the paper's proposed direction is confirmed.

Watch

Extended reading notes

Core claim

The central finding is that mutation-based fault localization transfers to quantum programs but its effectiveness collapses when faults are real. On a curated subset of 23 reproducible bugs from the Bugs4Q benchmark, the worst-case median EXAM score is 19.4%, and roughly 40% of those bugs get a worst-case EXAM of 100% — the faulty statement is no better than last in the ranking. On 305 artificially injected faults in Qiskit-Aqua programs, the same pipeline reaches a worst-case median EXAM of 1.2%. The paper attributes the gap to the expressiveness of the mutation operations: real faults such as a forgotten qubit-initialization pattern are not reversible or reachable by single-statement gate addition, deletion, or replacement, while artificial faults often are. A secondary result is that quantum mutation operations outperform classical ones in producing behavior-changing mutants, and that MBFL beats spectrum-based fault localization on the artificial benchmark in the worst-case tie scenario, while spectrum-based methods cannot even be run on Bugs4Q because each bug has only one failing test.

Load-bearing premise

The load-bearing premise is that the 23 Bugs4Q bugs that reproduce in the authors' environment, each paired with a single manually written test, represent real-world quantum program faults well enough for the poor EXAM scores to be a true property of the technique rather than of the subset.

Editorial extensions

If this is right

  • Evaluations of MBFL for quantum programs must include real-world faults; benchmarks built only from injected mutants overstate practical effectiveness.
  • Quantum mutation operations should be the focus of MBFL tooling for quantum programs, since they produced 93.9% of behavior-changing mutants on real bugs despite being 3 of 23 operation types.
  • When a bug has a single failing test, spectrum-based fault localization cannot rank statements at all, while MBFL still produces a suspiciousness ordering.
  • Enriching mutation operations with recurring quantum patterns, such as a missing qubit-initialization operation, is a concrete next step implied by the failure cases.
  • On the artificial benchmark, MBFL resolves suspiciousness ties more conservatively than SBFL, yielding a significantly better worst-case EXAM (p<0.01, large effect size).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the 19 excluded Bugs4Q bugs differ systematically from the 23 reproducible ones, the true gap between artificial and real-world difficulty could be larger or smaller than the reported 19.4% versus 1.2% median; the paper's validity section acknowledges this bias but does not bound it.
  • Adding a missing-initialization mutation operation and re-running the same pipeline on Bugs4Q would give a direct test of whether operator coverage, rather than test quality or fault complexity, explains the poor real-world scores.
  • The single-failing-test advantage of MBFL over SBFL may extend beyond quantum software to any domain where bug reports arrive with a single reproduction test.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper applies mutation-based fault localization (MBFL) to quantum programs written in Qiskit, using the QMutPy tool to generate mutants of buggy programs and the MUSE-style suspiciousness formula of Eq. (1) to rank statements. The evaluation uses 23 real-world faults from Bugs4Q and 305 artificial faults from a newly constructed benchmark, BugsAqua, which is built by injecting QMutPy mutants into Qiskit-Aqua programs. The central results are: (RQ1) real-world faults are harder for MBFL than artificial faults, with median worst-case EXAM scores of 19.4% versus 1.2%; (RQ2) quantum mutation operations produce more behavior-changing mutants per operator than classical operations; and a comparison showing MBFL outperforms SBFL in the worst-case scenario on BugsAqua. The authors conclude that MBFL shows promise for quantum programs but that existing mutation operations are limited for realistic faults, and they suggest enriching the set of quantum mutation operations.

Significance. If the findings hold, this is a useful contribution to the emerging area of quantum software testing and debugging. The paper is, to my knowledge, among the first to compare MBFL performance on real-world versus artificially injected quantum program faults, and it directly follows the methodology of Pearson et al. for classical programs. The use of publicly available benchmarks (Bugs4Q, QMutPy) and the explicit discussion of failure cases (e.g., Listing 2) are strengths. The paper also provides a falsifiable ranking result: quantum mutation operations dominate behavior-changing mutants for Bugs4Q, which is a concrete and checkable claim. However, the current support for the headline claim is bounded by methodological gaps: the instrumentation used to compute f_P(s) is not described, the artificial benchmark is constructed with the same mutation operators that MBFL uses for scoring, and a large fraction of both real and artificial faults is excluded. These issues do not automatically invalidate the direction of the result, but they currently prevent the paper from supporting its strong comparative conclusion.

major comments (5)
  1. [Section 3.3, Eq. (1)] The suspiciousness formula relies on f_P(s), defined as the set of failed test cases when statement s is executed in program P. The paper never describes how statement execution is tracked for quantum programs, which contain both classical Python control flow and quantum operations such as gates and measurements. Is the instrumentation done at the AST level, through Qiskit circuit inspection, or via some other mechanism? Without this description, the experimental procedure is not reproducible and the reader cannot assess whether statements that are not executed on some paths are handled correctly.
  2. [Section 3.2.2 and Section 4.1 (RQ1)] The BugsAqua benchmark is constructed by applying the same 23 QMutPy mutation operators that MBFL later uses to score statements. As the authors note in Section 4.1, many of these injected faults are 'reversible': a one-step replacement can restore the original passing program, which makes the faulty statement trivially suspicious in Eq. (1). The paper reports this as an explanation for the RQ1 gap, but it does not control for it. Since the headline result (median worst-case EXAM 19.4% vs. 1.2%) compares an artificial benchmark that is partly easy by construction against a real-world benchmark that is not, the comparison is confounded. I ask the authors to (a) quantify the proportion of BugsAqua faults that are reversible under the 23 operators, (b) repeat the RQ1 comparison on the subset of non-reversible artificial faults, or otherwise show that the conclusion is not an artifact of benchmark construction, and (c) clearly separate the 'intrinsic difficulty' claim from the 'benchmark construction' claim.
  3. [Section 3.2.2 and Section 4.1] The paper excludes 289 of 594 BugsAqua mutants solely because their execution time exceeds one hour, and it uses the remaining 305 as the artificial benchmark. This is a selection on execution time, which can correlate with program size, loop structure, or the number of quantum operations. The paper does not compare the included and excluded mutants in terms of mutation operator type, program size, or the resulting EXAM scores. A sensitivity analysis is needed to show that the median EXAM results are stable under this exclusion. At a minimum, report the distribution of execution times and the characteristics of the excluded mutants.
  4. [Section 4.2 and Section 1 (Contribution 2)] RQ2 is answered only by counting behavior-changing mutants (Table 1), but the paper's contribution statement says 'quantum mutation operations are effective for MBFL.' Behavior-changing mutants are a necessary input to Eq. (1), but they do not directly imply lower EXAM scores; a mutation operator could produce many behavior-changing mutants that all localize to the wrong statement. To support the effectiveness claim, the authors should report EXAM scores broken down by mutation-operation type, or at least show that the statements mutated by quantum operators are more often the faulty ones. As written, RQ2 is about mutation strength, not fault-localization effectiveness.
  5. [Section 3.2.1 and Section 4.1] The real-world evaluation relies on 23 of the 42 Bugs4Q bugs, with the other 19 excluded because they could not be reproduced. The paper acknowledges this, but it does not assess whether the excluded bugs differ systematically from the included ones (e.g., in fault type, quantum versus classical root cause, or program size). Given that the central RQ1 conclusion is about real-world faults, the representativeness of the 23-bug subset is load-bearing. I request an analysis of the excluded cases from the Bugs4Q replication package, or a clear argument that the non-reproduction is orthogonal to the fault-localization behavior.
minor comments (7)
  1. [Abstract and Section 3.2.2] The abstract reports '305 artificially induced faults,' but Section 3.2.2 states that QMutPy generated 2,361 mutants and 594 had at least one failing test, after which the timeout exclusion left 305. The wording in the abstract should distinguish faults from mutants and mention the filtering step.
  2. [Section 4.1] There is a typo in the Answer to RQ1: 'Additionaly' should be 'Additionally.'
  3. [Section 3.2.2] The phrase 'a level that satisfies approximately 10% of practitioners' with reference [12] is unclear. Reference [12] is about practitioners' expectations on fault localization, not about execution timeout thresholds; please clarify how the one-hour timeout was derived from that source.
  4. [Section 3.4] Equation (1) does not define the behavior when f_P(s) is empty, which can occur for statements not executed by any failing test. It would be helpful to state the convention used in that case.
  5. [Section 5.2] The statement that SBFL 'cannot be applied' to Bugs4Q because there is only one failing test case is too strong. Ochiai and Tarantula can still be computed with a single failing test and multiple passing tests; the issue is that the resulting scores may be degenerate. Please rephrase to say that SBFL is not meaningful or not reliably applicable in this setting.
  6. [Table 2] The 'Sig. level' column uses '**' and '-' without a legend in the table caption. Please add a footnote explaining that '-' means p > 0.05.
  7. [Section 6 (Threats to Validity)] The threats section covers many of the concerns raised above, but it does not mention the circularity/reversibility issue in the construction of BugsAqua. Since the paper explicitly identifies reversible mutants as the explanation for RQ1, this threat should be acknowledged here as well.

Circularity Check

1 steps flagged · score 6.0 of 10

BugsAqua's artificial faults are generated by the same QMutPy operators used for MBFL, so the RQ1 1.2%-vs-19.4% gap is partly built into the benchmark construction.

  1. self definitional [Sections 3.2.2, 3.3, Eq. (1), and Section 4.1]
    "We then apply QMutPy to these 24 programs, generating 2,361 mutants. Among them, 594 mutants have at least one failing test case, indicating that the faults are successfully injected. ... The first step in MBFL is to apply mutation operations to the buggy programs. We use QMutPy to generate mutants. ... an artificial fault injected by replacing a quantum gate can be fixed by applying a mutation operation that replaces it back with the original quantum gate."

    BugsAqua's buggy programs are created by applying QMutPy mutation operators to correct Qiskit-Aqua programs, and MBFL then scores statements by applying the same QMutPy operator set to the buggy programs. For any artificial fault that is a one-step replacement, the correct original statement is reachable as a mutant of the faulty statement (the inverse replacement), so that mutant turns the failing test into a passing test. By Eq. (1), this gives the faulty statement a maximum suspiciousness contribution. Thus the low EXAM scores on BugsAqua are partly determined by benchmark construction, not by an intrinsic property of the faults.

full rationale

The only substantive circularity is in the artificial-benchmark arm. BugsAqua faults are QMutPy mutants of correct programs, and MBFL scores statements by whether QMutPy mutants of the buggy program turn failing tests into passing tests. An injected one-step replacement is therefore 'locatable' by construction: the original statement is one of its own mutants, which is why the paper's own 'reversible mutant' explanation aligns with Eq. (1). The Bugs4Q arm is independent, externally sourced from Zhao et al., and the real-world finding (median worst-case EXAM 19.4%) stands even if BugsAqua were discarded; consequently the paper is not wholly circular. The 19/42 Bugs4Q exclusions and single-test-per-bug weakness are representativeness and validity concerns, not circularity. Self-citations to Sato and Katsube [18] and Ishimoto et al. [7] are not load-bearing for the central claim. The RQ2 comparison of quantum vs classical mutation operations is descriptive and does not reduce to the input operation set in the same way. Overall, one arm of the central comparison is partially self-confirming by construction, giving a partial circularity score of 6.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

No equations are derived; this is an empirical study. The MUSE suspiciousness formula is taken from prior work. Key assumptions are representativeness of the surviving benchmarks, simulator-based execution, and the adequacy of single-test-case Bugs4Q tests. One study-design threshold, the 1-hour timeout, acts as a free parameter controlling which artificial faults are included.

free parameters (1)
  • Execution timeout for artificial mutants = 1 hour
    The authors set a 1-hour cap on mutant execution time (Section 3.2.2). 289 of 594 behavior-changing mutants were excluded by this rule. The reported EXAM distribution for BugsAqua depends on this hand-chosen threshold and may not generalize to longer-running programs.
assumptions (5)
  • standard math The MUSE suspiciousness formula (Equation 1) is a valid fault-localization score.
    The formula is adopted from Moon et al. [14] without independent derivation or validation for quantum programs.
  • domain assumption Behavior-changing mutants are reliable evidence that a statement is faulty.
    This is the core MBFL premise; it fails for faults such as missing initialization patterns (Listing 2), as the paper itself shows.
  • domain assumption Simulator execution faithfully represents quantum program behavior for debugging purposes.
    Section 3.6 runs all programs on a classical simulator because of noise and availability of real quantum computers; results may differ on hardware.
  • ad hoc to paper The 23 reproduced Bugs4Q bugs are representative of real-world quantum bugs.
    19 of 42 Bugs4Q bugs were excluded due to version mismatches (Section 3.2.1); if excluded bugs differ, real-world conclusions may be biased.
  • domain assumption A single manually written test per Bugs4Q bug is sufficient to capture mutant behavior.
    The paper notes Bugs4Q provides only one test per bug, written by benchmark authors, and suggests weak tests may partly explain poor MBFL performance (Section 4.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Mutation-based Fault Localization for Quantum Programs." pith.science (2026). https://pith.science/paper/4NDTS7T6

@misc{pith2026250509059,
  author       = {Pith},
  title        = {Pith review of: Evaluating Mutation-based Fault Localization for Quantum Programs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4NDTS7T6}},
  note         = {Machine review of arXiv:2505.09059}
}
read the original abstract

Quantum computers leverage the principles of quantum mechanics to execute operations. They require quantum programs that define operations on quantum bits (qubits), the fundamental units of computation. Unlike traditional software development, the process of creating and debugging quantum programs requires specialized knowledge of quantum computation, making the development process more challenging. In this paper, we apply and evaluate mutation-based fault localization (MBFL) for quantum programs with the aim of enhancing debugging efficiency. We use quantum mutation operations, which are specifically designed for quantum programs, to identify faults. Our evaluation involves 23 real-world faults and 305 artificially induced faults in quantum programs developed with Qiskit(R). The results show that real-world faults are more challenging for MBFL than artificial faults. In fact, the median EXAM score, which represents the percentage of the code examined before locating the faulty statement (lower is better), is 1.2% for artificial benchmark and 19.4% for the real-world benchmark in the worst-case scenario. Our study highlights the potential and limitations of MBFL for quantum programs, considering different fault types and mutation operation types. Finally, we discuss future directions for improving MBFL in the context of quantum programming.

Figures

Figures reproduced from arXiv: 2505.09059 by the authors.

Figure 1
Figure 1. Our study design. Mutation Analysis for Quantum Programs. Fortunato et al. [6] developed QMutpy, a mutation analysis tool for quantum programs written in Qiskit®. It extends Mutpy, an existing mutation tool for Python® programs. They introduced five types of quantum mutation operations: addition, deletion, and replacement of quantum gates, as well as addition and deletion of qubit measurements. Since QMutpy is a for… view at source ↗
Figure 2
Figure 2. Empirical cumulative distribution function of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages

  1. [1]

    Rui Abreu, Peter Zoeteweij, and Arjan JC Van Gemund. 2007. On the accuracy of spectrum-based fault localization. In Testing: Academic and industrial conference practice and research techniques-MUTATION (TAICPART-MUTATION 2007). IEEE, 89–98

    2007

  2. [2]

    Shaukat Ali, Paolo Arcaini, Xinyi Wang, and Tao Yue. 2021. Assessing the effectiveness of input and output coverage criteria for testing quantum programs. In Proceedings of the 14th IEEE Conference on Software Testing, Verification and Validation. 13–23

    work page 2021

  3. [3]

    Qihong Chen, Rúben Câmara, José Campos, André Souto, and Iftekhar Ahmed

    work page

  4. [4]

    Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin 114, 3 (1993), 494

    work page 1993

  5. [5]

    William Jay Conover. 1999. Practical nonparametric statistics. john wiley & sons

    work page 1999

  6. [6]

    Daniel Fortunato, Jose Campos, and Rui Abreu. 2022. Mutation testing of quantum programs: A case study with Qiskit. IEEE Transactions on Quantum Engineering 3 (2022), 1–17

    work page 2022

  7. [7]

    Yuta Ishimoto, Yuto Nakamura, Ryota Katsube, Naoto Sato, Hideto Ogawa, Masa- nari Kondo, Yasutaka Kamei, and Naoyasu Ubayashi. 2024. An Empirical Study on Self-Admitted Technical Debt in Quantum Software. In Proceedings of the 31st Asia-Pacific Software Engineering Conference (APSEC) . 41–50

    work page 2024

  8. [8]

    Yue Jia and Mark Harman. 2009. Higher order mutation testing. Information and Software Technology 51, 10 (2009), 1379–1393

    work page 2009

Show all 23 references
  1. [9]

    James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the taran- tula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering . 273–282

    2005

  2. [10]

    Xiaolin Ju, Shujuan Jiang, Xiang Chen, Xingya Wang, Yanmei Zhang, and Heling Cao. 2014. HSFal: Effective fault localization using hybrid spectrum of full slices and execution slices. Journal of Systems and Software 90 (2014), 3–17

    2014

  3. [11]

    Barbara Kitchenham, Lech Madeyski, David Budgen, Jacky Keung, Pearl Brereton, Stuart Charters, Shirley Gibbs, and Amnart Pohthong. 2017. Robust statistical methods for empirical software engineering. Empirical Software Engineering 22 (2017), 579–630

    2017

  4. [12]

    Pavneet Singh Kochhar, Xin Xia, David Lo, and Shanping Li. 2016. Practitioners’ expectations on automated fault localization. In Proceedings of the 25th interna- tional symposium on software testing and analysis . 165–176

    2016

  5. [13]

    Eñaut Mendiluze, Shaukat Ali, Paolo Arcaini, and Tao Yue. 2021. Muskit: A mutation analysis tool for quantum software testing. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering . 1266– 1270

    2021

  6. [14]

    Seokhyeon Moon, Yunho Kim, Moonzoo Kim, and Shin Yoo. 2014. Ask the mutants: Mutating faulty programs for fault localization. InProceedings of the IEEE Seventh International Conference on Software Testing, Verification and Validation . 153–162

    2014

  7. [15]

    2010.Quantum computation and quantum information

    Michael A Nielsen and Isaac L Chuang. 2010.Quantum computation and quantum information. Cambridge university press

    2010

  8. [16]

    Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D Ernst, Deric Pang, and Benjamin Keller. 2017. Evaluating and improving fault localization. In Proceedings of the IEEE/ACM 39th International Conference on Software Engineering. 609–620

    2017

  9. [17]

    Neilson Carlos Leite Ramalho, Higor Amario de Souza, and Marcos Lordello Chaim. 2024. Testing and Debugging Quantum Programs: The Road to 2030. arXiv preprint arXiv:2405.09178 (2024)

    2024 arXiv

  10. [18]

    Naoto Sato and Ryota Katsube. 2024. Locating Buggy Segments in Quantum Program Debugging. InProceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results . 26–31

    2024

  11. [19]

    Ruslan Shaydulin, Caleb Thomas, and Paige Rodeghero. 2020. Making Quantum Computing Open: Lessons from Open Source Projects. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops . 451–455

    2020

  12. [20]

    Xinyi Wang, Tongxuan Yu, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022. Mutation-based test generation for quantum programs with multi-objective search. In Proceedings of the genetic and evolutionary computation conference . 1345–1353

    2022

  13. [21]

    W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707–740

    2016

  14. [22]

    Pengzhan Zhao, Zhongtao Miao, Shuhan Lan, and Jianjun Zhao. 2023. Bugs4Q: A benchmark of existing bugs to enable controlled testing and debugging studies for quantum programs. Journal of Systems and Software 205 (2023), 111805

    2023

  15. [2023]

    InProceedings of the IEEE/ACM 45th International Conference on Software Engineering

    The smelly eight: An empirical study on the prevalence of code smells in quantum computing. InProceedings of the IEEE/ACM 45th International Conference on Software Engineering. 358–370

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.