Pith. sign in

REVIEW 3 major objections 1 minor 1 cited by

Reproducibility of Bugs4Q quantum defect dataset dropped from 62.2% to 16.2% on newer Qiskit versions due to dependency changes.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-26 03:27 UTC pith:CILLZC5F

load-bearing objection Bugs4Q reproducibility falls sharply with Qiskit updates, mostly from dependencies, and the patched Bugs4Q-Robust lifts it back up, but the 37-artifact sample and manual labels are the parts that need checking. the 3 major comments →

arxiv 2606.27124 v1 pith:CILLZC5F submitted 2026-06-25 cs.SE

On the Reproducibility of Quantum Software Defect Datasets: A Case Study of Bugs4Q

Haruto Ohto , Yuta Ishimoto , Shinsuke Matsumoto , Shinji Kusumoto This is my paper
classification cs.SE
keywords reproducibilityquantum softwaredefect datasetsBugs4QQiskitdependency managementreplication studydataset maintenance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper replicates earlier findings on classical defect datasets by testing the Bugs4Q collection of real quantum program bugs. It ran 77,700 executions of 37 artifacts across 21 versions of the Qiskit library. Reproducibility fell sharply over time, with nearly all failures tied to dependency problems. Unlike classical cases, fixing versions alone rarely worked and source code edits were usually needed. The authors released a patched Bugs4Q-Robust that raised success to 78.4 percent on the newest library.

Core claim

The reproducibility of Bugs4Q dropped from 62.2% on Qiskit v0.20.1 to 16.2% on v2.3.1. A manual inspection showed 93.6% of the failures were dependency-related. Most reproduction failures require source-code modifications such as migrating import paths and API invocations rather than just adjusting dependency versions. Bugs4Q-Robust, a patched version, increases reproducibility to 78.4% on Qiskit v2.3.1.

What carries the argument

The set of 37 Bugs4Q artifacts executed on successive Qiskit versions, with manual root-cause classification of reproduction failures.

Load-bearing premise

The 37 chosen Bugs4Q artifacts stand in for the entire dataset and that manual classification correctly identifies dependency issues as the main cause without interference from test environments or missed API shifts.

What would settle it

Executing the full set of Bugs4Q artifacts on Qiskit v2.3.1 and finding that fewer than 80 percent of the non-reproducible cases are fixed by the proposed patches or that a different root cause dominates.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Quantum defect datasets require ongoing maintenance to stay usable as core libraries evolve.
  • Dependency version adjustments alone are insufficient for restoring reproducibility in quantum software bugs.
  • Source-level patches for import paths and API calls can restore most reproducibility in Bugs4Q.
  • Research results based on outdated quantum defect datasets may not hold on current library versions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar decay patterns likely affect other quantum software datasets beyond Bugs4Q.
  • Tools that automatically update quantum code for new library versions could reduce maintenance effort.
  • Dataset creators in rapidly changing domains should plan for periodic re-validation and patching from the start.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 1 minor

Summary. The paper conducts a replication study of reproducibility issues in quantum defect datasets using Bugs4Q. It reports results from 77,700 executions of 37 artifacts across 21 Qiskit versions, showing reproducibility declining from 62.2% on v0.20.1 to 16.2% on v2.3.1 (as of April 2026), with manual root-cause analysis attributing 93.6% of failures to dependency issues. The authors introduce Bugs4Q-Robust, a patched version that raises reproducibility to 78.4% on the latest Qiskit, and argue for continuous maintenance of such datasets.

Significance. If the central empirical claims hold, the work usefully extends classical SE reproducibility studies (e.g., on Defects4J) to quantum software, documenting both the expected decay and a distinctive pattern where dependency pinning alone is insufficient and source-level API migrations are required. The scale of the experiment (77,700 runs) and the release of a patched dataset constitute concrete contributions that could support follow-on quantum SE research.

major comments (3)
  1. [Abstract / §3] Abstract and §3 (Dataset and Artifact Selection): The 37 Bugs4Q artifacts are the basis for all reported percentages (62.2%, 16.2%, 93.6%, 78.4%), yet no selection protocol, sampling frame, or comparison to the full Bugs4Q corpus is provided; without this, the generalizability of the headline reproducibility figures cannot be assessed.
  2. [§5] §5 (Root Cause Analysis): The claim that 93.6% of failures are dependency-related rests on manual classification, but the manuscript supplies neither inter-rater agreement statistics nor explicit classification criteria or decision rules; this directly affects the reliability of the root-cause distribution that motivates the creation of Bugs4Q-Robust.
  3. [§4] §4 (Experimental Setup): The description of how the execution environment was held constant (Python version, secondary packages, random seeds, test harness) while only the core Qiskit library varied is absent; uncontrolled confounders could inflate the dependency-related failure count and undermine the attribution used for the 93.6% figure.
minor comments (1)
  1. [Abstract] The abstract states the latest version date as April 1, 2026; clarify whether this is a projected or actual date and ensure consistency with the Qiskit release timeline referenced in the methods.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback on our replication study. We address each major comment below, indicating where revisions will be made to improve clarity and rigor without altering the core empirical claims.

read point-by-point responses

  1. Referee: [Abstract / §3] Abstract and §3 (Dataset and Artifact Selection): The 37 Bugs4Q artifacts are the basis for all reported percentages (62.2%, 16.2%, 93.6%, 78.4%), yet no selection protocol, sampling frame, or comparison to the full Bugs4Q corpus is provided; without this, the generalizability of the headline reproducibility figures cannot be assessed.

    Authors: We agree that §3 lacks an explicit selection protocol. The 37 artifacts comprise the complete set of Bugs4Q bugs whose source code could be statically parsed and dynamically executed under our test harness (i.e., those without immediate non-version-related syntax issues). In the revision we will expand §3 with (1) the total size of the original Bugs4Q corpus, (2) the precise inclusion criteria, and (3) a brief comparison table of bug types and Qiskit API usage between the selected subset and the full corpus. This addition will directly address generalizability concerns. revision: yes

  2. Referee: [§5] §5 (Root Cause Analysis): The claim that 93.6% of failures are dependency-related rests on manual classification, but the manuscript supplies neither inter-rater agreement statistics nor explicit classification criteria or decision rules; this directly affects the reliability of the root-cause distribution that motivates the creation of Bugs4Q-Robust.

    Authors: The classification was performed by a single author using a rule-based scheme derived from error-message patterns (ImportError/ModuleNotFoundError and version-mismatch warnings counted as dependency-related; AttributeError on Qiskit objects counted as API-migration-related). We will add the full decision rules as an appendix and note the single-rater limitation in the revised §5. While inter-rater agreement statistics cannot be retroactively computed, the raw execution logs and classifications will be released with the artifact to permit independent verification. revision: partial

  3. Referee: [§4] §4 (Experimental Setup): The description of how the execution environment was held constant (Python version, secondary packages, random seeds, test harness) while only the core Qiskit library varied is absent; uncontrolled confounders could inflate the dependency-related failure count and undermine the attribution used for the 93.6% figure.

    Authors: We acknowledge the description in §4 is incomplete. All runs used a fixed Python 3.9.7 base, pinned secondary packages (numpy==1.23.5, scipy==1.9.3, etc.), and a Dockerized harness that set random seeds for any stochastic Qiskit operations. Only the Qiskit wheel was swapped via pip. The revised §4 will include the full environment specification, harness source, and a statement that no other variables were altered. This should eliminate the possibility of confounding the 93.6% attribution. revision: yes

Circularity Check

0 steps flagged

Empirical replication study with no circular derivations or self-referential steps

full rationale

This is a direct empirical replication study that reports measured reproducibility rates (62.2% to 16.2%) from 77,700 program executions across Qiskit versions, followed by manual root-cause classification (93.6% dependency-related) and creation of a patched dataset. No equations, fitted parameters, predictions derived from inputs, uniqueness theorems, or ansatzes appear. The sole external citation is to Zhu et al. (non-overlapping authors) for context on classical datasets; the present results rest on execution logs and inspection rather than any self-citation chain or definitional reduction. The selection of 37 artifacts and manual labels are methodological choices whose validity is external to any derivation, so the paper's claims do not reduce to their own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

This is an empirical case study and does not introduce new mathematical models or entities. It relies on the assumption that the chosen artifacts represent the dataset and that execution results accurately reflect reproducibility.

axioms (1)
  • domain assumption The 37 Bugs4Q artifacts are representative for assessing overall dataset reproducibility.
    The study uses these to draw general conclusions about the dataset.

pith-pipeline@v0.9.1-grok · 5837 in / 1341 out tokens · 65306 ms · 2026-06-26T03:27:42.032153+00:00 · methodology

0 comments
read the original abstract

The reproducibility of software defect datasets is essential for obtaining reliable and comparable research results. Zhu et al. have shown that defect datasets such as Defects4J suffer from reproduction failures (i.e., reported bugs become non-reproducible) as time passes since their creation. However, it remains unclear whether these findings generalize to quantum software defect datasets. We therefore conduct a replication study of the prior work using Bugs4Q, a widely used dataset of real-world bugs in quantum programs. Our analysis includes 77,700 quantum program executions of 37 Bugs4Q artifacts across 21 core-library versions. The experimental results showed that the reproducibility of Bugs4Q dropped from 62.2% on Qiskit v0.20.1 to 16.2% on v2.3.1, the latest version as of April 1, 2026. A manual inspection of the root causes further indicated that 93.6% of the failures were dependency-related. While these findings are consistent with those of the prior work, we also observed differences. In particular, most reproduction failures in Bugs4Q cannot be resolved merely by adjusting dependency versions; instead, they require source-code modifications such as migrating import paths and API invocations. Based on this observation, we curated Bugs4Q-Robust, a patched version of Bugs4Q to restore reproducibility. Bugs4Q-Robust increases reproducibility from 16.2% to 78.4% on Qiskit v2.3.1. Our findings highlight the importance of continuous dataset maintenance in the rapidly evolving quantum software ecosystem.

Figures

Figures reproduced from arXiv: 2606.27124 by Haruto Ohto, Shinji Kusumoto, Shinsuke Matsumoto, Yuta Ishimoto.

Figure 1
Figure 1. Figure 1: An overview of our experiment. Type Match. It additionally requires that the observed failure type matches the one described in the original source of each Bugs4Q artifact, such as the corresponding Stack Overflow post. For artifacts categorized as TE, Type Match is satisfied when the buggy version raises the same exception class as the one described in the source. For artifacts catego￾rized as WO, the exp… view at source ↗
Figure 2
Figure 2. Figure 2: Reproducibility for each snapshot under the three criteria. The value above each bar indicates the percentage of [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Number of newly broken artifacts for each criterion [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Benchmarking Large Language Models on Repairing Qiskit Programs using Bugs4Q

    cs.SE 2026-07 conditional novelty 6.0

    Bugs4Q validity is version-dependent; most LLM repair passes land on invalid entries, so version-pinned validation must precede quantum APR evaluation.

Reference graph

Works this paper leans on

52 extracted references · 6 canonical work pages · cited by 1 Pith paper · 3 internal anchors

  1. [1]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” inProceed- ings of the International Symposium on Software Testing and Analysis, 2014, pp. 437–440

    2014

  2. [2]

    The manybugs and introclass benchmarks for automated repair of c programs,

    C. Le Goues, N. Holtschulte, E. K. Smith, Y . Brun, P. Devanbu, S. Forrest, and W. Weimer, “The manybugs and introclass benchmarks for automated repair of c programs,”IEEE Transactions on Software Engineering, vol. 41, no. 12, pp. 1236–1256, 2015

    2015

  3. [3]

    Bugswarm: Mining and continuously growing a dataset of reproducible failures and fixes,

    D. A. Tomassi, N. Dmeiri, Y . Wang, A. Bhowmick, Y .-C. Liu, P. T. Devanbu, B. Vasilescu, and C. Rubio-Gonz ´alez, “Bugswarm: Mining and continuously growing a dataset of reproducible failures and fixes,” inProceedings of the 41st International Conference on Software Engi- neering, 2019, pp. 339–349

    2019

  4. [4]

    SWE-bench: Can Language Models Resolve Real-World GitHub Issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?”arXiv preprint arXiv:2310.06770, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges,

    S. Shamshiri, R. Just, J. M. Rojas, G. Fraser, P. McMinn, and A. Arcuri, “Do automatically generated unit tests find real faults? an empirical study of effectiveness and challenges,” inProceedings of the 30th IEEE/ACM International Conference on Automated Software Engineer- ing, 2015, pp. 201–211

    2015

  6. [6]

    Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults,

    M. Papadakis, D. Shin, S. Yoo, and D.-H. Bae, “Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults,” inProceedings of the 40th International Conference on Software Engineering, 2018, pp. 537–548

    2018

  7. [7]

    Defect prediction guided search-based software testing,

    A. Perera, A. Aleti, M. B ¨ohme, and B. Turhan, “Defect prediction guided search-based software testing,” inProceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, 2020, pp. 448–460

    2020

  8. [8]

    A survey on software fault localization,

    W. E. Wong, R. Gao, Y . Li, R. Abreu, and F. Wotawa, “A survey on software fault localization,”IEEE Transactions on Software Engineering, vol. 42, no. 8, pp. 707–740, 2016

    2016

  9. [9]

    A quantitative and qualitative evaluation of llm-based explainable fault localization,

    S. Kang, G. An, and S. Yoo, “A quantitative and qualitative evaluation of llm-based explainable fault localization,”Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 1424–1446, 2024

    2024

  10. [10]

    Soapfl: A standard operating procedure for llm-based method-level fault localization,

    Y . Qin, S. Wang, Y . Lou, J. Dong, K. Wang, X. Li, and X. Mao, “Soapfl: A standard operating procedure for llm-based method-level fault localization,”IEEE Transactions on Software Engineering, vol. 51, no. 4, pp. 1173–1187, 2025

    2025

  11. [11]

    Automatic software repair: A bibliography,

    M. Monperrus, “Automatic software repair: A bibliography,”ACM Computing Surveys, vol. 51, no. 1, pp. 1–24, 2018

    2018

  12. [12]

    Automated program repair,

    C. Le Goues, M. Pradel, and A. Roychoudhury, “Automated program repair,”Communications of the ACM, vol. 62, no. 12, pp. 56–65, 2019

    2019

  13. [13]

    Evolv- ing paradigms in automated program repair: Taxonomy, challenges, and opportunities,

    K. Huang, Z. Xu, S. Yang, H. Sun, X. Li, Z. Yan, and Y . Zhang, “Evolv- ing paradigms in automated program repair: Taxonomy, challenges, and opportunities,”ACM Computing Surveys, vol. 57, no. 2, pp. 1–43, 2024

    2024

  14. [14]

    On the reproducibility of software defect datasets,

    H.-N. Zhu and C. Rubio-Gonz ´alez, “On the reproducibility of software defect datasets,” inProceedings of the IEEE/ACM 45th International Conference on Software Engineering, 2023, pp. 2324–2335

    2023

  15. [15]

    M. A. Nielsen and I. L. Chuang,Quantum computation and quantum information. Cambridge university press, 2010

    2010

  16. [16]

    Quantum software engineering: Landscapes and horizons,

    J. Zhao, “Quantum software engineering: Landscapes and horizons,” arXiv preprint arXiv:2007.07047, 2020

    work page arXiv 2007

  17. [17]

    Testing and debugging quantum programs: The road to 2030,

    N. C. Leite Ramalho, H. Amario de Souza, and M. Lordello Chaim, “Testing and debugging quantum programs: The road to 2030,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, 2025

    2030

  18. [18]

    A comprehensive study of bug fixes in quantum programs,

    J. Luo, P. Zhao, Z. Miao, S. Lan, and J. Zhao, “A comprehensive study of bug fixes in quantum programs,” inProceedings of the IEEE Interna- tional Conference on Software Analysis, Evolution and Reengineering, 2022, pp. 1239–1246

    2022

  19. [19]

    Bugs4q: A benchmark of existing bugs to enable controlled testing and debugging studies for quantum programs,

    P. Zhao, Z. Miao, S. Lan, and J. Zhao, “Bugs4q: A benchmark of existing bugs to enable controlled testing and debugging studies for quantum programs,”Journal of Systems and Software, vol. 205, no. 111805, 2023

    2023

  20. [20]

    QMutBench: A Dataset of Quantum Circuit Mutants

    E. M. Usandizaga, T. Laurent, P. Arcaini, and S. Ali, “Qmutbench: A dataset of quantum circuit mutants,”arXiv preprint arXiv:2604.15870, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  21. [21]

    The smelly eight: An empirical study on the prevalence of code smells in quan- tum computing,

    Q. Chen, R. C ˆamara, J. Campos, A. Souto, and I. Ahmed, “The smelly eight: An empirical study on the prevalence of code smells in quan- tum computing,” inProceedings of the IEEE/ACM 45th International Conference on Software Engineering, 2023, pp. 358–370

    2023

  22. [22]

    An empirical study into the effects of transpilation on quantum circuit smells,

    M. D. Stefano, D. D. Nucci, F. Palomba, and A. D. Lucia, “An empirical study into the effects of transpilation on quantum circuit smells,”Empirical Software Engineering, vol. 29, no. 3, p. 61, 2024

    2024

  23. [23]

    Technical debts and faults in open-source quantum software systems: An empirical study,

    M. Openja, M. M. Morovati, L. An, F. Khomh, and M. Abidi, “Technical debts and faults in open-source quantum software systems: An empirical study,”Journal of Systems and Software, vol. 193, p. 111458, 2022

    2022

  24. [24]

    An empirical study on self-admitted technical debt in quantum software,

    Y . Ishimoto, Y . Nakamura, R. Katsube, N. Sato, H. Ogawa, M. Kondo, Y . Kamei, and N. Ubayashi, “An empirical study on self-admitted technical debt in quantum software,” inProceedings of the 31st Asia- Pacific Software Engineering Conference, 2024, pp. 41–50

    2024

  25. [25]

    Identifying bug patterns in quantum pro- grams,

    P. Zhao, J. Zhao, and L. Ma, “Identifying bug patterns in quantum pro- grams,” inProceedings of the IEEE/ACM 2nd International Workshop on Quantum Software Engineering, 2021, pp. 16–21

    2021

  26. [26]

    Bugs in quantum computing platforms: an empirical study,

    M. Paltenghi and M. Pradel, “Bugs in quantum computing platforms: an empirical study,”Proceedings of the ACM on Programming Languages, vol. 6, no. OOPSLA1, pp. 1–27, 2022

    2022

  27. [27]

    Understanding replication of experiments in software engineering: A classification,

    O. S. G ´omez, N. Juristo, and S. Vegas, “Understanding replication of experiments in software engineering: A classification,”Information and Software Technology, vol. 56, no. 8, pp. 1033–1048, 2014

    2014

  28. [28]

    Automatic repair of quantum programs via unitary operation,

    Y . Li, H. Pei, L. Huang, B. Yin, and K.-Y . Cai, “Automatic repair of quantum programs via unitary operation,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 6, pp. 1–43, 2024

    2024

  29. [29]

    On repairing quantum programs using chatgpt,

    X. Guo, J. Zhao, and P. Zhao, “On repairing quantum programs using chatgpt,” inProceedings of the 5th ACM/IEEE International Workshop on Quantum Software Engineering, 2024, pp. 9–16

    2024

  30. [30]

    Hornbro: Homotopy-like method for automated quantum program repair,

    S. Tan, L. Lu, D. Xiang, T. Chu, C. Lang, J. Chen, X. Hu, and J. Yin, “Hornbro: Homotopy-like method for automated quantum program repair,”Proceedings of the ACM on Software Engineering, vol. 2, no. FSE, pp. 734–756, 2025

    2025

  31. [31]

    Bug-locating method based on statistical testing for quantum programs,

    N. Sato and R. Katsube, “Bug-locating method based on statistical testing for quantum programs,”IEEE Transactions on Software Engi- neering, vol. 51, no. 10, pp. 2804–2829, 2025

    2025

  32. [32]

    Evaluating mutation-based fault localization for quantum programs,

    Y . Ishimoto, M. Kondo, N. Ubayashi, Y . Kamei, R. Katsube, N. Sato, and H. Ogawa, “Evaluating mutation-based fault localization for quantum programs,” inProceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, 2025, pp. 666– 671

    2025

  33. [33]

    Leveraging mutation analysis for llm-based repair of quantum programs,

    C. Yoshida, Y . Ishimoto, O. Nourry, M. Kondo, M. Matsushita, Y . Kamei, and Y . Higo, “Leveraging mutation analysis for llm-based repair of quantum programs,” inProceedings of the 33rd International Conference on Software Analysis, Evolution and Reengineering, 2026, pp. 92–97

    2026

  34. [34]

    Quantum computing with Qiskit

    A. Javadi-Abhari, M. Treinish, K. Krsulich, C. J. Wood, J. Lishman, J. Gacon, S. Martiel, P. D. Nation, L. S. Bishop, A. W. Crosset al., “Quantum computing with qiskit,”arXiv preprint arXiv:2405.08810, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  35. [35]

    Identifying flakiness in quantum programs,

    L. Zhang, M. Radnejad, and A. Miranskyy, “Identifying flakiness in quantum programs,” inProceedings of the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2023, pp. 1–7

    2023

  36. [36]

    Identifying flaky tests in quantum code: A machine learning approach,

    K. Kaur, D. Kim, A. Jamshidi, and L. Zhang, “Identifying flaky tests in quantum code: A machine learning approach,” inProceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering-Companion, 2025, pp. 158–165

    2025

  37. [37]

    A practical guide for using statistical tests to assess randomized algorithms in software engineering,

    A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” inProceedings of the 33rd International Conference on Software Engineering, 2011, pp. 1–10

    2011

  38. [38]

    A survey of flaky tests,

    O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, “A survey of flaky tests,”ACM Transactions on Software Engineering and Methodol- ogy, vol. 31, no. 1, pp. 1–74, 2021

    2021

  39. [39]

    Cardinal, I

    J. Cardinal, I. Benzarti, C. Pereet al., “Migrating qaoa from qiskit 1. x to 2. x: An experience report,”arXiv preprint arXiv:2512.08245, 2025

    work page arXiv 2025

  40. [40]

    Taxonomy of migration scenarios for qiskit refactoring using llms,

    J. M. Su ´arez, L. M. Bibb ´o, J. Bogado, and A. Fernandez, “Taxonomy of migration scenarios for qiskit refactoring using llms,”arXiv preprint arXiv:2506.07135, 2025

    work page arXiv 2025

  41. [41]

    Qiskit 1.0 migration guide,

    IBM Quantum, “Qiskit 1.0 migration guide,” https://docs.quantum.ibm .com/migration-guides/qiskit-1.0, 2024

    2024

  42. [42]

    Qiskit v2.0 migration guide,

    ——, “Qiskit v2.0 migration guide,” https://quantum.cloud.ibm.com/do cs/migration-guides/qiskit-2.0, 2025

    2025

  43. [43]

    Charmaz,Constructing grounded theory: A practical guide through qualitative analysis

    K. Charmaz,Constructing grounded theory: A practical guide through qualitative analysis. sage, 2006

    2006

  44. [44]

    Measuring nominal scale agreement among many raters

    J. L. Fleiss, “Measuring nominal scale agreement among many raters.” Psychological bulletin, vol. 76, no. 5, p. 378, 1971

    1971

  45. [45]

    Understanding interobserver agreement: the kappa statistic,

    A. J. Viera, J. M. Garrettet al., “Understanding interobserver agreement: the kappa statistic,”Fam med, vol. 37, no. 5, pp. 360–363, 2005

    2005

  46. [46]

    Statistical assertions for validating patterns and finding bugs in quantum programs,

    Y . Huang and M. Martonosi, “Statistical assertions for validating patterns and finding bugs in quantum programs,” inProceedings of the 46th International Symposium on Computer Architecture, 2019, pp. 541–553

    2019

  47. [47]

    From bugs to benchmarks: A comprehensive survey of software defect datasets,

    H.-N. Zhu, R. Furth, M. Pradel, and C. Rubio-Gonz ´alez, “From bugs to benchmarks: A comprehensive survey of software defect datasets,” ACM Computing Surveys, vol. 58, no. 10, 2026

    2026

  48. [48]

    An empirical study of bugs in quantum machine learning frameworks,

    P. Zhao, X. Wu, J. Luo, Z. Li, and J. Zhao, “An empirical study of bugs in quantum machine learning frameworks,” inProceedings of the IEEE International Conference on Quantum Software (QSW), 2023, pp. 68–75

    2023

  49. [49]

    Mutation testing of quantum programs: A case study with qiskit,

    D. Fortunato, J. Campos, and R. Abreu, “Mutation testing of quantum programs: A case study with qiskit,”IEEE Transactions on Quantum Engineering, vol. 3, pp. 1–17, 2022

    2022

  50. [50]

    Reproducing and improving the bugsinpy dataset,

    F. Aguilar, S. Grayson, and D. Marinov, “Reproducing and improving the bugsinpy dataset,” inProceedings of the IEEE 23rd International Working Conference on Source Code Analysis and Manipulation, 2023, pp. 260–264

    2023

  51. [51]

    Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies,

    R. Widyasari, S. Q. Sim, C. Lok, H. Qi, J. Phan, Q. Tay, C. Tan, F. Wee, J. E. Tan, Y . Yiehet al., “Bugsinpy: a database of existing bugs in python programs to enable controlled testing and debugging studies,” inProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ...

    2020

  52. [52]

    When quantum meets classical: Characterizing hybrid quantum-classical issues discussed in developer forums,

    J. Zappin, T. Stalnaker, O. Chaparro, and D. Poshyvanyk, “When quantum meets classical: Characterizing hybrid quantum-classical issues discussed in developer forums,” inProceedings of the IEEE/ACM 47th International Conference on Software Engineering, 2025, pp. 2931– 2943

    2025