Pith. sign in

REVIEW 2 major objections 4 minor 41 references

Only about a quarter of quantum computing papers share runnable code, and most of those packages fail in a clean environment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 08:56 UTC pith:JM3R6BQA

load-bearing objection Solid empirical baseline: ~25% code availability and ~65% clean-environment failure in QC papers, with manual and automated numbers that line up and a concrete failure taxonomy. the 2 major comments →

arxiv 2607.08348 v1 pith:JM3R6BQA submitted 2026-07-09 quant-ph

Works on My QPU: Reproducibility in Quantum Computing Research

Dominik K\"oster , Maja Franz , Benjamin Zec , Nicole Hoess , Ralf Ramsauer , Wolfgang Mauerer This is my paper
classification quant-ph
keywords reproducibilityquantum computingquantum softwareresearch artefactssoftware environmentsempirical studyenvironment specificationexecutability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Quantum computing research relies on complex, fast-changing software stacks and specialised hardware, yet published results are often hard to re-run. This paper measures that gap with a five-question checklist applied by hand to 127 carefully filtered NISQ-era papers and by automated screening to nearly 5000 papers in the broader quantum algorithm and software literature. It finds that only roughly one quarter of papers supply accessible code, that many of those repositories omit machine-readable environment specifications, and that among the packages that could be tested, nearly two-thirds fail to execute successfully even in a freshly provisioned environment. The failures are traced to concrete, recurring problems: unpinned or missing dependencies, hardcoded local paths, incomplete documentation, non-pinned container tags, and missing hardware or licence details. The authors argue that these problems are not only long-term drift but often incomplete capture of the original development context, and they offer practical recommendations plus a reusable reproduction package template to make environments explicit and re-creatable.

Core claim

Reproducibility is not yet consistently achieved in quantum computing research. Across a manual sample of 127 papers only 24.4 percent provide code artefacts, and 64.5 percent of those artefacts fail to run in a clean environment; a large-scale automated screen of nearly 5000 papers yields a matching code-availability rate of 26.8 percent and shows that roughly one-third of accessible repositories lack machine-readable environment specifications.

What carries the argument

A five-question reproducibility assessment framework (code availability, environment specification, documentation, hardware description, and executability) applied first by manual validation on a filtered sample and then by automated full-text and repository screening on a much larger corpus.

Load-bearing premise

The multi-stage filters used to build the manual sample (NISQ keyword, DOI as peer-review proxy, presence in both literature databases, experimental-language keywords) do not systematically bias code-sharing or executability rates away from the broader quantum computing literature.

What would settle it

Re-run the same five-question framework on a new random sample of recent experimental quantum computing papers drawn without the NISQ or dual-database filters; if code availability substantially exceeds roughly 25 percent and clean-environment success substantially exceeds roughly 35 percent, the central empirical claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper presents a combined manual and automated empirical study of reproducibility practices in quantum computing research. From a multi-stage filtered sample of 127 NISQ-era experimental papers (drawn from a 4966-paper corpus spanning 2021–2026), the authors apply a five-question framework (code availability, environment specification, documentation, hardware specification, executability). Only 24.4% of the sample provide accessible code; of those packages, 64.5% fail clean-environment execution. An automated screen of the full corpus yields a consistent code-availability rate of 26.8% and shows that roughly one-third of repositories with code lack machine-readable environment files. Six recurring failure modes are catalogued, and the authors supply practical recommendations plus a reusable reproduction package (Makefile/Docker plus an exploratory Nix flake) that operationalises those recommendations.

Significance. If the measured rates hold, the work supplies the first large-scale, dual-method quantification of artefact availability and short-term executability in QC software research. The close agreement between the curated manual sample (24.4%) and the unfiltered automated corpus (26.8%) strengthens the claim that the headline figures are not artefacts of the NISQ/DOI/dual-database filters. The explicit failure taxonomy and the shipped analysis pipeline (GitHub) make the results independently re-verifiable and give the community concrete, actionable targets. The recommendations and template package further convert diagnosis into practice, which is of immediate value for authors, reviewers, and programme committees.

major comments (2)
  1. Section II-C.5 (RQ5) and the six failure modes: the executability claim rests on 31 packages, of which only a subset satisfied both RQ2 and RQ3 before the clean-environment attempt. The paper should state the exact number of packages that were actually executed (and the success/failure counts for each of the six modes) so that readers can judge the statistical weight of the 64.5% figure. A short table or appendix listing the 31 repositories and their RQ2–RQ5 outcomes would make the result fully auditable without altering the central claim.
  2. Section II-B.2 and Fig. 1: the multi-stage filter (NISQ keyword, DOI proxy, dual-database presence, experimental-language keywords) is acknowledged, yet the paper never reports a sensitivity check that relaxes any single filter and recomputes the code-availability rate. While the automated 4966-paper screen already mitigates the most serious bias concern, a one-paragraph sensitivity note (or a supplementary table) would close the residual sampling objection cleanly.
minor comments (4)
  1. Table II: the “N/A” column mixes “no code available” with “not applicable because classical simulation only.” A footnote clarifying the two meanings would improve readability.
  2. Section IV: the Nix flake is described as “exploratory.” A single sentence stating whether the flake was used to re-execute any of the 31 packages (or only the authors’ own pipeline) would set expectations correctly.
  3. References [36] and [17] are arXiv preprints dated 2026; ensure the final version cites the most recent stable identifiers or DOIs if they become available.
  4. Fig. 1 pipeline diagram: the arrow labels “249” and “127” are slightly hard to parse at a glance; adding the filter names next to the numbers would help.

Circularity Check

0 steps flagged

No significant circularity: headline rates are direct empirical counts from PDF/repository screening, not derived by construction from fitted inputs or load-bearing self-citations.

full rationale

The paper's central claims (code availability 24.4%/26.8%, ~64.5% of available packages failing clean-environment execution, ~1/3 lacking machine-readable environment specs) are obtained by keyword/full-text screening of arXiv+Semantic Scholar corpora, manual validation of a 127-paper NISQ subset, and automated checks on ~4966 papers (Section II, Fig. 1, Tab. II). These are observational tallies of presence/absence of artefacts and of whether packages ran under the authors' clean-environment protocol; they do not rest on any equation that defines a quantity in terms of itself, any parameter fitted to a subset then re-presented as a prediction, or any uniqueness theorem. Self-citations (e.g., prior Docker/Nix reproducibility templates by overlapping authors) appear only as background motivation and as the authors' own recommended mitigation; they are not used to force or justify the measured rates. The multi-stage filter is a sampling choice whose possible bias is cross-checked by the unfiltered large-scale run yielding a nearly identical availability rate; that check is independent evidence, not circular. No step reduces by construction to its inputs. Honest non-finding: score 0, empty steps.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The paper is an empirical measurement study. It inherits standard assumptions of literature mining and clean-environment testing but introduces no free parameters fitted to data and no new physical or mathematical entities. The main load-bearing premises are domain conventions about what constitutes a usable reproduction package.

axioms (3)
  • domain assumption Public code plus a machine-readable environment specification is a necessary precondition for computational reproducibility of QC software experiments.
    Stated in the introduction and used to justify the five-question framework (Table I).
  • domain assumption A DOI is a reasonable proxy for peer review when filtering the paper corpus.
    Explicitly used in the paper-filtering stage (Section II-B).
  • ad hoc to paper Keyword presence of 'NISQ', 'experiment', 'numerical', etc., sufficiently isolates experimental QC papers for the manual sample.
    Defines the 127-paper curated set; the authors note it is a pragmatic filter rather than a gold-standard classifier.

pith-pipeline@v1.1.0-grok45 · 15184 in / 2020 out tokens · 24740 ms · 2026-07-10T08:56:16.242137+00:00 · methodology

0 comments
read the original abstract

Quantum computing research increasingly depends on complex software stacks, yet the reproducibility of published results does not receive the priority and longevity mandated by recommendations of large international scientific bodies and best practices in software-centric systems research. In this paper, we present a combined manual and automated large-scale analysis of the reproducibility landscape in quantum computing research, quantify shortcomings, and derive actionable steps forward. We manually evaluate a curated sample of 127 papers using a five-question framework that covers code availability, environment specification, documentation, hardware description, and executability. To place these findings in a broader context, we conduct an automated large-scale screening of nearly 5000 quantum computing papers for the same reproducibility indicators. Our manual analysis reveals that only 24.4% of the sampled papers provide code artefacts, and among those, 64.5% fail to execute successfully in a clean environment. This assessment is corroborated by a large-scale automated analysis that yields a consistent code availability rate of 26.8%. Further, it shows that approximately one-third of the papers with accessible code lack machine-readable environment specifications. The results in this paper indicate that reproducibility is not yet consistently achieved in quantum computing research. In response, we outline a set of practical recommendations that address the observed failure modes and illustrate how reproducibility can be improved in practice.

Figures

Figures reproduced from arXiv: 2607.08348 by Benjamin Zec, Dominik K\"oster, Maja Franz, Nicole Hoess, Ralf Ramsauer, Wolfgang Mauerer.

Figure 1
Figure 1. Figure 1: Papers are retrieved via a structured query from arXiv and Semantic Scholar, merged and deduplicated. Subsequently [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 41 canonical work pages · 4 internal anchors

  1. [1]

    J. M. Gonz ´alez-Barahona and G. Robles, On the re- producibility of empirical software engineering studies based on data retrieved from development repositories, EMSE, 2012

    work page 2012

  2. [2]

    Mauerer, S

    W. Mauerer, S. Klessinger, and S. Scherzinger, Beyond the badge: Reproducibility engineering as a lifetime skill, inProc. SEENG, 2022

    work page 2022

  3. [3]

    Trautsch, S

    F. Trautsch, S. Herbold, P. Makedonski, et al., Ad- dressing problems with replicability and validity of repository mining studies through a smart data platform, EMSE, 2018

    work page 2018

  4. [4]

    J. M. Gonzalez-Barahona and G. Robles, Revisiting the reproducibility of empirical software engineering studies based on data retrieved from development repos- itories,IST, 2023

    work page 2023

  5. [5]

    J. T. Liang, C. Badea, C. Bird, et al., Can GPT-4 Replicate Empirical Software Engineering Research? PACMSE, 2024

    work page 2024

  6. [6]

    T. Yue, W. Mauerer, S. Ali, et al., Challenges and oppor- tunities in quantum software architecture, inSoftware Architecture: Research Roadmaps from the Community, 2023

    work page 2023

  7. [7]

    Senapati, Z

    P. Senapati, Z. Wang, W. Jiang, et al., Towards Redefin- ing the Reproducibility in Quantum Computing: A Data Analysis Approach on NISQ Devices, inProc. QCE, 2023

    work page 2023

  8. [8]

    Carbonelli, M

    C. Carbonelli, M. Felderer, M. Jung, et al., Challenges for quantum software engineering: An industrial ap- plication scenario perspective, inQuantum Software: Aspects of Theory and System Design, 2024

    work page 2024

  9. [9]

    Munasinghe, K

    T. Munasinghe, K. A. Cornell, J. Hendler, et al., A Knowledge-Based System for Managing Hardware De- pendency and Reproducibility in Quantum Machine Learning Workflows, inProc. IEEE BigData, 2025

    work page 2025

  10. [10]

    Mauerer and S

    W. Mauerer and S. Scherzinger, 1-2-3 Reproducibility for Quantum Software Experiments, inProc. SANER, 2022

    work page 2022

  11. [11]

    Malka, S

    J. Malka, S. Zacchiroli, and T. Zimmermann, Repro- ducibility of Build Environments through Space and Time, inProc. ICSE-NIER, 2024

    work page 2024

  12. [12]

    Ramsauer and W

    R. Ramsauer and W. Mauerer, Towards system-level quantum-accelerator integration, inProc. QCE, 2025

    work page 2025

  13. [13]

    Younis and C

    E. Younis and C. Iancu, Quantum Circuit Optimization and Transpilation via Parameterized Circuit Instantia- tion, inProc. QCE, 2022

    work page 2022

  14. [14]

    Greiwe, T

    F. Greiwe, T. Kr ¨uger, and W. Mauerer, Effects of imper- fections on quantum algorithms: A software engineering perspective, inProc. QSW, 2023

    work page 2023

  15. [15]

    Kr ¨uger and W

    T. Kr ¨uger and W. Mauerer, Out of the Loop: Structural Approximation of Optimisation Landscapes and non- Iterative Quantum Optimisation,Quantum, 2025

    work page 2025

  16. [16]

    Schmidbauer, E

    L. Schmidbauer, E. Lobe, I. Schaefer, et al., It’s quick to be square: Fast quadratisation for quantum toolchains, TQC, 2026

    work page 2026

  17. [17]

    I. M. Veiga and E. H ¨anggi, Reproducible builds for quantum computing, arXiv:2510.02251, 2025

    work page arXiv 2025

  18. [18]

    Y . Shi, P. Gokhale, P. Murali, et al., Resource-Efficient Quantum Computing by Breaking Abstractions,Proc. IEEE, 2020

    work page 2020

  19. [19]

    Dasgupta and T

    S. Dasgupta and T. Humble, Impact of Unreliable Devices on Stability of Quantum Computations,TQC, 2024

    work page 2024

  20. [20]

    Thelen, H

    S. Thelen, H. Safi, and W. Mauerer, Approximating under the influence of quantum noise and compute power, inProc. QCE, 2024

    work page 2024

  21. [21]

    S. R. Maschek, J. Schwittalla, M. Franz, et al., Make some noise! measuring noise model quality in real- world quantum software, inProc. QSW, 2025

    work page 2025

  22. [22]

    Quantum computing with Qiskit

    A. Javadi-Abhari, M. Treinish, K. Krsulich, et al., Quan- tum computing with qiskit, arXiv:2405.08810, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  23. [23]

    PennyLane: Automatic differentiation of hybrid quantum-classical computations

    V . Bergholm, J. Izaac, M. Schuld, et al., Pennylane: Automatic differentiation of hybrid quantum-classical computations, arXiv:1811.04968, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  24. [24]

    Cirq Developers, Cirq, 2025

    work page 2025

  25. [25]

    R. S. Smith, M. J. Curtis, and W. J. Zeng, A practical quantum instruction set architecture, arXiv:1608.03355, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  26. [26]

    Schulz, M

    M. Schulz, M. Ruefenacht, D. Kranzlm ¨uller, et al., Accelerating HPC With Quantum Computing: It Is a Software Challenge Too,CiSE, 2022

    work page 2022

  27. [27]

    Zhong, S

    Z. Zhong, S. He, H. Wang, et al., An Empirical Study on Package-Level Deprecation in Python Ecosystem, in Proc. ICSE, 2025

    work page 2025

  28. [28]

    D. B. Bose, T. Chan, M. Trimble, et al., AutoPyDep: A Recommendation System for Python Dependency Management Utilizing Graph-Based Analytics, inProc. FSE Companion, 2025

    work page 2025

  29. [29]

    Fern ´andez-Osuna, R

    M. Fern ´andez-Osuna, R. P ´erez-Castillo, J. A. Cruz- Lemus, et al., Exploring design patterns in quantum software: A case study,Computing, 2025

    work page 2025

  30. [30]

    L. J. Kitt and M. B. Cohen, MorphQ++: A Repro- ducibility Study of Metamorphic Testing on Quantum Compilers, inProc. ASE Workshops, 2024

    work page 2024

  31. [31]

    Cardinal, I

    J. Cardinal, I. Benzarti, G. E. boussaidi, et al., Migrating QAOA from Qiskit 1.x to 2.x: An experience report, arXiv:2512.08245, 2026

    work page arXiv 2026

  32. [32]

    Malka, S

    J. Malka, S. Zacchiroli, and T. Zimmermann, Does Functional Package Management Enable Reproducible Builds at Scale? Yes. InProc. MSR, 2025

    work page 2025

  33. [33]

    Devresse, F

    A. Devresse, F. Delalondre, and F. Sch ¨urmann, Nix based fully automated workflows and ecosystem to guarantee scientific result reproducibility across soft- ware environments and systems, inProc. SE-HPCCSE, 2015

    work page 2015

  34. [34]

    Open Archives Initiative, arXiv API, 2026

    work page 2026

  35. [35]

    The Semantic Scholar Open Data Platform

    R. Kinney, C. Anastasiades, R. Authur, et al., The semantic scholar open data platform, arXiv:2301.10140, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    Malka, S

    J. Malka, S. Zacchiroli, and T. Zimmermann, Docker does not guarantee reproducibility, arXiv:2601.12811, 2026

    work page arXiv 2026

  37. [37]

    Sch ¨onberger, M

    M. Sch ¨onberger, M. Franz, S. Scherzinger, et al., Peel — pile? cross-framework portability of quantum soft- ware, inProc. ICSA-C, 2022

    work page 2022

  38. [38]

    Vallet, D

    N. Vallet, D. Michonneau, and S. Tournier, Toward practical transparent verifiable and long-term repro- ducible research using guix,Scientific Data, 2022

    work page 2022

  39. [39]

    Court `es and R

    L. Court `es and R. Wurmus, Reproducible and User- Controlled Software Environments in HPC with Guix, inProc. Euro-Par Workshops, 2015

    work page 2015

  40. [40]

    Hausch, S

    M. Hausch, S. Hauser, and B. Uekermann, Improving reproducibility of scientific software using nix/nixos: A case study on the precice ecosystem,ECEASST, 2025

    work page 2025

  41. [41]

    Dolstra, M

    E. Dolstra, M. De Jonge, E. Visser, et al., Nix: A safe and policy-free system for software deployment. InLISA, 2004

    work page 2004