pith. sign in

arxiv: 2605.19736 · v1 · pith:AN7QZDIWnew · submitted 2026-05-19 · 🪐 quant-ph · cs.SE

QUTest: A Native Testing Framework for Quantum Programs

Jos\'e Campos This is my paper

Pith reviewed 2026-05-20 06:12 UTC · model grok-4.3

classification 🪐 quant-ph cs.SE
keywords quantum testingOpenQASMtest frameworkquantum circuitspragma directivesArrange Act Assertcontinuous integrationquantum software development
0
0 comments X

The pith

Tests for quantum programs can be written directly in standard OpenQASM files.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

QUTest lets developers keep both the quantum circuit and its tests in the same .qasm file. Configuration and assertions go into pragma comments that existing tools ignore. This setup follows the classic Arrange/Act/Assert pattern but stays native to the quantum description language. Readers would care if it reduces the friction of testing quantum code and makes it easier to maintain tests alongside the program.

Core claim

The central claim is that a testing framework can be built where tests are standard .qasm files, using pragma comments (//%) to encode configuration, runtime requirements, and assertions without altering the OpenQASM language or breaking compatibility with current parsers and tools. It supports 12 assertion types covering deterministic, statistical, quantum-state, and structural checks, includes a linter, and offers an environment-aware mode plus CLI tools for discovery and reporting.

What carries the argument

Pragma comments starting with //% that embed test logic inside .qasm files while remaining invisible to standard OpenQASM processors.

If this is right

  • Quantum tests become portable with the circuit code itself.
  • The same test file works across different quantum runtimes when using the environment-aware mode.
  • Integration with continuous integration becomes straightforward via XML reports and automatic discovery.
  • Twelve different assertion types allow checks on probabilities, states, and structure.
  • Linting can catch issues in test definitions before running on quantum hardware.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This native approach could reduce the need to learn separate testing libraries in Python or other hosts.
  • Pragma-based assertions might be adapted for other quantum languages or circuit formats.
  • Running tests in isolated environments could help identify version-specific bugs in quantum simulators or compilers.
  • Future evaluations with mutation testing would show how effective the assertions are at detecting faults in quantum programs.

Load-bearing premise

Pragma comments can encode everything needed for testing without requiring changes to the OpenQASM standard or causing compatibility problems with existing tools.

What would settle it

Processing a sample .qasm test file through a standard OpenQASM parser or compiler and verifying that the pragmas are ignored while the circuit runs normally, or finding that some assertion cannot be expressed this way.

read the original abstract

Quantum programs are often shared as OpenQASM 3 circuits, but tests are still written in host languages such as Python with Qiskit. We present QUTest, a native framework in which both programs and tests are standard .qasm files. Tests follow the Arrange / Act / Assert pattern, while configuration, runtime requirements, and assertions are encoded as pragma comments (//%), preserving compatibility with existing OpenQASM tools. QUTest provides 12 assertion types spanning deterministic, statistical, quantum-state, and structural checks, plus a linter and an environment-aware mode for running the same test across selected runtime versions in isolated environments. Its CLI supports automatic test discovery, runtime compatibility checks, and XML reports for continuous integration. We describe the pragma language, implementation, and a planned evaluation using coverage and mutation testing. QUTest is available at https://github.com/QBugs/qutest. Video demo: https://youtu.be/FvgvsiAXuW0.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 2 minor

Summary. The manuscript presents QUTest, a native testing framework for quantum programs in which both the program under test and the tests themselves are expressed as standard .qasm files. Tests follow the Arrange/Act/Assert pattern, with configuration, runtime requirements, and assertions encoded via //% pragma comments that preserve compatibility with existing OpenQASM parsers and tools. The framework supplies 12 assertion types (deterministic, statistical, quantum-state, and structural), a linter, an environment-aware execution mode, and a CLI supporting automatic discovery, compatibility checks, and XML reporting for CI.

Significance. If the described mechanism functions as claimed, QUTest would address a practical gap in quantum software engineering by allowing tests to remain in the native circuit language rather than requiring host-language wrappers. The pragma-based encoding is a pragmatic solution that avoids language extensions while enabling richer testing capabilities; combined with CI support and multiple assertion categories, the approach could improve reproducibility and reliability of quantum program development.

major comments (1)
  1. The manuscript describes the intended design, pragma language, and implementation at a high level but supplies no concrete syntax examples, usage snippets, or execution traces. Without these, it is not possible to verify that the 12 assertion types can be expressed within //% comments while remaining valid OpenQASM 3 and that the runner correctly extracts and applies them.
minor comments (2)
  1. The abstract states that an evaluation using coverage and mutation testing is planned; if any preliminary results or methodology details are present in the full text, they should be moved into the main body rather than left as future work.
  2. A small example .qasm test file illustrating the pragma syntax for at least one quantum-state assertion would substantially improve clarity.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the positive evaluation of QUTest's potential impact and for the constructive major comment. We address the point below and will incorporate revisions to strengthen the manuscript.

read point-by-point responses

  1. Referee: The manuscript describes the intended design, pragma language, and implementation at a high level but supplies no concrete syntax examples, usage snippets, or execution traces. Without these, it is not possible to verify that the 12 assertion types can be expressed within //% comments while remaining valid OpenQASM 3 and that the runner correctly extracts and applies them.

    Authors: We agree that the manuscript would be improved by the inclusion of concrete syntax examples, usage snippets, and sample execution traces. In the revised version we will add a new section (or subsection) that presents multiple complete .qasm test files illustrating the Arrange/Act/Assert pattern together with representative examples from each of the four assertion categories. Each example will be accompanied by the corresponding CLI invocation and output, thereby demonstrating both syntactic validity under OpenQASM 3 and the extraction/execution behavior of the runner. These additions will directly address the referee's concern while remaining within the page limits. revision: yes

Circularity Check

0 steps flagged

No significant circularity in tool-description paper

full rationale

This manuscript presents a software framework (QUTest) for embedding tests inside OpenQASM files via pragma comments. It contains no equations, fitted parameters, predictions, or first-principles derivations that could reduce to their own inputs. The central mechanism—treating //% lines as directives while preserving valid OpenQASM syntax—rests on the external, pre-existing behavior of OpenQASM parsers that discard comments, which is independent of the present work. All other elements (assertion types, CLI features, linter) are descriptive implementation choices rather than claims that loop back to self-defined or self-cited quantities. The paper is therefore self-contained with no circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is a software framework paper with no mathematical free parameters, axioms, or invented physical entities. The design rests on the engineering assumption that pragma comments are sufficient to express test intent while preserving parser compatibility.

pith-pipeline@v0.9.0 · 5693 in / 1134 out tokens · 38924 ms · 2026-05-20T06:12:03.930456+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages · 2 internal anchors

  1. [1]

    2002.Test-Driven Development: By Example

    Kent Beck. 2002.Test-Driven Development: By Example. Addison-Wesley Prof

    work page 2002

  2. [2]

    Julien Cardinal, Imen Benzarti, Ghizlane El boussaidi, and Christophe Pere

    work page

  3. [3]

    arXiv:2512.08245 [cs.SE] https://arxiv.org/abs/2512.08245

    Migrating QAOA from Qiskit 1.x to 2.x: An experience report. arXiv:2512.08245 [cs.SE] https://arxiv.org/abs/2512.08245

    work page arXiv

  4. [4]

    Cirq Developers. 2026. Cirq: An open source framework for programming quantum computers. https://quantumai.google/cirq Accessed: 2026-05

    work page 2026

  5. [5]

    Andrew Cross and et. al. 2022. OpenQASM 3: A Broader and Deeper Quantum Assembly Language.ACM Transactions on Quantum Computing3, 3, Article 12 (Sept. 2022), 50 pages. doi:10.1145/3505636

    work page doi:10.1145/3505636 2022

  6. [6]

    Gadi Aleksandrowicz et al. 2019. Qiskit: An Open-source Framework for Quan- tum Computing. (Jan. 2019). doi:10.5281/zenodo.2562111

    work page doi:10.5281/zenodo.2562111 2019

  7. [7]

    Ville Bergholm et al. 2022. PennyLane: Automatic differentiation of hybrid quantum-classical computations. https://arxiv.org/abs/1811.04968

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  8. [8]

    Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs: A Case Study With Qiskit.IEEE Transactions on Quantum Engineering3 (2022), 1–17. doi:10.1109/TQE.2022.3195061

    work page doi:10.1109/tqe.2022.3195061 2022

  9. [9]

    Daniel Fortunato, José Campos, and Rui Abreu. 2026. Probabilistic Condition, Decision and Path Coverage of Circuit-based Quantum Programs. https://arxiv. org/abs/2604.26609

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  10. [10]

    Tiancheng Jin, Shangzhou Xia, and Jianjun Zhao. 2025. NovaQ: Improving Quantum Program Testing through Diversity-Guided Test Case Generation. In IEEE/ACM International Conference on Automated Software Engineering (ASE). 3881–3885. doi:10.1109/ASE63991.2025.00335

    work page doi:10.1109/ase63991.2025.00335 2025

  11. [11]

    Kanguk Lee. 2024. Standard Gates not recognized by qiskit.qasm2.loads. https: //github.com/Qiskit/qiskit/issues/12124 Accessed: 2026-05

    work page 2024

  12. [12]

    Yuechen Li, Kai-Yuan Cai, and Beibei Yin. 2026. A Dynamic Test Oracle for Quantum Programs With Separable Output States.IEEE Transactions on Software Engineering52, 4 (2026), 1568–1591. doi:10.1109/TSE.2026.3670211

    work page doi:10.1109/tse.2026.3670211 2026

  13. [13]

    Eñaut Mendiluze, Shaukat Ali, Paolo Arcaini, and Tao Yue. 2021. Muskit: A Mutation Analysis Tool for Quantum Software Testing. InIEEE/ACM In- ternational Conference on Automated Software Engineering (ASE). 1266–1270. doi:10.1109/ASE51524.2021.9678563

    work page doi:10.1109/ase51524.2021.9678563 2021

  14. [14]

    Matteo Paltenghi and Michael Pradel. 2022. Bugs in Quantum computing plat- forms: an empirical study.Proc. ACM Program. Lang.6, OOPSLA1, Article 86, 27 pages. doi:10.1145/3527330

    work page doi:10.1145/3527330 2022

  15. [15]

    Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022. QuSBT: Search- Based Testing of Quantum Programs. InACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-C). 173–177. doi:10.1145/ 3510454.3516839

    work page arXiv 2022

  16. [16]

    Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2023. QuCAT: A Combina- torial Testing Tool for Quantum Software. InIEEE/ACM International Conference on Automated Software Engineering (ASE). 2066–2069. doi:10.1109/ASE56229. 2023.00062

    work page doi:10.1109/ase56229 2023

  17. [17]

    Jiaming Ye, Shangzhou Xia, Fuyuan Zhang, Paolo Arcaini, Lei Ma, Jianjun Zhao, and Fuyuki Ishikawa. 2023. QuraTest: Integrating Quantum Specific Features in Quantum Program Testing. InIEEE/ACM International Conference on Automated Software Engineering (ASE). 1149–1161. doi:10.1109/ASE56229.2023.00196

    work page doi:10.1109/ase56229.2023.00196 2023