REVIEW 3 major objections 6 minor 1 cited by
Enhancing Quantum Software Development Process with Experiment Tracking
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read This paper shows that MLflow can serve as the foundation for quantum experiment tracking by logging run parameters, result figures, and calibration data from a real 50-qubit quantum processor.
desk verdict A credible small demo of MLflow for quantum run tracking, whose conclusions run ahead of the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the MLflow tracking workflow shown in Listing 1: a named experiment, a start_run context, and the set_tag, log_param, log_figure, log_text, and log_dict calls that attach run-specific artifacts, followed by search_runs in Listing 2 to retrieve experiments as a data frame. MLflow's artifact store is what carries the calibration data, and run tags and parameters carry the provenance-like metadata. The paper's key move is mapping QProv's provenance categories onto these generic MLflow primitives rather than developing a quantum-specific schema.
What would settle it
Run the same MLflow tracking workflow with two different calibration sets on the same device and then try to reconstruct, from only the logged artifacts, the exact circuit, compiler settings, and hardware state that produced each run; if the logged fields are insufficient to distinguish the runs or recover the execution conditions, the paper's claim that MLflow captures what quantum experiment tracking needs fails.
Extended reading notes
Core claim
The paper's central claim is that a general-purpose experiment tracking platform built for machine learning, MLflow, is already capable of serving as the foundation for quantum experiment tracking. On the paper's own terms, the demonstration consists of wrapping a quantum program in an MLflow run context, logging the number of shots as a parameter, storing the measurement histogram as a figure artifact, and storing the calibration set identifier plus the calibration data as text and JSON artifacts; afterward, a search query returns those runs as a pandas data frame for further analysis. The paper interprets this as evidence that the four QProv provenance categories (quantum circuit, quantum computer, compilation, execution) can be captured with MLflow primitives, and that the maturity of the MLflow ecosystem makes it preferable to building standalone provenance systems.
Load-bearing premise
The load-bearing assumption is that the few MLflow primitives shown in the demo—tags, one parameter, a histogram figure, a calibration-set identifier, and a calibration JSON file—are enough to capture what quantum experiment tracking and provenance require, a claim the paper supports with a single simple run and leaves full schema automation to future work.
Editorial extensions
If this is right
- If MLflow becomes the tracking layer, quantum teams inherit its search, web UI, and ecosystem instead of building a bespoke provenance tool.
- Every logged run becomes a reconstruction point, since parameters, result figure, and calibration data are stored together in one run record.
- Because the calibration set identifier and calibration JSON are stored per run, hardware drift between runs becomes visible when historical runs are compared.
- A single tracking server can cover both the machine-learning and quantum parts of a hybrid workflow, aligning practices across the two domains.
- Search results returned as a data frame let developers turn run history into progress reports and comparison plots with standard data tools.
Reading between the lines
- We infer a natural next test: serialize the full quantum provenance schema into MLflow tags, parameters, and artifacts and check whether a provenance document can be reconstructed from a logged run, since the paper explicitly leaves this automation as future work.
- We infer the pattern probably transfers to other quantum back ends because the code only uses the MLflow client API and a provider object, but nothing in the paper shows that search and artifact retrieval stay reliable at scale or with many users.
- We infer that the practical reproducibility payoff depends on teams agreeing on a minimal, stable set of tag and artifact names, a convention the paper does not propose.
- We infer that the approach could make hardware vendors and software teams share a common run record, with calibration data attached to each execution, though the paper demonstrates only a single organization's setup.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that MLflow, an open-source ML experiment tracking platform, can be used to track quantum software experiments, and presents a demonstration in which a quantum program executed on VTT's IQM 50-qubit QPU via the QX service logs a tag, a shot count parameter, a result histogram, and calibration artifacts to an MLflow server. The paper claims that MLflow's tagging, parameter, metric, and artifact capabilities can capture the QProv quantum provenance model, and concludes that MLflow is well-suited to quantum research, supporting development practices, reproducibility, decision-making, and collaboration. The manuscript includes a setup diagram and two code listings, but no quantitative evaluation or multi-user study.
Significance. If the claims are validated, the paper would provide a low-cost path to structured experiment tracking in quantum R&D by leveraging a mature tool, and the specific integration with the VTT QX service and logging of calibration_set_id is a useful concrete recipe. The paper also correctly identifies QProv as a relevant provenance model. However, the evidence is limited to a single simple run; there is no demonstration that MLflow can faithfully represent QProv's relational schema, no measurement of search or scalability, and no collaboration scenario. As it stands, the paper is more an experience report than a validated system contribution, and its significance is correspondingly modest.
major comments (3)
- [Section III (Listing 1)] The assertion that MLflow's capabilities 'can capture the QProv attributes' is not supported by the evidence. QProv (Ref. [6]) defines a provenance model over four categories (circuit, computer, compilation, execution) with explicit relationships. Listing 1 logs a tag, one parameter ('shots'), a histogram figure, a calibration_set_id text file, and a calibration JSON; it does not map any of these to QProv entities or relations, nor does Listing 2 show a query that reconstructs those relations. An explicit mapping table or a structured search example is needed before this claim can be accepted.
- [Section IV] The conclusion that 'MLflow is well-suited to quantum research' is too broad for the evidence presented. The demonstration consists of one 50-qubit run logged by a single user, with no repeated runs, no multi-user collaboration scenario, and no comparison with alternative tracking approaches. To make the conclusion load-bearing, the paper should either add an evaluation (e.g., reproduce a result from logged artifacts, query across multiple runs) or explicitly narrow the conclusion to 'a promising foundation'.
- [Listing 2, Section III] The claim that the search functionality supports 'informed decision-making' is not demonstrated. Listing 2 retrieves a DataFrame from mlflow.search_runs but provides no example provenance query, no evaluation of search performance at scale, and no indication of how the returned data maps to the QProv categories. A concrete query, such as retrieving all runs with a given calibration_set_id and compiler configuration, would substantiate the claim.
minor comments (6)
- [Section I] In the third paragraph, 'its’s' appears in 'By leveraging its’s existing capabilities'; this should be 'its'.
- [Figure 2 caption] The caption contains a doubled phrase: 'executes a program on the on a quantum computer'; it should be 'executes a program on a quantum computer'.
- [Listing 1, line 4] The tag key 'Training info' is ML-oriented and would be clearer as 'toolkit' or 'software_stack'; consider aligning key names with the QProv categories if a mapping is intended.
- [Section III] The manuscript refers to 'Listing. 1' and 'Listing. 2' with a period after 'Listing'; the standard form 'Listing 1' and 'Listing 2' is preferable.
- [Section III] The paper cites a GitHub repository for the full program but does not pin a version or commit; adding a revision identifier would improve reproducibility.
- [Section II] The description of QProv is too brief to support the later mapping claim; a short enumeration of the four categories and example attributes would help the reader assess the claimed fit.
Circularity Check
No significant circularity: the paper makes no fitted or derived prediction, and the one claim that could resemble an imported premise (MLflow tags/parameters/artifacts 'can capture the QProv attributes') is explicitly deferred by the paper's own future-work statement.
full rationale
The paper is an experience/demo report, not a derivation: Listing 1 logs an MLflow run with a tag, one parameter, a histogram figure, a calibration_set_id text file, and a calibration JSON from a single IQM 50-qubit run via VTT QX. There are no equations, fitted parameters, or quantitative predictions whose inputs could coincide with outputs. The only load-bearing-looking assertion is that MLflow's tags, parameters, metrics, and free-form artifacts 'can capture the QProv attributes' (Section III). This is not circular: QProv is an external provenance schema (Ref. [6], by different authors), MLflow is an external platform, and the paper does not define MLflow's capabilities in terms of QProv or redefine QProv in terms of MLflow. The paper itself limits the claim by stating 'As next steps, we plan to automate the collection of data according to QProv format' (Section IV), so the authors explicitly do not present the flat-logging demo as a full QProv mapping. The self-citation [1] (Kinanen et al.) is used only for motivational statements about toolchains and development stages, not as the justification for the demonstration's validity. No uniqueness theorem, ansatz, or known-result renaming is invoked. The conclusion that MLflow is 'well-suited to quantum research' restates the demonstration's framing, which is a weakness in evidentiary strength rather than circularity. Therefore the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Experiment tracking workflows from ML/AI transfer to quantum software development.
- domain assumption MLflow's tag, parameter, and artifact logging is sufficient to capture quantum provenance attributes described by QProv.
- domain assumption The VTT QX IQM 50-qubit demo is representative of quantum experiment tracking needs.
Cite this review
Pith. "Pith review of Enhancing Quantum Software Development Process with Experiment Tracking." pith.science (2026). https://pith.science/paper/N6GMMFSN
@misc{pith2026250706990,
author = {Pith},
title = {Pith review of: Enhancing Quantum Software Development Process with Experiment Tracking},
year = {2026},
howpublished = {\url{https://pith.science/paper/N6GMMFSN}},
note = {Machine review of arXiv:2507.06990}
}
read the original abstract
As quantum computing advances from theoretical promise to experimental reality, the need for rigorous experiment tracking becomes critical. Drawing inspiration from best practices in machine learning (ML) and artificial intelligence (AI), we argue that reproducibility, scalability, and collaboration in quantum research can benefit significantly from structured tracking workflows. This paper explores the application of MLflow in quantum research, illustrating how it enables better development practices, experiment reproducibility, decision making, and cross-domain integration in an increasingly hybrid classical-quantum landscape.
Figures
Forward citations
Cited by 1 Pith paper
-
Toward Standardized Quantum Provenance: A Cross-Provider Analysis, Unified API, and Reference Prototype
A cross-provider analysis of 15 quantum platforms finds fragmented provenance access, and a unified OpenAPI contract with provider adapters is shown to normalize representative records while preserving evidence origins.
Reference graph
Works this paper leans on
- [6]
-
[1]
O. Kinanen, A. D. Mu ˜noz-Moller, V . Stirbu, J. M. Murillo, and T. Mikko- nen. Toolchain for faster iterations in quantum software development. Computing, 107(4):99, Mar 2025
work page 2025
-
[2]
J. M. Murillo, J. Garcia-Alonso, E. Moguel, J. Barzen, F. Leymann, S. Ali, T. Yue, P. Arcaini, R. P ´erez-Castillo, I. Garc ´ıa Rodr ´ıguez de Guzm ´an, M. Piattini, A. Ruiz-Cort ´es, A. Brogi, J. Zhao, A. Miranskyy, and M. Wimmer. Quantum software engineering: Roadmap and challenges ahead. ACM Trans. Softw. Eng. Methodol. , Jan. 2025
work page 2025
- [3]
-
[4]
P. Senapati, Z. Wang, W. Jiang, T. S. Humble, B. Fang, S. Xu, and Q. Guan. Towards redefining the reproducibility in quantum computing: A data analysis approach on nisq devices. In 2023 IEEE International Conference on Quantum Computing and Engineering (QCE) , volume 01, pages 468–474, Sep. 2023
work page 2023
-
[5]
B. Weder, J. Barzen, F. Leymann, M. Salm, and D. Vietz. The quantum software lifecycle. In Proceedings of the 1st ACM SIGSOFT International Workshop on Architectures and Paradigms for Engineering Quantum Software, APEQS 2020, page 2–9, New York, NY , USA, 2020. Association for Computing Machinery
work page 2020
-
[7]
M. Zaharia, A. Chen, A. Davidson, A. Ghodsi, S. A. Hong, A. Konwinski, S. Murching, T. Nykodym, P. Ogilvie, M. Parkhe, et al. Accelerating the machine learning lifecycle with mlflow. IEEE Data Eng. Bull. , 41(4):39– 45, 2018. 6https://github.com/AqueductHub
work page 2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.