{"id":"b3b49035-c3a2-4bb5-9d8c-99a924c362ae","arxiv_id":"2507.06990","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A small demonstration shows MLflow can log parameters, results, and calibration data from an IQM 50-qubit quantum computer, and the authors argue this should become standard practice.","lead":"This paper proposes using MLflow, a popular machine learning experiment tracker, to record quantum computing runs, including hardware calibration data. It argues that borrowing tracking practices from ML/AI can improve reproducibility and collaboration in quantum software development.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claim that MLflow 'can capture the QProv attributes' is unsupported: the demo logs flat artifacts with no demonstrated mapping to QProv's four provenance categories or their relationships.","rationale":"The reader's conditional verdict is fair. I agree that the weakest assumption is the sufficiency of the small MLflow primitive set for quantum provenance. My stress-test focuses on provenance semantics: QProv is not a bag of attributes but a model with entities and relations; MLflow's run-centric key/value/artifact model does not natively support typed relations between circuit, computer, compilation, and execution. The paper's own future-work sentence ('automate the collection of data according to QProv format') admits the mapping is not done. The demo is credible as a thin feasibility test, and the code is available, which counts in its favor. However, the central claim as worded overstates what is shown. A conditional acceptance requiring a schema mapping or explicit narrowing of the claim is appropriate. No independent evidence, formal verification, or evaluation section exists to strengthen it. My concrete test would settle whether the QProv-conformance gap is real by attempting to answer QProv queries from the logged MLflow data; until that passes, the broad conclusion should not be accepted as demonstrated. Because the reader already reached CONDITIONAL on essentially this basis, I do not change the verdict.","tokens_in":3243,"tokens_out":3504,"duration_ms":38168,"concrete_test":"Use the logged runs from the linked repository (github.com/qubernetes-dev/q8s-examples) and attempt to construct a QProv provenance document for one run that satisfies the QProv schema for circuit, computer, compilation, and execution. Then pose three representative provenance queries from QProv (e.g., 'which calibration set was used for the execution that produced results.png?', 'which transpiler/compiler version was used?', 'which qubits were used?') and check whether they can be answered using only the MLflow search API and the artifacts in Listing 1. If any query is unanswerable without adding new fields or relational links, the conclusion must be narrowed to a feasibility demonstration, not a QProv-capable foundation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the assertion in Section III that MLflow's tags, parameters, metrics, and free-form artifacts 'can capture the QProv attributes,' combined with the Section IV conclusion that MLflow is 'well-suited to quantum research.' The evidence is Listing 1, which logs a tag, one parameter, a histogram figure, a calibration_set_id text file, and a calibration JSON from a single 50-qubit run. QProv (Ref. [6]) defines a provenance model over four categories—circuit, computer, compilation, execution—with relationships between them. A flat MLflow run record does not by itself encode those relationships, and the paper neither maps the logged fields to QProv entities/relations nor evaluates whether a provenance query such as 'which calibration set and compiler configuration produced this result' can be answered from the logged data. Since the paper explicitly defers QProv-format automation to future work, the central claim currently rests on an unverified equivalence between logging key-value artifacts and capturing structured provenance. This is not an internal contradiction; it is an unsupported extrapolation from a minimal demo.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that MLflow, an open-source ML experiment tracking platform, can be used to track quantum software experiments, and presents a demonstration in which a quantum program executed on VTT's IQM 50-qubit QPU via the QX service logs a tag, a shot count parameter, a result histogram, and calibration artifacts to an MLflow server. The paper claims that MLflow's tagging, parameter, metric, and artifact capabilities can capture the QProv quantum provenance model, and concludes that MLflow is well-suited to quantum research, supporting development practices, reproducibility, decision-making, and collaboration. The manuscript includes a setup diagram and two code listings, but no quantitative evaluation or multi-user study.","tokens_in":3565,"tokens_out":4901,"duration_ms":48815,"significance":"If the claims are validated, the paper would provide a low-cost path to structured experiment tracking in quantum R&D by leveraging a mature tool, and the specific integration with the VTT QX service and logging of calibration_set_id is a useful concrete recipe. The paper also correctly identifies QProv as a relevant provenance model. However, the evidence is limited to a single simple run; there is no demonstration that MLflow can faithfully represent QProv's relational schema, no measurement of search or scalability, and no collaboration scenario. As it stands, the paper is more an experience report than a validated system contribution, and its significance is correspondingly modest.","major_comments":[{"comment":"The assertion that MLflow's capabilities 'can capture the QProv attributes' is not supported by the evidence. QProv (Ref. [6]) defines a provenance model over four categories (circuit, computer, compilation, execution) with explicit relationships. Listing 1 logs a tag, one parameter ('shots'), a histogram figure, a calibration_set_id text file, and a calibration JSON; it does not map any of these to QProv entities or relations, nor does Listing 2 show a query that reconstructs those relations. An explicit mapping table or a structured search example is needed before this claim can be accepted.","section":"Section III (Listing 1)"},{"comment":"The conclusion that 'MLflow is well-suited to quantum research' is too broad for the evidence presented. The demonstration consists of one 50-qubit run logged by a single user, with no repeated runs, no multi-user collaboration scenario, and no comparison with alternative tracking approaches. To make the conclusion load-bearing, the paper should either add an evaluation (e.g., reproduce a result from logged artifacts, query across multiple runs) or explicitly narrow the conclusion to 'a promising foundation'.","section":"Section IV"},{"comment":"The claim that the search functionality supports 'informed decision-making' is not demonstrated. Listing 2 retrieves a DataFrame from mlflow.search_runs but provides no example provenance query, no evaluation of search performance at scale, and no indication of how the returned data maps to the QProv categories. A concrete query, such as retrieving all runs with a given calibration_set_id and compiler configuration, would substantiate the claim.","section":"Listing 2, Section III"}],"minor_comments":[{"comment":"In the third paragraph, 'its’s' appears in 'By leveraging its’s existing capabilities'; this should be 'its'.","section":"Section I"},{"comment":"The caption contains a doubled phrase: 'executes a program on the on a quantum computer'; it should be 'executes a program on a quantum computer'.","section":"Figure 2 caption"},{"comment":"The tag key 'Training info' is ML-oriented and would be clearer as 'toolkit' or 'software_stack'; consider aligning key names with the QProv categories if a mapping is intended.","section":"Listing 1, line 4"},{"comment":"The manuscript refers to 'Listing. 1' and 'Listing. 2' with a period after 'Listing'; the standard form 'Listing 1' and 'Listing 2' is preferable.","section":"Section III"},{"comment":"The paper cites a GitHub repository for the full program but does not pin a version or commit; adding a revision identifier would improve reproducibility.","section":"Section III"},{"comment":"The description of QProv is too brief to support the later mapping claim; a short enumeration of the four categories and example attributes would help the reader assess the claimed fit.","section":"Section II"}],"recommendation":"major_revision","confidential_remarks":"This is a very short paper that might be better suited to a workshop or short-paper track, but under the standard journal bar it is not acceptable without revision. The central problem is overclaiming relative to evidence: the QProv capture claim and the broad 'well-suited' conclusion both need either additional support or explicit qualification. A minimal added evaluation—say, one or two more runs and a cross-run search—would substantially strengthen the paper, and a clear statement of which QProv attributes are and are not covered would resolve the main technical concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a small, credible feasibility demo — MLflow logging an IQM 50-qubit run on VTT's QX service — with conclusions that run ahead of the evidence. Worth reading for the concrete integration details, not for the strong claims.\n\nWhat's new: the specific integration is new in the cited literature. The demo logs a tag, one parameter, a histogram, a calibration_set_id text file, and a calibration JSON from a real QPU backend. MLflow's API usage is correct and the setup diagram is clear. That's a legitimate data point for the quantum software engineering community.\n\nThe soft spot is the QProv claim. The paper says MLflow's tags/parameters/metrics/artifacts 'can capture the QProv attributes.' The stress-test note has this right: QProv defines four provenance categories — circuit, computer, compilation, execution — with relationships. A flat MLflow run record does not encode those relationships, and the paper never maps logged fields to QProv entities or tests whether a provenance query can be answered from the logged data. The authors explicitly defer QProv-format automation to future work, so the Section IV conclusion that MLflow is 'well-suited to quantum research' is an unsupported extrapolation from a minimal demo. That's fixable: either narrow the claim to 'MLflow can log these artifacts' or add a real provenance mapping and a query exercise.\n\nNo other serious problems. No evaluation of search at scale or multi-user collaboration, but that's clearly future work and the paper says so. Novelty is low but acceptable for a feasibility study. Self-citation in the intro is not an issue.\n\nWho's it for: quantum software engineering researchers, especially those thinking about experiment tracking and provenance. It reads like a short workshop paper, not a major result. A serious referee could get value out of pushing the authors on the QProv mapping. I'd send it to review — conditional accept with revisions.\n\nOn the structured axes: I'd bring it to a reading group as a discussion piece, but I wouldn't cite it myself. Serious thinker, yes; the reasoning is coherent and honest, just overreaching.","headline":"A credible small demo of MLflow for quantum run tracking, whose conclusions run ahead of the evidence.","tokens_in":3929,"tokens_out":2402,"would_cite":false,"duration_ms":25127,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that MLflow can serve as the foundation for quantum experiment tracking by logging run parameters, result figures, and calibration data from a real 50-qubit quantum processor.","keywords":["experiment tracking","MLflow","quantum software engineering","reproducibility","quantum provenance","NISQ devices","calibration data","hybrid classical-quantum workflows"],"falsifier":"Run the same MLflow tracking workflow with two different calibration sets on the same device and then try to reconstruct, from only the logged artifacts, the exact circuit, compiler settings, and hardware state that produced each run; if the logged fields are insufficient to distinguish the runs or recover the execution conditions, the paper's claim that MLflow captures what quantum experiment tracking needs fails.","tokens_in":3070,"feed_emoji":"⚛️","tokens_out":7633,"duration_ms":79135,"temperature":0.7,"pith_summary":"Quantum software developers face noisy hardware whose behavior can shift between runs, so they need records of what was executed and under what conditions. This paper proposes reusing MLflow, a widely used machine-learning experiment tracker, for quantum experiment tracking instead of building a separate provenance system. It demonstrates the idea by running a small quantum program on a 50-qubit IQM quantum computer through the QX service, logging the shot count, a measurement histogram, the calibration set identifier, and a calibration JSON file for one run. The paper argues that because MLflow already handles experiment naming, tags, parameters, artifacts, and search, it can support reproducibility, progress tracking, and collaboration in hybrid classical-quantum development. The evidence is a single simple example; automatic capture of a full quantum provenance schema is left as future work.","feed_headline":"MLflow can track real 50-qubit quantum experiments","feed_subtitle":"Run parameters, result figures, and calibration data stored per run make quantum results reproducible and searchable.","key_machinery":"The central mechanism is the MLflow tracking workflow shown in Listing 1: a named experiment, a start_run context, and the set_tag, log_param, log_figure, log_text, and log_dict calls that attach run-specific artifacts, followed by search_runs in Listing 2 to retrieve experiments as a data frame. MLflow's artifact store is what carries the calibration data, and run tags and parameters carry the provenance-like metadata. The paper's key move is mapping QProv's provenance categories onto these generic MLflow primitives rather than developing a quantum-specific schema.","core_discovery":"The paper's central claim is that a general-purpose experiment tracking platform built for machine learning, MLflow, is already capable of serving as the foundation for quantum experiment tracking. On the paper's own terms, the demonstration consists of wrapping a quantum program in an MLflow run context, logging the number of shots as a parameter, storing the measurement histogram as a figure artifact, and storing the calibration set identifier plus the calibration data as text and JSON artifacts; afterward, a search query returns those runs as a pandas data frame for further analysis. The paper interprets this as evidence that the four QProv provenance categories (quantum circuit, quantum computer, compilation, execution) can be captured with MLflow primitives, and that the maturity of the MLflow ecosystem makes it preferable to building standalone provenance systems.","pith_inferences":["We infer a natural next test: serialize the full quantum provenance schema into MLflow tags, parameters, and artifacts and check whether a provenance document can be reconstructed from a logged run, since the paper explicitly leaves this automation as future work.","We infer the pattern probably transfers to other quantum back ends because the code only uses the MLflow client API and a provider object, but nothing in the paper shows that search and artifact retrieval stay reliable at scale or with many users.","We infer that the practical reproducibility payoff depends on teams agreeing on a minimal, stable set of tag and artifact names, a convention the paper does not propose.","We infer that the approach could make hardware vendors and software teams share a common run record, with calibration data attached to each execution, though the paper demonstrates only a single organization's setup."],"forward_implications":["If MLflow becomes the tracking layer, quantum teams inherit its search, web UI, and ecosystem instead of building a bespoke provenance tool.","Every logged run becomes a reconstruction point, since parameters, result figure, and calibration data are stored together in one run record.","Because the calibration set identifier and calibration JSON are stored per run, hardware drift between runs becomes visible when historical runs are compared.","A single tracking server can cover both the machine-learning and quantum parts of a hybrid workflow, aligning practices across the two domains.","Search results returned as a data frame let developers turn run history into progress reports and comparison plots with standard data tools."],"supporting_citations":[{"why":"Establishes the noisy intermediate-scale quantum context that makes per-run hardware data necessary.","marker":"[3]"},{"why":"Documents reproducibility challenges on noisy devices, the problem the tracking workflow addresses.","marker":"[4]"},{"why":"Defines the quantum software lifecycle in which experiment tracking is situated.","marker":"[5]"},{"why":"Supplies the quantum provenance schema whose categories the paper maps onto MLflow primitives.","marker":"[6]"},{"why":"Introduces MLflow, the experiment tracking platform whose existing capabilities the paper reuses.","marker":"[7]"},{"why":"Motivates data-guided iteration in quantum software development, the workflow the demo supports.","marker":"[1]"}],"fun_headline_variants":["Quantum experiments get MLflow tracking","MLflow: a new home for quantum run data","Reproducible quantum runs with MLflow","Quantum meets ML: tracking with MLflow","Store, search, and share quantum runs with MLflow"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the few MLflow primitives shown in the demo—tags, one parameter, a histogram figure, a calibration-set identifier, and a calibration JSON file—are enough to capture what quantum experiment tracking and provenance require, a claim the paper supports with a single simple run and leaves full schema automation to future work.","fun_headline_variants_meta":{"raw":{"variants":["Quantum experiments get MLflow tracking","MLflow: a new home for quantum run data","Reproducible quantum runs with MLflow","Quantum meets ML: tracking with MLflow","Store, search, and share quantum runs with MLflow"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1277,"prompt_tokens":760,"completion_tokens":517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":376,"completion_tokens_details":{"reasoning_tokens":447}},"tokens_in":376,"tokens_out":517,"duration_ms":5579,"temperature":1.0,"reasoning_tokens":447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:49:01.961429+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same MLflow tracking workflow with two different calibration sets on the same device and then try to reconstruct, from only the logged artifacts, the exact circuit, compiler settings, and hardware state that produced each run; if the logged fields are insufficient to distinguish the runs or recover the execution conditions, the paper's claim that MLflow captures what quantum experiment tracking needs fails.","supporting_citations":[{"cited_title":"Preskill","cited_arxiv_id":null,"evidence_quote":"Establishes the noisy intermediate-scale quantum context that makes per-run hardware data necessary."},{"cited_title":"Senapati, Z","cited_arxiv_id":null,"evidence_quote":"Documents reproducibility challenges on noisy devices, the problem the tracking workflow addresses."},{"cited_title":"Weder, J","cited_arxiv_id":null,"evidence_quote":"Defines the quantum software lifecycle in which experiment tracking is situated."},{"cited_title":"Weder, J","cited_arxiv_id":null,"evidence_quote":"Supplies the quantum provenance schema whose categories the paper maps onto MLflow primitives."},{"cited_title":"Zaharia, A","cited_arxiv_id":null,"evidence_quote":"Introduces MLflow, the experiment tracking platform whose existing capabilities the paper reuses."},{"cited_title":"Kinanen, A","cited_arxiv_id":null,"evidence_quote":"Motivates data-guided iteration in quantum software development, the workflow the demo supports."}],"review_version":1}