{"id":"c81e4390-6b80-459f-8a0e-f1e03c28b727","arxiv_id":"2608.05041","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"An autologging framework for Qiskit automatically captures compiler provenance, including pass-level details, and stores it in MLflow for visualization and comparison.","lead":"This paper presents a software tool that automatically records the inner workings of the Qiskit quantum compiler, capturing each compilation step and its timing without requiring programmers to add logging code. It stores this provenance in an MLflow server, making quantum compilation experiments easier to compare and reproduce.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Per-pass provenance completeness is not demonstrated: Qiskit flow controllers and alternate entry points can bypass a callback-based logger, so the 'complete compilation process' claim is unverified.","rationale":"The reader's CONDITIONAL verdict is appropriate. The paper is a useful demonstration, but the central automatic-observability claim needs evidence that the recorded pass trace is complete and non-perturbing. I looked for a single concrete flaw rather than simply asking for more evaluation. The most specific risk is that C7 captures are incomplete when passes are nested inside flow controllers, or when an application uses a different entry point than the demonstrated generate_preset_pass_manager().run() path. The text explicitly promises that C7 records 'each compiler pass executed' (Section II-C), so a silent omission would falsify the core contribution. This is not an internal inconsistency; it is an unverified mechanism whose failure mode is concrete and checkable. The paper does provide open-source code and a working example, so the concern is about generalizability and completeness, not about the demonstration being fabricated. The reader flagged the same general area (missing passes and perturbation), but I am more specific about the flow-controller/callback mechanism. I therefore recommend that the verdict remain CONDITIONAL, pending the completeness test described above.","tokens_in":4789,"tokens_out":6440,"duration_ms":70772,"concrete_test":"Run autolog on a PassManager containing a DoWhileController (or ConditionalController) with two known passes inside, e.g., CXCancellation and Depth, under Qiskit 1.x. Compare the C7 Passes records against the actual pass execution order obtained by adding Qiskit's own callback to the same PassManager or by instrumenting the inner task list. If the nested passes are missing, mislabeled, or counted only once, the completeness claim in Section II-C fails for exactly the control-flow constructs that C6 Plan claims to expose; if they are present, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that C7 Passes records 'the outcome of each compiler pass executed' and that this allows 'the complete compilation process to be reconstructed' (Section II-C and Section IV). The paper does not specify how pass-level events are hooked, only that the layer 'transparently intercepts key stages' (Section II-B). If the hook is Qiskit's per-pass callback on PassManager.run, passes executed inside FlowController objects (ConditionalController/DoWhileController) are not necessarily individually reported, because the callback fires for each top-level task in the pass list and a flow controller is one task containing nested passes. Since the preset optimization levels use exactly this kind of iterative control flow, the recorded C7 entities and Figure 4 may be a partial trace rather than a complete one. No completeness measurement (logged pass count compared with a Qiskit-internal trace) and no overhead measurement for the per-pass circuit-metric computation are reported, so the 'comprehensive provenance' and 'without perturbing' claims rest on design intent rather than demonstrated behavior.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an MLflow-inspired autologging framework for Qiskit that automatically captures compiler provenance, including transpilation stage plans, per-pass execution data, circuit metrics, backend characteristics, and execution results. The framework extends the QProv provenance model with two new entities (C6 Plan and C7 Passes), stores records in an MLflow Tracking Server, and provides visualizations such as a transpilation timeline. The demonstration uses a QFT benchmark transpiled with a preset pass manager and executed on an IQM simulator, and shows a comparison of optimization levels 2 and 3.","tokens_in":4920,"tokens_out":3519,"duration_ms":39811,"significance":"If the central capability holds, the tool is a useful community resource: it lowers the barrier to collecting detailed compiler provenance, supports reproducible comparison of compilation workflows, and builds on the mature MLflow infrastructure. The paper's evidence is, however, anecdotal and does not validate the claimed completeness of pass-level capture or the absence of performance perturbation. The open-source availability and the explicit extension of a published provenance model are strengths that make the work worth publishing after the validation gaps are addressed.","major_comments":[{"comment":"The claim that C7 Passes records 'the outcome of each compiler pass executed' and that this allows 'the complete compilation process to be reconstructed' is not supported by any completeness validation. The paper does not specify how pass-level events are hooked, only that the autologging layer 'transparently intercepts key stages' (§II-B). If the hook is Qiskit's per-pass callback on PassManager.run, passes executed inside FlowController objects (e.g., ConditionalController or DoWhileController) may not be reported individually, because Qiskit invokes the callback for each top-level task and a flow controller is one such task. Since the preset optimization levels use flow controllers for iterative optimization, the recorded C7 trace in Figure 4 could be a partial trace. The authors should describe the exact interception mechanism and provide a completeness check, for example by comparing the logged pass sequence with a Qiskit-internal trace for optimization levels 0–3 on representative circuits.","section":"§II-C, §IV, Fig. 4"},{"comment":"The paper claims that autologging is 'transparent' and that provenance is collected without perturbing the compilation process, but no overhead measurements are reported. Recording circuit metrics (Q4–Q7 and C5) after every pass requires per-pass circuit analysis whose cost grows with circuit size, potentially distorting the very pass timings the tool is designed to observe. An experiment reporting compilation time and memory usage with and without autologging for circuits of varying qubit count and optimization level would substantiate the non-perturbation claim.","section":"§II-A, §II-C"},{"comment":"The demonstration exercises only one application path: generating a preset pass manager, calling manager.run, and then backend.run on a fake backend. The framework's central claim is that it 'transparently intercepts key stages of the Qiskit transpilation and execution APIs' without modifying application code. To support that generality, the authors should either demonstrate or explicitly state coverage of other common entry points, such as the standalone transpile() function, direct PassManager construction, and execution on a QPU or simulator through the Qiskit IBM Runtime or Aer primitives, as these paths may require different interception logic.","section":"§III-A, Fig. 3"}],"minor_comments":[{"comment":"'an unified experiment record' should be 'a unified experiment record' (the same wording appears in Section IV).","section":"Abstract"},{"comment":"There is a typo in 'TheC7 Passesentity captures': it should read 'The C7 Passes entity captures'.","section":"§II-C"},{"comment":"The bold formatting used to mark the new C6 Plan and C7 Passes entities is not visible in the printed figure; please use color or a distinct label.","section":"Fig. 2"},{"comment":"The timeline label 'pre_init' is not a standard Qiskit preset pass manager stage name (the standard names are init, layout, routing, translation, optimization, and scheduling). Clarify whether this label comes from the tool's own stage detection or is a deviation from Qiskit naming.","section":"Fig. 4"},{"comment":"The sentence 'The collected provenance data and derived visualization is collected as artifacts' has a subject-verb agreement error ('data' is plural; consider 'are collected').","section":"§III-B"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to target a software/demonstration venue where a single worked example may be acceptable, but for a journal the load-bearing claims of comprehensive provenance and non-perturbing instrumentation need direct validation. The skeptic's concern about FlowController bypass is technically plausible and should be addressed head-on, either by describing a hook that captures nested passes or by providing a completeness measurement. The open-source code is a positive sign, and the revision path is clear."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look for anyone doing Qiskit compilation benchmarking. The new thing here is not MLflow or QProv but the integration: a single-call autologging layer that intercepts Qiskit transpilation and execution, records pass-level data, and stores it as an MLflow run. The QProv extensions (C6 Plan, C7 Passes) are reasonable and the timeline visualization is genuinely useful for seeing where time goes.\n\nThe paper is honest about being a demonstration. It doesn't claim a scientific result; it claims a tool that works and is available. That is fine. The code and package are public, which is the right kind of evidence for a software paper.\n\nThe soft spot is the 'complete compilation process' claim. The paper says C7 Passes records the outcome of every pass executed and that this lets the full process be reconstructed. But the mechanism is only described as 'transparently intercepting key stages,' and no completeness check is reported. The stress-test concern is real: if the interception relies on Qiskit's pass callback, passes inside flow controllers (DoWhileController etc.) may not be individually reported. The timeline in Fig. 4 shows repeated blocks that look like loop iterations, so maybe the hook does catch them, but the paper doesn't show that, and Qiskit's internals vary across versions. A simple test comparing the logged pass list against a reference trace would close this. Overhead is also not measured; computing circuit metrics for every pass could add noticeable time for large circuits.\n\nNone of this kills the tool. For a demonstration paper, the evaluation level is about right, but the completeness claim needs either a caveat or a measurement. I'd send it to peer review in a software/demo track and ask for that addition.\n\nGood for people who want to adopt provenance collection in their own Qiskit workflows, and for anyone building similar tools. I'd cite it.","headline":"A genuinely useful integration that makes Qiskit compiler provenance easy to capture, but the 'complete process' claim needs a completeness check before it can be trusted.","tokens_in":5481,"tokens_out":2670,"would_cite":true,"duration_ms":27321,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that a single autolog() call can turn Qiskit transpilation runs into complete, machine-readable provenance records, giving pass-level insight into why a compiler produced the circuit it did.","keywords":["quantum compilers","observability","experiment tracking","autologging","provenance","Qiskit","MLflow","transpilation"],"falsifier":"Run a known Qiskit program, such as the QFT benchmark from MQT Bench, with autologging enabled, and independently instrument the transpiler by wrapping each pass with an external timer that logs pass identity and order. If the autologged C7 record misses any pass that the independent instrument observed, or if the recorded per-pass durations differ materially from the external timer's timings for the same run, the claim of complete, unbiased pass-level provenance fails.","tokens_in":4557,"feed_emoji":"⚙️","tokens_out":5295,"duration_ms":50831,"temperature":0.7,"pith_summary":"The paper claims that compiler observability in Qiskit can be achieved without manual instrumentation. It presents an MLflow-inspired autologging layer that intercepts transpilation and execution APIs, recording the transpilation plan, each pass's execution time and resulting circuit, backend details, configuration, and results into MLflow experiments. The framework extends the QProv provenance model with two new entities, C6 Plan and C7 Passes, so the recorded data stays compatible with existing quantum provenance schemas while capturing compiler-specific information. If the approach works as described, researchers can compare compilation configurations and reproduce experiments by looking at a structured log and visualizations such as transpilation timelines, rather than instrumenting code by hand.","feed_headline":"Qiskit compiler runs can now be tracked automatically, pass by pass","feed_subtitle":"One autolog() call records stages, pass times, and circuit changes, making compilation runs reproducible and comparable.","key_machinery":"The load-bearing mechanism is an autologging instrumentation layer that wraps Qiskit's transpilation and execution APIs. It records two extensions to the QProv model: the C6 Plan entity captures the transpilation stage plan constructed before execution (stages, passes, and control-flow constructs), and the C7 Passes entity records each executed pass with its name, stage, resulting circuit, and circuit metrics. These records are written to an MLflow Tracking Server, whose run comparison interface and artifact storage provide analysis and visualization. The pass timeline visualization in the paper is a direct rendering of C7 records, showing pass runtime against transpilation stage.","core_discovery":"The paper's central claim is that the Qiskit compilation workflow can be made self-documenting. With one autolog() call, each transpilation run is recorded in an MLflow experiment that contains the preset pass manager's stage plan, per-pass execution data (pass name, stage, resulting circuit, and circuit metrics), backend characteristics, compiler configuration, and execution results. These records extend the QProv provenance model with two entities, C6 Plan and C7 Passes, so the full compilation process can be reconstructed and analyzed. The demonstration shows how this turns a conventional experiment comparison into pass-level attribution: for the same QFT circuit, optimization levels 2 and 3 can be compared by seeing which passes changed circuit depth and which dominated compilation time.","pith_inferences":["Beyond the paper, if pass-level timing is captured without perturbing the process, the same provenance could serve as a regression benchmark for compiler improvements: a new pass version should show a predictable change in the C7 timeline.","Beyond the paper, the C6 Plan / C7 Passes split also enables behavioral comparison of different SDKs' compilers under a common schema, letting researchers ask whether Qiskit's routing stage and a hypothetical Qrisp pipeline make comparable trade-offs.","Beyond the paper, the recorded pass-level circuit metrics could be mined to identify which transpiler passes are responsible for systematic depth or gate-count regressions across a benchmark suite, an analysis the paper demonstrates only for a single QFT instance.","Beyond the paper, because all runs are stored in MLflow, continuous integration could automatically flag any change in Qiskit's pass manager that alters the pass sequence for a fixed benchmark, acting as a compiler smoke test."],"forward_implications":["Researchers comparing optimization levels can see, not just final depth and gate count, but exactly which passes caused the improvement and which consumed runtime.","Compilation experiments become reproducible artifacts: the recorded QProv-compatible metadata plus pass records allow another team to replay the same transpilation configuration and check pass-for-pass agreement.","The QProv model gains a compiler-observability layer (C6 and C7) that future tools can consume without breaking existing QProv-based analyses.","The same autologging pattern can be carried to other quantum SDKs, giving a unified experiment log across Qiskit, Qrisp, and PennyLane workflows.","Visualizations like the transpilation timeline become standard debugging views, turning compiler behavior from a black box into an inspectable sequence of transformations."],"supporting_citations":[{"why":"It supplies the experiment tracking infrastructure: the MLflow Tracking Server, run comparison UI, and artifact storage that the autologging layer writes to.","marker":"[1]"},{"why":"It defines the QProv provenance model that the framework extends with the C6 Plan and C7 Passes entities for compiler information.","marker":"[2]"},{"why":"It supplies the benchmark circuits used in the demonstration, specifically the QFT benchmark generated with MQT Bench at algorithm level.","marker":"[3]"}],"fun_headline_variants":["One autolog call reveals every Qiskit compiler pass","Automatic pass-by-pass tracking for Qiskit compilation","Autolog makes Qiskit transpiler runs self-recording","Track Qiskit compiler passes automatically with autolog"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The automatic instrumentation layer intercepts every Qiskit transpilation and execution call without missing a pass and without changing the compilation process or its timing.","fun_headline_variants_meta":{"raw":{"variants":["One autolog call reveals every Qiskit compiler pass","Automatic pass-by-pass tracking for Qiskit compilation","Autolog makes Qiskit transpiler runs self-recording","Track Qiskit compiler passes automatically with autolog"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":2909,"prompt_tokens":780,"completion_tokens":2129,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":396,"completion_tokens_details":{"reasoning_tokens":2062}},"tokens_in":396,"tokens_out":2129,"duration_ms":16242,"temperature":1.0,"reasoning_tokens":2062,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:43:06.927770+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a known Qiskit program, such as the QFT benchmark from MQT Bench, with autologging enabled, and independently instrument the transpiler by wrapping each pass with an external timer that logs pass identity and order. If the autologged C7 record misses any pass that the independent instrument observed, or if the recorded per-pass durations differ materially from the external timer's timings for the same run, the claim of complete, unbiased pass-level provenance fails.","supporting_citations":[{"cited_title":"Accelerating the machine learning lifecycle with mlflow","cited_arxiv_id":null,"evidence_quote":"It supplies the experiment tracking infrastructure: the MLflow Tracking Server, run comparison UI, and artifact storage that the autologging layer writes to."},{"cited_title":"The MQT handbook: A summary of design automation tools and software for quantum computing,","cited_arxiv_id":null,"evidence_quote":"It supplies the benchmark circuits used in the demonstration, specifically the QFT benchmark generated with MQT Bench at algorithm level."}],"review_version":1}