{"id":"594e5391-a596-4716-a696-2a19d4708e6c","arxiv_id":"2508.01690","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A simulated variational quantum circuit policy, trained offline on a learned model, balances a physical cart-pole in most tested regions, but only under local simulation because cloud quantum latency exceeds real-time limits.","lead":"A team replaced the neural network in an offline reinforcement learning controller with a quantum-style circuit and tested the trained policy on a real cart-pole robot. The policy balances the pole for most tested starting positions when run on a local simulator, but cloud-based quantum execution is far too slow for real-time control.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 6 reports bin-averaged steps, not per-episode outcomes; 'most bins exceed 200' can hold even if most runs fail the paper's own 200-step success criterion.","rationale":"The reader's strongest claim and weakest assumption point to the sim-to-real transfer of the offline model, which is a real limitation the authors also acknowledge. I read the paper as an engineering feasibility study: the central claim is that a pure VQC policy selected by model-based offline policy search can balance a physical cart-pole with local simulated inference within a 15 ms budget. The most load-bearing gap I find is in the evidence for that claim, not the training pipeline: Sec. V-b states a 200-consecutive-step episode criterion, but the only hardware figure reports per-bin averages over 10 runs, and averages are compatible with a majority of failed episodes. This can be settled by releasing per-run step counts, so the concern is not a rejection of the result but a condition on how the result should be reported. The cloud-latency comparison is secondary and single-point, but it does not undercut the local real-time claim as directly. I therefore keep the reader's CONDITIONAL verdict: acceptance should require per-episode success statistics, and ideally a classical MOOSE control on the same hardware protocol.","tokens_in":7641,"tokens_out":7703,"duration_ms":87604,"concrete_test":"Re-analyze the raw per-episode step counts behind Figure 6. For each start-position bin, report (i) the number of the 10 runs that reached 200 steps, (ii) the median and interquartile range of episode lengths, and (iii) the same statistics for a classical MOOSE policy run under the identical 10-runs-per-bin hardware protocol. If, in most bins, fewer than half of the QMOOSE runs reach 200 steps, the statement that 'most bins exceed the baseline threshold' is a bin-average artifact, and the hardware-balancing claim should be weakened to 'some runs can balance,' which would require a revised verdict.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. V-b defines the baseline success criterion as 'the ability of the quantum policy to balance the real cart-pole for at least 200 consecutive steps.' The supporting hardware evidence, however, is Figure 6, which plots the average number of balancing steps over 10 runs per start-position bin, and the text concludes that 'most bins exceed the baseline threshold of 200 steps.' A bin average above 200 does not imply that individual episodes meet the criterion: one or two long runs can lift the mean above 200 while the majority of the 10 runs fail early. Because this figure is the only quantitative hardware evidence for the central claim that QMOOSE 'can balance the hardware cart-pole,' the conclusion is vulnerable to an averaging artifact unless per-run counts are provided. The paper gives no per-bin variance, medians, or success fractions, and no classical MOOSE comparison under the same hardware protocol, so neither the robustness nor the relative value of the balancing result can be assessed. This is an internal mismatch between the stated criterion and the reported metric; it is separable from, and logically prior to, the sim-to-real transfer issue the authors acknowledge.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents an offline model-based reinforcement learning pipeline in which the policy is a variational quantum circuit (VQC) with trainable input-scaling and output weights, called QMOOSE. The policy is trained on a fixed dataset collected from a physical cart-pole system, using an ensemble of learned transition models to generate synthetic rollouts. The trained quantum policy is then deployed on a Raspberry Pi/Arduino cart-pole setup, and the authors compare local simulated inference with remote IBM Cloud QPU execution. The central claims are that QMOOSE with trainable weights can balance the physical cart-pole for at least 200 steps in most start-position bins, that local simulated inference (4 +/- 1 ms) satisfies a 15 ms real-time budget, and that cloud QPU execution (about 3.7 s) does not.","tokens_in":7858,"tokens_out":7149,"duration_ms":82227,"significance":"If the central claims hold, this is a valuable empirical contribution: it is one of the few demonstrations of a pure VQC-based policy, trained via model-based offline RL, deployed on physical control hardware, and it provides a concrete latency characterization of local versus cloud quantum execution. The paper is also commendably candid about its limitations, including the sim-to-real gap and the restriction to simulated quantum circuits. However, the evidence as reported is incomplete. The hardware success criterion is defined per episode, but the only quantitative result is presented as bin averages without per-run counts, error bars, or success fractions; no classical MOOSE policy is run on the same hardware protocol; and key preprocessing and reward details are underspecified. These gaps currently prevent the reader from fully assessing the headline claim, but they are addressable within the scope of a revision.","major_comments":[{"comment":"The paper defines the baseline success criterion as the ability to balance the real cart-pole for at least 200 consecutive steps, but the only quantitative hardware evidence is Figure 6, which plots the average number of balancing steps over 10 runs per start-position bin. A bin average above 200 does not imply that individual episodes satisfy the criterion; one or two long runs can lift the mean while most runs fail early. Please report per-run counts, or at least medians, quartiles, and the fraction of runs exceeding 200 steps for each bin, along with confidence intervals. Without these, the central claim that QMOOSE 'can balance the hardware cart-pole' is not supported by the reported metric.","section":"Sec. V-b, Fig. 6"},{"comment":"The evaluation section announces a comparison against classical baselines, but no classical MOOSE policy is run on the physical hardware under the same binned protocol. Table I reports only inference times, and Figure 5 is entirely in the learned surrogate. A classical MOOSE hardware baseline is needed to interpret the absolute 200-step threshold and to assess the relative value of the quantum policy; otherwise the reader cannot tell whether the QMOOSE hardware behavior reflects a meaningful control capability or merely the permissiveness of the test protocol.","section":"Sec. V-b, Table I"},{"comment":"The preprocessing step that removes 'invalid or extreme observations, such as out-of-bounds positions or discontinuous angles' is never specified. The exact thresholds for out-of-bounds and discontinuity, and the fraction of the dataset removed, are load-bearing because the offline dataset determines both the ensemble transition model and the initial-state distribution used in Eq. (1). Without these details, the reported edge-case failures in Figure 6 cannot be attributed to data coverage, and the training pipeline cannot be reproduced.","section":"Sec. IV-A"},{"comment":"The surrogate evaluation compares classical MOOSE and QMOOSE using the same learned transition models that generated the training rollouts. This is a circular evaluation: the policy is optimized inside those models and then scored inside the same models, so the surrogate results are expected to be optimistic. The authors should state this limitation explicitly and, ideally, evaluate on held-out transition models or report disagreement across ensemble members. This does not invalidate the hardware result, but it weakens the motivation that 'positive results in the surrogate' justify deployment.","section":"Sec. V-a, Fig. 5"},{"comment":"Section III-A states that the physical system operates with an average action loop of approximately 20 ms, while Section V-b defines real-time control as requiring inference below 15 ms and Table I reports QMOOSE inference at 4 +/- 1 ms. If the 20 ms figure is the actual closed-loop control period, then the deployed system exceeds the stated 15 ms budget; if the 15 ms budget applies only to policy inference rather than the full action loop, this distinction must be stated explicitly. The manuscript should clarify which number defines the real-time requirement and report the full action-loop latency on hardware, because the claim that 'local simulated execution meets real-time requirements' is load-bearing.","section":"Sec. III-A and Sec. V-b, Table I"}],"minor_comments":[{"comment":"The statement that 'all trainable parameters are contained within the quantum circuit' is misleading: the weights w_i scale classical inputs before encoding, and w_out rescales the classical measurement outcome. The paper should say 'no classical neural network layers' rather than implying that all trainable parameters are quantum gates.","section":"Sec. II-B"},{"comment":"The reward function is described qualitatively but its coefficients are never given; a concrete equation with all coefficients is needed for reproducibility.","section":"Sec. III-B-c"},{"comment":"The final training losses are reported without noting that a lower numerical value for QMOOSE w/o TW (-18) is worse under the negative-reward convention; consider reporting returns or success rates to avoid confusion.","section":"Table I"},{"comment":"The QPU queue time is omitted due to variability; the paper should at least report a typical range, since queue time can dominate end-to-end cloud latency and the 3.7 s figure is therefore not a stable characterization.","section":"Table II"},{"comment":"The shaded standard deviation bands are described as 'across repeated episodes', but the number of repeated episodes is only 5 and no success criterion is applied; please state whether all 5 runs completed the full 200 steps.","section":"Fig. 5"},{"comment":"The hyperparameter search is described, but the search ranges and the criterion for selecting the final configuration are not reported; this limits reproducibility of the Optuna results.","section":"Sec. IV-B"},{"comment":"The paper would benefit from a data and code availability statement; the manuscript currently gives no indication that the offline dataset, preprocessing scripts, or trained circuit parameters will be released.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The empirical setup is a genuine contribution, and the paper is generally honest about limitations. The main concern is reporting: the paper should show per-run hardware data and a classical hardware baseline before the headline claim is accepted. I do not see grounds for rejection, but the manuscript needs a major revision to make the central claims fully supported and reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper reports the first deployment (as far as I can tell from the citations) of a variational-quantum-circuit policy on a physical cart-pole, trained offline with model-based policy search (QMOOSE). The genuinely new results are the hardware integration and the latency numbers: local simulated inference at roughly 4 ms per action meets the real-time budget, while IBM cloud QPU execution comes in at about 3.7 seconds per action and does not. That is a legitimate, modest engineering contribution, and the authors are appropriately careful: they state plainly that all quantum execution is simulated and that real-device evaluation remains future work.\n\nThe main thing to check before accepting the central claim is Figure 6. The text defines success as balancing the cart-pole for at least 200 consecutive steps, then supports the claim that \"most bins exceed the baseline threshold\" using bin-averaged steps over 10 runs per start position. An average above 200 does not tell you how many individual runs actually survived; a few long runs can pull the mean up while most episodes fail early. The paper provides no per-run counts, medians, variances, or success fractions, and there is no classical MOOSE baseline on the same hardware. The result is plausible, but the reported evidence is weaker than the language suggests. The surrogate evaluation in Figure 5 also uses the same learned transition model that generated the training rollouts, so that part is partly circular.\n\nThe cloud latency comparison is a single-point measurement with queue time omitted, so treat the exact 3.7 s figure as indicative, not precise. The lack of dataset and code release makes independent verification harder, though the setup sounds reasonably reproducible.\n\nOverall, this is a fair engineering demonstration, not a quantum-advantage claim. The averaging mismatch is the softest spot, and it is fixable with additional reporting. I would send it to a serious referee in quantum reinforcement learning or applied control, and I would ask for per-run success data and a same-hardware classical baseline before treating the headline claim as strong.","headline":"Deploying a VQC policy on real hardware is a useful first step, but the bin-averaged success metric in Figure 6 undercuts the central claim until per-episode data are reported.","tokens_in":8394,"tokens_out":3628,"would_cite":true,"duration_ms":39730,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A purely variational quantum circuit, trained offline as a policy, can balance a physical cart-pole when executed on a fast local simulator, but cloud quantum execution is too slow for real-time control.","keywords":["variational quantum circuits","quantum reinforcement learning","offline reinforcement learning","model-based policy search","cart-pole control","real-time control","sim-to-real transfer","latency analysis"],"falsifier":"Re-running the trained QMOOSE policy on the physical cart-pole with many episodes from every 10-unit start-position bin, while measuring per-action inference latency on the same local hardware setup, would settle the claim: the central result fails if most bins do not average 200 balancing steps or if the local inference time consistently exceeds 15 ms.","tokens_in":7438,"feed_emoji":"⚛️","tokens_out":6371,"duration_ms":71638,"temperature":0.7,"pith_summary":"This paper asks whether a quantum circuit can serve as the decision-making policy in a real-time control loop, not just in simulation. The authors train a variational quantum circuit (VQC) purely offline, using model-based policy search with an ensemble of learned dynamics models, and then deploy it on a physical cart-pole. They report that the trained quantum policy balances the cart for more than 200 steps in most starting-position bins, and that local simulated inference takes about 4 ms, well within the 15 ms real-time budget. Cloud-based quantum execution, by contrast, takes about 3.7 seconds per action, ruling it out for closed-loop control. The result matters because it shows a quantum model with no classical neural-network layers can produce usable continuous control from offline data, while identifying latency on real quantum hardware as the current bottleneck.","feed_headline":"Quantum circuit policy balances a real cart-pole","feed_subtitle":"A quantum circuit, trained offline, meets the 15 ms real-time budget locally; cloud quantum processors take seconds.","key_machinery":"The load-bearing object is the VQC policy: eight qubits, each input component $x_i$ encoded as a trainable rotation $R_X(w_i x_i)$, followed by two strongly entangling layers of $R_Z$–$R_Y$–$R_Z$ single-qubit rotations with CNOT gates in a ring, and a computational-basis measurement scaled by a trainable output weight $w_{\\mathrm{out}}$. This circuit replaces the classical policy network inside MOOSE, so policy gradients flow through fixed ensemble transition models during offline training. The trainable input weights let the circuit learn how to embed each feature, and the output weight maps the quantum expectation value to a continuous action without any classical postprocessing layer; this design is what makes the pure quantum policy trainable enough to transfer to hardware.","core_discovery":"On the paper's own terms, the central discovery is that a pure variational quantum circuit, with all trainable parameters inside the circuit, can learn a cart-pole balancing policy from a fixed offline dataset and transfer that policy to real hardware. The policy, called QMOOSE, replaces the neural network in the MOOSE offline reinforcement learning algorithm; it encodes the eight-dimensional state through trainable rotations, passes it through two strongly entangling layers, and scales the measured expectation value by a trainable output weight. In hardware evaluation, most track-position bins average at least 200 balancing steps, meeting the authors' success criterion, while local simulated inference at 4±1 ms satisfies the 15 ms real-time constraint. The same policy without trainable input-output weights fails to learn, and cloud-based QPU execution adds roughly 3.7 seconds per action, so the paper concludes that only local simulation is currently viable for latency-sensitive control.","pith_inferences":["Because the 200-step threshold at a 20 ms loop spans only about 4 seconds of balancing, the result should be read as a feasibility demonstration; longer-horizon industrial benchmarks would likely expose the edge-region failures more sharply.","The success of trainable input scaling suggests a general recipe for VQC policies in continuous control: let the encoding and the output scale be learned parameters, independent of the circuit depth.","The latency breakdown implies a design rule for quantum control: a plant whose control period exceeds the end-to-end quantum inference time could, in principle, be driven by a real QPU; the cart-pole's 15–20 ms period is far too fast.","Since the circuit is simulated rather than run on quantum hardware, the demonstration isolates algorithmic feasibility; real-device noise and measurement overhead would likely degrade both training transfer and latency, so the positive result is not yet evidence of a quantum advantage."],"forward_implications":["If the central claim holds, purely quantum policies can be trained offline and deployed in real-time control, so quantum models are not confined to simulation benchmarks.","Trainable input and output weights become a necessary ingredient for this circuit class: without them the same architecture cannot learn the balancing task.","Real-time closed-loop control with cloud-based quantum processors is ruled out at current latencies (about 3.7 s per action), so near-term deployments must use local simulation or reduce end-to-end quantum latency by orders of magnitude.","The sim-to-real gap shown in the hardware evaluation means model-based offline training should be tested across the full state space, especially edge regions, before trusting surrogate evaluations."],"supporting_citations":[{"why":"Supplies the MOOSE model-based offline policy search whose neural policy is replaced by the VQC.","marker":"[12]"},{"why":"Source of the trainable input and output weights that make the pure VQC trainable.","marker":"[13]"},{"why":"Defines the StronglyEntanglingLayers circuit template used for the two variational layers.","marker":"[14]"},{"why":"Provides the fast local quantum-circuit simulator that meets the 15 ms inference budget.","marker":"[18]"},{"why":"Underlies the real cart-pole hardware setup and figure used for deployment evaluation.","marker":"[15]"}],"fun_headline_variants":["Quantum policy balances cart-pole in real-time locally","VQC control works, but only on simulated quantum","Offline quantum policy masters hardware cart-pole","Quantum circuit passes 15ms control test locally","Real cart-pole tamed by simulated quantum policy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole result rests on the offline dataset and the learned ensemble of dynamics models being accurate enough that a policy trained inside them works on the real cart-pole; the authors' own comparison shows the surrogate overestimates hardware performance, so if the data or models are biased, the edge failures in hardware follow.","fun_headline_variants_meta":{"raw":{"variants":["Quantum policy balances cart-pole in real-time locally","VQC control works, but only on simulated quantum","Offline quantum policy masters hardware cart-pole","Quantum circuit passes 15ms control test locally","Real cart-pole tamed by simulated quantum policy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000535,"raw_usage":{"total_tokens":2525,"prompt_tokens":855,"completion_tokens":1670,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":1596}},"tokens_in":471,"tokens_out":1670,"duration_ms":14279,"temperature":1.0,"reasoning_tokens":1596,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:26:04.939938+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the trained QMOOSE policy on the physical cart-pole with many episodes from every 10-unit start-position bin, while measuring per-action inference latency on the same local hardware setup, would settle the claim: the central result fails if most bins do not average 200 balancing steps or if the local inference time consistently exceeds 15 ms.","supporting_citations":[{"cited_title":"Overcoming model bias for robust offline deep reinforcement learn- ing,","cited_arxiv_id":null,"evidence_quote":"Supplies the MOOSE model-based offline policy search whose neural policy is replaced by the VQC."},{"cited_title":"Quantum agents in the Gym: A variational quantum algorithm for deep Q-learning,","cited_arxiv_id":null,"evidence_quote":"Source of the trainable input and output weights that make the pure VQC trainable."},{"cited_title":"Circuit-centric quantum classifiers,","cited_arxiv_id":null,"evidence_quote":"Defines the StronglyEntanglingLayers circuit template used for the two variational layers."},{"cited_title":"Tensorcircuit: A quantum software framework for the nisq era,","cited_arxiv_id":null,"evidence_quote":"Provides the fast local quantum-circuit simulator that meets the 15 ms inference budget."},{"cited_title":"Inter- pretable control by reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Underlies the real cart-pole hardware setup and figure used for deployment evaluation."}],"review_version":1}