REVIEW 3 major objections 5 minor 23 references
From Classical Data to Quantum Advantage -- Quantum Policy Evaluation on Quantum Hardware
T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read This paper claims that a complete quantum reinforcement-learning pipeline—learning environment parameters from classical offline data with gradient-free QML, then using those parameters in quantum policy evaluation—can run on today's quantu
desk verdict First hardware run of QPE is a genuine engineering milestone, but the QML-to-QPE integration claim is unauditable because Section IV never reports the actual theta angles used in the QPE circuit. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrier of the argument is a small parameterized two-qubit circuit for a two-armed bandit. The action qubit, prepared in |0⟩ or |1⟩, selects an arm; a controlled Ry rotation on the reward qubit, with angle θ← or θ→, sets that arm's reward probability through P(reward)=sin²(θ/2). Learning reduces to finding the two angles that make the circuit's measured reward frequencies match the classical data. The second piece is quantum policy evaluation (QPE): a quantum phase estimation routine that, given a unitary encoding of the policy and environment, returns a quantized estimate of the policy value using O(1/ϵ) quantum samples—quadratically fewer than classical Monte Carlo. The paper's contrib
What would settle it
Compare QPE output distributions for the same environment and hardware using (a) the learned angles from Table I and (b) the ideal angles derived from the true 70%/20% winning probabilities; if the two distributions are statistically indistinguishable, the claimed end-to-end learning pipeline is not evidenced by the QPE results.
Extended reading notes
Core claim
The central discovery is a working two-stage pipeline. Classical bandit data (empirical left/right winning frequencies) train the rotation angles θ← and θ→ of a two-qubit environment circuit, using a gradient-free optimizer with a noise-mitigation wrapper; on real hardware the angles end up within 0.015 of the theoretical ideal for true reward probabilities 70%/20%. The learned environment then goes into QPE, run on a trapped-ion quantum computer: with n=3 phase-estimation qubits, estimated policy values concentrate near the true value v=0.45 for a randomized policy; n=4 is visibly corrupted by noise. A deterministic policy is almost unaffected because the environment collapses to one rotati
Load-bearing premise
The load-bearing premise is that the QPE circuits run with the rotation angles the QML step learned from the batch data, not with the known true reward probabilities inserted directly; the paper states the environment is 'as learned in Section III' but does not report those angles for the QPE circuit, so that link cannot be checked.
Editorial extensions
If this is right
- If the pipeline holds up, offline classical RL data can be used to instantiate a quantum environment on hardware, removing the need for hand-crafted environment circuits in QPE.
- QPE's quadratic sample-complexity advantage becomes practically accessible only when hardware noise is controlled enough to run the deeper n=4 and higher circuits; currently only the shallowest circuit gives reliable estimates.
- The noise robustness of deterministic policies suggests that QPE on current hardware will be most reliable in settings where the policy collapses the environment dynamics to a small number of gates.
- The small training deviation (around 0.015) indicates that gradient-free QML with error suppression can learn environment parameters well enough for simple benchmarks, so the learning stage is not the current bottleneck.
Reading between the lines
- A natural next test is to report the exact learned angles used in the QPE circuits and to compare against circuits built from the ideal angles; that would separate the QML learning quality from the QPE hardware behavior, and it is not auditable from the paper as written.
- The same two-stage recipe could be tried on multi-state MDPs, but circuit depth grows quickly with state and action count, so hardware noise—not the learning rule—will likely set the practical limit.
- The paper's framing suggests that once error mitigation improves, the quadratic qsample advantage could be demonstrated by increasing n and showing the QPE estimate converging to the true value faster than classical Monte Carlo on equivalent samples; that experiment would operationalize 'quantum advantage' for this benchmark.
- If the learned and ideal angles produce indistinguishable QPE outputs, then the current results would be evidence of hardware noise behavior rather than of learning; a reader should treat the end-to-end claim as conditional on that check.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports an end-to-end integration of quantum machine learning (QML) and quantum policy evaluation (QPE) on real quantum hardware. In Section III, the authors learn the rotation parameters of a two-armed bandit environment from classical batch data using gradient-free COBYLA optimization with Q-CTRL noise mitigation on IBM hardware. In Section IV, they implement QPE on IonQ's Forte-1 device, claiming the environment is 'as learned in Section III,' and estimate the values of two policies. The results show learned parameters close to the theoretical inverse-sine formulas (Eqs. 8-9), and QPE histograms that peak near the true values for n=3 but degrade for deeper circuits under noise. The paper concludes that, despite hardware limitations, the QML-QPE integration shows promise for quantum advantage in reinforcement learning.
Significance. If the end-to-end link is fully supported, this is a useful first hardware demonstration of QPE and of using QML to construct quantum environment models from classical data. The paper's strengths include the use of actual quantum hardware for both stages, the choice of a gradient-free optimizer appropriate for noisy devices, the explicit formulas in Eqs. (8)-(9) that allow direct cross-checking of learned parameters, and an honest discussion of noise-induced degradation. However, the central integration claim is currently unauditable because the QPE circuit parameters are not reported, no error bars accompany the hardware histograms, and the loss values in Section III appear inconsistent with the stated MSE cost. The theoretical quantum-advantage claim is inherited from prior work [6]; the present experiments do not themselves compare against classical baselines, so the title's 'quantum advantage' should be read prospectively rather than as a demonstrated result.
major comments (3)
- [Section IV.B / Table I] The QPE circuits are said to use the environment 'as learned in Section III,' but the exact rotation angles inserted into the circuits are never reported. Table I gives learned values θ←=1.96, θ→=0.91 for the 70%/20% environment, while the exact theoretical values from Eqs. (8)-(9) are θ←≈1.982, θ→≈0.927. These are close enough that the histograms in Fig. 5 would be nearly indistinguishable, so the figures cannot confirm that the learned parameters were actually used. The abstract's claim that 'the learned quantum environment is then applied in QPE' is load-bearing; without reporting the circuit angles (or providing code/data), the end-to-end QML-to-QPE link cannot be audited. Please state the exact θ values used for each QPE run and, ideally, provide the circuit construction details as supplementary material.
- [Section III.C and III.F] The cost function is defined in Section III.C as the MSE between measured and empirical relative frequencies, which for two probabilities in [0,1] can be at most 2. Yet Section III.F reports L_opt ≈ 111.3 for the 70%/20% case and L_opt ≈ 69.3 for the 0%/50% case. These values are incompatible with the stated MSE. Please define the actual loss function (e.g., count-based, weighted, or rescaled MSE), and report the empirical batch frequencies f_data used for each run. Table I's 'Empirical' columns (1.98 and 0.93) coincide exactly with the inverse-sine of the true probabilities 0.7 and 0.2; please clarify whether these are finite-sample estimates or the exact theoretical values, and if finite, provide the sample sizes and measured frequencies.
- [Section IV.C / Figure 5] The QPE results are presented as count histograms over 300 runs, but no error bars or confidence intervals are given. Given that each bin count is a binomial random variable, standard errors are easy to compute and would allow the reader to judge whether the differences between n=3 and n=4, and between simulator and hardware, are statistically meaningful. Without these, the qualitative comparison in Section IV.C (e.g., 'more shots are farther away from the true value') is not quantitatively supported. Please add error bars or per-bin counts with uncertainties.
minor comments (5)
- [Section II / IV.A] The abbreviation 'QPE' is used both for 'quantum policy evaluation' and 'quantum phase estimation' in Section IV.A. Please disambiguate to avoid confusion.
- [Section III.F] The text says 'the deviation is 0.015 for both parameters' but does not specify whether this is the absolute difference from the theoretical values in Eqs. (8)-(9) or from the empirical values in Table I. Please clarify.
- [Section IV.B] Minor typos: '300times' should be '300 times', and 'noticable' should be 'noticeable' in Section IV.C.
- [Section IV.B] The hardware name is given as 'forte-1' but elsewhere as 'Forte-1' (and the reference uses 'Forte'). Please standardize the capitalization.
- [Conclusion / Abstract] The phrase 'quantum advantage' in the title and abstract is stronger than what the experiments demonstrate. The conclusion appropriately says 'potential quantum advantage,' so please align the abstract/title with that more cautious framing, or add a classical baseline comparison.
Circularity Check
No significant circularity: the QML fit is an in-sample benchmark and the QPE experiment is a hardware execution test with known ground-truth values; the only issue is an audit gap in reporting QPE theta, which is not a circular step.
full rationale
The paper's derivation chain is: (i) classical reward frequencies f_data define the target; (ii) QML adjusts theta so that sin^2(theta/2) matches f_data (Section III.C); (iii) QPE is executed with an environment circuit of the same form to estimate the policy value v = Pi(<-)*sin^2(theta_<-/2) + Pi(->)*sin^2(theta_->/2). Step (ii) is an ordinary fit: the cost is MSE against f_data, and the direct formulas (8)-(9) are explicitly labeled as reference points. The learned parameters are compared to those reference values in Table I, so the QML claim is benchmarked against an external analytic expression. Step (iii) is a hardware benchmark of a known algorithm: the target values v50=0.45 and v0=0.2 are computed from the known bandit probabilities (70%/20%), not from the learned theta, and Figure 5 compares QPE output distributions to these nominal values. Running QPE on a quantum computer to compute a known function of the environment is not a tautology; it is an experimental test of the implementation under noise. The citation to [6] (Wiedemann et al.) for the QPE algorithm is a normal citation to prior published work by a co-author, and it is not used to forbid alternatives or to supply an unverified premise; the present experiments execute that algorithm rather than deriving it. The manuscript has a genuine reproducibility gap: Section IV.B does not report the theta values actually loaded into the QPE circuit, so an external reviewer cannot verify that the 'learned' rather than 'exact' angles were used. That gap affects the end-to-end integration claim and should be addressed, but it is an auditability/correctness issue, not a circular step: no equation or fitted parameter is renamed as an independent prediction. Therefore no circularity is established under the required standard.
Assumptions & free parameters
free parameters (2)
- theta_left =
1.96 (70%/20% case); -0.08 (0%/50% case)
- theta_right =
0.91 (70%/20% case); 1.55 (0%/50% case)
assumptions (4)
- domain assumption The two-armed bandit is exactly representable by the two-qubit circuit with policy and environment unitary segments (Fig. 2, Eqs. 5-7).
- domain assumption QPE approximation error bound P(|v_tilde - v| <= epsilon(n)) >= 8/pi^2 with epsilon(n) in O(1/2^n) (Eq. 11).
- domain assumption Measurement statistics on real hardware are unbiased estimators of the ideal circuit probabilities up to noise treated by Q-CTRL mitigation.
- domain assumption COBYLA converges to a satisfactory local minimum for this two-parameter noisy optimization.
Cite this review
Pith. "Pith review of From Classical Data to Quantum Advantage -- Quantum Policy Evaluation on Quantum Hardware." pith.science (2026). https://pith.science/paper/Y7V7EEIR
@misc{pith2026250907614,
author = {Pith},
title = {Pith review of: From Classical Data to Quantum Advantage -- Quantum Policy Evaluation on Quantum Hardware},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y7V7EEIR}},
note = {Machine review of arXiv:2509.07614}
}
read the original abstract
Quantum policy evaluation (QPE) is a reinforcement learning (RL) algorithm which is quadratically more efficient than an analogous classical Monte Carlo estimation. It makes use of a direct quantum mechanical realization of a finite Markov decision process, in which the agent and the environment are modeled by unitary operators and exchange states, actions, and rewards in superposition. Previously, the quantum environment has been implemented and parametrized manually for an illustrative benchmark using a quantum simulator. In this paper, we demonstrate how these environment parameters can be learned from a batch of classical observational data through quantum machine learning (QML) on quantum hardware. The learned quantum environment is then applied in QPE to also compute policy evaluations on quantum hardware. Our experiments reveal that, despite challenges such as noise and short coherence times, the integration of QML and QPE shows promising potential for achieving quantum advantage in RL.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[6]
S. Wiedemann et al., “Quantum policy iteration via am- plitude estimation and Grover search – towards quantum advantage for reinforcement learning,”Transactions on Machine Learning Research, 2023
work page 2023
-
[1]
A survey on quantum reinforcement learning,
N. Meyer et al., “A survey on quantum reinforcement learning,”arXiv preprint, 2022
work page 2022
-
[2]
M. A. Nielsen and I. L. Chuang,Quantum computation and quantum information. Cambridge univ. press, 2010
work page 2010
-
[3]
Quantum algorithms for reinforcement learning with a generative model,
D. Wang et al., “Quantum algorithms for reinforcement learning with a generative model,” inInternational Conference on Machine Learning, PMLR, 2021
work page 2021
-
[4]
Quantum reinforcement learning via policy iteration,
E. A. Cherrat, I. Kerenidis, and A. Prakash, “Quantum reinforcement learning via policy iteration,”Quantum Machine Intelligence, 2023
work page 2023
-
[5]
Quantum- enhanced machine learning,
V . Dunjko, J. M. Taylor, and H. J. Briegel, “Quantum- enhanced machine learning,”Phys. rev. let., 2016
work page 2016
-
[7]
Noisy intermediate-scale quantum algorithms,
K. Bharti et al., “Noisy intermediate-scale quantum algorithms,”Reviews of Modern Physics, 2022
work page 2022
-
[8]
Variational quantum circuits for deep reinforcement learning,
S. Y .-C. Chen et al., “Variational quantum circuits for deep reinforcement learning,”IEEE access, 2020
work page 2020
Show all 23 references
-
[9]
Model-based offline quantum rein- forcement learning,
S. Eisenmann et al., “Model-based offline quantum rein- forcement learning,” inIEEE International Conference on Quantum Computing and Engineering, QCE, 2024
2024
-
[10]
A fast quantum mechanical algorithm for database search,
L. K. Grover, “A fast quantum mechanical algorithm for database search,” inProceedings of the twenty-eighth annual ACM symposium on Theory of computing, 1996
1996
-
[11]
Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer,
P. W. Shor, “Polynomial-time algorithms for prime factorization and discrete logarithms on a quantum computer,”SIAM review, 1999
1999
-
[12]
Reinforcement learning with quantum variational circuit,
O. Lockwood and M. Si, “Reinforcement learning with quantum variational circuit,” inProc. of the AAAI conf. on AI and interactive digital entertainment, 2020
2020
-
[13]
Comparing quantum hybrid reinforcement learning to classical methods,
M. Moll and L. Kunczik, “Comparing quantum hybrid reinforcement learning to classical methods,”Human- Intelligent Systems Integration, 2021
2021
-
[14]
Quantum agents in the Gym: A variational quantum algorithm for deep Q-learning,
A. Skolik, S. Jerbi, and V . Dunjko, “Quantum agents in the Gym: A variational quantum algorithm for deep Q-learning,”Quantum, 2022
2022
-
[15]
Parametrized quantum policies for re- inforcement learning,
S. Jerbi et al., “Parametrized quantum policies for re- inforcement learning,”Advances in Neural Information Processing Systems, 2021
2021
-
[16]
Framework for learning agents in quantum environments,
V . Dunjko, J. M. Taylor, and H. J. Briegel, “Framework for learning agents in quantum environments,”arXiv preprint, 2015
2015
-
[17]
Exponential improvements for quantum-accessible reinforcement learning,
V . Dunjko et al., “Exponential improvements for quantum-accessible reinforcement learning,”arXiv preprint, 2017
2017
-
[18]
Quantum- accessible reinforcement learning beyond strictly epochal environments,
A. Hamann, V . Dunjko, and S. W ¨olk, “Quantum- accessible reinforcement learning beyond strictly epochal environments,”Quantum Mach. Int., 2021
2021
-
[19]
Quantum circuit evolution on NISQ devices,
L. Franken et al., “Quantum circuit evolution on NISQ devices,” in2022 IEEE Congress on Evolutionary Com- putation (CEC), 2022
2022
-
[20]
Benchmark- ing of different optimizers in the variational quantum algorithms for applications in quantum chemistry,
H. Singh, S. Majumder, and S. Mishra, “Benchmark- ing of different optimizers in the variational quantum algorithms for applications in quantum chemistry,”The Journal of Chemical Physics, 2023
2023
-
[21]
M. J. Powell,A direct search optimization method that models the objective and constraint functions by linear interpolation. Springer, 1994
1994
-
[22]
Experimental benchmarking of an automated deterministic error-suppression workflow for quantum algorithms,
P. S. Mundada et al., “Experimental benchmarking of an automated deterministic error-suppression workflow for quantum algorithms,”Physical Review Applied, 2023
2023
-
[23]
Benchmarking a trapped-ion quantum computer with 30 qubits,
J.-S. Chen et al., “Benchmarking a trapped-ion quantum computer with 30 qubits,”Quantum, 2024
2024
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.