REVIEW 3 major objections 5 minor 2 cited by
Automated quantum programming via reinforcement learning for combinatorial optimization
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Reinforcement learning agent writes quantum programs that beat QAOA
desk verdict The RL-based quantum program synthesis is a genuine step forward, but the headline 'beats QAOA' claim rests on an early-stopping scoring asymmetry that needs a control before the claim is credible. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the reinforcement-learning environment defined as a partially observed Markov decision process. The agent's observation is a $10\times 10$ binary array of measurement shots plus the vectorized problem weights; its actions are single-qubit X/Y/Z rotations by multiples of $\pi/4$ and CNOT gates; the reward is the normalized expectation of the problem Hamiltonian estimated from ten shots. The agent is trained with proximal policy optimization (PPO), a clipped policy-gradient method, and constructs programs incrementally until either the normalized reward exceeds 0.8 or the program reaches 25 uncompiled instructions. This incremental prefix-reward structure is what lets the agent learn to stop early at good states, and is also the source of the scoring asymmetry against QAOA.
What would settle it
Rerun the agent and single-step QAOA on the same test instances using a single scoring rule, either final-circuit reward for both or best-prefix reward for both, and compare the mean episode scores; if the agent no longer leads across all problem types and both resources, the central claim fails.
Extended reading notes
Core claim
The central discovery is that a PPO-trained agent, given only sampled bitstrings and problem weights as observations, learns gate-sequences whose mean test performance exceeds $p=1$ QAOA across all three problem classes on both simulated and physical quantum resources. The agent's episode score is the maximum normalized expectation of the cost Hamiltonian over any prefix of the generated program, while QAOA is scored at its final circuit; the advantage appears in every problem/resource combination tested. On the physical device, the QPU-trained agent maintains comparable episode scores to the simulator-trained agent while cutting compiled program lengths from over 100 instructions to under 50 for MAXCUT. The authors also report that the learned policies favor CNOT, $\mathrm{RX}(\pi)$, and $\mathrm{RY}(\pi)$ gates, supporting their hypothesis that optimal programs for computational-basis-diagonal Hamiltonians amount to preparing a solution bitstring with essentially X-type flips.
Load-bearing premise
The load-bearing premise is that it is fair to score the trained agent by its maximum reward over any program prefix while scoring QAOA only by its final-circuit reward.
Editorial extensions
If this is right
- Trained policies transfer from a noiseless simulator to a physical processor with similar expected episode scores, so this style of synthesis does not require training directly on the target hardware to be useful.
- Training on the hardware itself shortens compiled programs (e.g., MAXCUT compiled lengths drop below 50 instructions) without sacrificing episodic reward, meaning device noise can act as an implicit regularizer for program length.
- Learned programs are substantially shorter than $p=1$ QAOA circuits, which the authors expect to make them less vulnerable to decoherence on near-term devices.
- The agent generalizes to held-out instances of all three problem classes and is less sensitive to problem type than QAOA, suggesting the learned policy captures a common structure in these diagonal-Hamiltonian optimizations.
Reading between the lines
- This reader's inference: the headline comparison is sensitive to the scoring rule; if QAOA were scored by its best intermediate expectation value instead of its final value, or the agent were forced to report only its final reward, the reported mean advantage could shrink or reverse.
- This reader's inference: because the reward and observation are tailored to computational-basis measurements, the method is most naturally a compiler of bitstring-preparation circuits; for problems requiring superposition or phase information, the observation space and gate set would need to be enriched, and the paper notes such extensions are untested.
- This reader's inference: an immediate empirical extension is to apply the same environment to a Hamiltonian that is not diagonal in the computational basis and check whether the learned policies begin to use non-Clifford rotations; the paper's own gate-frequency analysis suggests such a change would be detectable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a reinforcement-learning framework, based on PPO, that incrementally constructs short quantum programs from a discrete gate set to solve 10-variable MAXCUT, MAXQP, and QUBO instances. Agents are trained on a shuffled mixture of 50,000 instances, validated on 12,000, and tested on 3,000 instances per problem type, using both the Rigetti QVM simulator and the Aspen QPU. The central claim, stated in Section 3.2, is that the trained agents achieve higher mean episode scores than both untrained agents and p=1 QAOA on both simulated and physical quantum resources. The paper also reports that the trained agents generate shorter programs than QAOA and presents action-frequency statistics and sample generated programs.
Significance. If the central claim is supported, this is a useful demonstration that reinforcement learning can synthesize short, problem-specific quantum programs for small combinatorial optimization instances and that such policies can transfer from a simulator to real hardware. The study is clearly described, uses a standard RL algorithm, and reports sample programs and hyperparameter details, which aids reproducibility. However, the headline comparison currently rests on an asymmetric evaluation metric and a noise-limited baseline; as reported, the evidence does not yet convincingly establish that the trained agents outperform QAOA in an apples-to-apples sense.
major comments (3)
- [Section 3.2, episode-score definitions] The evaluation metric is asymmetric: ES_agent is defined as the maximum reward over all program prefixes, while ES_QAOA is the reward of the final p=1 ansatz. Because the reward sequence is non-monotonic, as the supplementary program examples show (e.g., the reward dips after repeated RX(pi/4) gates before later rising), max-over-prefix can exceed the reward of the program executed as a whole. This gives the RL agent an early-stopping advantage that QAOA is not granted. Please report ES_agent using the final reward r_m, or equivalently allow QAOA to be scored by its best intermediate state, and re-evaluate the central claim with the symmetric metric.
- [Section 3.2, QAOA baseline on the QPU] The QPU comparison is confounded by program length and noise. QAOA programs compile to more than 102 instructions on the QPU, while the trained agent programs are much shorter, and the paper itself states that the untrained agent outperforms QAOA on the QPU. This indicates that the p=1 QAOA baseline is noise-limited on hardware, so the observation that the trained agent beats QAOA on the QPU does not establish that the synthesized programs are better quantum algorithms. The claim should be qualified to the simulator, or a noise-matched or instruction-length-matched baseline should be provided.
- [Section 3.2 and Figures 2-4] The central empirical claim is made on the basis of mean episode scores, but the paper reports no confidence intervals, standard errors, paired comparisons, or significance tests for the differences between the trained agent, untrained agent, and QAOA. Because the test set contains 3,000 instances per problem type, such statistics are straightforward to compute and should be reported to support the claim that the trained agent's performance 'exceeds' the baselines.
minor comments (5)
- [Supplementary Information, Section 2] There is a typo: 'With the except of the hyperparameters' should read 'With the exception of the hyperparameters.'
- [Supplementary Information, Sections 1-2] The symbol m is used for both the number of measurement shots in the supplementary information and the final instruction index in Section 3.2 of the main text; these overlapping notations should be disambiguated.
- [Section 2, Eq. (1)] The discounted-return expression in Eq. (1) is written as a sum over k with reward r_{t+k+1}; the indexing should be made consistent with the discount factor, for example by writing sum_{k=0}^\infty \gamma^k r_{t+k+1} explicitly.
- [Section 4] The statement that 'we cannot use the policy improvement theorem to improve upon this policy' is unclear: the policy improvement theorem concerns policies, not circuit families, and the reasoning should be expanded or rephrased.
- [General] No code or data availability statement is provided; making the Gym environment, trained models, and test-instance generation scripts available would materially improve reproducibility.
Circularity Check
No circularity: the agent-vs-QAOA comparison is an empirical benchmark, not a derivation from fitted inputs; the max-over-prefix episode score is an explicit scoring convention, not a construction that forces the result.
full rationale
The paper's central claim (Section 3.2) is an empirical comparison of mean episode scores for trained RL agents, untrained agents, and p=1 QAOA on held-out MAXCUT, MAXQP, and QUBO instances. The reward normalization r_i = (\langle\psi_i|H_C|\psi_i\rangle - m)/(M-m) uses the exact classical min/max as a benchmark scale; the paper explicitly acknowledges that M is 'precisely what we set out to compute,' but this is a normalization constant, not a fitted parameter, and it is applied identically to the agent and QAOA. The episode-score definitions ES_agent := max_i r_i and ES_QAOA := r_m are an asymmetric early-stopping convention: the agent is credited with its best prefix reward while QAOA is scored by its final circuit. This is a methodological comparability limitation that could affect the empirical gap, but it is not a circular reduction because the comparison is measured, not derived: the agent score is not defined to equal the QAOA score, and the paper does not claim the advantage follows from the definitions. PPO hyperparameters were tuned on a validation set, which is standard practice, and the QAOA parameters were optimized by exact grid search per instance. No load-bearing self-citation or imported uniqueness theorem appears; references to Rigetti's QVM and Quilc are implementation tools, not premises that assume the target result. Therefore no step in the derivation chain reduces to its own inputs.
Assumptions & free parameters
free parameters (4)
- win_threshold =
0.8
- max_uncompiled_program_length =
25
- measurement_shots =
10
- PPO_hyperparameters =
lambda=0.999, n_steps=512, batch_size=512, clip=0.2
assumptions (5)
- standard math The environment can be modeled as a POMDP and PPO converges to a near-optimal policy with sufficient training.
- domain assumption Quantum state evolution is unitary for ideal gates, measurement samples follow Born probabilities, and device noise is reflected in sampled bitstrings.
- domain assumption For n=10, exact minimum and maximum cost Hamiltonian expectation values can be computed classically by enumerating all 2^10 bitstrings.
- domain assumption The three cost Hamiltonians considered are diagonal in the computational basis, so computational-basis measurements yield candidate solutions directly.
- ad hoc to paper Training on a shuffled mixture of MAXCUT, MAXQP, and QUBO instances is sufficient to learn a policy that transfers to held-out instances of all three problem types.
Cite this review
Pith. "Pith review of Automated quantum programming via reinforcement learning for combinatorial optimization." pith.science (2026). https://pith.science/paper/ANLADOVX
@misc{pith2026190808054,
author = {Pith},
title = {Pith review of: Automated quantum programming via reinforcement learning for combinatorial optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/ANLADOVX}},
note = {Machine review of arXiv:1908.08054}
}
read the original abstract
We develop a general method for incentive-based programming of hybrid quantum-classical computing systems using reinforcement learning, and apply this to solve combinatorial optimization problems on both simulated and real gate-based quantum computers. Relative to a set of randomly generated problem instances, agents trained through reinforcement learning techniques are capable of producing short quantum programs which generate high quality solutions on both types of quantum resources. We observe generalization to problems outside of the training set, as well as generalization from the simulated quantum resource to the physical quantum resource.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 2 Pith papers
-
Quantum Architecture Search for Solving Quantum Machine Learning Tasks
A reinforcement learning framework (RL-QAS) discovers compact variational quantum circuit architectures for Iris and binary MNIST classification, outperforming a simple strongly-entangling-layer baseline.
-
Incorporating Quantum Advantage in Quantum Circuit Generation through Genetic Programming
Two fitness functions that reward fewer oracle calls and require superposition or entanglement gates evolve Bernstein-Vazirani and 3-qubit search circuits faster than the baseline.
Reference graph
Works this paper leans on
-
[1]
Ashley Hill et al. Stable Baselines. https://github.com/hill-a/stable-baselines . 2018
work page 2018
-
[2]
A practical quantum instruction set architecture
Robert S Smith, Michael J Curtis, and William J Zeng. “A practical quantum instruction set architecture”. In: arXiv preprint arXiv:1608.03355 (2016). 5
arXiv 2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.