{"id":"062cc079-5b62-419a-bdd8-3f22fc5c9850","arxiv_id":"1908.08054","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A reinforcement learning agent writes short quantum programs that outperform p=1 QAOA on random 10-variable MAXCUT, MAXQP, and QUBO instances on both a simulator and a physical quantum processor.","lead":"This paper trains a reinforcement learning agent to write short quantum gate sequences that solve three combinatorial optimization problems (MAXCUT, MAXQP, QUBO) on a simulator and on Rigetti's real quantum processor. A generalist might read it to see whether machine learning can replace hand-designed quantum optimization circuits in near-term hardware.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The agent-vs-QAOA comparison uses max-over-prefix reward for the agent but final-circuit reward for QAOA; if final-state reward is used, the claimed advantage may shrink or disappear.","rationale":"The paper's empirical demonstration is internally consistent: the environment, action space, reward, and training procedure are described in enough detail that the reported experiments are plausible, and the paper openly acknowledges in Section 5 that scaling behavior is not investigated. The central claim, however, is defined by a specific metric comparison, and the metric is asymmetric: the agent gets credit for its best intermediate reward, while QAOA is scored only at the end of its fixed ansatz. This is not a minor implementation detail; it is the literal definition of the dependent variable in the headline result. The Supplementary Information's sample programs confirm that reward trajectories are non-monotonic, so the max-over-prefix choice can materially inflate agent scores. A simple reanalysis of the existing trajectories would settle whether the claim survives, and no retraining is required. Secondary concerns, such as the weak p=1 QAOA baseline and the absence of error bars, are real but less decisive: even a stronger baseline would not change the fact that the reported comparison metric is asymmetric. Because the reader already identified this same load-bearing assumption and issued a conditional verdict, my assessment does not move the verdict; it reinforces the condition under which the paper should be accepted: the authors should rerun the analysis with a symmetric scoring protocol, or explicitly justify max-over-prefix as the intended program output semantics.","tokens_in":12037,"tokens_out":5250,"duration_ms":63091,"concrete_test":"Recompute the test-set mean episode scores for all three problem types on both QVM and QPU using ES_agent := r_m (the reward at the final action of the episode, or at the termination condition) instead of max_i r_i, keeping ES_QAOA := r_m unchanged. Compare the means and bootstrap confidence intervals for the agent-versus-QAOA gap. If the corrected agent mean no longer exceeds QAOA on both resources, the headline comparison fails; if it still exceeds, the early-stopping concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 3.2 rests entirely on the episode-score definitions: ES_agent := max_{1<=i<=m} r_i, while ES_QAOA := r_m, where m is the final ansatz instruction. This is an early-stopping asymmetry: the RL agent is credited with the best reward achieved at any prefix of its program, whereas QAOA is scored only at the end of its p=1 circuit. The sample programs in the Supplementary Information show that the reward along an episode is non-monotonic (e.g., after sequences of RX(pi/4) gates, r decreases before later increases), so max-over-prefix can exceed the reward of the program considered as a whole. QAOA is given no analogous option to stop at a partial ansatz or to report its best intermediate state. Since the headline result is a comparison of these two metrics, the reported advantage could be a protocol artifact rather than a statement about the quality of the synthesized programs. The same normalization is used for both methods, so the asymmetry is specifically in the output-selection rule, not in the reward scale. This is the most load-bearing assumption because if the agent were required to report only the reward of its final program, or if QAOA were allowed the same early-stopping privilege, the mean gap could shrink or reverse. The paper does not report the final-reward analogue or any control for this effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reinforcement-learning framework, based on PPO, that incrementally constructs short quantum programs from a discrete gate set to solve 10-variable MAXCUT, MAXQP, and QUBO instances. Agents are trained on a shuffled mixture of 50,000 instances, validated on 12,000, and tested on 3,000 instances per problem type, using both the Rigetti QVM simulator and the Aspen QPU. The central claim, stated in Section 3.2, is that the trained agents achieve higher mean episode scores than both untrained agents and p=1 QAOA on both simulated and physical quantum resources. The paper also reports that the trained agents generate shorter programs than QAOA and presents action-frequency statistics and sample generated programs.","tokens_in":12346,"tokens_out":4692,"duration_ms":50418,"significance":"If the central claim is supported, this is a useful demonstration that reinforcement learning can synthesize short, problem-specific quantum programs for small combinatorial optimization instances and that such policies can transfer from a simulator to real hardware. The study is clearly described, uses a standard RL algorithm, and reports sample programs and hyperparameter details, which aids reproducibility. However, the headline comparison currently rests on an asymmetric evaluation metric and a noise-limited baseline; as reported, the evidence does not yet convincingly establish that the trained agents outperform QAOA in an apples-to-apples sense.","major_comments":[{"comment":"The evaluation metric is asymmetric: ES_agent is defined as the maximum reward over all program prefixes, while ES_QAOA is the reward of the final p=1 ansatz. Because the reward sequence is non-monotonic, as the supplementary program examples show (e.g., the reward dips after repeated RX(pi/4) gates before later rising), max-over-prefix can exceed the reward of the program executed as a whole. This gives the RL agent an early-stopping advantage that QAOA is not granted. Please report ES_agent using the final reward r_m, or equivalently allow QAOA to be scored by its best intermediate state, and re-evaluate the central claim with the symmetric metric.","section":"Section 3.2, episode-score definitions"},{"comment":"The QPU comparison is confounded by program length and noise. QAOA programs compile to more than 102 instructions on the QPU, while the trained agent programs are much shorter, and the paper itself states that the untrained agent outperforms QAOA on the QPU. This indicates that the p=1 QAOA baseline is noise-limited on hardware, so the observation that the trained agent beats QAOA on the QPU does not establish that the synthesized programs are better quantum algorithms. The claim should be qualified to the simulator, or a noise-matched or instruction-length-matched baseline should be provided.","section":"Section 3.2, QAOA baseline on the QPU"},{"comment":"The central empirical claim is made on the basis of mean episode scores, but the paper reports no confidence intervals, standard errors, paired comparisons, or significance tests for the differences between the trained agent, untrained agent, and QAOA. Because the test set contains 3,000 instances per problem type, such statistics are straightforward to compute and should be reported to support the claim that the trained agent's performance 'exceeds' the baselines.","section":"Section 3.2 and Figures 2-4"}],"minor_comments":[{"comment":"There is a typo: 'With the except of the hyperparameters' should read 'With the exception of the hyperparameters.'","section":"Supplementary Information, Section 2"},{"comment":"The symbol m is used for both the number of measurement shots in the supplementary information and the final instruction index in Section 3.2 of the main text; these overlapping notations should be disambiguated.","section":"Supplementary Information, Sections 1-2"},{"comment":"The discounted-return expression in Eq. (1) is written as a sum over k with reward r_{t+k+1}; the indexing should be made consistent with the discount factor, for example by writing sum_{k=0}^\\infty \\gamma^k r_{t+k+1} explicitly.","section":"Section 2, Eq. (1)"},{"comment":"The statement that 'we cannot use the policy improvement theorem to improve upon this policy' is unclear: the policy improvement theorem concerns policies, not circuit families, and the reasoning should be expanded or rephrased.","section":"Section 4"},{"comment":"No code or data availability statement is provided; making the Gym environment, trained models, and test-instance generation scripts available would materially improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest exploratory study, but the headline comparison is under-supported by the current evaluation protocol. The main fix is straightforward: report a symmetric episode-score metric and add statistical rigor. I do not see grounds for rejection if the authors can re-run or re-analyze the comparison."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is application-level synthesis: an RL agent that outputs a whole gate sequence for a combinatorial optimization problem, trained on a simulator and then transferred to a real QPU. Earlier RL work mostly targeted gate-level control, state preparation, or error correction, not the full program. Running on Aspen across three problem types (MAXCUT, MAXQP, QUBO) and showing sim-to-device transfer is a real step. The paper also does useful things: it reports the compiled instruction counts, shows the agent's programs are shorter than QAOA's after compilation, and honestly flags the lack of a scaling study in Section 5. The sample programs in the SI are a nice touch.\n\nThe soft spot is exactly the one the stress-test flags. ES_agent is max reward over prefixes; ES_QAOA is final reward. That asymmetry is load-bearing, because the central claim in Section 3.2 is a comparison of these two metrics. The SI reward traces show non-monotonic behavior, so max-over-prefix can beat final-state reward by a lot. The paper never reports the agent's final-reward analogue. If the agent is forced to report only its final program's reward, or QAOA is allowed to stop at its best prefix, the reported advantage might shrink or disappear. This needs a control before the 'exceeds QAOA' claim should be published. Also, p=1 QAOA with a coarse gamma-beta grid is a weak baseline—no classical heuristic or higher-depth QAOA is compared—and there are no error bars or significance tests. The paper is otherwise competent and the authors were honest about the scaling limitation; the lack of code/data is a reproducibility cost but not disqualifying.\n\nThis deserves a serious referee. The method is plausible and the QPU evidence is rare. But I would want a control with symmetric scoring rules and a strong classical baseline before accepting. Conditional accept with a required revision.","headline":"The RL-based quantum program synthesis is a genuine step forward, but the headline 'beats QAOA' claim rests on an early-stopping scoring asymmetry that needs a control before the claim is credible.","tokens_in":12833,"tokens_out":1918,"would_cite":true,"duration_ms":22068,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reinforcement learning agent writes quantum programs that beat QAOA","keywords":["reinforcement learning","quantum program synthesis","combinatorial optimization","QAOA","MAXCUT","MAXQP","QUBO","near-term quantum computing"],"falsifier":"Rerun the agent and single-step QAOA on the same test instances using a single scoring rule, either final-circuit reward for both or best-prefix reward for both, and compare the mean episode scores; if the agent no longer leads across all problem types and both resources, the central claim fails.","tokens_in":11846,"feed_emoji":"⚛️","tokens_out":8740,"duration_ms":77548,"temperature":0.7,"pith_summary":"This paper tries to establish that a reinforcement learning agent can act as a practical quantum program synthesizer for combinatorial optimization. Given a problem instance, the agent builds a quantum circuit gate by gate, receiving rewards from measured samples of the prepared state, and learns a policy that produces short, high-reward programs. On held-out random instances of MAXCUT, MAXQP, and QUBO, the trained agent's mean episode score exceeds both an untrained random agent and single-step QAOA, on a simulator and on a physical superconducting quantum processor. The authors also find that programs trained on the simulator retain their score when run on the hardware, and that further training on hardware produces shorter compiled programs. If the claim holds, reinforcement learning offers a general incentive-based route to writing quantum programs for near-term devices without hand-designing an ansatz.","feed_headline":"Reinforcement learning agent writes quantum programs that beat QAOA","feed_subtitle":"On held-out MAXCUT, MAXQP, and QUBO instances, it out-scores single-step QAOA on simulator and hardware.","key_machinery":"The load-bearing object is the reinforcement-learning environment defined as a partially observed Markov decision process. The agent's observation is a $10\\times 10$ binary array of measurement shots plus the vectorized problem weights; its actions are single-qubit X/Y/Z rotations by multiples of $\\pi/4$ and CNOT gates; the reward is the normalized expectation of the problem Hamiltonian estimated from ten shots. The agent is trained with proximal policy optimization (PPO), a clipped policy-gradient method, and constructs programs incrementally until either the normalized reward exceeds 0.8 or the program reaches 25 uncompiled instructions. This incremental prefix-reward structure is what lets the agent learn to stop early at good states, and is also the source of the scoring asymmetry against QAOA.","core_discovery":"The central discovery is that a PPO-trained agent, given only sampled bitstrings and problem weights as observations, learns gate-sequences whose mean test performance exceeds $p=1$ QAOA across all three problem classes on both simulated and physical quantum resources. The agent's episode score is the maximum normalized expectation of the cost Hamiltonian over any prefix of the generated program, while QAOA is scored at its final circuit; the advantage appears in every problem/resource combination tested. On the physical device, the QPU-trained agent maintains comparable episode scores to the simulator-trained agent while cutting compiled program lengths from over 100 instructions to under 50 for MAXCUT. The authors also report that the learned policies favor CNOT, $\\mathrm{RX}(\\pi)$, and $\\mathrm{RY}(\\pi)$ gates, supporting their hypothesis that optimal programs for computational-basis-diagonal Hamiltonians amount to preparing a solution bitstring with essentially X-type flips.","pith_inferences":["This reader's inference: the headline comparison is sensitive to the scoring rule; if QAOA were scored by its best intermediate expectation value instead of its final value, or the agent were forced to report only its final reward, the reported mean advantage could shrink or reverse.","This reader's inference: because the reward and observation are tailored to computational-basis measurements, the method is most naturally a compiler of bitstring-preparation circuits; for problems requiring superposition or phase information, the observation space and gate set would need to be enriched, and the paper notes such extensions are untested.","This reader's inference: an immediate empirical extension is to apply the same environment to a Hamiltonian that is not diagonal in the computational basis and check whether the learned policies begin to use non-Clifford rotations; the paper's own gate-frequency analysis suggests such a change would be detectable."],"forward_implications":["Trained policies transfer from a noiseless simulator to a physical processor with similar expected episode scores, so this style of synthesis does not require training directly on the target hardware to be useful.","Training on the hardware itself shortens compiled programs (e.g., MAXCUT compiled lengths drop below 50 instructions) without sacrificing episodic reward, meaning device noise can act as an implicit regularizer for program length.","Learned programs are substantially shorter than $p=1$ QAOA circuits, which the authors expect to make them less vulnerable to decoherence on near-term devices.","The agent generalizes to held-out instances of all three problem classes and is less sensitive to problem type than QAOA, suggesting the learned policy captures a common structure in these diagonal-Hamiltonian optimizations."],"supporting_citations":[{"why":"Defines the quantum approximate optimization algorithm that serves as the baseline the trained agents must beat.","marker":"[17]"},{"why":"Provides the proximal policy optimization algorithm used to train the agents.","marker":"[34]"},{"why":"Supplies the quantum virtual machine simulator and instruction-set language used for training and simulated inference.","marker":"[36]"},{"why":"Documents the parametrically activated entangling-gate hardware on which the physical-device experiments run.","marker":"[12]"},{"why":"Provides the environment interface used to implement the agent-quantum-resource interaction.","marker":"[9]"}],"fun_headline_variants":["RL agent writes quantum circuits that beat QAOA on real hardware","Quantum programming by RL: shorter circuits, higher scores than QAOA","Reinforcement learning codes quantum programs that outperform QAOA","Agent learns to program quantum computer, beats QAOA on simulator and QPU","RL-trained agent designs quantum circuits: better than QAOA, runs on hardware"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that it is fair to score the trained agent by its maximum reward over any program prefix while scoring QAOA only by its final-circuit reward.","fun_headline_variants_meta":{"raw":{"variants":["RL agent writes quantum circuits that beat QAOA on real hardware","Quantum programming by RL: shorter circuits, higher scores than QAOA","Reinforcement learning codes quantum programs that outperform QAOA","Agent learns to program quantum computer, beats QAOA on simulator and QPU","RL-trained agent designs quantum circuits: better than QAOA, runs on hardware"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1209,"prompt_tokens":803,"completion_tokens":406,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":314}},"tokens_in":419,"tokens_out":406,"duration_ms":4906,"temperature":1.0,"reasoning_tokens":314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:50:48.886311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the agent and single-step QAOA on the same test instances using a single scoring rule, either final-circuit reward for both or best-prefix reward for both, and compare the mean episode scores; if the agent no longer leads across all problem types and both resources, the central claim fails.","supporting_citations":[],"review_version":1}