Pith. sign in

REVIEW 4 major objections 6 minor 12 references

Hybrid-Quantum Neural Architecture Search for The Proximal Policy Optimization Algorithm

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An evolutionary search over 666 PPO architectures places the best hybrid quantum model 11th, behind every classical network.

desk verdict A small, honest negative result on hybrid quantum PPO that needs more seeds and a check on its search-space assumptions before the conclusion can stand. read the letter →

arxiv 2501.10673 v1 pith:N5NL2XXR submitted 2025-01-18 quant-ph cs.LGcs.NE

Moustafa Zada This is my paper · ORCID
classification quant-phcs.LGcs.NE
keywords quantummachinelearninghybridquantum-classicalmodelsneuralarchitecturesearchproximalpolicyoptimizationvariationalcircuitsregularizedevolutionreinforcementCartPole
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a variational quantum circuit used as a layer in a reinforcement-learning network gives a measured advantage over a purely classical network, and it reports no for the CartPole-v1 task under an architecture search intended to reduce human bias. Running Regularized Evolution for more than 1000 iterations produced 666 unique PPO actor-critic architectures, 390 of them purely classical and the rest containing one to four quantum layers. The ten best models were classical, with average rewards between 340.8 and 442.6 out of 500, while the best hybrid model, a two-qubit strongly entangled quantum layer embedded in a deep classical network, scored 339.7 and placed eleventh. The author concludes that variational quantum circuits as quantum layers in hybrid models, in their current form, do not hint at any advantage over well-designed classical models, and gives architectural observations about entanglement, qubit count, and training instability that could guide future hybrid design.

What carries the argument

The load-bearing mechanism is Regularized Evolution acting on a DNA encoding of a full PPO actor-critic architecture. The algorithm keeps a population of $P$ architectures, samples $S$ candidates, mutates the best-scoring one, trains it, and deletes the oldest member; ten mutation operators add or remove classical and quantum layers, change neuron or qubit counts, ansatz repetitions, entanglement type, activation function, or do nothing. The quantum layer is a variational circuit used as a differentiable layer: angle encoding maps inputs to qubit rotations, BasicEntanglingLayer or StrongEntanglingLayer defines the entangling structure, and the layer outputs either $n$ expectation values or $2^n$ bitstring probabilities depending on whether the next layer is quantum or classical. The search space is bounded by at most ten layers, at most 64 neurons, and at most ten qubits, and the PPO clipped objective serves as the training signal that assigns fitness to each architecture.

What would settle it

Rerun the same Regularized Evolution search on CartPole-v1 with the 'Add a Quantum Layer' edge case fixed, with the same budget and environment, and check whether any hybrid enters the top ten; if a fixed search yields a hybrid with average reward above the best classical score of 442.6/500, the paper's negative conclusion is overturned. A smaller controlled check: train the best classical architecture and the best hybrid architecture matched for parameter count and random seeds; if the hybrid consistently matches or exceeds the classical model across many seeds, the claimed absence of advantage fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that an unbiased evolutionary search over hybrid classical-quantum architectures for PPO does not produce a quantum-enhanced winner: every one of the top ten unique models is classical. The leading classical model reaches 442.6 out of 500 average reward, and the first hybrid appears in eleventh place at 339.7; the paper ties the hybrid's lower rank to inconsistent scores across evaluation episodes. The explanation offered is structural: quantum layers whose information flows mainly through entanglement, especially with many unmeasured qubits, are hard to train, while small qubit counts with strong entanglement behave better; larger quantum layers and higher ansatz repetitions were not favoured by the search. From this it concludes that quantum variational circuits as quantum layers in hybrid models, in their current form, do not hint at any advantage over well-designed classical networks.

Load-bearing premise

The conclusion that quantum layers offer no advantage depends on the search space and mutation operators actually being able to reach competitive hybrid architectures; if the mutation set, the random initialization, or the disclosed edge-case bug in adding a quantum layer kept promising hybrids from being tried, the absence of hybrids at the top could reflect search failure rather than a lack of quantum benefit.

Editorial extensions

If this is right

  • If the result holds, adding VQC layers to PPO networks should not be expected to improve CartPole-level control tasks; effort is better spent on classical architecture tuning or on different quantum embeddings.
  • The paper's observation that unmeasured qubits and entanglement-dependent information flow hurt training implies a design rule: every quantum parameter should contribute directly to the loss through its measured output.
  • The search's preference for few qubits, strong entanglement, and low ansatz repetitions gives a concrete starting point for hand-designed hybrid models.
  • The negative result motivates testing the same search across harder environments before generalizing the conclusion, since CartPole-v1 is a simple task and one run took 20 hours.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A corrected rerun of the search, fixing the disclosed edge-case bug in the 'Add a Quantum Layer' mutation, is the natural sharper version of this experiment; if the fix lets good hybrids into the search space, the leaderboard could change.
  • The evaluation ranks by mean reward over ten episodes; using worst-case or variance would give different weight to the hybrid's unstable episode scores and might change which architectures are judged best.
  • The paper's hypothesis that information flow via entanglement rather than direct parameter contribution causes poor training is testable: compare gradient norms or effective dimension of quantum layers with measured versus unmeasured qubits.
  • If the negative result generalizes, the bottleneck may be the generic layer-insertion approach to hybrid quantum machine learning rather than quantum circuits themselves; better ansatz search or task-specific embeddings might still yield advantage.
Share X Bluesky LinkedIn Reddit HN

Formalized claims in Lean

  1. Claim #1: On the paper's own terms, the central discovery is that an unbiased evolutionary search over hybrid classical-quantum architectures for PPO does not produce a quantum-enhanced winner: every one of the top ten unique models is classical. The leading classical model reaches 442.6 out of 500 average reward, and the first hybrid appears in eleventh place at 339.7; the paper ties the hybrid's lower ran

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper applies the Regularized Evolution neural architecture search algorithm to find a hybrid classical-quantum architecture for the Proximal Policy Optimization (PPO) algorithm on the CartPole-v1 environment. The search runs for over 1000 iterations, produces 666 unique architectures, and ranks them by average reward over 10 episodes. The best hybrid model places 11th behind classical-only models, and the author concludes that variational quantum circuits used as layers in hybrid models currently show no advantage over well-designed classical models. The paper also offers several post hoc observations about why some architectures performed poorly, and it makes code and data available on GitHub.

Significance. If the negative result were established with adequate statistical and search-robustness evidence, it would provide a useful empirical data point for hybrid quantum-classical reinforcement learning, a field that frequently relies on hand-picked ansätze rather than systematic search. The paper has the merit of being fully transparent: the mutation set, constraints, and code are described, and the author discloses a known bug in the 'Add a Quantum Layer' mutation. However, the central claim currently exceeds what the reported evidence supports, because the evaluation is based on a single PPO run per architecture with no error bars, and because the search procedure itself contains acknowledged biases against hybrid architectures. The explanatory conclusions in Section 5 are post hoc interpretations of the same run and are not independently tested.

major comments (4)
  1. [Sec. 4.3 and Abstract] The central empirical claim that the best hybrid model placed 11th and that hybrid architectures show no advantage is not supported with statistical confidence. Each architecture is scored by a single average over 10 episodes from one PPO training run; no multiple seeds, standard deviations, or error bars are reported. PPO training is stochastic, and the gap between the top classical score (442.6/500) and the best hybrid score (339.7/500) cannot be judged statistically meaningful without repeated runs. The paper should either provide repeated-seed statistics or temper the conclusion to a descriptive statement about this particular search run.
  2. [Acknowledgments and Sec. 4.1] The author discloses in the Acknowledgments that the 'Add a Quantum Layer' mutation is bugged in an edge case, but dismisses the impact without investigation. Because the conclusion is a negative claim about the hybrid architecture space, any bug that biases this mutation against producing or inserting quantum layers directly weakens the inference. The manuscript should analyze the bug's effect on the search (e.g., by checking whether certain parent DNAs can never receive a quantum layer) or fix the bug and rerun the search. A negative result obtained from a defective mutation operator is not a reliable negative result about hybrids in general.
  3. [Sec. 4.2] The constraint that adjacent quantum layers must satisfy log2(n2) = n1, which the authors themselves say 'might have negatively affected the results, which is unclear,' severely restricts multi-quantum-layer architectures. In the search, only 34 models had two quantum layers, 4 had three, and 2 had four. The paper's sweeping conclusion about hybrid architectures in Section 5 is not justified if the search space disproportionately penalizes or excludes a large class of hybrid designs. The authors should either relax this constraint and rerun, or explicitly limit the conclusion to the constrained search space actually explored.
  4. [Sec. 5] The bullet-point explanations, such as 'Quantum layers with many qubits proved to be harder to train' and the hand-mutation of the best hybrid showing that Basic entanglement halves the reward, are post hoc interpretations of the same single search run that produced the leaderboard. These are not validated on independent data and should be clearly framed as speculative hypotheses rather than empirical conclusions. For example, the hand-mutation result is based on one model and one set of evaluation episodes without error bars, so it cannot support a general claim about entanglement type.
minor comments (6)
  1. [Abstract] The abstract contains grammar and spacing issues (e.g., 'results,and') that should be corrected.
  2. [Sec. 2] Reference [4] is cited as Lockwood and Si (2021) at NeurIPS 2020 Workshop, but the citation appears with an incomplete publisher location ('???'); this should be fixed.
  3. [Table 1] The DNA sequences in Table 1 contain inconsistent formatting, such as 'C, 1' versus 'C 1' and 'C, 1' in the first and fifth rows; these should be made uniform.
  4. [Sec. 4.2] The paper uses 'ADAM' and 'Pennylane' instead of the standard 'Adam' and 'PennyLane'; please correct the spelling.
  5. [General] No hyperparameter values for PPO (learning rate, clip epsilon, GAE lambda, number of epochs, batch size) are reported anywhere in the manuscript, despite these being essential for reproducibility. These should be listed in Section 4 or in an appendix.
  6. [Sec. 3.2] The description of Regularized Evolution would benefit from a formal specification of the parameters P and S, which are introduced but never given numerical values in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is an empirical leaderboard, not a quantity derived from its own inputs.

full rationale

The paper's central claim is that, after a Regularized Evolution search over 1000+ iterations, the best hybrid quantum-classical PPO model ranked eleventh among unique models on CartPole-v1. This is a measured outcome of a training-and-evaluation pipeline, not a predicted quantity that is equivalent to a fitted input or to a self-citation. The reported leaderboard values in Table 1 are direct evaluation scores, and the conclusion in Sec. 5 that variational quantum circuits 'do not hint at any advantage' is an interpretation of that measured ranking. No equation in the paper defines a target in terms of itself, no fitted parameter is relabeled as a prediction, and no load-bearing result is imported from the author's prior work. The self-citation to the author's GitHub repository [9] is only a code-availability pointer, not an argumentative premise. The acknowledged mutation bug and the logarithmic qubit-count constraint are threats to the external validity of the conclusion, but they do not make the derivation circular; they concern whether the search space was broad enough to find competitive hybrids, which is an evidence-strength question rather than a self-reference problem. The paper is therefore self-contained as an empirical study, and the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central empirical claim rests on the search implementation and the single environment; the paper does not introduce free parameters in a derivation sense, but it leaves key search and training hyperparameters unreported. The main assumptions are that the mutation space covers good hybrid designs and that CartPole-v1 is a meaningful probe. No invented physical entities are introduced.

free parameters (3)
  • NAS population size P and tournament sample size S = not reported
    Regularized Evolution performance depends on these hyperparameters; the paper does not report their values, so results cannot be exactly reproduced from text.
  • PPO hyperparameters (learning rate, clip epsilon, GAE lambda, epochs, batch size) = not reported
    Paper refers to [8] for implementation details but does not list the exact values used, which affect all model scores.
  • Mutation probabilities for each of the 10 mutation types = not reported
    Determines search dynamics and which architectures are visited; not reported in the paper.
assumptions (4)
  • domain assumption CartPole-v1 performance is a meaningful proxy for reinforcement learning capability
    Sec.4.2 and 4.3 use a single Gym environment; conclusions about quantum advantage generalize only if this task is representative.
  • domain assumption The mutation set and layer constraints define an adequate search space for hybrid architectures
    Sec.4.1 and 4.2; negative result assumes good hybrids could appear within this space.
  • standard math PPO implementation follows standard details in [8] and is correctly adapted to hybrid layers
    Sec.3.1; paper relies on cited PPO implementation without specifying full hyperparameters.
  • domain assumption Backpropagation through quantum layers via PennyLane is reliable
    Sec.4.2; training assumes gradients through VQC layers are correct.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hybrid-Quantum Neural Architecture Search for The Proximal Policy Optimization Algorithm." pith.science (2026). https://pith.science/paper/N5NL2XXR

@misc{pith2026250110673,
  author       = {Pith},
  title        = {Pith review of: Hybrid-Quantum Neural Architecture Search for The Proximal Policy Optimization Algorithm},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N5NL2XXR}},
  note         = {Machine review of arXiv:2501.10673}
}
read the original abstract

Recent studies in quantum machine learning advocated the use of hybrid models to assist with the limitations of the currently existing Noisy Intermediate Scale Quantum (NISQ) devices, but what was missing from most of them was the explanations and interpretations of the choices that were made to pick those exact architectures and the differentiation between good and bad hybrid architectures, this research attempts to tackle that gap in the literature by using the Regularized Evolution algorithm to search for the optimal hybrid classical-quantum architecture for the Proximal Policy Optimization (PPO) algorithm, a well-known reinforcement learning algorithm, ultimately the classical models dominated the leaderboard with the best hybrid model coming in eleventh place among all unique models, while we also try to explain the factors that contributed to such results,and for some models to behave better than others in hope to grasp a better intuition about what we should consider good practices for designing an efficient hybrid architecture.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 5 canonical work pages

  1. [1]

    Killoran, N.: Hybrid Quantum-Classical Machine Learning. Xanadu. https://www.youtube.com/watch?v= t9ytqPTij7k Accessed 2020-11-17

    work page 2020

  2. [2]

    Chen, S.Y.-C.: Asynchronous training of quantum reinforcement learning. arXiv. arXiv:2301.05096 [quant-ph] (2023). https://doi.org/10.48550/arXiv.2301.05096 . http://arxiv.org/abs/2301.05096 Accessed 2024-11-26

    work page Pith review arXiv doi:10.48550/arxiv.2301.05096 2023

  3. [3]

    Chen, H.-Y., Chang, Y.-J., Liao, S.-W., Chang, C.-R.: Deep-Q Learning with Hybrid Quantum Neural Network on Solving Maze Problems. arXiv. arXiv:2304.10159 [quant-ph] (2023). https://doi.org/10.48550/arXiv.2304. 10159 . http://arxiv.org/abs/2304.10159 Accessed 2024-11-22

    arXiv doi:10.48550/arxiv.2304.10159 2023

  4. [4]

    In: Bertinetto, L., Henriques, J.F., Albanie, S., Paganini, M., Varol, G

    Lockwood, O., Si, M.: Playing atari with hybrid quantum-classical reinforcement learning. In: Bertinetto, L., Henriques, J.F., Albanie, S., Paganini, M., Varol, G. (eds.) NeurIPS 2020 Workshop on Pre-registration in Machine Learning. Proceedings of Machine Learning Research, vol. 148, pp. 285–301. PMLR, ??? (2021). https://proceedings.mlr.press/v148/lockw...

    work page 2021

  5. [5]

    Dr˘ agan, T.-A., Monnet, M., Mendl, C.B., Lorenz, J.M.: Quantum Reinforcement Learning for Solving a Stochas- tic Frozen Lake Environment and the Impact of Quantum Architecture Choices. arXiv. arXiv:2212.07932 [quant-ph] (2022). https://doi.org/10.48550/arXiv.2212.07932 . http://arxiv.org/abs/2212.07932 Accessed 2024- 11-26

    arXiv doi:10.48550/arxiv.2212.07932 2022

  6. [6]

    Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized Evolution for Image Classifier Architecture Search. arXiv. arXiv:1802.01548 [cs] (2019). https://doi.org/10.48550/arXiv.1802.01548 . http://arxiv.org/abs/1802. 01548 Accessed 2024-11-22

    arXiv doi:10.48550/arxiv.1802.01548 2019

  7. [7]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal Policy Optimization Algorithms. arXiv. arXiv:1707.06347 [cs] (2017). https://doi.org/10.48550/arXiv.1707.06347 . http://arxiv.org/abs/1707. 06347 Accessed 2024-11-22

    arXiv doi:10.48550/arxiv.1707.06347 2017

  8. [8]

    In: ICLR Blog Track (2022)

    Huang, S., Dossa, R.F.J., Raffin, A., Kanervisto, A., Wang, W.: The 37 implementation details of proximal policy optimization. In: ICLR Blog Track (2022). https://iclr-blog-track.github.io/2022/03/25/ppo-implementation- details/. https://iclr-blog-track.github.io/2022/03/25/ppo-implementation-details/

    work page 2022

Show all 12 references
  1. [9]

    https:/github.com/moustafa7zada/Quantum-Hybrid-NAS-via-Regularized-Evolution/

    Zada, M. https:/github.com/moustafa7zada/Quantum-Hybrid-NAS-via-Regularized-Evolution/

  2. [10]

    Bergholm, V., Izaac, J., Schuld, M., Gogolin, C., Ahmed, S., Ajith, V., Alam, M.S., Alonso-Linaje, G., Akash- Narayanan, B., Asadi, A., Arrazola, J.M., Azad, U., Banning, S., Blank, C., Bromley, T.R., Cordier, B.A., Ceroni, J., Delgado, A., Di Matteo, O., Dusko, A., Garg, T., ...

    2022 arXiv

  3. [11]

    arXiv preprint arXiv:1606.01540 (2016)

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym. arXiv preprint arXiv:1606.01540 (2016)

    2016 arXiv

  4. [12]

    White, C., Safari, M., Sukthanker, R., Ru, B., Elsken, T., Zela, A., Dey, D., Hutter, F.: Neural Architecture Search: Insights from 1000 Papers. arXiv. arXiv:2301.08727 [cs, stat] (2023). https://doi.org/10.48550/arXiv. 2301.08727 . http://arxiv.org/abs/2301.08727 Accessed 202...

    2023 doi arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.