{"id":"4a4f2fb7-5faa-45b9-a00d-13def02abaeb","arxiv_id":"2501.10673","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"In a 1000-iteration evolutionary search over hybrid quantum-classical PPO architectures on CartPole, the best hybrid model ranked 11th, behind eight classical models.","lead":"The paper used an evolutionary architecture search to find good hybrid classical-quantum neural networks for the PPO reinforcement learning algorithm on the CartPole task, and found that classical-only models outperformed every hybrid model. It is a concrete test of whether small quantum layers help reinforcement learning, and the answer on this benchmark is no, with lessons about which hybrid designs fail.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Best hybrid ranked 11th is not evidence of no quantum advantage: the acknowledged 'Add a Quantum Layer' edge-case bug and single-seed/no-error-bar fitness can both make hybrids look worse than they are.","rationale":"The reader's weakest assumption (search-space breadth) is real and worth keeping. I would put the emphasis on the combined effect of a self-admitted mutation bug and single-seed/no-error-bar evaluation, because the strongest claim is a leaderboard claim and every leaderboard entry rests on one noisy fitness value per architecture. The paper itself flags both the mutation bug (Acknowledgments) and the unusual adjacent-quantum-layer constraint (Sec.4.2) as potentially harmful, so these are not external speculations. Because the code was not accessible for independent verification, these flags cannot be dismissed. I agree with the CONDITIONAL verdict; this stress-test does not move it. The conditions for acceptance should include: fix the mutation, relax or justify the log2 constraint, and provide multi-seed error bars. Without those, the absence of hybrids near the top cannot be separated from search failure/evaluation noise. The paper deserves credit for honestly reporting a negative result, reporting the bug, and giving the top-11 DNA sequences; those are useful, but they do not yet support the general statement in Sec.5 that hybrid VQC layers in their current form show no advantage.","tokens_in":6552,"tokens_out":10607,"duration_ms":114816,"concrete_test":"Patch the acknowledged 'Add a Quantum Layer' edge case in the repository [9], relaunch Regularized Evolution on CartPole-v1 with the same settings for 5 independent seeds (a reduced 200-iteration pilot is acceptable if the full 1000+ run is prohibitive), and evaluate every unique architecture with 10 PPO seeds x 100 episodes; report the best hybrid score with a 95% CI against the best classical score. If the intervals overlap or any hybrid enters the top 10, the central negative conclusion is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that, after 1000+ Regularized Evolution iterations on CartPole-v1, the best hybrid placed 11th, supporting the conclusion that hybrid VQC layers show no advantage (Abstract; Sec.4.3; Sec.5). This inference requires the search and the fitness ranking to be unbiased or at least robust. Neither requirement is met. First, the Acknowledgements admit the 'Add a Quantum Layer' mutation (4.1) is bugged in an edge case; if that bug prevents quantum-layer insertion from certain parent DNAs, a whole class of hybrid architectures is never explored. Second, Sec.4.2 imposes n1 = log2(n2) for adjacent quantum layers and the authors themselves note this 'might have negatively affected the results, which is unclear'; this constraint sharply limits multi-quantum-layer designs and is applied on top of an already rough mutation set. Third, all architectures are ranked by a single average over 10 episodes with no multiple PPO seeds and no error bars; PPO training is stochastic, so a 442.6 vs 339.7 gap between the top classical model and the best hybrid is not shown to be statistically meaningful. The conclusion is therefore not yet established: the absence of hybrids near the top could reflect search failure or evaluation noise rather than a fundamental lack of quantum advantage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies the Regularized Evolution neural architecture search algorithm to find a hybrid classical-quantum architecture for the Proximal Policy Optimization (PPO) algorithm on the CartPole-v1 environment. The search runs for over 1000 iterations, produces 666 unique architectures, and ranks them by average reward over 10 episodes. The best hybrid model places 11th behind classical-only models, and the author concludes that variational quantum circuits used as layers in hybrid models currently show no advantage over well-designed classical models. The paper also offers several post hoc observations about why some architectures performed poorly, and it makes code and data available on GitHub.","tokens_in":6829,"tokens_out":2965,"duration_ms":30453,"significance":"If the negative result were established with adequate statistical and search-robustness evidence, it would provide a useful empirical data point for hybrid quantum-classical reinforcement learning, a field that frequently relies on hand-picked ansätze rather than systematic search. The paper has the merit of being fully transparent: the mutation set, constraints, and code are described, and the author discloses a known bug in the 'Add a Quantum Layer' mutation. However, the central claim currently exceeds what the reported evidence supports, because the evaluation is based on a single PPO run per architecture with no error bars, and because the search procedure itself contains acknowledged biases against hybrid architectures. The explanatory conclusions in Section 5 are post hoc interpretations of the same run and are not independently tested.","major_comments":[{"comment":"The central empirical claim that the best hybrid model placed 11th and that hybrid architectures show no advantage is not supported with statistical confidence. Each architecture is scored by a single average over 10 episodes from one PPO training run; no multiple seeds, standard deviations, or error bars are reported. PPO training is stochastic, and the gap between the top classical score (442.6/500) and the best hybrid score (339.7/500) cannot be judged statistically meaningful without repeated runs. The paper should either provide repeated-seed statistics or temper the conclusion to a descriptive statement about this particular search run.","section":"Sec. 4.3 and Abstract"},{"comment":"The author discloses in the Acknowledgments that the 'Add a Quantum Layer' mutation is bugged in an edge case, but dismisses the impact without investigation. Because the conclusion is a negative claim about the hybrid architecture space, any bug that biases this mutation against producing or inserting quantum layers directly weakens the inference. The manuscript should analyze the bug's effect on the search (e.g., by checking whether certain parent DNAs can never receive a quantum layer) or fix the bug and rerun the search. A negative result obtained from a defective mutation operator is not a reliable negative result about hybrids in general.","section":"Acknowledgments and Sec. 4.1"},{"comment":"The constraint that adjacent quantum layers must satisfy log2(n2) = n1, which the authors themselves say 'might have negatively affected the results, which is unclear,' severely restricts multi-quantum-layer architectures. In the search, only 34 models had two quantum layers, 4 had three, and 2 had four. The paper's sweeping conclusion about hybrid architectures in Section 5 is not justified if the search space disproportionately penalizes or excludes a large class of hybrid designs. The authors should either relax this constraint and rerun, or explicitly limit the conclusion to the constrained search space actually explored.","section":"Sec. 4.2"},{"comment":"The bullet-point explanations, such as 'Quantum layers with many qubits proved to be harder to train' and the hand-mutation of the best hybrid showing that Basic entanglement halves the reward, are post hoc interpretations of the same single search run that produced the leaderboard. These are not validated on independent data and should be clearly framed as speculative hypotheses rather than empirical conclusions. For example, the hand-mutation result is based on one model and one set of evaluation episodes without error bars, so it cannot support a general claim about entanglement type.","section":"Sec. 5"}],"minor_comments":[{"comment":"The abstract contains grammar and spacing issues (e.g., 'results,and') that should be corrected.","section":"Abstract"},{"comment":"Reference [4] is cited as Lockwood and Si (2021) at NeurIPS 2020 Workshop, but the citation appears with an incomplete publisher location ('???'); this should be fixed.","section":"Sec. 2"},{"comment":"The DNA sequences in Table 1 contain inconsistent formatting, such as 'C, 1' versus 'C 1' and 'C, 1' in the first and fifth rows; these should be made uniform.","section":"Table 1"},{"comment":"The paper uses 'ADAM' and 'Pennylane' instead of the standard 'Adam' and 'PennyLane'; please correct the spelling.","section":"Sec. 4.2"},{"comment":"No hyperparameter values for PPO (learning rate, clip epsilon, GAE lambda, number of epochs, batch size) are reported anywhere in the manuscript, despite these being essential for reproducibility. These should be listed in Section 4 or in an appendix.","section":"General"},{"comment":"The description of Regularized Evolution would benefit from a formal specification of the parameters P and S, which are introduced but never given numerical values in the text.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant question and is commendably transparent, but the single-seed evaluation and the acknowledged mutation bug are load-bearing for the negative conclusion. As written, the manuscript would not convince a careful reader that the absence of hybrid architectures at the top of the leaderboard is due to a fundamental lack of quantum advantage rather than search or evaluation artifacts. A major revision that adds repeated-seed statistics, fixes or analyzes the mutation bug, and narrows the conclusions to the actual search space would be needed before publication in a serious journal. The paper may also be a better fit for a workshop or a more empirical venue if the authors do not wish to extend the experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one with a cup of coffee. It's a small, honest empirical study: the author uses Regularized Evolution to search hybrid quantum-classical architectures for PPO on CartPole-v1, runs 1000+ iterations, and finds that the best hybrid places 11th behind classical models. The new thing is the search method applied to hybrid quantum RL; prior work hand-picked ansatze. The paper also transparently discloses a bug in the 'Add a Quantum Layer' mutation and a constraint on adjacent quantum layers that might have hurt hybrids. That transparency earns credit.\n\nThe method section is basic but sufficient: 10 mutations, clear experimental constraints, and a leaderboard of 666 unique models. The author attempts to explain why some architectures did better, which is more than most negative-result papers do. The conclusion is cautious: 'do not hint at any advantage,' with a call for more environments.\n\nNow the soft spots. The central negative claim rests on two assumptions that are untested. First, the search space defined by the mutations must be broad enough to actually include competitive hybrids; the acknowledged 'Add a Quantum Layer' bug could have suppressed promising hybrids, and the log2(n2)=n1 constraint for adjacent quantum layers adds another restriction. Second, the fitness ranking uses a single PPO run averaged over 10 episodes, with no error bars or multiple seeds. PPO is stochastic; a 442.6 vs 339.7 gap is not shown to be statistically robust. The post hoc explanations in Sec.5 are heuristics, not validated findings.\n\nThese are real concerns but not fatal to the paper's value. The author does not overclaim; the abstract states the factual leaderboard, and the interpretive section is explicitly exploratory. The scope is limited to one simple environment, so the general conclusion about quantum advantage should be read narrowly.\n\nThis paper is for researchers in hybrid quantum RL who want a first data point on evolutionary NAS for PPO architectures and a set of design heuristics. It is not a rigorous benchmark, but it is an honest, reproducible-in-principle study (GitHub repo, though I couldn't access it). I'd send it to peer review with major revision requirements: multiple seeds, error bars, full hyperparameter reporting, and an analysis of the mutation bug's impact. A desk reject would be too harsh; a conditional accept with revisions could make this a useful negative result.","headline":"A small, honest negative result on hybrid quantum PPO that needs more seeds and a check on its search-space assumptions before the conclusion can stand.","tokens_in":7329,"tokens_out":2553,"would_cite":false,"duration_ms":25560,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An evolutionary search over 666 PPO architectures places the best hybrid quantum model 11th, behind every classical network.","keywords":["quantum machine learning","hybrid quantum-classical models","neural architecture search","proximal policy optimization","variational quantum circuits","regularized evolution","quantum reinforcement learning","CartPole"],"falsifier":"Rerun the same Regularized Evolution search on CartPole-v1 with the 'Add a Quantum Layer' edge case fixed, with the same budget and environment, and check whether any hybrid enters the top ten; if a fixed search yields a hybrid with average reward above the best classical score of 442.6/500, the paper's negative conclusion is overturned. A smaller controlled check: train the best classical architecture and the best hybrid architecture matched for parameter count and random seeds; if the hybrid consistently matches or exceeds the classical model across many seeds, the claimed absence of advantage fails.","tokens_in":6345,"feed_emoji":"⚛️","tokens_out":11094,"duration_ms":107668,"temperature":0.7,"pith_summary":"This paper asks whether a variational quantum circuit used as a layer in a reinforcement-learning network gives a measured advantage over a purely classical network, and it reports no for the CartPole-v1 task under an architecture search intended to reduce human bias. Running Regularized Evolution for more than 1000 iterations produced 666 unique PPO actor-critic architectures, 390 of them purely classical and the rest containing one to four quantum layers. The ten best models were classical, with average rewards between 340.8 and 442.6 out of 500, while the best hybrid model, a two-qubit strongly entangled quantum layer embedded in a deep classical network, scored 339.7 and placed eleventh. The author concludes that variational quantum circuits as quantum layers in hybrid models, in their current form, do not hint at any advantage over well-designed classical models, and gives architectural observations about entanglement, qubit count, and training instability that could guide future hybrid design.","feed_headline":"Best hybrid quantum model places 11th behind classical nets","feed_subtitle":"A 1,000+ iteration search over 666 architectures finds the best hybrid in 11th place, behind all classical models.","key_machinery":"The load-bearing mechanism is Regularized Evolution acting on a DNA encoding of a full PPO actor-critic architecture. The algorithm keeps a population of $P$ architectures, samples $S$ candidates, mutates the best-scoring one, trains it, and deletes the oldest member; ten mutation operators add or remove classical and quantum layers, change neuron or qubit counts, ansatz repetitions, entanglement type, activation function, or do nothing. The quantum layer is a variational circuit used as a differentiable layer: angle encoding maps inputs to qubit rotations, BasicEntanglingLayer or StrongEntanglingLayer defines the entangling structure, and the layer outputs either $n$ expectation values or $2^n$ bitstring probabilities depending on whether the next layer is quantum or classical. The search space is bounded by at most ten layers, at most 64 neurons, and at most ten qubits, and the PPO clipped objective serves as the training signal that assigns fitness to each architecture.","core_discovery":"On the paper's own terms, the central discovery is that an unbiased evolutionary search over hybrid classical-quantum architectures for PPO does not produce a quantum-enhanced winner: every one of the top ten unique models is classical. The leading classical model reaches 442.6 out of 500 average reward, and the first hybrid appears in eleventh place at 339.7; the paper ties the hybrid's lower rank to inconsistent scores across evaluation episodes. The explanation offered is structural: quantum layers whose information flows mainly through entanglement, especially with many unmeasured qubits, are hard to train, while small qubit counts with strong entanglement behave better; larger quantum layers and higher ansatz repetitions were not favoured by the search. From this it concludes that quantum variational circuits as quantum layers in hybrid models, in their current form, do not hint at any advantage over well-designed classical networks.","pith_inferences":["A corrected rerun of the search, fixing the disclosed edge-case bug in the 'Add a Quantum Layer' mutation, is the natural sharper version of this experiment; if the fix lets good hybrids into the search space, the leaderboard could change.","The evaluation ranks by mean reward over ten episodes; using worst-case or variance would give different weight to the hybrid's unstable episode scores and might change which architectures are judged best.","The paper's hypothesis that information flow via entanglement rather than direct parameter contribution causes poor training is testable: compare gradient norms or effective dimension of quantum layers with measured versus unmeasured qubits.","If the negative result generalizes, the bottleneck may be the generic layer-insertion approach to hybrid quantum machine learning rather than quantum circuits themselves; better ansatz search or task-specific embeddings might still yield advantage."],"forward_implications":["If the result holds, adding VQC layers to PPO networks should not be expected to improve CartPole-level control tasks; effort is better spent on classical architecture tuning or on different quantum embeddings.","The paper's observation that unmeasured qubits and entanglement-dependent information flow hurt training implies a design rule: every quantum parameter should contribute directly to the loss through its measured output.","The search's preference for few qubits, strong entanglement, and low ansatz repetitions gives a concrete starting point for hand-designed hybrid models.","The negative result motivates testing the same search across harder environments before generalizing the conclusion, since CartPole-v1 is a simple task and one run took 20 hours."],"supporting_citations":[{"why":"Supplies the Regularized Evolution search algorithm that defines the architecture search procedure.","marker":"[6]"},{"why":"Defines the PPO algorithm whose clipped objective is the training signal for every candidate architecture.","marker":"[7]"},{"why":"Provides the PPO implementation details and the shared actor-critic architecture used as the experimental baseline.","marker":"[8]"},{"why":"Is the code repository that specifies the DNA encoding and the ten mutations, and holds the experimental data.","marker":"[9]"},{"why":"Is the quantum library used to build and execute the variational quantum layers in the hybrid models.","marker":"[10]"},{"why":"Provides the CartPole-v1 environment whose average rewards are used to rank the models.","marker":"[11]"},{"why":"Previously compared 19 hand-picked ansatze for quantum PPO, the study this search extends by removing the human choice of ansatz.","marker":"[5]"},{"why":"Is a prior hybrid quantum-classical reinforcement learning study on Atari; the paper uses its dimension-reduction explanation as context for its own results.","marker":"[4]"}],"fun_headline_variants":["Quantum hybrid search: classical still wins, 11th place","Evolution fails to find quantum edge in PPO architectures","Top ten models all classical; best quantum hybrid only 11th","Quantum layers no match for classical in PPO search","Hybrid quantum nets rank 11th; classical dominate top ten"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that quantum layers offer no advantage depends on the search space and mutation operators actually being able to reach competitive hybrid architectures; if the mutation set, the random initialization, or the disclosed edge-case bug in adding a quantum layer kept promising hybrids from being tried, the absence of hybrids at the top could reflect search failure rather than a lack of quantum benefit.","fun_headline_variants_meta":{"raw":{"variants":["Quantum hybrid search: classical still wins, 11th place","Evolution fails to find quantum edge in PPO architectures","Top ten models all classical; best quantum hybrid only 11th","Quantum layers no match for classical in PPO search","Hybrid quantum nets rank 11th; classical dominate top ten"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000306,"raw_usage":{"total_tokens":1719,"prompt_tokens":873,"completion_tokens":846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":489,"completion_tokens_details":{"reasoning_tokens":762}},"tokens_in":489,"tokens_out":846,"duration_ms":7554,"temperature":1.0,"reasoning_tokens":762,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:00:55.154748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same Regularized Evolution search on CartPole-v1 with the 'Add a Quantum Layer' edge case fixed, with the same budget and environment, and check whether any hybrid enters the top ten; if a fixed search yields a hybrid with average reward above the best classical score of 442.6/500, the paper's negative conclusion is overturned. A smaller controlled check: train the best classical architecture and the best hybrid architecture matched for parameter count and random seeds; if the hybrid consistently matches or exceeds the classical model across many seeds, the claimed absence of advantage fails.","supporting_citations":[{"cited_title":"In: ICLR Blog Track (2022)","cited_arxiv_id":null,"evidence_quote":"Provides the PPO implementation details and the shared actor-critic architecture used as the experimental baseline."},{"cited_title":"https:/github.com/moustafa7zada/Quantum-Hybrid-NAS-via-Regularized-Evolution/","cited_arxiv_id":null,"evidence_quote":"Is the code repository that specifies the DNA encoding and the ten mutations, and holds the experimental data."},{"cited_title":"In: Bertinetto, L., Henriques, J.F., Albanie, S., Paganini, M., Varol, G","cited_arxiv_id":null,"evidence_quote":"Is a prior hybrid quantum-classical reinforcement learning study on Atari; the paper uses its dimension-reduction explanation as context for its own results."}],"review_version":1}