Pith. sign in

REVIEW 2 major objections 5 minor 17 references

A Reinforcement Learning approach for Quantum State Engineering

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A reinforcement learning agent discovers measurement sequences that prepare Bell states with fidelity above 0.99, matching and generalizing a known human solution.

desk verdict Fixed-state RL sequence discovery for Bell states is real and worth knowing; the generalisation claim is undermined by local-max checkpoint selection and missing evaluation over starting-state distributions. read the letter →

arxiv 1908.05981 v1 pith:5JT253XX submitted 2019-08-16 quant-ph

Jelena Mackeprang , Durga Bhaktavatsala Rao Dasari , Jörg Wrachtrup This is my paper · ORCID
classification quant-ph PACS 03.67.Ac03.67.Bg42.50.Dv07.05.Mh
keywords reinforcementlearningquantumstateengineeringmeasurement-basedcontrolBellstatesdeepQ-networknitrogen-vacancycenterspinbathMarkovdecisionprocess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that a classical deep reinforcement learning agent can design measurement-based control sequences for quantum state engineering. In a model of a central electron spin coupled to two nuclear spins, the agent learns sequences of free evolutions and projective measurements that take the nuclear spins from a completely mixed state to each of the four Bell states with fidelity above 0.99. For the singlet state $|\Psi^-\rangle$, the agent rediscovers the known repeated $P_{x+}$ purification sequence and also finds alternative sequences built from projections onto $|x\pm\rangle$ or $|y\pm\rangle$ subspaces. The authors formulate the problem as a Markov decision process whose state is the full density matrix, with reward based on reaching a fidelity threshold, and discuss how this formulation scales to larger spin systems.

What carries the argument

The load-bearing machinery is the Markov decision process formulation: the state is the full density matrix of central spin plus bath, decomposed into independent real and imaginary parts; the actions are projective measurements $P_{z\pm}$, $P_{x\pm}$, $P_{y\pm}$ on the central spin or the identity (doing nothing), each preceded by free evolution over a fixed time $\tau$. Because the full density matrix contains all information needed for later dynamics, this encoding makes the quantum control problem Markovian despite the underlying non-Markovian evolution. A deep Q-network (or double DQN) with epsilon-greedy exploration approximates the action-value function $q(s,a)$, and the reward function uses a fidelity threshold rather than instantaneous fidelity, reflecting the observed non-monotonic behavior of fidelity along successful sequences.

What would settle it

Retrain the agent from several random seeds and evaluate its policy at every checkpoint, not just local maxima, over many episodes; the central claim would be undercut if checkpoints yielding fidelity above 0.99 are rare or isolated. A complementary test is to implement the learned sequences on an NV-center experiment and check whether the measured Bell-state fidelities and success rates match the simulation.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that deep reinforcement learning (DQN and DDQN) can solve measurement-based quantum state engineering in a continuous, high-dimensional state space. The environment state is the vector of independent entries of the three-spin density matrix (70 real inputs), the actions are six projection choices on the central spin plus doing nothing after a fixed free-evolution time, and the reward is +10 when the reduced nuclear state reaches fidelity above 0.99 with the target, -1 otherwise, with an extra penalty for unphysical outcomes. The trained agent returns sequences for $|\Phi^+\rangle$, $|\Phi^-\rangle$, $|\Psi^+\rangle$, and $|\Psi^-\rangle$ with reported fidelities from about 0.9916 to 1.0000 and success rates around 1--25 percent. The agent's $|\Psi^-\rangle$ solution coincides with the known repeated $P_{x+}$ sequence and generalizes it through other repeated superposition-subspace projections, a pattern that persists when the central spin starts in a random pure state.

Load-bearing premise

The load-bearing assumption is that the agent copies evaluated for the random-start case are representative of a stable learned policy, even though they were selected at training steps where performance happened to peak; if those checkpoints are transient fluctuations, the reported generalization may not reflect a generalizable strategy.

Editorial extensions

If this is right

  • A trained agent can rediscover a known human-designed purification protocol from the exact quantum dynamics, which validates reinforcement learning as a discovery tool for this class of measurement-based control problems.
  • The alternative sequences show that preparation of $|\Psi^-\rangle$ is not unique: repeated projections onto any of the superposition subspaces $|x\pm\rangle$ or $|y\pm\rangle$ can drive the two nuclear spins to the same entangled state, generalizing the known $P_{x+}$ solution.
  • The same recipe of full-density-matrix states, projective-measurement actions, and threshold rewards transfers directly to other small quantum control tasks that have a trial-and-error structure.
  • Scaling to larger spin baths is limited by the exponential growth of density-matrix entries with system size; the authors propose reducing the state to observations and using a partially observable MDP formulation as the next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the learned policies are stable beyond the selected checkpoints, the same approach could be aimed at graph or cluster states in larger nuclear-spin registers, once a compressed state representation replaces the full density matrix.
  • The dominance of repeated projections onto one superposition subspace suggests a general mechanism: fixed-basis measurement backaction combined with free evolution acts as a dissipative purification step, a principle that might transfer to other central-spin platforms.
  • A direct experimental check is feasible with an NV-center device: run the discovered five-step sequences and compare the measured Bell-state fidelity and success probability with the simulated values, which would quantify the sensitivity to decoherence and readout errors.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper applies deep reinforcement learning (DQN and DDQN) to measurement-based quantum state engineering in a central-spin model with one central electron spin and two nuclear spins. The agent's actions are time evolutions of fixed duration followed by projective measurements of the central spin (or no measurement), and the reward is based on whether the fidelity of the reduced nuclear-spin state to a target Bell state exceeds a threshold. For a fixed initial central-spin state |x+>, the agent is reported to find sequences preparing all four Bell states with fidelities above 0.99, including a repeated-Px+ sequence for |Ψ−> that matches the known analytical solution of Greiner et al. A second set of experiments with randomly initialized pure central-spin states is used to claim generalization of these solutions. The paper concludes with a discussion of scalability and possible extensions.

Significance. If the claims hold, the paper is a useful proof-of-concept that RL can discover nontrivial measurement sequences for preparing entangled states, and the recovery of the known Greiner sequence provides a sanity check. The physical model is clearly specified, the simulation uses exact unitary dynamics, and the comparison against a random policy is an appropriate baseline. The authors are also explicit about the exponential growth of the density-matrix state space, which frames the scalability discussion honestly. However, the evidence for the broader generalization claim is presently not conclusive, because the random-state analysis is based on manually selected checkpoints from a single unstable training run without multi-seed statistics or held-out evaluation. The fixed-state results are more robust and are the paper's strongest contribution; with additional statistical evidence the work would be a solid proof-of-concept for the quantum-control/ML community.

major comments (2)
  1. [Random (pure) central spin starting state, Figs. 8–10 and Tables 2–3] The generalization claim—that the agent learns measurement sequences that work for arbitrary pure central-spin starting states—is not supported by the evidence as presented. The agent copies used for analysis are extracted at training steps 1900, 2000, 2290, and 2500 where its performance has local maxima, and the authors describe the learning curve in Fig. 8 as highly unstable. Tables 2 and 3 show that the step-1900 copy is unsuccessful for |x+> and the step-2000 copy is unsuccessful for |x−>, so the successes are conditional on favorable checkpoint selection. No multiple-seed statistics, error bars, or evaluation of a final ε=0.01 policy over a held-out distribution of random pure starting states are given. Without such evaluation, the preference histograms in Fig. 10 cannot distinguish a stable learned policy from lucky rollouts of an overfit network. I recommend training several independent seeds, reporting mean and variance of returns, and evaluating the final (or best) policy on a fixed test set of random starting states with per-state success rates.
  2. [Construction of the Bell states, Figs. 2–4] The central 'learning feature' claim rests on single-run learning curves without error bars, and the trained-versus-random comparison in Fig. 4 is qualitative. The number of episodes per training step is not stated, and no confidence intervals or effect sizes are reported. Since the abstract claims that learning is clearly displayed, the authors should provide multi-run statistics, for example mean ± standard deviation over independent seeds, and a quantitative comparison of the trained policy's success rate or average return against the random policy.
minor comments (5)
  1. [Quantum state engineering, Eqs. (6)–(7)] The time-evolution operator is written as U(τ)=exp(−i(t−τ)H) in Eq. (6) but as U(τ)=exp(−iHτ) in Eq. (7); please remove the inconsistent t−τ expression.
  2. [Quantum state engineering, Eq. (5)] The symbol /BD for the identity is not defined before use, and the tensor-product structure of the Zeeman term is typeset confusingly; please clean up the notation.
  3. [Table 1] The heading 'Sequence fidelity ·10^2' is ambiguous; write 'fidelity (×10^2)' or 'fidelity (%)' and state explicitly that 99.673 corresponds to 0.99673.
  4. [Random (pure) central spin starting state] The text after Fig. 8 attributes the instability to 'over-fitting of the underlying neural networks', but no evidence is offered for this diagnosis; it should be worded as a conjecture or omitted.
  5. [Modelling and Construction of the Bell states] The paper does not provide the network architecture, learning rate, replay-memory size, or number of episodes per training step, which are needed to reproduce the reported learning curves.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the reward and fidelities are independently computed, and the only self-citation (Greiner et al.) is a model/baseline source, not a load-bearing circular input.

full rationale

The claimed results are action sequences produced by DQN/DDQN agents and evaluated by the fidelity defined in Eq. (9) and the reward in Eq. (13). The reward is the objective being optimized, not an input later relabeled as a prediction; no parameter is fitted to a subset of data and then used to predict a closely related quantity. The 'generalisation' of Greiner et al.'s Px+ sequence is supported by Table 1, where the agent independently recovers the sequence U(2τ)-Px+-U(τ)-Px+-U(τ)-Px+-U(τ)-Px+-U(τ)-Px+ with fidelity 99.454%, and by additional sequences with comparable fidelities. The Greiner citation (which shares authors Dasari and Wrachtrup) supplies the physical Hamiltonian and the baseline human solution, but the paper's own simulation verifies that baseline, so the self-citation is not load-bearing. The random-starting-state generalization (Section 'random (pure) central spin starting state') is weakened by the authors' own statement that learning is 'highly unstable' and by the extraction of agent copies 'where its performance has local maxima' (Fig. 8 and Tables 2–3); this is a robustness/selection-bias concern, not a circularity, because the reported fidelities and success rates are still computed from the simulation rather than assumed. No step reduces by construction to its inputs; score 1 reflects only the minor shared-author citation.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The central claim depends on the chosen model parameters and algorithmic settings. No new physical entities are introduced. The main axioms are the standard quantum measurement postulate and the Markov property of the density-matrix state representation.

free parameters (6)
  • nuclear Zeeman frequency ω = 1/2 (relative units)
    Chosen by hand; all simulation results use this value.
  • free evolution time τ = 1 (relative units)
    Chosen by hand; the action set includes waiting for this fixed interval.
  • spin-bath coupling vector gk = (1,0,0) for all k
    Chosen by hand to model a linear chain along the x-axis; results may depend on this.
  • fidelity threshold θ = 0.99
    Defines success and termination; directly shapes the reward and learned policy.
  • reward values (r+, r-, rfatal) = 10, -1, -51
    Chosen by hand; set the RL objective and discourage fatal measurement outcomes.
  • epsilon-greedy decay schedule = not fully specified
    Exploration schedule is described as linear decay to a minimum, but exact values are not stated.
assumptions (3)
  • domain assumption The time evolution of the spin system is governed by the Hamiltonian in Eq. (5) and can be solved exactly.
    The paper relies on the model from Greiner et al. to simulate dynamics.
  • domain assumption The state after a measurement is fully described by the normalised projected density matrix in Eq. (8), and the transition obeys the Markov property when the full density matrix is used as state.
    This justifies the MDP formulation; the paper argues it but does not prove it in general.
  • domain assumption The DQN/DDQN neural networks can approximate the optimal action-value function over the continuous density-matrix state space with the given training procedure.
    Convergence is assumed based on prior RL literature, not demonstrated here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Reinforcement Learning approach for Quantum State Engineering." pith.science (2026). https://pith.science/paper/5JT253XX

@misc{pith2026190805981,
  author       = {Pith},
  title        = {Pith review of: A Reinforcement Learning approach for Quantum State Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5JT253XX}},
  note         = {Machine review of arXiv:1908.05981}
}
read the original abstract

Machine learning (ML) has become an attractive tool in information processing, however few ML algorithms have been successfully applied in the quantum domain. We show here how classical reinforcement learning (RL) could be used as a tool for quantum state engineering (QSE). We employ a measurement based control for QSE where the action sequences are determined by the choice of the measurement basis and the reward through the fidelity of obtaining the target state. Our analysis clearly displays a learning feature in QSE, for example in preparing arbitrary two-qubit entangled states. It delivers successful action sequences, that generalise previously found human solutions from exact quantum dynamics. We provide a systematic algorithmic approach for using RL algorithms for quantum protocols that deal with non-trivial continuous state (parameter) space, and discuss on scaling of these approaches for preparation of arbitrarily large entangled (cluster) states.

Figures

Figures reproduced from arXiv: 1908.05981 by the authors.

Figure 1
Figure 1. Depiction of our RL scheme. The agent influences the centra [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Average over the total sum of all received rewards during t [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Average over the total sum of all received rewards during t [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Comparison of a randomly acting agent (ǫ = 1, red dots) to a trained one (ǫ = 0.1, green triangles) for different target states. Each point or triangle represents the total sum of received rewards during an episode. The agent receives a reward of 10 and terminates the …
Figure 5
Figure 5. Figure 5: Purity of the nuclear spin state, the fidelity and trace distan [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Purity of the nuclear spin state, the fidelity and trace distan [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Counts of appearances of action combinations in the unique, [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Average over the total sum of all received rewards during t [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Comparison of the agent’s copies (ǫ = 0.01) at different training steps during the learning process shown in figure 8 to untrained ones (ǫ = 1). The trained agents receive more rewards per episode than the untrained ones [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]
Figure 10
Figure 10. Figure 10: Counts of appearances of action combinations in 10000 seq [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 4 canonical work pages

  1. [1]

    Scientific Reports 2:400, doi:10.1038/srep00400, ://www.nature.com/articles/srep00400

    Briegel HJ, De las Cuevas G (2012) Projective simulation for artificial intelligence. Scientific Reports 2:400, doi:10.1038/srep00400, ://www.nature.com/articles/srep00400

    doi:10.1038/srep00400 2012

  2. [2]

    Physical Review X 8(3):031086, doi:10.1103/PhysRevX.8.031086, ://link.aps.org/doi/10.1103/PhysRevX.8.031086

    Bukov M, Day AG, Sels D, Weinberg P, Polkovnikov A, Mehta P (2018) Reinforcement Learning in Different Phases of Quantum Control . Physical Review X 8(3):031086, doi:10.1103/PhysRevX.8.031086, ://link.aps.org/doi/10.1103/PhysRevX.8.031086

    doi:10.1103/physrevx.8.031086 2018

  3. [3]

    Physics Reports 528(1):1--45, doi:10.1016/j.physrep.2013.02.001, ://arxiv.org/abs/1302.3288, arXiv: 1302.3288

    Doherty MW, Manson NB, Delaney P, Jelezko F, Wrachtrup J, Hollenberg LCL (2013) The nitrogen-vacancy colour centre in diamond. Physics Reports 528(1):1--45, doi:10.1016/j.physrep.2013.02.001, ://arxiv.org/abs/1302.3288, arXiv: 1302.3288

    arXiv 2013

  4. [4]

    Scientific Reports 7, doi:10.1038/s41598-017-00603-z, ://www.ncbi.nlm.nih.gov/pmc/articles/PMC5428879/

    Greiner JN, Dasari DBR, Wrachtrup J (2017) Purification of an unpolarized spin ensemble into entangled singlet pairs. Scientific Reports 7, doi:10.1038/s41598-017-00603-z, ://www.ncbi.nlm.nih.gov/pmc/articles/PMC5428879/

    work page doi:10.1038/s41598-017-00603-z 2017

  5. [5]

    In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , AAAI Press, Phoenix, Arizona, AAAI '16, pp 2094--2100, ://dl.acm.org/citation.cfm?id=3016100.3016191

    Hasselt Hv, Guez A, Silver D (2016) Deep Reinforcement Learning with Double Q - Learning . In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , AAAI Press, Phoenix, Arizona, AAAI '16, pp 2094--2100, ://dl.acm.org/citation.cfm?id=3016100.3016191

    arXiv 2016

  6. [6]

    arXiv:150706527 [cs] ://arxiv.org/abs/1507.06527, arXiv: 1507.06527

    Hausknecht M, Stone P (2015) Deep Recurrent Q - Learning for Partially Observable MDPs . arXiv:150706527 [cs] ://arxiv.org/abs/1507.06527, arXiv: 1507.06527

    arXiv 2015

  7. [7]

    Neural Information Processing Systems 25, doi:10.1145/3065386

    Krizhevsky A, Sutskever I, E Hinton G (2012) ImageNet Classification with Deep Convolutional Neural Networks . Neural Information Processing Systems 25, doi:10.1145/3065386

    doi:10.1145/3065386 2012

  8. [8]

    arXiv:180407755 [cs] ://arxiv.org/abs/1804.07755, arXiv: 1804.07755

    Lample G, Ott M, Conneau A, Denoyer L, Ranzato M (2018) Phrase- Based & Neural Unsupervised Machine Translation . arXiv:180407755 [cs] ://arxiv.org/abs/1804.07755, arXiv: 1804.07755

    arXiv 2018

Show all 17 references
  1. [9]

    arXiv:180308823 [cond-mat, physics:physics, stat] ://arxiv.org/abs/1803.08823, arXiv: 1803.08823

    Mehta P, Bukov M, Wang CH, Day AGR, Richardson C, Fisher CK, Schwab DJ (2018) A high-bias, low-variance introduction to Machine Learning for physicists. arXiv:180308823 [cond-mat, physics:physics, stat] ://arxiv.org/abs/1803.08823, arXiv: 1803.08823

    2018 arXiv

  2. [10]

    Proceedings of the National Academy of Sciences 115(6):1221--1226, doi:10.1073/pnas.1714936115, ://www.pnas.org/content/115/6/1221

    Melnikov AA, Nautrup HP, Krenn M, Dunjko V, Tiersch M, Zeilinger A, Briegel HJ (2018) Active learning machine learns to create new quantum experiments. Proceedings of the National Academy of Sciences 115(6):1221--1226, doi:10.1073/pnas.1714936115, ://www.pnas.org/content/115/6/1221

    2018 doi

  3. [11]

    Nature 518(7540):529--533, ://www.nature.com/articles/nature14236

    Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D (2015) Human-level control through deep reinforcement learning. ...

    2015

  4. [12]

    arXiv:170301041 [cs] ://arxiv.org/abs/1703.01041

    Real E, Moore S, Selle A, Saxena S, Suematsu YL, Tan J, Le Q, Kurakin A (2017) Large- Scale Evolution of Image Classifiers . arXiv:170301041 [cs] ://arxiv.org/abs/1703.01041

    2017 arXiv

  5. [13]

    Science 362(6419):1140--1144, doi:10.1126/science.aar6404, ://science.sciencemag.org/content/362/6419/1140

    Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, Lillicrap T, Simonyan K, Hassabis D (2018) A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362(6419):1140--1144, ...

    2018 doi

  6. [14]

    Sutton RS, Barto AG (1998) Reinforcement Learning : An Introduction

    1998

  7. [15]

    Journal of Magnetic Resonance 269:225--236, doi:10.1016/j.jmr.2016.06.017

    Wrachtrup J, Finkler A (2016) Single spin magnetic resonance. Journal of Magnetic Resonance 269:225--236, doi:10.1016/j.jmr.2016.06.017

    2016 doi

  8. [16]

    , " * write output.state after.block = add.period write newline

    ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mi...

  9. [17]

    write newline

    " write newline "" before.all 'output.state := FUNCTION add.period duplicate empty 'skip "." * add.blank if FUNCTION if.digit duplicate "0" = swap duplicate "1" = swap duplicate "2" = swap duplicate "3" = swap duplicate "4" = swap duplicate "5" = swap duplicate "6" = swap dupl...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.