Reinforcement learning for spin torque oscillator tasks

Jakub Mojsiejuk; S{\l}awomir Zi\k{e}tek; Witold Skowro\'nski

REVIEW 3 major objections 6 minor 22 references

Reinforcement learning for spin torque oscillator tasks

T0 review · 3 major / 6 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reinforcement-learning agents can be trained in simulation to synchronise a spin torque oscillator to a target frequency, reaching near-perfect success and smoother, more energy-efficient control.

desk verdict A reasonable simulation-only proof of concept for RL-based STO synchronization, undercut by the per-step reset that turns the task into static parameter selection. read the letter →

arxiv 2509.10057 v1 pith:FDAW2N7J submitted 2025-09-12 physics.app-ph cs.AIcs.LG

Jakub Mojsiejuk , S{l}awomir Ziętek , Witold Skowroński This is my paper · ORCID

classification physics.app-phcs.AIcs.LG

keywords reinforcementlearningspintorqueoscillatorsynchronisationLLGSequationmagnetictunneljunctionrewardshapingactor-criticmacrospinsimulation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that automatic synchronisation of a spintronic oscillator (STO) can be learned by reinforcement learning instead of tuned by hand. Using a numerical macrospin solution of the Landau-Lifshitz-Gilbert-Slonczewski equation, the authors train two actor-critic agents to choose the current density and magnetic field (magnitude and angles) so that a randomly sampled magnetic tunnel junction oscillates at a target frequency within a fixed number of steps. After about 1500 evaluation steps, these agents reach almost 100% success. The paper shows that simple reward shaping—a reward proportional to frequency error, a penalty for large action changes, and a bonus for high Q-factor—makes synchronisation smoother and more energy-efficient, and preserves high oscillation quality at the cost of extra steps. If this result holds, a single pretrained controller could operate devices with a wide parameter spread without per-device re-tuning, which matters for wireless and neuromorphic applications of STOs.

What carries the argument

Load-bearing is the simulated RL environment built on the macrospin Landau-Lifshitz-Gilbert-Slonczewski equation, which produces resistance time series through the tunnelling magnetoresistance formula. The environment extracts FFT-based observations—peak frequency, frequency error to target, and derivatives of peak frequency with respect to current density, field magnitude, polar angle, and azimuthal angle. The agent's normalised action tuple is current density plus the three field parameters, and two off-policy actor-critic algorithms (TD3 and SAC) train against it. Reward shaping—Rδf proportional to frequency error, Rδa squared action derivative, and R_Q* normalised Q-factor at synchronisa

What would settle it

Evaluate the trained agents in the same simulator with the magnetisation carried over between steps instead of reset, and with episodes starting from varied initial states; if synchronisation success falls well below the reported ~100%, the fixed reset state is responsible for the result.

Watch

Extended reading notes

Core claim

The paper claims that a reinforcement-learning controller can learn the complex mapping from current density and magnetic field (magnitude and angles) to the oscillation frequency of a spin torque oscillator, using only resistance-derived FFT observations. In a macrospin LLGS simulator, two actor-critic agents synchronise randomly sampled magnetic tunnel junctions to randomly chosen target frequencies, reaching almost 100% success after about 1500 evaluation steps. Adding reward shaping—proportional frequency error, squared action-change penalty, and a normalised Q-factor bonus—yields smoother, more energy-efficient synchronisation and a higher final Q-factor, at the cost of more steps.

Load-bearing premise

The load-bearing premise is that the per-step reset of the magnetisation to the same semi-stable initial state does not distort what the controller learns, even though a real device carries its magnetisation state between control updates.

Editorial extensions

If this is right

A single pretrained controller can tune STO devices across a wide spread of magnetic parameters without per-device re-tuning, because the policy learns the implicit relation between control inputs and oscillation frequency.
A squared-change penalty on the controller's actions reduces drastic field and current jumps, lowering energy use and the risk of sample damage during synchronisation.
A Q-factor bonus in the reward raises the coherence of the synchronised oscillation, at the price of slower synchronisation.
The observation space is deliberately restricted to measurable resistance-derived frequency features, so the setup is compatible with an experimental bias-tee measurement chain.
The same environment and reward design can be extended to other tunable spintronic devices, such as VCMA-based field sensors, with different control parameters and observations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the trained policy is to remove the per-step magnetisation reset and let the state carry over between control updates; the resulting success rate would show how strongly the reported result depends on that reset.
The weight on the Q-factor bonus could be scheduled during training—start at zero to lock quickly, then increase to improve coherence—potentially softening the speed-versus-quality trade-off observed in the paper.
The derivative-based observation features could be estimated on hardware by dithering current and field by small amounts, giving an experimental route to the same observation space.
The same controller architecture could be extended from fixed-target synchronisation to frequency tracking, where the target changes mid-episode and the agent must re-lock without a reset.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a clean, modest proof of concept - TD3 and SAC can find current/field settings that make a macrospin STO hit a target frequency in simulation. The core result in Fig. 1b is believable. But the authors reset the magnetization to the same semi-stable state every step, which means the policy is choosing points on a static action-to-frequency map rather than driving a dynamical system. They openly acknowledge this won't happen in experiment, so the near-100% success rate is real only for the reset task. That's the main caveat; it's not a hidden flaw, but it sharply limits what the paper can claim about real STO control.

What's genuinely new: this is, as far as the citations go, the first application of TD3/SAC to STO frequency synchronization. The reward shaping terms - frequency error, action-derivative penalty, and Q-factor reward - are tailored to oscillator physics and the comparison among them is a useful starting point. Using the cmtj macrospin simulator as the environment is sensible and the setup mimics an experimental bias-tee configuration. They also don't oversell: the outlook is honest that this is a simulation framework.

Soft spots: the 'energy efficiency' improvement in the abstract is never measured as energy. Eq. 8 penalizes action derivatives and Fig. 2 then shows the agent takes smaller action steps - that's partly by construction, not an independent metric. The Q-factor improvement is less circular, since Q comes from FFT spectra, but it's averaged over only 8 runs with no error bars and the reward weights zeta1-zeta3 and tolerance epsilon0 are not reported, so the sensitivity of the results to those constants is unknown. The introduction says RL has an advantage over PID controllers but no PID baseline is ever run. Also, the observation features include derivatives like fdiff/dj computed from short FFT windows after 1-2 ns; that's optimistic for experimental frequency estimation, but in simulation it's fine.

The reset issue is the most substantial. The authors justify it as a way to avoid numerical transients and say in experiment the pause between actions gives relaxation time. That's plausible only if the oscillator settles to a unique steady state within the pause and has no history dependence. They don't test that. So the transfer claim rests on an untested assumption. That said, they flag it explicitly, and the paper doesn't pretend to have done hardware.

Who should read it: people working on RL for device control or STO-based computing who want a starting point for reward design. It deserves a serious referee - the question of whether per-step reset is acceptable in sim-to-real for oscillators is a real one and this paper gives a clean testbed. Recommendation: send to peer review with a request to add a continuous-control variant (no reset), report the missing constants, and replace the energy-efficiency language with what was actually measured.

Referee Report

3 major / 6 minor

Summary. The paper uses a macrospin LLGS simulator (cmtj library) to create an RL environment for a magnetic tunnel junction spin-torque oscillator. A TD3 or SAC agent chooses normalized actions (current density, field magnitude, and two field angles) and is rewarded when the measured STO peak frequency enters a tolerance band around a randomly sampled target frequency. The authors report that both agents reach near-100% synchronization success after about 1500 evaluation steps (Fig. 1b). They then test reward-shaping terms: Rδf proportional to frequency error (Eq. 7), Rδa penalizing squared normalized action derivatives (Eq. 8), and a Q-factor reward based on amplitude and linewidth (Eq. 9), combined in Eq. (10). The paper claims improvements in convergence smoothness, energy efficiency, and Q-factor, and discusses extension to other tunable magnetic devices.

Significance. If the simulation result transfers, the paper would demonstrate a useful RL-based auto-tuning method for spintronic oscillators, potentially reducing the need for device-specific manual tuning and enabling pretraining for experimental deployment. The work has concrete strengths: it uses an established open-source macrospin solver, compares two modern actor-critic algorithms, and reports a quantitative success-rate curve (Fig. 1b) rather than only anecdotal trajectories. However, the central practical claim of automatic synchronization is weakened by the environment's per-step reset of magnetization, which turns the task into static operating-point selection rather than closed-loop control of a dynamical system. The energy-efficiency claim is also not supported by any physical energy measurement. These issues are acknowledged in part by the authors, but they are load-bearing for the paper's stated goals.

major comments (3)

[Section 3, Environment design] The per-step reset of magnetization ('At each step, the magnetisation of the agent is first reset to the same initial, semi-stable condition') and the later admission ('the concept of restarting the initial conditions does not exist') mean every action is evaluated from the same initial condition. The frequency returned for each action is therefore a point in a static action-to-frequency map; no oscillator state (magnetization orientation, phase, amplitude, transient) carries over between control updates. The observations fdiff/dj, fdiff/dh, fdiff/dθ, and fdiff/dϕ are finite differences of this reset map, not true state-dependent responses. Consequently the near-100% success in Fig. 1(b) is for the reset task, not for continuous STO control. Since the introduction motivates pretraining for deployment, this acknowledged limitation is load-bearing. I recommend training without the reset (d
[Abstract and §4.1, Eq. (8)] The abstract claims 'improvement in ... energy efficiency', but Rδa = (at − at−1)^2 is a normalized action-derivative penalty, not a physical energy or power measurement. No energy, power, or integrated Joule heating is computed anywhere in the paper. The smoother trajectories in Fig. 2(d) are partly by construction, since Rδa directly rewards smaller action changes. The energy-efficiency statement should either be supported by an actual energy metric or removed; at minimum the wording should say 'smoother control actions' rather than 'energy efficiency'.
[Figs. 2–3 and §4] The comparative claims (smoother convergence, higher Q, trade-off between Q-factor and synchronization time) rely on averages over 8 runs with no error bars, confidence intervals, or significance tests. In Fig. 3, the synchronization-time difference appears modest, and the reward weights ζ1, ζ2, ζ3 are not reported in the text. Without this information, the reader cannot judge whether the Q-factor improvement is robust or an artifact of a particular weighting. This weakens the quantitative content of the reward-shaping comparison, although the basic synchronization success in Fig. 1(b) is less affected.

minor comments (6)

[Eq. (10)] The sentence 'R rewards are applied only on the synchronisation event' is unclear: Rδf and Rδa are earlier described as step-wise punishments. Please clarify which terms are active at non-synchronized steps and which are active only on successful synchronization.
[Figs. 1–3] Please add shaded confidence intervals or standard deviations for the 8-run averages; currently the plots show only mean curves.
[§4] Please report the actual values used for reward weights ζ1, ζ2, ζ3, the tolerance ε0, and the episode lengths (50 or 100 steps are mentioned, but not which is used in each figure).
[Eq. (7)] Rδf = |ε* − ε0| is an unusual form for a 'difference between target and achieved frequency' penalty, since ε0 is a tolerance. Please clarify whether ε0 is subtracted inside the absolute value or whether a separate tolerance is applied after computing |ε*|.
[Section 3, observation space] The notation fdiff/dj, fdiff/dh, fdiff/dθ, fdiff/dϕ is ambiguous: it should state explicitly that these are finite-difference quotients computed from successive reset-step measurements, and how the small denominators are regularized.
[Throughout] Minor typographical issues include 'aharmonic frequencies' (should be 'anharmonic frequencies') and inconsistent use of hats on reward terms in Eq. (10).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the RL synchronization results are emergent from training, and the reward-shaping ablations report designed objectives rather than imported predictions.

full rationale

The central claim—that TD3/SAC agents can synchronize a randomly sampled MTJ to a target frequency—is an empirical RL result. The reward only gives a positive signal when the frequency error is below threshold; the near-100% success rate in Fig. 1b must be learned and is not encoded in the state or reward definitions. The reward-shaping variants (Rδf, Rδa, R_Q*) are explicit objectives: reporting that the agent exhibits smaller action derivatives or higher Q-factors when these quantities are rewarded is a verification of the reward design, not a circular prediction. The paper also reports an emergent trade-off (longer synchronization times with Q-factor reward) that is not present in the reward. The simulation uses the authors' cmtj library [13], but this is a standard tool citation for LLGS integration; no result from that paper is invoked to define the present conclusions. The per-step magnetization reset is an acknowledged simplifiability/limitation and does not create an input-output identity: the agent still must discover the action-to-frequency map. No equation or fitted parameter is reused as a prediction, so no load-bearing circularity is present.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The environment rests on the macrospin LLGS approximation and on modeling choices (per-step magnetization reset, clean FFT-derived frequency features, target sampling within the achievable range) that a real controller would not enjoy. The main liability is the free parameters: reward weights, tolerance ε0, and sampling range are unreported, so the reward-shaping comparisons cannot be reproduced exactly. No new physical entities are introduced.

free parameters (5)

reward weights ζ1, ζ2, ζ3 = unspecified
Weights in eq. (10) that balance frequency-error, action-derivative, and Q-factor rewards. Never given in the paper, so the reward-shaping comparison is underdetermined.
synchronization tolerance ε0 = unspecified
Threshold in Section 4 that triggers the sync reward when ε* < ε0. Its value relative to the ~100 MHz FFT bin width determines the reported success rates.
target frequency sampling range = not quantified
Section 4 samples f* 'within the achievable frequency range under specified action limits', which presumes knowledge of the device response manifold that a real controller would have to explore blindly.
RL hyperparameters = learning rate 0.002, discount 0.98, network [64,64]
Section 4: fixed hand-chosen hyperparameters with no sensitivity analysis.
integration time and simulation window = 1 ps step, ~10 ns per action
Section 3: chosen 'to ensure rapid training'; the resulting ~100 MHz frequency resolution is coarse relative to GHz-scale targets and never discussed against ε0.

assumptions (5)

domain assumption The macrospin LLGS equation (2) with fields (5) faithfully models STO dynamics
Section 2: the entire environment and all reported success rates depend on this model.
domain assumption Single-domain macrospin approximation is adequate for this device
Section 2-3: spatial modes are neglected, which can matter in real STOs.
domain assumption Resetting magnetization to the same initial state each step does not invalidate the learned policy
Section 3: the authors acknowledge that in experiment the reset 'does not exist', so the policy may exploit the artificial reset.
domain assumption The observation features fdiff/dj, fdiff/dh, fdiff/dθ, fdiff/dϕ are measurable and informative
Section 3: these derivative features require reliable frequency estimates from short windows, which is harder experimentally than in simulation.
domain assumption FFT peak frequency over a ~10 ns window estimates the instantaneous frequency
Section 3: with 1 ps integration and ~10 ns window, resolution is ~100 MHz; the relation of this resolution to the sync tolerance ε0 is not analyzed.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Reinforcement learning for spin torque oscillator tasks." pith.science (2026). https://pith.science/paper/FDAW2N7J

@misc{pith2026250910057,
  author       = {Pith},
  title        = {Pith review of: Reinforcement learning for spin torque oscillator tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FDAW2N7J}},
  note         = {Machine review of arXiv:2509.10057}
}

read the original abstract

We address the problem of automatic synchronisation of the spintronic oscillator (STO) by means of reinforcement learning (RL). A numerical solution of the macrospin Landau-Lifschitz-Gilbert-Slonczewski equation is used to simulate the STO and we train the two types of RL agents to synchronise with a target frequency within a fixed number of steps. We explore modifications to this base task and show an improvement in both convergence and energy efficiency of the synchronisation that can be easily achieved in the simulated environment.

Figures

Figures reproduced from arXiv: 2509.10057 by the authors.

**Figure 2.** The effect of the reward shaping strategies on the sample trajectories. (a) shows the RL agent [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** The effect of the Q-factor reward shaping, averaged over 8 runs per each variant. (a-b) present [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 2 linked inside Pith

[1]

Chapter Four - Spin-Torque Oscillators

J.-V. Kim. “Chapter Four - Spin-Torque Oscillators”. Ed. by R. E. Camley and R. L. Stamps. Vol. 63. Solid State Physics. ISSN: 0081-1947. Academic Press, 2012, pp. 217–294

1947
[2]

Topologically protected vortex structures for low-noise magnetic sensors with high linear range

D. Suess et al. “Topologically protected vortex structures for low-noise magnetic sensors with high linear range”.Nature Electronics16 (2018), pp. 362–370

2018
[3]

Spin nano–oscillator–based wireless communication

H. S. Choi et al. “Spin nano–oscillator–based wireless communication”.Scientific Reports41 (2014), p. 5486

2014
[4]

Scaling up electrically synchronized spin torque oscillator networks

S. Tsunegi et al. “Scaling up electrically synchronized spin torque oscillator networks”.Scientific Reports81 (2018), p. 13475

2018
[5]

Vowel recognition with four coupled spin-torque nano-oscillators

M. Romera et al. “Vowel recognition with four coupled spin-torque nano-oscillators”.Nature563 7730 (2018), pp. 230–234

2018
[6]

Binding events through the mutual synchronization of spintronic nano-neurons

M. Romera et al. “Binding events through the mutual synchronization of spintronic nano-neurons”. Nature Communications131 (2022), p. 883

2022
[7]

Magnetic control of tokamak plasmas through deep reinforcement learning

J. Degrave et al. “Magnetic control of tokamak plasmas through deep reinforcement learning”. Nature6027897 (2022), pp. 414–419

2022
[8]

Avoiding fusion plasma tearing instability with deep reinforcement learning

J. Seo et al. “Avoiding fusion plasma tearing instability with deep reinforcement learning”.Nature 6268000 (2024), pp. 746–751

2024

Show all 22 references

[9]

Classics in Magnetics A Phenomenological Theory of Damping in Ferromagnetic Ma- terials

T. Gilbert. “Classics in Magnetics A Phenomenological Theory of Damping in Ferromagnetic Ma- terials”.IEEE Transactions on Magnetics406 (2004), pp. 3443–3449

2004
[10]

Spin transfer torques

D. Ralph and M. Stiles. “Spin transfer torques”.Journal of Magnetism and Magnetic Materials320 7 (2008), pp. 1190–1216

2008
[11]

Current-driven excitation of magnetic multilayers

J. Slonczewski. “Current-driven excitation of magnetic multilayers”.Journal of Magnetism and Magnetic Materials1591-2 (1996), pp. L1–L7

1996
[12]

Currents and torques in metallic magnetic multilayers

J. Slonczewski. “Currents and torques in metallic magnetic multilayers”.Journal of Magnetism and Magnetic Materials2473 (2002), pp. 324–338

2002
[13]

cmtj: Simulation package for analysis of multilayer spintronic devices

J. Mojsiejuk et al. “cmtj: Simulation package for analysis of multilayer spintronic devices”.npj Computational Materials91 (2023), p. 54

2023
[14]

The design and verification of MuMax3

A. Vansteenkiste et al. “The design and verification of MuMax3”.AIP Advances410 (2014), p. 107133

2014
[15]

Analysis of phase noise in a spin torque oscillator stabilized by phase locked loop

S. Tamaru et al. “Analysis of phase noise in a spin torque oscillator stabilized by phase locked loop”. Applied Physics Express95 (2016), p. 053005

2016
[16]

Spin-Torque and Spin-Hall Nano-Oscillators

T. Chen et al. “Spin-Torque and Spin-Hall Nano-Oscillators”.Proceedings of the IEEE10410 (2016), pp. 1919–1945

2016
[17]

Electrical detection of magnetization dynamics via spin recti- fication effects

M. Harder, Y. Gui, and C.-M. Hu. “Electrical detection of magnetization dynamics via spin recti- fication effects”.Physics Reports661(2016), pp. 1–59

2016
[18]

Fujimoto, H

S. Fujimoto, H. van Hoof, and D. Meger.Addressing Function Approximation Error in Actor-Critic Methods. arXiv:1802.09477 [cs, stat]. 2018

2018 arXiv
[19]

Haarnoja et al.Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

T. Haarnoja et al.Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv:1801.01290 [cs, stat]. 2018

2018 arXiv
[20]

Voltage-induced perpendicular magnetic anisotropy change in magnetic tunnel junctions

T. Nozaki et al. “Voltage-induced perpendicular magnetic anisotropy change in magnetic tunnel junctions”.Applied Physics Letters962 (2010), p. 022506

2010
[21]

Magnetic field sensor with voltage-tunable sensing properties

W. Skowroński et al. “Magnetic field sensor with voltage-tunable sensing properties”.Applied Physics Letters10119 (2012), p. 192401

2012
[22]

Effect of CoFeB electrode compositions on low frequency magnetic noise in tunneling magnetoresistance sensors

P. Wisniowski et al. “Effect of CoFeB electrode compositions on low frequency magnetic noise in tunneling magnetoresistance sensors”.Journal of Applied Physics12221 (2017), p. 213906

2017

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Chapter Four - Spin-Torque Oscillators

J.-V. Kim. “Chapter Four - Spin-Torque Oscillators”. Ed. by R. E. Camley and R. L. Stamps. Vol. 63. Solid State Physics. ISSN: 0081-1947. Academic Press, 2012, pp. 217–294

1947

[2] [2]

Topologically protected vortex structures for low-noise magnetic sensors with high linear range

D. Suess et al. “Topologically protected vortex structures for low-noise magnetic sensors with high linear range”.Nature Electronics16 (2018), pp. 362–370

2018

[3] [3]

Spin nano–oscillator–based wireless communication

H. S. Choi et al. “Spin nano–oscillator–based wireless communication”.Scientific Reports41 (2014), p. 5486

2014

[4] [4]

Scaling up electrically synchronized spin torque oscillator networks

S. Tsunegi et al. “Scaling up electrically synchronized spin torque oscillator networks”.Scientific Reports81 (2018), p. 13475

2018

[5] [5]

Vowel recognition with four coupled spin-torque nano-oscillators

M. Romera et al. “Vowel recognition with four coupled spin-torque nano-oscillators”.Nature563 7730 (2018), pp. 230–234

2018

[6] [6]

Binding events through the mutual synchronization of spintronic nano-neurons

M. Romera et al. “Binding events through the mutual synchronization of spintronic nano-neurons”. Nature Communications131 (2022), p. 883

2022

[7] [7]

Magnetic control of tokamak plasmas through deep reinforcement learning

J. Degrave et al. “Magnetic control of tokamak plasmas through deep reinforcement learning”. Nature6027897 (2022), pp. 414–419

2022

[8] [8]

Avoiding fusion plasma tearing instability with deep reinforcement learning

J. Seo et al. “Avoiding fusion plasma tearing instability with deep reinforcement learning”.Nature 6268000 (2024), pp. 746–751

2024