Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

Sample-Efficient Reinforcement Learning Controller for Deep Brain Stimulation in Parkinson's Disease

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SEA-DBS claims to make adaptive deep brain stimulation learnable with far fewer patient interactions.

desk verdict A reasonable engineering exercise, but the headline sample-efficiency result is an artifact of an unsound Q-target that doubles the reward. read the letter →

arxiv 2507.06326 v1 pith:PUDX2EQQ submitted 2025-07-08 cs.LG cs.AIcs.SYeess.SYq-bio.NC

Harsh Ravivarapu , Gaurav Bagwe , Xiaoyong Yuan , Chunxiu Yu , Lan Zhang This is my paper
classification cs.LGcs.AIcs.SYeess.SYq-bio.NC
keywords DeepBrainStimulationAdaptiveDBSReinforcementLearningSampleEfficiencyGumbel-SoftmaxBeta-bandOscillationsActor-CriticParkinson'sDisease
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SEA-DBS is a reinforcement-learning controller for adaptive deep brain stimulation in Parkinson's disease. The paper's central claim is that combining a predictive reward model with Gumbel-Softmax exploration lets a DDPG actor-critic learn to suppress pathological beta-band oscillations far more sample-efficiently than standard DDPG. This matters because clinical calibration cannot afford long interaction trials and because implantable stimulators have tight memory and power budgets. The authors validate the claim on a biologically realistic basal-ganglia simulation, reporting faster convergence, stronger beta suppression, and resilience to FP16 post-training quantization.

What carries the argument

Two mechanisms carry the argument. The first is the predictive reward model $f_\theta$, a learned function that maps a state-action pair to an estimated immediate reward $\hat r_t$; the paper inserts this estimate directly into the Q-target so the critic is updated with $r_t + \hat r_t + \gamma Q'$. The second is Gumbel-Softmax exploration, which perturbs action logits with Gumbel noise and softmaxes them at an annealed temperature, supplying a differentiable approximation of sampling from the binary action set $\{0,1\}$.

What would settle it

Run the DDPG baseline on the same basal-ganglia simulation with the reward function multiplied by 2 (the approximate inflation when the predictor is accurate) and compare its beta-power and reward curves to SEA-DBS; if the curves overlap, the claimed sample-efficiency gain is a reward-scaling artifact rather than an effect of predictive modeling.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that a DDPG-based controller can be made practical for adaptive DBS by altering the Q-target and the exploration scheme. SEA-DBS sets the critic target to $r_t + \hat r_t + \gamma Q_{\phi'}(s_{t+1}, \pi_{\theta'}(s_{t+1}))$, where $\hat r_t = f_\theta(s_t, a_t)$ is produced by a reward-predictive model trained with mean-squared error against the observed reward $r_t$. Actions are drawn by adding Gumbel noise to actor logits and applying a temperature-annealed softmax, making binary selection differentiable. In the simulation, this combination outperforms the DDPG baseline on $\beta$-band power and reward across environment-change intervals of 10, 20, 50, and 75 steps, and the trained policy retains its performance after FP16 quantization.

Load-bearing premise

The load-bearing premise is that adding the model's predicted reward to the observed reward in the training target gives the controller genuinely new information; if the predictor is accurate, the target simply contains roughly twice the same reward, and the apparent speed-up could be a scaling artifact.

Editorial extensions

If this is right

  • If SEA-DBS works as reported, aDBS controllers can be calibrated with far fewer stimulation episodes, shrinking the interaction budget from the millions of steps typical of model-free RL.
  • The same policy survives FP16 post-training quantization, reducing the model from 65 MB to 33 MB and bringing it closer to implantable-device memory limits.
  • Gumbel-Softmax exploration provides a differentiable path through binary actions, eliminating the need for continuous action noise or epsilon-greedy schedules in $\{0,1\}$ stimulation decisions.
  • The predictive-reward stream supplies its largest gains early in training, exactly when real feedback is scarcest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the predictive model is trained to approximate the same immediate reward $r_t$, an accurate predictor makes the Q-target carry roughly $2r_t$; a baseline DDPG trained with its reward doubled may reproduce the reported convergence curves, which would indicate that part of the sample-efficiency gain is reward rescaling rather than new information.
  • The two components should transfer to other binary closed-loop neuromodulation problems where a low-dimensional biomarker is the state and the reward is learnable, such as epilepsy or chronic pain stimulation.
  • A decisive ablation would hold the predicted reward out of the Q-target and instead use it only as an auxiliary loss; if convergence slows down, the paper's placement of $\hat r_t$ inside the temporal-difference target is doing the work, not the information it encodes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes SEA-DBS, a DDPG-based reinforcement learning controller for adaptive deep brain stimulation in a simulated Parkinsonian basal ganglia model. The framework augments DDPG with two components: a predictive reward model whose output is added to the observed reward in the Q-target, and Gumbel-Softmax-based exploration for binary stimulation actions. The authors report that SEA-DBS converges faster, suppresses beta-band power more strongly, and remains accurate after FP16 post-training quantization compared with a DDPG baseline.

Significance. If the sample-efficiency mechanism were valid, the work would be practically relevant for closed-loop neuromodulation, where interaction budgets are limited and on-device memory is constrained. The paper is clearly structured and includes an explicit algorithm, ablations, and a hardware-deployment angle via FP16 quantization. However, the central algorithmic contribution is not a legitimate sample-efficiency mechanism: the predictive reward model is trained to predict the same immediate reward that is still observed at every step, and adding it to the Q-target approximately doubles the reward signal when the predictor is accurate. The reported experiments also lack error bars, multiple seeds, or a direct sample-complexity measurement, so the empirical support for the core claim is not established.

major comments (3)
  1. [Section IV-B, Eqs. (8)-(10), Algorithm 1] The proposed predictive-reward mechanism does not provide a valid sample-efficiency gain. The predictor f_theta is trained by Eq. (10) to minimize the MSE against the immediate reward r_t, and Eq. (9) adds the predicted reward r_hat_t to the observed r_t in the Q-target. When the predictor is accurate, r_hat_t is approximately r_t, so the target becomes approximately 2 r_t + gamma Q'. Such a uniform rescaling of the reward does not add information about environment dynamics and cannot reduce the number of interactions needed to learn the optimal policy. Moreover, Algorithm 1 still observes and stores the true reward r_t at every step (lines 8-10), directly contradicting the abstract's claim that the method reduces reliance on real-time feedback. The faster convergence in Section V could be reproduced by simply doubling the reward for the baseline, and the paper offers no control experiment for this confound.
  2. [Section V, Table II and Figures 4-7] The empirical claims are not supported by the reported experiments. Table II gives single point estimates per condition with no error bars, no number of seeds, and no significance tests, and Figures 4-7 likewise appear to show individual training curves or single inference traces. The central claim of sample efficiency is never directly measured: the paper does not report the number of environment interactions required to reach a given performance threshold. Without a quantitative interaction-budget comparison and multiple seeds, the observed 'faster convergence' cannot be distinguished from noise or from the reward-doubling artifact in Eq. (9).
  3. [Section V-B, environment description] The evaluation uses the same computational PD model as the authors' prior work [18], with no comparison against an independent simulator or external benchmark. Because the baseline DDPG may be disadvantaged by the binary action handling or by the unmodified reward scale, the experiments do not establish that the predictive model, rather than reward inflation, drives the reported advantage. At minimum, the authors should include a control baseline in which the predicted reward is replaced by the observed reward itself (r_hat_t = r_t) to test whether the improvement is merely a doubling of the reward scale, and should validate on an independent neural model or a recorded neural-signal benchmark.
minor comments (5)
  1. [Section IV-C, Eq. (13)] The definition z_i = log(pi_theta(s_i)) + g_i / tau is inconsistent with Eq. (11), where the temperature divides the sum log(pi_i) + g_i. Please clarify the intended Gumbel-Softmax logit transformation and ensure the implementation matches the stated annealing schedule.
  2. [Section III-V, Table I] The observation window length n_obs in Eq. (4) and the reward threshold beta_t in Eq. (7) are not listed in Table I, even though both are free parameters that affect the state representation and reward scale.
  3. [Table II] The table layout is confusing: the column headings place 'Avg PSD' and 'Avg Reward' under each method without making it clear which sub-column belongs to which condition, and the units of the reward, given as x10^-2, are not derived from the quadratic reward function in Eq. (7).
  4. [Figures 4-7] The figure captions are too terse. They do not state whether curves are averaged over multiple runs or seeds, and Figure 5's panels are labeled by stimulation frequency while the text describes them as inference comparisons; please expand the captions to specify the plotted quantities and any error bars or confidence intervals.
  5. [Section V-B] The reported model-size reduction appears as '65M B to 33M B'; this should be written as '65 MB to 33 MB' or '65 MB to 33 MB' for clarity.

Circularity Check

1 steps flagged · score 6.0 of 10

Predictive reward model is fit to the observed reward and then added to it in the Q-target, making the claimed sample-efficiency gain a reward-rescaling artifact rather than a reduction in environment interactions.

  1. fitted input called prediction [Section IV-B, Eq. (9) and Eq. (10); Algorithm 1, Line 8]
    "Qtarget = rt + ˆrt + γQϕ′(st+1, πθ′(st+1)), (9) ... The predictive model is trained concurrently by minimizing the mean squared error (MSE) between predicted and actual rewards: Lpred = 1 B P B i=1 (ri − ˆri)2. (10)"

    Equation (10) fits r̂ to the immediate reward r_t. When the fit is accurate, r̂_t ≈ r_t, so Eq. (9) becomes Qtarget ≈ 2 r_t + γQ′. This is a constant rescaling of the reward signal: it preserves the optimal policy but adds no information about the environment. The method still requires the observed r_t at every step—Algorithm 1, Line 8 records it and Eq. (10) trains on it—so the stated 'reduced reliance on real-time feedback' and the claimed sample-efficiency gain do not follow from the equations. Faster convergence in Section V is equally consistent with simply multiplying the baseline reward by 2, and the paper provides no control for reward scaling.

full rationale

SEA-DBS's central novelty is the predictive reward model. The circular step is in Eqs. (8)-(10): f_theta is trained by MSE against r_t and its output is then added to r_t in the critic target. When f_theta is accurate, the target is approximately 2 r_t + γQ', a positive scaling that cannot change the optimal policy and does not reduce the number of environment interactions. The algorithm still observes r_t every step (Algorithm 1, Line 8) and trains f_theta on it (Eq. 10), so the claim that the model 'reduces reliance on real-time feedback' is not supported by the equations. The reported faster convergence and ablation advantage are therefore at least partly attributable to reward amplification; the paper does not compare against a reward-doubled DDPG baseline. The environment from [18] is a self-citation (four co-authors overlap), but using one's own simulator as a testbed is not itself a circular derivation; it mainly limits external validation. The Gumbel-Softmax component is a standard tool and its use is not circular. Because the central sample-efficiency claim is partially reduced by construction to reward rescaling, the score is 6.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the inherited simulator, the beta-power objective, the unverified reward-augmentation rule, and the unspecified exploration schedule. None of these are established by the paper itself, and several are drawn from the authors' prior work.

free parameters (5)
  • Reward threshold beta_t = 0.35
    Hand-set in Eq. 7; fixes when stimulation is rewarded, so it directly shapes all measured rewards.
  • Reward scale factor = 10
    Multiplicative constant in Eq. 7; scales reward magnitudes and therefore the magnitude of the Q-target and gradients.
  • Predictive reward weight = 1.0 (implicit)
    Eq. 9 adds predicted reward to observed reward with no tunable coefficient or theoretical justification; this determines the double-counting effect.
  • Gumbel-Softmax annealing parameters = not reported
    Eq. 14 defines tau_t but tau_0, tau_min, and lambda_tau are never given; exploration and convergence depend on them.
  • Observation window length n_obs = not reported
    Eq. 4 uses a fixed-length window of beta values but n_obs is unspecified.
assumptions (5)
  • domain assumption The basal ganglia model from Mehregan et al. [18] reproduces Parkinsonian beta-band pathology and therapeutically relevant DBS response.
    All training and evaluation happen inside this simulator; no independent validation against patient data or other models is provided.
  • domain assumption Beta-band power is a sufficient proxy for PD motor symptoms and a valid reward signal.
    State, reward, and all reported metrics are computed from GPi beta power; if beta power is not the clinical target, the policy maximizes the wrong objective.
  • ad hoc to paper Adding a learned reward predictor trained on the same reward into the Bellman target is a legitimate sample-efficiency mechanism.
    Eq. 9 includes the predicted reward; the paper gives no bias correction, no potential-based shaping guarantee, and no proof that this accelerates true learning.
  • standard math Gumbel-Softmax relaxation provides an unbiased or sufficiently low-bias policy gradient for the binary DBS action space.
    The paper cites Jang et al. [35] but does not analyze bias at finite temperature; the annealed schedule is unspecified.
  • domain assumption The baseline DDPG is implemented with comparable capacity and tuning.
    No code or baseline hyperparameters are provided; only one baseline is compared.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sample-Efficient Reinforcement Learning Controller for Deep Brain Stimulation in Parkinson's Disease." pith.science (2026). https://pith.science/paper/PUDX2EQQ

@misc{pith2026250706326,
  author       = {Pith},
  title        = {Pith review of: Sample-Efficient Reinforcement Learning Controller for Deep Brain Stimulation in Parkinson's Disease},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUDX2EQQ}},
  note         = {Machine review of arXiv:2507.06326}
}
read the original abstract

Deep brain stimulation (DBS) is an established intervention for Parkinson's disease (PD), but conventional open-loop systems lack adaptability, are energy-inefficient due to continuous stimulation, and provide limited personalization to individual neural dynamics. Adaptive DBS (aDBS) offers a closed-loop alternative, using biomarkers such as beta-band oscillations to dynamically modulate stimulation. While reinforcement learning (RL) holds promise for personalized aDBS control, existing methods suffer from high sample complexity, unstable exploration in binary action spaces, and limited deployability on resource-constrained hardware. We propose SEA-DBS, a sample-efficient actor-critic framework that addresses the core challenges of RL-based adaptive neurostimulation. SEA-DBS integrates a predictive reward model to reduce reliance on real-time feedback and employs Gumbel Softmax-based exploration for stable, differentiable policy updates in binary action spaces. Together, these components improve sample efficiency, exploration robustness, and compatibility with resource-constrained neuromodulatory hardware. We evaluate SEA-DBS on a biologically realistic simulation of Parkinsonian basal ganglia activity, demonstrating faster convergence, stronger suppression of pathological beta-band power, and resilience to post-training FP16 quantization. Our results show that SEA-DBS offers a practical and effective RL-based aDBS framework for real-time, resource-constrained neuromodulation.

Figures

Figures reproduced from arXiv: 2507.06326 by the authors.

Figure 1
Figure 1. Schematic of the PD brain model, highlighting key [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Overall training process using predictive modeling [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Comparison of SEA-DBS model training. Steps Baseline (DDPG) SEA-DBS (Ours) Avg PSD ↓ Avg Reward ↑ (×10−2 ) Avg PSD ↓ Avg Reward ↑ (×10−2 ) 10 364 -1.96 326 5.76 20 366 -2.56 309 16.81 50 332 3.24 304 21.16 75 328 4.84 302 23.00 TABLE II: Performance comparison between Baseline (DDPG) and SEA-DBS across different stimulation update intervals. SEA-DBS achieves lower PSD and higher average reward, indicating more effec… view at source ↗
Figures from the paper (3 more)
Figure 7
Figure 7. Figure 7: PSD reduction over 10 stimulation steps for Baseline, [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]
Figure 5
Figure 5. Figure 5: Inference comparison of SEA-DBS vs Baseline. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Performance of SEA-DBS vs Baseline models after [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Inverse Reinforcement Learning for Interpretable Keystroke Biomarkers in Parkinson's Disease

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    IRL on keystroke data from the neuroQWERTY dataset yields a speed-preference reward weight correlating with UPDRS-III severity (r=-0.607) after correcting collinearity in a three-parameter model, with replication and ...

  2. Inverse Reinforcement Learning for Interpretable Keystroke Biomarkers in Parkinson's Disease

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Inverse reinforcement learning recovers per-subject reward weights from keystroke timing that correlate with UPDRS-III severity (r=-0.607) on the neuroQWERTY dataset after correcting a collinear four-parameter model t...

  3. Inverse Reinforcement Learning for Interpretable Keystroke Biomarkers in Parkinson's Disease

    cs.LG 2026-06 conditional novelty 6.0 of 10

    The IRL-derived speed-preference weight from keystroke data correlates with Parkinson's motor severity and improves prediction beyond raw typing speed.

Reference graph

Works this paper leans on

44 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [18]

    Enhancing adaptive deep brain stimulation via efficient reinforcement learning,

    J. Mehregan, X. Yuan, G. Bagwe, J. Jackson, E. Mohammed, L. Zhang, and C. Yu, “Enhancing adaptive deep brain stimulation via efficient reinforcement learning,” in 2024 IEEE Intelligent Mobile Computing (MobileCloud). IEEE, 2024, pp. 38–45

    work page 2024

  2. [1]

    Cell biology of parkin- son’s disease: Mechanisms of synaptic, lysosomal, and mitochondrial dysfunction,

    S. M. Brooker, G. E. Naylor, and D. Krainc, “Cell biology of parkin- son’s disease: Mechanisms of synaptic, lysosomal, and mitochondrial dysfunction,” Current Opinion in Neurobiology, vol. 85, p. 102841, Apr 2024

    work page 2024

  3. [2]

    The cell biology of parkinson’s disease,

    N. Panicker, P. Ge, V . L. Dawson, and T. M. Dawson, “The cell biology of parkinson’s disease,” Journal of Cell Biology , vol. 220, no. 4, Apr 5 2021

    work page 2021

  4. [3]

    Pharmacological treatment of parkinson disease: A review,

    B. S. Connolly and A. E. Lang, “Pharmacological treatment of parkinson disease: A review,” JAMA: The Journal of the American Medical Association, vol. 311, no. 16, pp. 1670–1683, 2014

    work page 2014

  5. [4]

    Circuit mechanisms of parkinson’s disease,

    M. M. McGregor and A. B. Nelson, “Circuit mechanisms of parkinson’s disease,” Neuron, vol. 101, no. 6, pp. 1042–1056, 2019

    work page 2019

  6. [5]

    Biomarkers for closed-loop deep brain stimulation in parkinson disease and beyond,

    W. Bouthour, P. Megevand, J. Donoghue, C. Luscher, N. Birbaumer, and P. Krack, “Biomarkers for closed-loop deep brain stimulation in parkinson disease and beyond,” Nature Reviews Neurology , vol. 15, no. 6, pp. 343–352, Jun 2019

    work page 2019

  7. [6]

    New perspectives of deep brain stimulation indications for parkinson’s disease: A critical review,

    R. M. G. Barbosa, M. C. Soares, D. M. M. C. Portela, T. G. Guimarães, and R. G. Cury, “New perspectives of deep brain stimulation indications for parkinson’s disease: A critical review,”Brain Sciences, vol. 14, no. 7,

    work page

  8. [7]

    Parkinson’s disease dbs: what, when, who and why? the time has come to tailor dbs targets,

    M. S. Okun and K. D. Foote, “Parkinson’s disease dbs: what, when, who and why? the time has come to tailor dbs targets,” Expert Review of Neurotherapeutics , vol. 10, no. 12, pp. 1847–1857, Dec 2010. [Online]. Available: http://www.ncbi.nlm.nih.gov/pubmed/21384698

    work page arXiv 2010

Show all 44 references
  1. [8]

    Translational principles of deep brain stimulation,

    M. L. Kringelbach, N. Jenkinson, S. L. Owen, and T. Z. Aziz, “Translational principles of deep brain stimulation,” Nature Reviews Neuroscience, vol. 8, no. 8, pp. 623–635, 2007

    2007

  2. [9]

    Adaptive deep brain stimulation in advanced parkinson disease,

    S. Little, A. Pogosyan, S. Neal, B. Zavala, L. Zrinzo, M. Hariz, T. Foltynie, P. Limousin, K. Ashkan, J. FitzGerald, A. Green, T. Aziz, and P. Brown, “Adaptive deep brain stimulation in advanced parkinson disease,” Annals of Neurology , vol. 74, no. 3, pp. 449–457, 2013

    2013

  3. [10]

    Chronic adaptive deep brain stimulation versus conventional stimulation in parkinson’s disease: a blinded randomized feasibility trial,

    C. R. Oehrn, S. Cernera, L. H. Hammer, M. Shcherbakova, J. Yao, A. Hahn, S. Wang, J. L. Ostrem, S. Little, and P. A. Starr, “Chronic adaptive deep brain stimulation versus conventional stimulation in parkinson’s disease: a blinded randomized feasibility trial,” Nature Medicine...

    2024

  4. [11]

    Adaptive deep brain stimulation in parkinson’s disease: A delphi consensus study,

    M. Guidetti, T. Bocci, M. D. P. Del Álamo, G. Deuschl, A. Fasano, R. M. Fernandez, C. Gasca-Salas, C. Hamani, J. K. Krauss, A. A. Kühn et al., “Adaptive deep brain stimulation in parkinson’s disease: A delphi consensus study,” medRxiv, 2024

    2024

  5. [12]

    Sensing data and methodology from the adaptive dbs algorithm for personalized therapy in parkinson’s disease (adapt-pd) clinical trial,

    S. Stanslaski, R. L. Summers, L. Tonder, Y . Tan, M. Case, R. S. Raike, N. Morelli, T. M. Herrington, M. Beudel, J. L. Ostrem et al., “Sensing data and methodology from the adaptive dbs algorithm for personalized therapy in parkinson’s disease (adapt-pd) clinical trial,” npj P...

    2024

  6. [13]

    The origin of abnormal beta oscillations in the parkinsonian corticobasal ganglia circuits,

    A. Asadi, M. Madadi Asl, A. Vahabie, and A. Valizadeh, “The origin of abnormal beta oscillations in the parkinsonian corticobasal ganglia circuits,” Parkinson’s Disease, vol. 2022, p. 7524066, 2022

    2022

  7. [14]

    Oscillatory beta dynamics inform biomarker-driven treatment optimization for parkinson’s disease,

    E. Radcliffe, A. Baumgartner, D. Kern, M. Al Borno, S. Ojemann, D. Kramer, and J. Thompson, “Oscillatory beta dynamics inform biomarker-driven treatment optimization for parkinson’s disease,” Jour- nal of Neurophysiology , vol. 129, no. 6, pp. 1492–1504, 2023

    2023

  8. [15]

    The modulatory effect of adaptive deep brain stimulation on beta bursts in parkinson’s disease,

    G. Tinkhauser, A. Pogosyan, S. Little, M. Beudel, D. Herz, H. Tan, and P. Brown, “The modulatory effect of adaptive deep brain stimulation on beta bursts in parkinson’s disease,” Brain, vol. 140, no. 4, pp. 1053– 1067, 2017

    2017

  9. [16]

    Reinforcement learning framework for deep brain stimulation study,

    D. Krylov, R. Tachet, R. Laroche, M. Rosenblum, and D. V . Dylov, “Reinforcement learning framework for deep brain stimulation study,” arXiv preprint arXiv:2002.10948 , 2020

    2002 arXiv

  10. [17]

    Closed-loop deep brain stimulation with reinforcement learning and neural simulation,

    C.-C. Cho, P.-J. Huang, M.-C. Chen, and C.-W. Lin, “Closed-loop deep brain stimulation with reinforcement learning and neural simulation,” IEEE Transactions on Neural Systems and Rehabilitation Engineering , vol. 32, pp. 3615–3624, 2024

    2024

  11. [19]

    Neu- rostimulation for parkinson’s disease with early motor complications,

    W. Schuepbach, J. Rau, K. Knudsen, J. V olkmann, P. Krack, L. Tim- mermann, T. Hälbig, H. Hesekamp, S. Navarro, N. Meier et al., “Neu- rostimulation for parkinson’s disease with early motor complications,” New England Journal of Medicine , vol. 368, no. 7, pp. 610–622, 2013

    2013

  12. [20]

    The mds- updrs tracks motor and non-motor improvement due to subthalamic nucleus deep brain stimulation in parkinson disease,

    K. L. Chou, J. L. Taylor, and P. G. Patil, “The mds- updrs tracks motor and non-motor improvement due to subthalamic nucleus deep brain stimulation in parkinson disease,” Parkinsonism & related disorders , vol. 19, no. 11, pp. 966–969, 2013

    2013

  13. [21]

    A framework for translational therapy development in deep brain stimulation,

    J. Chen, J. V olkmann, and C. Ip, “A framework for translational therapy development in deep brain stimulation,” NPJ Parkinson’s Disease , vol. 10, no. 1, p. 216, Nov 8 2024

    2024

  14. [22]

    Offline learning of closed-loop deep brain stimulation controllers for parkinson disease treatment,

    Q. Gao, S. L. Schmidt, A. Chowdhury, G. Feng, J. J. Peters, K. Genty, W. M. Grill, D. A. Turner, and M. Pajic, “Offline learning of closed-loop deep brain stimulation controllers for parkinson disease treatment,” in Proceedings of the ACM/IEEE 14th International Conference on ...

    2023

  15. [23]

    A simulated environment for early development stages of reinforcement learning algorithms for closed-loop deep brain stimulation,

    S. Castaño-Candamil, M. Vaihinger, and M. Tangermann, “A simulated environment for early development stages of reinforcement learning algorithms for closed-loop deep brain stimulation,” in 2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biolog...

    2019

  16. [24]

    Reinforcement learning with foundation priors: Let the embodied agent efficiently learn on its own,

    W. Ye, Y . Zhang, H. Weng, X. Gu, S. Wang, T. Zhang, M. Wang, P. Abbeel, and Y . Gao, “Reinforcement learning with foundation priors: Let the embodied agent efficiently learn on its own,” arXiv preprint arXiv:2310.02635, 2023

    arXiv

  17. [25]

    Utilizing simulation for reinforcement learning and curiosity driven exploration in robotics,

    H. Åström, V . Krueger, and E. A. Topp, “Utilizing simulation for reinforcement learning and curiosity driven exploration in robotics,” in SAIS 2019, 31st Workshop of the Swedish AI Society , 2019

    2019

  18. [26]

    Curl: Contrastive unsupervised representations for reinforcement learning,

    M. Laskin, A. Srinivas, and P. Abbeel, “Curl: Contrastive unsupervised representations for reinforcement learning,” in International conference on machine learning . PMLR, 2020, pp. 5639–5650

    2020

  19. [27]

    Data-efficient reinforcement learning with self-predictive representations,

    M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman, “Data-efficient reinforcement learning with self-predictive representations,” arXiv preprint arXiv:2007.05929 , 2020

    2007 arXiv

  20. [28]

    Model-based reinforcement learning for atari,

    B. Osinski, C. Finn, D. Erhan, G. Tucker, H. Michalewski, K. Czechowski, L. M. Kaiser, M. Babaeizadeh, P. Kozakowski, P. Milos et al., “Model-based reinforcement learning for atari,” ICLR, vol. 1, p. 2, 2020

    2020

  21. [29]

    Dream to control: Learn- ing behaviors by latent imagination,

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to control: Learn- ing behaviors by latent imagination,” arXiv preprint arXiv:1912.01603 , 2019

    1912 arXiv

  22. [30]

    Mastering diverse do- mains through world models,

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse do- mains through world models,” arXiv preprint arXiv:2301.04104 , 2023

    2023 arXiv

  23. [31]

    Temporal difference learning for model predictive control,

    N. Hansen, X. Wang, and H. Su, “Temporal difference learning for model predictive control,” arXiv preprint arXiv:2203.04955 , 2022

    2022 arXiv

  24. [32]

    Td-mpc2: Scalable, robust world models for continuous control,

    N. Hansen, H. Su, and X. Wang, “Td-mpc2: Scalable, robust world models for continuous control,” arXiv preprint arXiv:2310.16828, 2023

    2023 arXiv

  25. [33]

    Mastering atari games with limited data,

    W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y . Gao, “Mastering atari games with limited data,” Advances in neural information processing systems, vol. 34, pp. 25 476–25 488, 2021

    2021

  26. [34]

    Efficientzero v2: Mastering discrete and continuous control with limited data,

    S. Wang, S. Liu, W. Ye, J. You, and Y . Gao, “Efficientzero v2: Mastering discrete and continuous control with limited data,” arXiv preprint arXiv:2403.00564, 2024

    2024 arXiv

  27. [35]

    Categorical reparameterization with gumbel-softmax,

    E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” arXiv preprint arXiv:1611.01144 , 2016

    2016 arXiv

  28. [36]

    Gumbel-softmax score and flow matching for discrete biological sequence generation,

    S. Tang, Y . Zhang, A. Tong, and P. Chatterjee, “Gumbel-softmax score and flow matching for discrete biological sequence generation,” in ICLR 2025 Workshop on Deep Generative Model in Machine Learning: Theory, Principle and Efficacy , 2025

    2025

  29. [37]

    Seizure control in a com- putational model using a reinforcement learning stimulation paradigm,

    V . Nagaraj, A. Lamperski, and T. I. Netoff, “Seizure control in a com- putational model using a reinforcement learning stimulation paradigm,” International journal of neural systems , vol. 27, no. 07, p. 1750012, 2017

    2017

  30. [38]

    Adaptive optimal tracking control of an underactuated surface vessel using actor–critic reinforcement learning,

    L. Chen, S.-L. Dai, and C. Dong, “Adaptive optimal tracking control of an underactuated surface vessel using actor–critic reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems, 2022

    2022

  31. [39]

    The role of multisensor data fusion in neuromuscular control of a sagittal arm with a pair of muscles using actor-critic reinforcement learning method,

    V . Golkhou, M. Parnianpour, and C. Lucas, “The role of multisensor data fusion in neuromuscular control of a sagittal arm with a pair of muscles using actor-critic reinforcement learning method,” Technology and Health Care , vol. 12, no. 6, pp. 425–438, 2004

    2004

  32. [40]

    Neuromuscular control of the point to point and oscillatory movements of a sagittal arm with the actor–critic reinforcement learning method,

    ——, “Neuromuscular control of the point to point and oscillatory movements of a sagittal arm with the actor–critic reinforcement learning method,” Computer Methods in Biomechanics and Biomedical Engineer- ing, vol. 8, no. 2, pp. 103–113, 2005

    2005

  33. [41]

    Complexity of subthalamic 13-35 hz oscillatory activity directly correlates with clinical impairment in patients with parkinson’s disease,

    C. Chen, Y . Hsu, H. Chan, S. Chiou, P. Tu, S. Lee, C. Tsai, C. Lu, and P. Brown, “Complexity of subthalamic 13-35 hz oscillatory activity directly correlates with clinical impairment in patients with parkinson’s disease,” Experimental Neurology, vol. 224, no. 1, pp. 234–240, 2010

    2010

  34. [42]

    Actor-critic reinforcement learning for control with stability guarantee,

    M. Han, L. Zhang, J. Wang, and W. Pan, “Actor-critic reinforcement learning for control with stability guarantee,” IEEE Robotics and Au- tomation Letters, vol. 5, no. 4, pp. 6217–6224, 2020

    2020

  35. [43]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” in International conference on machine learning . Pmlr, 2018, pp. 1861–1870

    2018

  36. [2024]

    Available: https://www.mdpi.com/2076-3425/14/7/638

    [Online]. Available: https://www.mdpi.com/2076-3425/14/7/638

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.