Pith. sign in

REVIEW 4 major objections 4 minor 105 references

Foundations of Reinforcement Learning and Control:Connections and New Perspectives

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fixed Soft Actor-Critic policy survives a sudden loss of joint damping when a low-level MRAC loop retunes the feedback gains online.

desk verdict A solid tutorial with an original but under-evidenced hybrid experiment: the SAC+MRAC result is honestly hedged yet only demonstrated on a matched damping perturbation with a single run. read the letter →

arxiv 2608.02433 v1 pith:JPRD4RQ5 submitted 2026-08-03 cs.LG

Claire Vernade , Onno Eberhard , Martha White , Florian Dörfler , Csaba Szepesvári , Miroslav Krstic , Michael Muehlebach This is my paper
classification cs.LG
keywords reinforcementlearningadaptivecontrolmodel-referenceactor-criticmethodssoftdynamicprogrammingroboticlocomotionhybrid
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning and adaptive control are often treated as rival approaches to data-driven control, but this tutorial argues they are complementary answers to the same problem. Its original contribution is a layered architecture: a Soft Actor-Critic (SAC) policy, trained in a static Half-Cheetah simulation, is used as a high-level planner that outputs desired joint angles rather than torques; a model-reference adaptive controller (MRAC) then adjusts the low-level feedback gains so the real joints track those angles. When the damping coefficients of all six joints are halved mid-deployment, pure SAC and SAC with a fixed low-level controller lose performance dramatically, while the SAC+MRAC combination keeps the policy effective. The mechanism, the paper states, is that the adaptive controller restores the low-level dynamics to the nominal dynamics used during training, without retraining the policy. The tutorial also maps the conceptual vocabulary of the two fields—Lyapunov functions, value functions, dynamic programming, adaptation, exploration—onto each other to make the bridge usable.

What carries the argument

The load-bearing object is the low-level MRAC layer with a reference model identified from data, together with the reinterpretation of the policy output as a desired trajectory. MRAC's matching assumption—that some ideal feedback gains can make the closed loop reproduce the reference model—is what gives the architecture its ability to compensate for parametric drift; the adaptation law updates only the damping-compensating gain k_thetadot, using a one-step prediction error to avoid the large, uninformative errors that nonlinear contact dynamics would otherwise produce.

What would settle it

Run the same hybrid pipeline under a dynamics change that alters joint stiffness or actuator scaling while leaving damping fixed: the adaptation is frozen in the k_r and k_theta directions, so if the paper's account is right, performance should collapse; if it does not, the restoration effect is not specific to damping. A second check: replace the one-step prediction error with the standard model-state error used in Section 3.1; the paper predicts this will fail on the nonlinear contact dynamics, so a successful run would undercut the mechanism.

Watch

Extended reading notes

Core claim

The central claim is that a model-free policy and an adaptive controller can be composed across a semantics boundary: the policy emits reference trajectories, and a low-level MRAC loop enforces them. Concretely, the authors train and freeze an SAC policy on Half-Cheetah, then add a low-level controller a = k_r r + k_theta theta + k_thetadot thetadot. A linear reference model is fit by least squares from 200,000 steps of the nominal system, and MRAC is allowed to adapt only k_thetadot, with Gamma entries for k_r and k_theta zeroed, a step size of 10^-7, and k_thetadot clamped to [-0.3, 0.3]. The injected failure—halving all six joint dampings—is exactly the matched case in which only k_thetad

Load-bearing premise

The demonstration depends on the failure being caused only by a change in the joint damping coefficient, and on the swapped-in one-step prediction error staying stable even though the standard MRAC guarantees no longer apply.

Editorial extensions

If this is right

  • A pretrained RL policy can be made robust to a class of low-level dynamics changes without any gradient updates, simply by inserting an adaptive tracking layer underneath it.
  • The learned policy no longer needs to encode the plant's physical parameters; it only needs to encode the intended behavior, which is why the same gait remains valid after a damping change.
  • The reference model for the low-level layer can be obtained from data via a least-squares fit from logged transitions, so the hybrid scheme does not require an analytic model of the robot.
  • The same two-level split should carry over to any system with separable high-level planning and low-level stabilization, such as legged robots on changing terrain, manipulators with changing payloads, exoskeletons, and process plants under varying loads.
  • Conceptually, the comparison suggests that policy evaluation/improvement loops and Lyapunov-based adaptive laws are not competing metaphors but different tools for the same dynamics, and that optimality and stability can be aligned rather than traded off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not in the paper: if the policy's outputs are semantically trajectories, the same frozen policy should work across a family of low-level controllers as long as each holds the reference; this could be tested by swapping the MRAC layer for a different tracking controller and checking whether the SAC policy's reward stays high.
  • The paper leaves the one-step prediction-error modification without a stability proof; a useful next step would be to characterize, for nonlinear systems, when this reset preserves boundedness of the closed loop, giving the hybrid method a certificate beyond the matched linear case.
  • The adaptation is deliberately frozen along k_r and k_theta; for shifts that also change joint stiffness or input scaling, the same architecture would need those directions active, so the experiment implicitly predicts where the hybrid approach will fail.
  • A practical consequence of the design is that the high-level policy can be trained once in nominal conditions and deployed under parametric drift, suggesting a cheap robustness recipe: separate behavioral learning from plant compensation instead of entangling them in one network.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This tutorial paper aims to bridge reinforcement learning and adaptive control by introducing a unified notation, reviewing MRAC (§3.1) and actor-critic/SAC (§3.3), and proposing an original hybrid architecture in §3.4: a fixed SAC policy outputs desired joint angles, and a low-level MRAC controller with gains k_r, k_θ, k_θ̇ tracks them. The central empirical claim is in §3.4 and Figure 3: after a deployment-time halving of joint damping in Half-Cheetah, SAC+MRAC maintains running performance, whereas pure SAC and SAC with a fixed low-level controller K0 degrade. The authors candidly disclose two key limitations: the experiment perturbs only the damping direction, for which the single adaptive parameter k_θ̇ can compensate (Eq. (11)), and the MRAC update is modified to a one-step prediction error that 'does not preserve theoretical guarantees' from §3.1. The paper's broader value is pedagogical, but the novel empirical contribution rests on a matched-perturbation, single-run demonstration.

Significance. If the hybrid-architecture claim in §3.4 is substantiated, the paper would provide a useful, reproducible proof-of-concept (code is released) for combining model-free RL with low-level adaptive control, and the tutorial exposition itself is well-structured and honest about limitations. The MRAC derivation in §3.1 is standard and appropriately flagged as approximate in discrete time. However, the current evidence for the central restoration claim is narrow: a single trajectory, a perturbation perfectly aligned with the sole adapted parameter, and a modified update law without stability guarantees. These features make the result a heuristic illustration rather than an established capability. The significance would be considerably strengthened by multi-seed experiments, mismatched perturbations, and either analysis or systematic sensitivity studies of the modified MRAC update.

major comments (4)
  1. [§3.4, Fig. 3] The load-bearing sentence 'the adaptive controller restores the low-level dynamics to the nominal dynamics' is supported only by the orange line in Figure 3, which appears to be a single run with no seeds, error bars, or confidence intervals. Because SAC is stochastic and the adaptation law is modified, one trajectory cannot establish restoration. Please report multiple independent runs (at least 5) with confidence intervals, and ideally directly measure the tracking error or the closed-loop input/output behavior before and after adaptation rather than inferring restoration solely from total reward.
  2. [§3.4, Eq. (11) and Γ setting] The experiment is a matched-perturbation test. The text sets the Γ components for k_r and k_θ to zero, leaving only k_θ̇ adaptive, and the injected fault is exactly a factor-2 reduction in damping in all joints—the one direction that Eq. (11) shows k_θ̇ can compensate. This demonstrates that an integral controller in one scalar direction can undo a shift along that same direction; it does not support the broader statements in §1 and §3.4 about adapting to 'changing low-level dynamics,' changing terrain, uncertain payloads, or other mismatches. Please either add mismatched perturbations (e.g., inertia, actuator gain, contact friction) or explicitly scope the claim to damping-direction changes.
  3. [§3.4, MRAC update modification] Replacing the MRAC error e by a one-step prediction error, with s_m reset to s at every timestep, is a significant change to the update derived in §3.1; the paper itself states that this 'does not preserve theoretical guarantees.' The stability and performance of the modified update on the nonlinear contact-rich Half-Cheetah are therefore unaccounted for by the Lyapunov argument, and Figure 3 is the only evidence that it works. Please provide either a stability/robustness analysis of the modified update (even linearized), or a systematic empirical sensitivity study over the adaptation gain, the clamp on k_θ̇, fault magnitude, and the effect of the nonlinear W^T φ term that is omitted. Without this, the restoration claim remains a tuned heuristic.
  4. [§3.4, reference-model fitting] The reference model (A, B) is obtained by least-squares fitting on 200,000 steps of SAC–K0 interaction, and the same fitted B is used as the 'real' input matrix in the MRAC update. The paper reports no fit quality (e.g., residuals, explained variance, or stability of the fitted A) and omits several experimental details needed for reproducibility (episode length, reward function, full SAC hyperparameters, and the exact simulation timestep). Because the entire low-level restoration depends on this fit, the paper should at least report the fitted matrices or their accuracy and specify all simulation settings, even if the code is public.
minor comments (4)
  1. [§1, structure paragraph] Typo: 'actor-criticalgorithms' should read 'actor-critic algorithms'.
  2. [§3.1, Fig. 2] The label 'Reference model (open-loop)' in Figure 2 is confusing: the reference model is driven by r(t) and defines the desired closed-loop behavior, not an open-loop plant. Clarify the label or the caption.
  3. [§3.1 vs §3.4, adaptation gain] The paper uses Γ=100I for the simple second-order illustration in §3.1 but Γ=10^{-7} for the k_θ̇ component in §3.4. The huge difference is explained by the nonlinear contact dynamics, but a one-sentence remark on scaling/normalization would help readers.
  4. [§3.3, Eq. (10)] The advantage term q_w(s,a)-v(s) uses a value estimate v(s) that is not defined before Eq. (10). Specify how v(s) is computed (e.g., as the expectation of q_w under the current policy, or by a separate value network).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the MRAC derivation is self-contained given its stated matching assumptions, and the Sec. 3.4 combination claim rests on an external simulation result with the paper's own caveat about the modified update.

full rationale

The paper's derivation chain is not circular. Section 3.1 derives the MRAC update from an explicit matching assumption (A - BΛK_s* = A_m, BΛK_r* = B_m) and a Lyapunov function U(e,Θ)=e^T P e + tr(Θ^T Γ^{-1} Θ Λ); the displayed approximate decrease U(e_{k+1},Θ_{k+1}) - U(e_k,Θ_k) ≈ -e_k^T Q e_k is a consequence of the update law, not an input. The paper assumes B is known and cites the standard matching conditions, so the derivation is self-contained given those assumptions. Section 3.4's reference model is obtained by least-squares system identification from 200k steps of the nominal SAC-K0 closed loop; this is a model-fitting input, but the MRAC controller's ability to track that reference under a damping change is an independent empirical result on the MuJoCo Half-Cheetah. The paper explicitly flags the one-step-error modification as 'a significant change to the MRAC update derived in Section 3.1, which does not preserve theoretical guarantees,' so the Sec. 3.4 orange-line result is not claimed to follow from the tutorial's own Lyapunov analysis. The damping-only perturbation and the zeroing of Γ components for k_r and k_θ narrow the demonstration's scope, but they are disclosed and constitute a controlled experiment rather than a circular reduction. Self-citations (e.g., [25], [26], [46], [67], [83], [104], [105]) are background references or extensions of standard adaptive control; none is invoked as a load-bearing uniqueness or existence theorem to force the paper's conclusions. No quantity in the paper reduces by construction to a fitted value or to the target claim.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central experiment rests on: (i) standard MRAC assumptions (matching condition, known B) imported from §3.1; (ii) a reference model fit to the controlled system's own nominal data; (iii) hand-tuned hyperparameters (Γ = 10^-7, clamp ±0.3, K0 gains) that the paper admits are needed to offset the neglected nonlinearity; and (iv) an injected perturbation that matches the single adaptation direction. The tutorial's expository content adds no free parameters beyond the standard MRAC/RL machinery.

free parameters (5)
  • Reference-model matrices A, B (MRAC target dynamics) = least-squares fit on 200,000 SAC–K0 interaction steps
    Section 3.4: 'We then find a least-squares fit for the matrices A and B'; the fitted B doubles as the 'real' plant input matrix in the adaptation law, so the controller's target is an empirical fit.
  • Adaptation step size for k_θ̇ (Γ component) = 10^-7
    Section 3.4: 'Such a small step size is necessary to prevent instabilities that arise due to the neglected nonlinear nature of the system.' Chosen by hand, and the system's nonlinearity is exactly what makes linear-theory gains unstable.
  • Clamp on adapted gain k_θ̇ = [−0.3, 0.3]
    Section 3.4: 'Similarly, we clamp k_θ̇ to the interval [−0.3, 0.3]' — a hand-chosen bound required for stability.
  • Fixed low-level controller K0 = k_θ̇=0, k_r=−k_θ=1
    Section 3.4: the SAC–K0 baseline that defines nominal performance and provides the data for the reference-model fit; gains chosen by the authors.
  • SAC hyperparameters = minibatch 256, replay buffer 100,000, exploration 100 steps
    Section 3.3 lists these as the Half-Cheetah settings; the figure's exact training configuration is not fully pinned (no seed, no entropy-coefficient value).
assumptions (6)
  • domain assumption Matching assumption: there exist ideal gains K*_s, K*_r (and W*) satisfying A − BΛK*_s = A_m and BΛK*_r = B_m
    Eq. (7), Section 3.1; standard MRAC realizability condition that the closed loop can reproduce the reference model. The whole adaptive-control derivation depends on it.
  • domain assumption The input matrix B (action directions) is known exactly
    Figure 1 and Section 3.1: B is listed under 'known'; the adaptation law uses B in the update Θ_{k+1} = Θ_k − Γω_k B^T P e_k.
  • domain assumption Each Half-Cheetah joint is approximately a second-order linear system with small disturbance η
    Eq. (1), Section 2.1: 'This is a useful model if η is small.' The MRAC design, the K0 analysis, and Eq. (11) all use this linear model.
  • ad hoc to paper The deployment-time dynamics change affects only the damping coefficient ζ
    Section 3.4: only then is adapting k_θ̇ alone sufficient; Γ components for k_r and k_θ are set to zero. The experiment injects exactly this perturbation.
  • standard math Discrete-time Lyapunov analysis remains valid when higher-order terms in the sampling interval and Γ are neglected
    Section 3.1: 'Neglecting these terms, the update yields the approximate decrease... Rigorous discrete-time analyses account for the higher-order terms explicitly...' The tutorial's update-law justification is approximate.
  • ad hoc to paper The least-squares reference model trained on SAC–K0 rollouts faithfully represents the nominal closed-loop dynamics
    Section 3.4: the fitted A, B define the tracking target and the 'nominal dynamics'; contact-rich nonlinearities are absorbed into the fit residuals.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Foundations of Reinforcement Learning and Control:Connections and New Perspectives." pith.science (2026). https://pith.science/paper/JPRD4RQ5

@misc{pith2026260802433,
  author       = {Pith},
  title        = {Pith review of: Foundations of Reinforcement Learning and Control:Connections and New Perspectives},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JPRD4RQ5}},
  note         = {Machine review of arXiv:2608.02433}
}
read the original abstract

Reinforcement learning and control theory are two adjacent scientific fields that focus on optimizing the controller of unknown dynamical systems using feedback. While both fields have common roots in dynamic programming, they have evolved with distinct methodologies, goals, and cultures. Despite decades of mutual influence, a significant gap persists between the two communities. This tutorial introduces adaptive control, actor-critic reinforcement algorithms, and a new way to combine these two paradigms for data-driven decision making on a classical locomotion control problem. Our aim is to provide a foundation for understanding the core differences between the two approaches and insights to help experts in each field better understand and engage with the tools and approaches of the other.

Figures

Figures reproduced from arXiv: 2608.02433 by the authors.

Figure 1
Figure 1. Architecture of a model-reference adaptive controller. The adaptation loop is shown in gray. [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Illustration of model-reference adaptive control on a second-order system with a dynamics [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. SAC in the Half-Cheetah environment. The left plot shows the first [PITH_FULL_IMAGE:figures/full_fig_p016_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

105 extracted references · 10 linked inside Pith

  1. [1]

    A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada. Control barrier function based quadratic programs for safety critical systems.IEEE Transactions on Automatic Control, 62(8):3861–3876, 2017

    2017

  2. [2]

    Aminmansour, T

    F. Aminmansour, T. Jafferjee, E. Imani, E. J. Talvitie, M. Bowling, and M. White. Mitigating value hallucination in Dyna-style planning via multistep predecessor models.Journal of Artificial Intelligence Research, 80:441–473, 2024

    2024

  3. [3]

    B. D. Anderson, R. R. Bitmead, C. R. Johnson Jr, P. V. Kokotovic, R. L. Kosut, I. M. Mareels, L. Praly, and B. D. Riedle.Stability of adaptive systems: Passivity and averaging analysis. MIT Press, 1986

    1986

  4. [4]

    B. D. O. Anderson and J. B. Moore.Linear Optimal Control. Prentice Hall, 1971

    1971

  5. [5]

    A. M. Annaswamy. Adaptive Control and Intersections with Reinforcement Learning.Annual Review of Control, Robotics, and Autonomous Systems, 6:65–93, 2023

    2023

  6. [6]

    A. M. Annaswamy and A. L. Fradkov. A historical perspective of adaptive control and learning.Annual Reviews in Control, 52:18–41, 2021

    2021

  7. [7]

    K. Åström. History of adaptive control.Encyclopedia of Systems and Control, pages 902–909, 2021

    2021

  8. [8]

    K. J. Åström and B. Wittenmark.Adaptive Control. Dover Publications, 2nd edition, 2013

    2013

Show all 105 references
  1. [9]

    P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time Analysis of the Multi-armed Bandit Problem.Machine Learning, 47:235–256, 2002

    2002

  2. [10]

    Bellemare, S

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation.Advances in Neural Information Processing Systems, 29, 2016

    2016

  3. [11]

    M. G. Bellemare, S. Candido, P. S. Castro, J. Gong, M. C. Machado, S. Moitra, S. S. Ponda, and Z. Wang. Autonomous navigation of stratospheric balloons using reinforcement learning.Nature, 588(7836):77–82, 2020. 21

    2020

  4. [12]

    Bellman.Dynamic Programming

    R. Bellman.Dynamic Programming. Princeton University Press, 1957

    1957

  5. [13]

    D. P. Bertsekas.Dynamic Programming and Optimal Control. Athena Scientific, 4th edition, 2017

    2017

  6. [14]

    D. P. Bertsekas and J. N. Tsitsiklis.Neuro-Dynamic Programming. Athena Scientific, 1996

    1996

  7. [15]

    S. Boyd, L. El Ghaoui, E. Feron, and V. Balakrishnan.Linear Matrix Inequalities in System and Control Theory. SIAM, 1994

    1994

  8. [16]

    W. I. Caldwell. Control system with automatic response adjustment, 1950. US Patent 2,517,081

    1950

  9. [17]

    A. Chan, H. Silva, S. Lim, T. Kozuno, A. R. Mahmood, and M. White. Greedification operators for policy optimization: Investigating forward and reverse KL divergences.Journal of Machine Learning Research, 23 (253):1–79, 2022

    2022

  10. [18]

    Chen and L

    X. Chen and L. Zhao. On the convergence of continuous single-timescale actor-critic. InInternational Conference on Machine Learning, 2025

    2025

  11. [19]

    Asurveyofsim-to-realmethods in RL: Progress, prospects and challenges with foundation models.arXiv preprint arXiv:2502.13187, 2025

    L.Da,J.Turnau,T.P.Kutralingam,A.Velasquez,P.Shakarian,andH.Wei. Asurveyofsim-to-realmethods in RL: Progress, prospects and challenges with foundation models.arXiv preprint arXiv:2502.13187, 2025

    2025 arXiv

  12. [20]

    S. Dean, H. Mania, N. Matni, B. Recht, and S. Tu. Sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics, 20:633–679, 2020

    2020

  13. [21]

    Degrave, F

    J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de Las Casas, et al. Magnetic control of Tokamak plasmas through deep reinforcement learning.Nature, 602(7897):414–419, 2022

    2022

  14. [22]

    F. Dörfler. Data-driven control: Part two of two: Hot take: Why not go with models?IEEE Control Systems Magazine, 43(6):27–31, 2023

    2023

  15. [23]

    J. C. Doyle. Guaranteed Margins for LQG Regulators.IEEE Transactions on Automatic Control, 23(4): 756–757, 1978

    1978

  16. [24]

    Challengesofreal-worldreinforcement learning: Definitions, benchmarks and analysis.Machine Learning, 110(9):2419–2468, 2021

    G.Dulac-Arnold,N.Levine,D.J.Mankowitz,J.Li,C.Paduraru,etal. Challengesofreal-worldreinforcement learning: Definitions, benchmarks and analysis.Machine Learning, 110(9):2419–2468, 2021

    2021

  17. [25]

    Pinknoiseisallyouneed: Colorednoiseexploration in deep reinforcement learning

    O.Eberhard,J.Hollenstein,C.Pinneri,andG.Martius. Pinknoiseisallyouneed: Colorednoiseexploration in deep reinforcement learning. InInternational Conference on Learning Representations, 2023

    2023

  18. [26]

    Eberhard, M

    O. Eberhard, M. Muehlebach, and C. Vernade. Partially observable reinforcement learning with memory traces. InInternational Conference on Machine Learning, volume 267, pages 14934–14949, 2025

    2025

  19. [27]

    Eberhard, C

    O. Eberhard, C. Vernade, and M. Muehlebach. Commit to the bit: Reactive reinforcement learning done right. InInternational Conference on Machine Learning, volume 306, 2026

    2026

  20. [28]

    Elelimy, A

    E. Elelimy, A. White, M. Bowling, and M. White. Real-time recurrent learning using trace units in reinforcement learning.Advances in Neural Information Processing Systems, 37:17006–17043, 2024

    2024

  21. [29]

    Elmkaiel, S

    G. Elmkaiel, S. Schmitt, and M. Muehlebach. Embodied intelligence for sustainable flight: A soaring robot with active morphological control.npj Robotics, 2026

    2026

  22. [30]

    Fujimoto, H

    S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587–1596, 2018

    2018

  23. [31]

    D.Gabor,W.Wilby,andR.Woodcock. Auniversalnon-linearfilter,predictorandsimulatorwhichoptimizes itself by a learning process.Proceedings of the IEE-Part B: Electronic and Communication Engineering, 108(40):422–435, 1961

    1961

  24. [32]

    M. Gaur, A. S. Bedi, D. Wang, and V. Aggarwal. Closing the gap: Achieving global convergence (last iterate) of actor-critic under Markovian sampling with neural network parametrization.arXiv preprint arXiv:2405.01843, 2024

    2024 arXiv

  25. [33]

    Golowich, A

    N. Golowich, A. Moitra, and D. Rohatgi. Planning and learning in partially observable systems via filter stability. InACM Symposium on Theory of Computing, pages 349–362, 2023

    2023

  26. [34]

    Goodwin and K

    G. Goodwin and K. Sin.Adaptive Filtering Prediction and Control. Prentice-Hall, 1984

    1984

  27. [35]

    Graves, E

    E. Graves, E. Imani, R. Kumaraswamy, and M. White. Off-policy actor-critic with emphatic weightings. Journal of Machine Learning Research, 24(146):1–63, 2023

    2023

  28. [36]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational Conference on Machine Learning, pages 1861–1870, 2018

    2018

  29. [37]

    Hafner, W

    D. Hafner, W. Yan, and T. Lillicrap. Training agents inside of scalable world models.arXiv preprint arXiv:2509.24527, 2025. 22

    2025 arXiv

  30. [38]

    Hansen, H

    N. Hansen, H. Su, and X. Wang. TD-MPC2: Scalable, robust world models for continuous control.arXiv preprint arXiv:2310.16828, 2023

    2023 arXiv

  31. [39]

    Z. He, S. Bolognani, F. Dörfler, and M. Muehlebach. Decision-dependent stochastic optimization: The role of distribution dynamics.arXiv preprint arXiv:2503.07324, 2025

    2025 arXiv

  32. [40]

    Hewing, J

    L. Hewing, J. Kabzan, and M. N. Zeilinger. Cautious model predictive control using Gaussian process regression.IEEE Transactions on Control Systems Technology, 28(6):2736–2743, 2020

    2020

  33. [41]

    K.-C. Hsu, H. Hu, and J. F. Fisac. The safety filter: A unified view of safety-critical control in autonomous systems.Annual Review of Control, Robotics, and Autonomous Systems, 7, 2023

    2023

  34. [42]

    P. A. Ioannou and J. Sun.Robust Adaptive Control. Dover Publications, 2012

    2012

  35. [43]

    Near-optimalregretboundsforreinforcementlearning.JournalofMachine Learning Research, 11(51):1563–1600, 2010

    T.Jaksch,R.Ortner,andP.Auer. Near-optimalregretboundsforreinforcementlearning.JournalofMachine Learning Research, 11(51):1563–1600, 2010

    2010

  36. [44]

    M. K. Janjua, H. Shah, M. White, E. Miahi, M. C. Machado, and A. White. Gvfs in the real world: making predictions online for water treatment.Machine Learning, 113(8):5151–5181, 2024

    2024

  37. [45]

    Ostrovski, J.Quan, R.Munos, andW.Dabney

    S.Kapturowski, G. Ostrovski, J.Quan, R.Munos, andW.Dabney. Recurrentexperiencereplayindistributed reinforcement learning. InInternational conference on learning representations, 2018

    2018

  38. [46]

    Karafyllis and M

    I. Karafyllis and M. Krstic.Robust adaptive control: deadzone-adapted disturbance suppression. SIAM, 2025

    2025

  39. [47]

    Karmakar and S

    P. Karmakar and S. Bhatnagar. Two time-scale stochastic approximation with controlled markov noise and off-policy temporal-difference learning.Mathematics of Operations Research, 43(1):130–151, 2018

    2018

  40. [48]

    H. K. Khalil.Nonlinear Systems. Prentice Hall, 3rd edition, 2002

    2002

  41. [49]

    Adam: Amethodforstochasticoptimization.arXivpreprintarXiv:1412.6980, 2014

    D.P.KingmaandJ.Ba. Adam: Amethodforstochasticoptimization.arXivpreprintarXiv:1412.6980, 2014

    2014 arXiv

  42. [50]

    Kocsis and C

    L. Kocsis and C. Szepesvári. Bandit based Monte-Carlo planning. InEuropean Conference on Machine Learning, pages 282—-293, 2006

    2006

  43. [51]

    Krishnamurthy, A

    A. Krishnamurthy, A. Agarwal, and J. Langford. PAC reinforcement learning with rich observations. Advances in Neural Information Processing Systems, 29, 2016

    2016

  44. [52]

    Krstić, I

    M. Krstić, I. Kanellakopoulos, and P. V. Kokotović.Nonlinear and Adaptive Control Design. Wiley, 1995

    1995

  45. [53]

    P. R. Kumar and A. Becker. A new family of optimal adaptive controllers for Markov chains.IEEE Trans. on Automatic Control, 27:137–146, 1982

    1982

  46. [54]

    Lai and H

    T. Lai and H. Robbins. Asymptotically efficient adaptive allocation rules.Advances in Applied Mathematics, 6(1):4–22, 1985. ISSN 0196-8858. doi: https://doi.org/10.1016/0196-8858(85)90002-8. URL https: //www.sciencedirect.com/science/article/pii/0196885885900028

    1985

  47. [55]

    Sample-basedplanningandlearningwithfunctionapproximation.Statistical Sciences, 40(4):517–545, 2025

    T.LattimoreandC.Szepesvári. Sample-basedplanningandlearningwithfunctionapproximation.Statistical Sciences, 40(4):517–545, 2025

    2025

  48. [56]

    Lauri, D

    M. Lauri, D. Hsu, and J. Pajarinen. Partially observable markov decision processes in robotics: A survey. IEEE Transactions on Robotics, 39(1):21–40, 2022

    2022

  49. [57]

    N. P. Lawrence, T. Banker, and A. Mesbah. Mpcritic: A plug-and-play MPC architecture for reinforcement learning. InIEEE Conference on Decision and Control, pages 1048–1054, 2025

    2025

  50. [58]

    F. L. Lewis and D. Vrabie. Reinforcement Learning and Adaptive Dynamic Programming for Feedback Control.IEEE Circuits and Systems Magazine, 9(3):32–50, 2009

    2009

  51. [59]

    Memorylesspolicies: Theoreticallimitationsandpracticalresults

    M.L.Littman. Memorylesspolicies: Theoreticallimitationsandpracticalresults. InInternationalConference on Simulation of Adaptive Behavior: From Animals to Animats, volume 3, pages 238–245, 1994

    1994

  52. [60]

    Q. Liu, A. Chung, C. Szepesvari, and C. Jin. When is partially observable reinforcement learning not scary? InConference on Learning Theory, volume 178, pages 5175–5220, 2022

    2022

  53. [61]

    T. Liu, Y. Song, L. Zhu, and D. J. Hill. Stability and control of power grids.Annual Review of Control, Robotics, and Autonomous Systems, 5(1):689–716, 2022

    2022

  54. [62]

    Reinforcementlearningwithmodel-basedfeedforward inputs for robotic table tennis.Autonomous Robots, 47(8):1387–1403, 2023

    H.Ma,D.Büchler,B.Schölkopf,andM.Muehlebach. Reinforcementlearningwithmodel-basedfeedforward inputs for robotic table tennis.Autonomous Robots, 47(8):1387–1403, 2023

    2023

  55. [63]

    Martius, R

    G. Martius, R. Der, and N. Ay. Information driven self-organization of complex robotic behaviors.PLOS One, 8(5):e63400, 2013

    2013

  56. [64]

    Mishkin and L

    E. Mishkin and L. Braun.Adaptive Control Systems. McGraw-Hill Publishing, 1961

    1961

  57. [65]

    Fidjeland, G

    V.Mnih,K.Kavukcuoglu,D.Silver,A.A.Rusu,J.Veness,M.G.Bellemare,A.Graves,M.Riedmiller,A.K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, 23 S. Legg, and D. Hassabis. Human-level control through deep reinforcement lea...

    2015

  58. [66]

    Mohri, A

    M. Mohri, A. Rostamizadeh, and A. Talwalkar.Foundations of Machine Learning. MIT Press, second edition, 2018

    2018

  59. [67]

    Muehlebach, Z

    M. Muehlebach, Z. He, and M. I. Jordan. The sample complexity of online reinforcement learning: A multi-model perspective.International Conference on Learning Representations, 2026

    2026

  60. [68]

    K. S. Narendra and A. M. Annaswamy.Stable Adaptive Systems. Prentice Hall, 1989

    1989

  61. [69]

    Osband and B

    I. Osband and B. Van Roy. Why is posterior sampling better than optimism for reinforcement learning? In International Conference on Machine Learning, pages 2701–2710, 2017

    2017

  62. [70]

    Osband, C

    I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped DQN.Advances in Neural Information Processing Systems, 29, 2016

    2016

  63. [71]

    Y. Pan, M. Zaheer, A. White, A. Patterson, and M. White. Organizing experience: a deeper look at replay mechanisms for sample-based planning in continuous state domains.International Joint Conference on Artificial Intelligence, 2018

    2018

  64. [72]

    C. H. Papadimitriou and J. N. Tsitsiklis. The complexity of Markov decision processes.Mathematics of Operations Research, 12(3):441–450, 1987

    1987

  65. [73]

    Patterson, A

    A. Patterson, A. White, and M. White. A generalized projected Bellman error for off-policy value estimation in reinforcement learning.Journal of Machine Learning Research, 23(145):1–61, 2022

    2022

  66. [74]

    Patterson, S

    A. Patterson, S. Neumann, M. White, and A. White. Empirical design in reinforcement learning.Journal of Machine Learning Research, 25(318):1–63, 2024

    2024

  67. [75]

    Piazza, G

    C. Piazza, G. Grioli, M. G. Catalano, and A. Bicchi. A century of robotic hands.Annual Review of Control, Robotics, and Autonomous Systems, 2(1):1–32, 2019

    2019

  68. [76]

    W. B. Powell.Approximate Dynamic Programming: Solving the Curses of Dimensionality. John Wiley & Sons, 2007

    2007

  69. [77]

    Rosenblatt

    F. Rosenblatt. The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 65(6):386, 1958

    1958

  70. [78]

    Sastry and M

    S. Sastry and M. Bodson.Adaptive control: stability, convergence and robustness. Prentice Hall, 1989

    1989

  71. [79]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  72. [80]

    Silver, G

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algorithms. InInternational Conference on Machine Learning, pages 387–395, 2014

    2014

  73. [81]

    D. e. a. Silver. Mastering the game of Go with deep neural networks and tree search.Nature, 529:484—-489, 2016

    2016

  74. [82]

    D. e. a. Silver. Mastering chess and Shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017

    2017 arXiv

  75. [83]

    Smyshlyaev and M

    A. Smyshlyaev and M. Krstic.Adaptive control of parabolic PDEs. Princeton University Press, 2010

    2010

  76. [84]

    M. J. A. Strens. A Bayesian framework for reinforcement learning. InICML, pages 943–950, 2000

    2000

  77. [85]

    Sukhija, M

    B. Sukhija, M. Turchetta, D. Lindner, A. Krause, S. Trimpe, and D. Baumann. GoSafeOpt: Scalable Safe Exploration for Global Optimization of Dynamical Systems.Artificial Intelligence, 320, 2023

    2023

  78. [86]

    R. S. Sutton. Learning to predict by the methods of temporal differences.Machine Learning, 3(1):9–44, 1988

    1988

  79. [87]

    R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. InAAAI Conference on Artificial Intelligence, pages 216–224, 1991

    1991

  80. [88]

    R. S. Sutton. The Bitter Lesson. Incomplete Ideas (Blog), 2019. URLhttp://www.incompleteideas. net/IncIdeas/BitterLesson.html

    2019

  81. [89]

    R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation.Advances in Neural Information Processing Systems, 12, 1999

    1999

  82. [90]

    R. S. Sutton, A. G. Barto, and R. J. Williams. Reinforcement Learning is Direct Adaptive Optimal Control. IEEE Control Systems Magazine, 12(2):19–22, 2002

    2002

  83. [91]

    C. Tang, B. Abbatematteo, J. Hu, R. Chandra, R. Martín-Martín, and P. Stone. Deep reinforcement learning for robotics: A survey of real-world successes.Annual Review of Control, Robotics, and Autonomous Systems, 8(1):153–188, 2025. 24

    2025

  84. [92]

    Tassa, Y

    Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite.arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  85. [93]

    Tayal, M

    M. Tayal, M. Tayal, A. Singh, S. Kolathaya, and R. Prakash. V-OCBF: Learning Safety Filters from Offline Data via Value-Guided Offline Control Barrier Functions.Transactions on Machine Learning, 2026

    2026

  86. [94]

    P. Thomas. Bias in natural actor-critic algorithms. InInternational Conference on Machine Learning, pages 441–448, 2014

    2014

  87. [95]

    Todorov, T

    E. Todorov, T. Erez, and Y. Tassa. MuJoCo: a physics engine for model-based control. InIEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033, 2012

    2012

  88. [96]

    Analysisoftemporal-diffferencelearningwithfunctionapproximation.Advances in Neural Information Processing Systems, 9, 1996

    J.TsitsiklisandB.VanRoy. Analysisoftemporal-diffferencelearningwithfunctionapproximation.Advances in Neural Information Processing Systems, 9, 1996

    1996

  89. [97]

    J. N. Tsitsiklis. Asynchronous stochastic approximation and q-learning.Machine learning, 16(3):185–202, 1994

    1994

  90. [98]

    H. P. Van Hasselt, M. Hessel, and J. Aslanides. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019

    2019

  91. [99]

    Y. Wang, S. S. Zhan, R. Jiao, Z. Wang, W. Jin, Z. Yang, Z. Wang, C. Huang, and Q. Zhu. Enforcing Hard Constraints with Soft Barriers: Safe Reinforcement Learning in Unknown Stochastic Environments. In International Conference on Machine Learning, 2023

    2023

  92. [100]

    R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. In Advances in Neural Information Processing Systems, volume 5, 1992

    1992

  93. [101]

    Y. Xie. A survey of safe reinforcement learning methods in robotics.ITM Web of Conferences, 2025

    2025

  94. [102]

    Zakka, B

    K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Y. Luo, A. Allshire, E. Frey, K. Sreenath, L. A. Kahrs, C. Sferrazza, Y. Tassa, and P. Abbeel. MuJoCo playground.arXiv preprint arXiv:2502.08844, 2025

    2025 arXiv

  95. [103]

    Zhang, B

    S. Zhang, B. Liu, H. Yao, and S. Whiteson. Provably convergent two-timescale off-policy actor-critic with function approximation. InInternational Conference on Machine Learning, pages 11204–11213, 2020

    2020

  96. [104]

    Y. Zhao, O. Eberhard, M. Khammassi, A. H. Sayed, and M. Muehlebach. Why linear recurrent memory works in partially observable reinforcement learning. InInternational Conference on Machine Learning, volume 306, 2026

    2026

  97. [105]

    Zhu and M

    Y. Zhu and M. Krstic.Delay-adaptive linear control. Princeton University Press, 2020. 25 A Notation and correspondence between RL and control Throughout this tutorial we adopt a unified notation whenever possible. Since RL and control theory have historically developed differe...

    2020

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.