Pith. sign in

REVIEW 3 major objections 9 minor 120 references

Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems

T0 review · 3 major / 9 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new reinforcement-learning method estimates value gradients, not values, to make physics-based policy training sample-efficient and scalable.

desk verdict A promising SHAC variant ruined as written by an adjoint-target equation that doesn't match the paper's own adjoint recursion, plus experiments too thin for the claims—but worth sending to a serious referee. read the letter →

arxiv 2607.16177 v1 pith:VN4YUIBQ submitted 2026-07-17 cs.LG math.OC

Matteo Tomasetto , Nicolò Botteghi , Gabriele Bruni , Andrea Manzoni This is my paper
classification cs.LGmath.OC MSC 68T0749M3790C4093B52
keywords physics-informedreinforcementlearningdifferentiablesimulationadjointmethodvaluegradientshort-horizonpolicyoptimizationautomaticdifferentiationflowcontrolparametric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PEARL is a reinforcement-learning algorithm for controlling systems whose dynamics are differentiable, such as those governed by ODEs and PDEs. It computes policy gradients by automatic differentiation over short horizons, then estimates the missing terminal sensitivity—the adjoint, or value gradient—with a neural network, rather than estimating the value function as prior short-horizon actors do. The paper argues, and demonstrates in two double-gyre navigation benchmarks, that learning value gradients directly is more sample-efficient, yields higher rewards than model-free baselines (PPO, TD3) and gradient-based baselines (BPTT, SHAC), and scales to high-dimensional distributed control without low-dimensional tricks or multi-agent decompositions. If right, the result tells controllers that in differentiable environments the right quantity to learn is the gradient of future return, not the return itself.

What carries the argument

The actor-adjoint method: policy parameters are updated with the gradient in Eq. (18), an AD-computed short-horizon policy gradient whose terminal adjoint λ̂_{k0+h} comes from Eq. (19) — the analytic final-payoff gradient plus the output of the adjoint network φ. The adjoint network is trained against TD-λ targets of Eq. (20), which use the same adjoint dynamics to propagate a mixture of the short-horizon analytic derivative and the network's own prediction. The workhorse identity is that the closed-loop adjoint variable equals the value-gradient, ∇_y V^π, so approximating the adjoint is approximating the correct long-horizon credit assignment signal.

What would settle it

Take a small control problem (e.g., the double-gyre leader-follower with a short horizon) where the exact value gradient can be computed by full BPTT or the adjoint method. If the trained φ's estimate deviates significantly from the exact ∇_y V^π along the policy's trajectory — or if freezing φ and using only exact short-horizon gradients performs similarly — then the claimed benefit of learned adjoint correction would be refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that short-horizon bias in physics-based policy optimization is best corrected at the level of the gradient terminal condition. PEARL trains a policy with automatic differentiation over h steps, and a target adjoint network φ supplies the adjoint terminal condition λ̂_{k0+h} = ∂φ_terminal + φ(y,µ). The adjoint network is trained with a TD-λ scheme in which targets are produced by the adjoint recursion itself, blending the one-step physics-based sensitivity with the network's own estimate. Empirically, in a leader-follower navigation task and its mean-field PDE counterpart with 2145 state and 4290 action dimensions, this actor-adjoint scheme outperforms PPO, TD3,

Load-bearing premise

The whole scheme leans on the assumption that the adjoint network φ can learn an accurate value gradient from TD-λ targets that themselves depend on φ — a bootstrapped loop with no convergence guarantee; if the network's gradient estimate drifts, the policy gradient is biased and the sample-efficiency advantage lacks a foundation.

Editorial extensions

If this is right

  • Policy learning in differentiable simulators can be treated as an optimization problem with O(1/ε) sample complexity, rather than model-free O(σ²/ε²), provided the adjoint network tracks the true value gradient.
  • Sparse-reward tasks that defeat model-free methods become tractable: PEARL outperformed PPO and TD3 on rewards that are essentially zero except near the target.
  • High-dimensional distributed control (PDE states, thousands of control channels) can be learned by a single parametric policy without dimensionality reduction or multi-agent decomposition.
  • A single policy trained with scenario parameters µ generalizes across initial conditions and time-varying parameter trajectories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The value-gradient objective could be reused in model-based RL where a learned differentiable surrogate replaces the true simulator; the same actor-adjoint correction would apply as long as the surrogate's adjoint is reliable.
  • Because the adjoint network learns a field over (state, parameter) space, it may serve as a reusable sensitivity map — e.g., for approximately solving nearby optimal control problems quickly, connecting the method to real-time MPC.
  • The bootstrap in Eq. (20) suggests a testable variant: replacing the network-in-the-target term with a Monte Carlo return over a longer horizon would isolate whether the self-referential TD-λ term helps or hurts; the paper's design predicts it helps sample efficiency.
  • The claim 'estimate gradients, not values' may transfer beyond control to other long-horizon differentiable optimization pipelines, such as trajectory optimization and PDE-constrained design, wherever the terminal sensitivity is hard to obtain.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. The paper proposes PEARL, a physics-enhanced RL algorithm for differentiable environments. It combines short-horizon policy optimization (truncated BPTT) with a neural-network approximation of the terminal adjoint variable, identified with the value gradient. The adjoint network is trained with a TD-λ target derived from the adjoint equation. The method is evaluated on two parametric navigation tasks in a double-gyre flow: a leader-follower problem and a high-dimensional mean-field version. The central claim is that PEARL outperforms PPO, TD3, BPTT, truncated BPTT, and SHAC in sample efficiency and final performance, and that it scales to high-dimensional state/action spaces without low-dimensional representations or multi-agent decomposition.

Significance. If the method is correct, it would be a valuable contribution to physics-informed RL, particularly for PDE-constrained and high-dimensional control. The connection between closed-loop adjoint variables and value gradients (Appendix D) is clean and useful, and the authors provide code. The two benchmark problems, especially the finite-element mean-field control task, are nontrivial and relevant. However, the current manuscript does not establish the validity of the core algorithmic step: the TD-λ target for the adjoint network is inconsistent with the adjoint equation, so the learned network is not the quantity needed for an unbiased policy gradient. In addition, the empirical evidence is too thin to support the strong comparative claims. The idea is promising, but the theoretical and empirical foundations need substantial revision.

major comments (3)
  1. [§III.A, Eq. (20)] The TD-λ target for the adjoint network does not match the adjoint equation (9). The exact discrete closed-loop adjoint recursion is λ_k = A_k^T λ_{k+1} + b_k, where A_k = ∂F/∂y_k + (∂F/∂u_k)(∂π/∂y_k) and b_k = (∂L/∂y_k + (∂L/∂u_k)(∂π/∂y_k))^T. Equation (20) instead defines ¯λ_k = A_k^T (∂L/∂y_{k+1} + λ ¯λ_{k+1} + (1−λ)φ(y_{k+1};ϑ̄)). Thus the local cost gradient b_k is absent, and the next-state cost gradient ∂L/∂y_{k+1} is multiplied by A_k^T. In the minimal case λ=1, φ≡0, h=1, the target reduces to A^T ∂ϕ/∂y_{k0+h}, whereas the exact one-step value gradient is A^T ∂ϕ/∂y_{k0+h} + b. These differ generically (e.g., in LQR whenever the state cost is nonzero). Since J_φ trains φ to match ¯λ_k, the adjoint network is not trained toward the true value gradient, and Eq. (18) is not a justified approximation to ∇θJ. The paper provides no convergence or consistency guarantee for this coupled o
  2. [Section IV, Figures 2 and 4] The central empirical claim that PEARL 'outperforms' PPO, TD3, BPTT, truncated BPTT, and SHAC is supported only by single training/evaluation runs on two synthetic tasks, each with 10 evaluation episodes and no error bars, no multiple seeds, and no significance tests. Figures 2 and 4 show individual curves without variability. For a journal-level methods claim, this is insufficient. The authors should report statistics across at least several seeds (mean ± std or confidence intervals), clearly describe baseline hyperparameter tuning, and, if the 'outperforms' claim is to be comparative, provide pairwise significance tests or effect sizes. The strong conclusions in Section V are not supported by the presented evidence.
  3. [Title, Introduction, Section V] The title and text repeatedly claim 'real-time' optimal control, but the paper contains no timing measurements, no wall-clock comparisons, and no analysis of inference latency or control-loop frequency. If real-time performance is a claimed advantage, it must be demonstrated or at least discussed quantitatively. As written, the real-time claim is unsubstantiated and does not follow from the method's architectural properties alone.
minor comments (9)
  1. [Abstract] Typo: 'differentibility' should be 'differentiability'.
  2. [§III.A] Typo: 'state-of-the-are' should be 'state-of-the-art'.
  3. [Conclusions] Typo: 'contorl' should be 'control'.
  4. [Appendix B] Unresolved reference 'Equation (??)' appears when selecting the discrete adjoint variables. The equation number is missing.
  5. [Appendix D] There is a stray symbol '↷' in the last paragraph; likely a LaTeX artifact that should be removed.
  6. [Notation, Eqs. (8), (9), (19), (20)] The row/column convention for gradients is inconsistent. For example, Eq. (9) uses transposes on b_k, Eq. (19) writes ∂ϕ/∂y without a transpose while λ is column, and Eq. (20) mixes row and column quantities. Please fix the convention globally.
  7. [§IV.B, Figure 4 caption] The caption of Figure 4 mentions 'dense and sparse rewards', but Section IV.B only describes a dense reward. This appears to be copied from Figure 2's caption.
  8. [§IV.B] The acronym 'SHAC-MOD' is used without definition. Clarify what 'MOD' refers to and how it differs from the original SHAC implementation.
  9. [§III.A, Eq. (20)] The symbol λ is used both for the TD parameter and for the adjoint variable λ_k. This is very confusing; consider renaming the TD parameter (e.g., η or β_TD).

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity; the only self-referential element is a standard TD-λ bootstrap, and the empirical claims rest on external baselines.

full rationale

The paper's core derivations are self-contained. Appendix C proves the equivalence of reverse-mode AD and the adjoint recursion, and Appendix D proves λ = ∇_y V^π by differentiating the HJB/Bellman equation and invoking standard ODE uniqueness; neither step imports a result from the authors' own prior work. The actor-adjoint update in Eq. (18) approximates the terminal adjoint with a network φ, and Eq. (20) trains φ against TD-λ targets that contain φ(y_{k+1}; ϑ̄) and λ̄_{k+1}. This is a bootstrapped, self-referential target, but it is the standard structure of TD/value-gradient learning rather than a fitted parameter being renamed as a prediction: the policy-gradient claim is not derived by definition from the same data used to fit φ. Performance claims are supported by comparisons against external algorithms (PPO, TD3, BPTT, SHAC) and by evaluation rollouts, so the result is not forced by a self-citation chain. There are self-citations ([16], [17]) in the bibliography, but they are not load-bearing for the core derivation. One non-circular correctness concern: Eq. (20) does not match the exact adjoint recursion Eq. (9) — the local gradient b_k is absent and ∂L/∂y_{k+1} is multiplied by A_k^T, so the learned φ may converge to a biased quantity and Eq. (18) may not approximate Eq. (8). That would undermine the claimed sample-efficiency advantage, but it is an algorithmic-bias issue, not a circularity. No convergence proof is provided, but that omission is a correctness gap, not a circularity signal.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. Its load-bearing assumptions are the deterministic-differentiable environment, the mathematical equivalence between adjoint and value gradient, and the heuristic that a bootstrapped adjoint network can supply accurate terminal value gradients. The free parameters are standard RL hyperparameters and reward weights, chosen by hand rather than learned from data.

free parameters (5)
  • short horizon h = 16
    Chosen for both experiments; directly controls the bias-variance tradeoff of the short-horizon gradient.
  • TD parameter λ = 0.95
    Chosen to blend adjoint-recursion targets with adjoint-network estimates in Eq. (20).
  • reward weights β and β_g = 0.2 and 0.1
    Hand-picked to balance tracking accuracy against control effort; not fitted by a rigorous procedure.
  • learning rates = 1e-4 (leader-follower) / 1e-5 (mean-field)
    Problem-specific hand tuning; no sensitivity analysis.
  • discount γ and target update α = γ=0.99, α=0.995
    Standard RL hyperparameters; values chosen per common practice for both SHAC and PEARL.
assumptions (4)
  • standard math The closed-loop adjoint variable equals the gradient of the value function, λ = ∇_y V^T.
    Proved in Appendix D under smoothness assumptions; a classical optimal-control fact.
  • domain assumption The environment is deterministic and differentiable.
    Equation (18) requires automatic differentiation through F; the paper does not address stochastic dynamics or non-differentiable simulators.
  • ad hoc to paper The adjoint network φ learns an accurate value-gradient despite bootstrapped TD-λ targets.
    Eq. (20) defines targets that depend on φ itself through the (1−λ)φ term; no convergence guarantee is given for this coupled system.
  • ad hoc to paper The short-horizon gradient with an approximate terminal adjoint is a valid ascent direction.
    The convergence-rate arguments in Section II apply to exact gradients; the biased approximate gradient lacks a formal convergence analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems." pith.science (2026). https://pith.science/paper/VN4YUIBQ

@misc{pith2026260716177,
  author       = {Pith},
  title        = {Pith review of: Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VN4YUIBQ}},
  note         = {Machine review of arXiv:2607.16177}
}
read the original abstract

Reinforcement learning (RL) has recently emerged as a promising feedback control strategy for nonlinear and complex dynamical systems. However, RL algorithms are sample inefficient and require a large number of interaction with the environment to synthesize optimal control strategies. Consequently, applications of RL are typically limited to sparse sensors and actuators due to the curse of dimensionality entailed by the exploration-exploitation dilemma in high-dimensional spaces. In this work, we bridge RL and traditional optimal control for dynamical system with a novel Physics-EnhAnced Reinforcement Learning (PEARL) paradigm tailored to the control of high-dimensional and parametric dynamical systems, exploiting the differentibility of their dynamics. Specifically, PEARL employs an actor-adjoint algorithm that leverages automatic differentiation to compute policy gradients over short horizons and adjoint-based sensitivities of future returns approximated via neural networks, significantly reducing the number of environment interactions, while mitigating long-term gradient instabilities. Through two challenging parametric navigation problems in unsteady flows, we show that PEARL (i) effectively exploits differentiable environments to outperform state-of-the-art RL algorithms, (ii) is sample efficient, thanks to the physics-guided policy learning, (iii) generalizes across multiple scenarios, which is crucial when dealing with parametric systems, and (iv) enables scaling RL to high-dimensional state and action spaces, without requiring low-dimensional state representations or multi-agent strategies.

Figures

Figures reproduced from arXiv: 2607.16177 by the authors.

Figure 1
Figure 1. FIG. 1. Graphical summary of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. FIG. 2 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. FIG. 3 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: FIG. 4 [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: FIG. 5 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: FIG. 6 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: FIG. 7 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: FIG. 8. One-step computational graph of a discrete-time optimal control problem. Starting from the state [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

120 extracted references · 7 linked inside Pith

  1. [1]

    Agarwal, S

    A. Agarwal, S. M. Kakade, J. D. Lee, and G. Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift, 2020

    2020

  2. [2]

    S. V. Albrecht, F. Christianos, and L. Sch¨ afer.Multi- agent reinforcement learning: Foundations and modern approaches. MIT Press, 2024

    2024

  3. [3]

    Arulkumaran, M

    K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath. Deep reinforcement learning: A brief survey.IEEE Signal Processing Magazine, 34(6):26–38, 2017

    2017

  4. [4]

    Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J. Clark, S. McCandlish, C. Olah, B. Mann, and J. Ka- plan....

    2022

  5. [5]

    Banerjee, K

    C. Banerjee, K. Nguyen, C. Fookes, and M. Raissi. A survey on physics informed reinforcement learn- ing: Review and open problems.arXiv preprint arXiv:2309.01909, 2023

    arXiv 2023

  6. [6]

    Bardi, I

    M. Bardi, I. C. Dolcetta, et al.Optimal control and vis- cosity solutions of Hamilton-Jacobi-Bellman equations, volume 12. Springer, 1997. 12 FIG. 4.Mean-field leader-follower game. Training and evaluation rewards obtained by the different competing agents in the leader-follower game with dense and sparse rewards

    1997

  7. [7]

    Barkley and D

    B. Barkley and D. Fridovich-Keil. Stealing that free lunch: Exposing the limits of dyna-style reinforcement learning. In A. Singh, M. Fazel, D. Hsu, S. Lacoste- Julien, F. Berkenkamp, T. Maharaj, K. Wagstaff, and J. Zhu, editors,Proceedings of the 42nd International Conference on Machine Learning, volume 267 ofPro- ceedings of Machine Learning Research, p...

  8. [8]

    A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learn- ing: a survey.Journal of machine learning research, 18(153):1–43, 2018

    2018

Show all 120 references
  1. [9]

    Beintema, A

    G. Beintema, A. Corbetta, L. Biferale, and F. Toschi. Controlling rayleigh–benard convection via reinforce- ment learning.Journal of Turbulence, 21(9-10):585–605, 2020

    2020

  2. [10]

    Bellman.Dynamic Programming

    R. Bellman.Dynamic Programming. Princeton Univer- sity Press, Princeton, NJ, USA, 1 edition, 1957

    1957

  3. [11]

    Bengio, P

    Y. Bengio, P. Simard, and P. Frasconi. Learning long- term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2):157–166, 1994

    1994

  4. [12]

    Bertsekas.Nonlinear Programming

    D. Bertsekas.Nonlinear Programming. Athena scientific optimization and computation series. Athena Scientific, 2016

    2016

  5. [13]

    Bertsekas.Reinforcement learning and optimal con- trol, volume 1

    D. Bertsekas.Reinforcement learning and optimal con- trol, volume 1. Athena Scientific, 2019

    2019

  6. [14]

    D. P. Bertsekas.Dynamic programming and optimal control. Athena Scientific, Belmont, MA, 1995

    1995

  7. [15]

    B¨ ottcher, N

    L. B¨ ottcher, N. Antulov-Fantulin, and T. Asikis. Ai pontryagin or how artificial neural networks learn to control dynamical systems.Nature Communications, 13(1), 2022

    2022

  8. [16]

    Botteghi, M

    N. Botteghi, M. Poel, and C. Brune. Unsupervised rep- resentation learning in deep reinforcement learning: A review.IEEE Control Systems, 45(2):26–68, 2025

    2025

  9. [17]

    Botteghi, M

    N. Botteghi, M. Tomasetto, U. Fasel, F. Braghin, and A. Manzoni. Hypemarl: Multi-agent reinforce- ment learning for high-dimensional, parametric, and distributed systems, 2025

    2025

  10. [18]

    Bradbury, R

    J. Bradbury, R. Frostig, P. Hawkins, M. J. John- son, Y. Katariya, C. Leary, D. Maclaurin, G. Nec- ula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018

    2018

  11. [19]

    S. L. Brunton and J. N. Kutz.Data-Driven Science and Engineering: Machine Learning, Dynamical Sys- tems, and Control. Cambridge University Press, 2022

    2022

  12. [20]

    S. L. Brunton, N. Zolman, J. N. Kutz, and U. Fasel. Machine learning for sparse nonlinear modeling and control.Annual Review of Control, Robotics, and Au- tonomous Systems, 8(Volume 8, 2025):127–152, 2025

    2025

  13. [21]

    M. A. Bucci, O. Semeraro, A. Allauzen, G. Wisniewski, L. Cordier, and L. Mathelin. Control of chaotic systems by deep reinforcement learning.Proceedings of the Royal Society A, 475(2231):20190351, 2019

    2019

  14. [22]

    Bu¸ soniu, R

    L. Bu¸ soniu, R. Babuˇ ska, and B. De Schutter. A com- prehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernet- ics, Part C (Applications and Reviews), 38(2):156–172, 2008

    2008

  15. [23]

    Bu¸ soniu, R

    L. Bu¸ soniu, R. Babuˇ ska, and B. De Schutter. Multi- agent reinforcement learning: An overview.Innovations in multi-agent systems and applications-1, pages 183– 221, 2010

    2010

  16. [24]

    Buzzicotti, L

    M. Buzzicotti, L. Biferale, F. Bonaccorso, P. Clark di Leoni, and K. Gustavsson.Optimal control of point-to- point navigation in turbulent time dependent flows using reinforcement learning. 2020

    2020

  17. [25]

    Camacho and C

    E. Camacho and C. Bordons.Model Predictive Control. Springer London, 2004

    2004

  18. [26]

    Courant and D

    R. Courant and D. Hilbert.Methods of mathematical physics: partial differential equations. John Wiley & Sons, 2008

    2008

  19. [27]

    Degrave, F

    J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning.Nature, 602(7897):414–419, 2022

    2022

  20. [28]

    M. P. Deisenroth and C. E. Rasmussen. Pilco: a model- based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on In- ternational Conference on Machine Learning, ICML’11, page 465–472, Madison, WI, USA, 2011. Omnipress

    2011

  21. [29]

    Eberhard, C

    O. Eberhard, C. Vernade, and M. Muehlebach. A pontryagin perspective on reinforcement learning. In N. Ozay, L. Balzano, D. Panagou, and A. Abate, ed- itors,Proceedings of the 7th Annual Learning for Dy- namics & Control Conference, volume 283 ofPro- ceedings of Machine Lea...

    2025

  22. [30]

    Fairbank

    M. Fairbank. Reinforcement learning by value gradients, 2008

    2008

  23. [31]

    D. Fan, L. Yang, Z. Wang, M. S. Triantafyllou, and G. E. Karniadakis. Reinforcement learning for bluff body active flow control in experiments and simula- tions.Proceedings of the National Academy of Sciences, 117(42):26091–26098, 2020

    2020

  24. [32]

    W. H. Fleming and H. M. Soner.Controlled Markov processes and viscosity solutions. Stochastic Modelling and Applied Probability. Springer New York, NY, 2006

    2006

  25. [33]

    Fran¸ cois-Lavet, P

    V. Fran¸ cois-Lavet, P. Henderson, R. Islam, M. G. Belle- mare, J. Pineau, et al. An introduction to deep rein- forcement learning.Foundations and Trends®in Ma- 14 0 seconds Test 1 Test 2 Test 3 1 second 2 seconds 5 seconds 10 seconds 20 seconds 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4...

    2018

  26. [34]

    C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mor- datch, and O. Bachem. Brax – a differentiable physics engine for large scale rigid body simulation, 2021

    2021

  27. [35]

    Fujimoto, H

    S. Fujimoto, H. Hoof, and D. Meger. Addressing func- tion approximation error in actor-critic methods. In International conference on machine learning, pages 1587–1596. PMLR, 2018

    2018

  28. [36]

    Garnier, J

    P. Garnier, J. Viquerat, J. Rabault, A. Larcher, A. Kuhnle, and E. Hachem. A review on deep rein- forcement learning for fluid mechanics.Computers & Fluids, 225:104973, 2021

    2021

  29. [37]

    Ghadimi and G

    S. Ghadimi and G. Lan. Stochastic first- and zeroth- order methods for nonconvex stochastic programming. 15 Episode 1 PPO Control – 10 seconds PEARL Control – 10 seconds PEARL State – 10 seconds Episode 2 Episode 3 Episode 5 Episode 10 Evaluation 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1....

    2013

  30. [38]

    M. Glavic. (deep) reinforcement learning for electric power system control and related problems: A short review and perspectives.Annual Reviews in Control, 48:22–35, 2019

    2019

  31. [39]

    Govinda, B

    S. Govinda, B. Brik, and S. Harous. A survey on deep reinforcement learning applications in autonomous sys- tems: Applications, open challenges, and future direc- tions.IEEE Transactions on Intelligent Transportation Systems, 26(7):11088–11113, 2025. 16

    2025

  32. [40]

    Greensmith, P

    E. Greensmith, P. Bartlett, and J. Baxter. Variance reduction techniques for gradient estimates in rein- forcement learning. In T. Dietterich, S. Becker, and Z. Ghahramani, editors,Advances in Neural Informa- tion Processing Systems, volume 14. MIT Press, 2001

    2001

  33. [41]

    Grondman, L

    I. Grondman, L. Busoniu, G. A. D. Lopes, and R. Babuska. A survey of actor-critic reinforcement learning: Standard and natural policy gradients.Trans. Sys. Man Cyber Part C, 42(6):1291–1307, 2012

    2012

  34. [42]

    S. Gu, E. Holly, T. Lillicrap, and S. Levine. Deep rein- forcement learning for robotic manipulation with asyn- chronous off-policy updates. In2017 IEEE international conference on robotics and automation (ICRA), pages 3389–3396. IEEE, 2017

    2017

  35. [43]

    Hafner, T

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination, 2020

    2020

  36. [44]

    Hafner, T

    D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels, 2019

    2019

  37. [45]

    Hafner, J

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Master- ing diverse control tasks through world models.Nature, 640(8059):647–653, 2025

    2025

  38. [46]

    Hernandez-Leal, B

    P. Hernandez-Leal, B. Kartal, and M. E. Taylor. A survey and critique of multiagent deep reinforcement learning.Autonomous Agents and Multi-Agent Systems, 33(6):750–797, 2019

    2019

  39. [47]

    P. Holl, N. Thuerey, and V. Koltun. Learning to control pdes with differentiable physics. InInternational Con- ference on Learning Representations, 2020

    2020

  40. [48]

    H¨ uttenrauch, A.ˇSoˇ si´ c, and G

    M. H¨ uttenrauch, A.ˇSoˇ si´ c, and G. Neumann. Deep rein- forcement learning for swarm systems.Journal of Ma- chine Learning Research, 20(54):1–31, 2019

    2019

  41. [49]

    Janner, J

    M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization, 2021

    2021

  42. [50]

    W. Jin, Z. Wang, Z. Yang, and S. Mou. Pontrya- gin differentiable programming: An end-to-end learning and control framework. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, pages 7979–7992. ...

    2020

  43. [51]

    Karnakov, L

    P. Karnakov, L. Amoudruz, and P. Koumoutsakos. Op- timal navigation in microfluidics via the optimization of a discrete loss.Phys. Rev. Lett., 134:044001, 2025

    2025

  44. [52]

    Kaufmann, P

    T. Kaufmann, P. Weng, V. Bengs, and E. H¨ ullermeier. A survey of reinforcement learning from human feed- back, 2025

    2025

  45. [53]

    D. E. Kirk.Optimal control theory: an introduction. Courier Corporation, 2004

    2004

  46. [54]

    Kober, J

    J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey.The International Jour- nal of Robotics Research, 32(11):1238–1274, 2013

    2013

  47. [55]

    J. N. Kutz.Data-driven modeling & scientific computa- tion: methods for complex systems & big data. Oxford University Press, 2013

    2013

  48. [56]

    Lample and D

    G. Lample and D. S. Chaplot. Playing fps games with deep reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017

    2017

  49. [57]

    F. L. Lewis and D. Liu.Reinforcement learning and approximate dynamic programming for feedback control. John Wiley & Sons, 2013

    2013

  50. [58]

    Y. Li. Deep reinforcement learning: An overview.arXiv preprint arXiv:1701.07274, 2017

    2017 arXiv

  51. [59]

    Lin.Reinforcement learning for robots using neural networks

    L. Lin.Reinforcement learning for robots using neural networks. Carnegie Mellon University, 1992

    1992

  52. [60]

    M. L. Littman. Markov games as a framework for multi- agent reinforcement learning. InMachine learning pro- ceedings 1994, pages 157–163. Elsevier, 1994

    1994

  53. [61]

    Liu and J

    X. Liu and J. F. MacArt. Adjoint-based machine learn- ing for active flow control.Phys. Rev. Fluids, 9:013901, 2024

    2024

  54. [62]

    Liu and J.-X

    X.-Y. Liu and J.-X. Wang. Physics-informed dyna-style model-based deep reinforcement learning for dynamic control.Proceedings of the Royal Society A: Mathemati- cal, Physical and Engineering Sciences, 477(2255), 2021

    2021

  55. [63]

    F.-M. Luo, T. Xu, H. Lai, X.-H. Chen, W. Zhang, and Y. Yu. A survey on model-based reinforcement learning. Science China Information Sciences, 67(2), 2024

    2024

  56. [64]

    L¨ ohning, M

    M. L¨ ohning, M. Reble, J. Hasenauer, S. Yu, and F. Allg¨ ower. Model predictive control using reduced or- der models: Guaranteed stability for constrained linear systems.Journal of Process Control, 24(11):1647–1659, 2014

    2014

  57. [65]

    Manjavacas, A

    A. Manjavacas, A. Campoy-Nieves, J. Jim´ enez-Raboso, M. Molina-Solana, and J. G´ omez-Romero. An experi- mental evaluation of deep reinforcement learning algo- rithms for hvac control.Artificial Intelligence Review, 57(7), 2024

    2024

  58. [66]

    Manzoni, A

    A. Manzoni, A. Quarteroni, and S. Salsa.Optimal con- trol of partial differential equations. Springer, 2021

    2021

  59. [67]

    C. C. Margossian. A review of automatic differentia- tion and its efficient implementation.Wiley interdisci- plinary reviews: data mining and knowledge discovery, 9(4):e1305, 2019

    2019

  60. [68]

    Mitusch, S

    S. Mitusch, S. Funke, and J. Dokken. dolfin-adjoint 2018.1: automated adjoints for fenics and firedrake. Journal of Open Source Software, 4(38):1292, 2019

    2018

  61. [69]

    V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  62. [70]

    V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning.Nature, 518(7540):529, 2015

    2015

  63. [71]

    Mokbel, C

    S. Mokbel, C. Lagemann, E. Lagemann, and S. L. Brun- ton. Controlling chaotic energy events in fluids with re- inforcement learning. InProceedings of the 16th ACM International Conference on Future and Sustainable En- ergy Systems, E-Energy ’25, page 954–958. ACM, 2025

    2025

  64. [72]

    M. A. Z. Mora, M. Peychev, S. Ha, M. Vechev, and S. Coros. Pods: Policy optimization via differentiable simulation. In M. Meila and T. Zhang, editors,Proceed- ings of the 38th International Conference on Machine Learning, volume 139 ofProceedings of Machine Learn- ing Research...

    2021

  65. [73]

    Nesterov.Introductory Lectures on Convex Optimiza- tion: A Basic Course, volume 87 ofApplied Optimiza- tion

    Y. Nesterov.Introductory Lectures on Convex Optimiza- tion: A Basic Course, volume 87 ofApplied Optimiza- tion. Springer Science & Business Media, New York, 2013

    2013

  66. [74]

    Nesterov and V

    Y. Nesterov and V. Spokoiny. Random gradient-free minimization of convex functions.Foundations of Com- putational Mathematics, 17(2):527–566, 2015

    2015

  67. [75]

    Orr and A

    J. Orr and A. Dutta. Multi-agent deep reinforcement learning for multi-robot applications: A survey.Sen- sors, 23(7):3625, 2023. 17

    2023

  68. [76]

    Paszke, S

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017

    2017

  69. [77]

    Peitz, J

    S. Peitz, J. Stenner, V. Chidananda, O. Wallscheid, S. L. Brunton, and K. Taira. Distributed control of partial differential equations using convolutional rein- forcement learning.Physica D: Nonlinear Phenomena, 461:134096, 2024

    2024

  70. [78]

    Peters and S

    J. Peters and S. Schaal. Reinforcement learning of motor skills with policy gradients.Neural Networks, 21(4):682–697, 2008

    2008

  71. [79]

    A. S. Polydoros and L. Nalpantidis. Survey of model- based reinforcement learning: Applications on robotics. Journal of Intelligent & Robotic Systems, 86(2):153– 173, 2017

    2017

  72. [80]

    Pontryagin, V

    L. Pontryagin, V. G. Boltyanskii, R. V. Gamkrelidze, and E. F. Mishchenko.The mathematical theory of op- timal processes.Wiley, NY, 1962

    1962

  73. [81]

    Rabault, M

    J. Rabault, M. Kuchta, A. Jensen, and U. R´ eglade. Ar- tificial neural networks trained through deep reinforce- ment learning discover control strategies for active flow control.Journal of Fluid Mechanics, 865:281–302, 2019

    2019

  74. [82]

    Rabault, F

    J. Rabault, F. Ren, W. Zhang, H. Tang, and H. Xu. Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization.Journal of Hydrodynamics, 32:234–246, 2020

    2020

  75. [83]

    F. Ren, C. Wang, and H. Tang. Bluff body uses deep-reinforcement-learning trained active flow control to achieve hydrodynamic stealth.Physics of Fluids, 33(9):093602, 2021

    2021

  76. [84]

    Schulman, P

    J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018

    2018

  77. [85]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  78. [86]

    Semeraro

    O. Semeraro. Reinforcement Learning for Fluid Me- chanics: an overview on Fundamentals from a Control Perspective. InMachine Learning for Fluid Dynamics. 2025

    2025

  79. [87]

    K. Shao, Z. Tang, Y. Zhu, N. Li, and D. Zhao. A survey of deep reinforcement learning in video games.arXiv preprint arXiv:1912.10944, 2019

    1912 arXiv

  80. [88]

    Silver, A

    D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mas...

    2016

  81. [89]

    Silver, T

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017

    2017

  82. [90]

    Silver, G

    D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. Deterministic policy gradient algo- rithms. InInternational conference on machine learn- ing, pages 387–395. Pmlr, 2014

    2014

  83. [91]

    Silver, S

    D. Silver, S. Singh, D. Precup, and R. S. Sutton. Reward is enough.Artificial Intelligence, 299:103535, 2021

    2021

  84. [92]

    Su´ arez, F

    P. Su´ arez, F. Alc´ antara-´Avila, A. Mir´ o, J. Rabault, B. Font, O. Lehmkuhl, and R. Vinuesa. Active flow con- trol for drag reduction through multi-agent reinforce- ment learning on a turbulent cylinder at r e d= 3900. Flow, Turbulence and Combustion, pages 1–25, 2025

    2025

  85. [93]

    H. J. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do differentiable simulators give better policy gradi- ents? In K. Chaudhuri, S. Jegelka, L. Song, C. Szepes- vari, G. Niu, and S. Sabato, editors,Proceedings of the 39th International Conference on Machine Learn- ing, volume...

    2022

  86. [94]

    R. S. Sutton and A. G. Barto.Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998

    1998

  87. [95]

    R. S. Sutton, A. G. Barto, and R. J. Williams. Rein- forcement learning is direct adaptive optimal control. IEEE control systems magazine, 12(2):19–22, 1992

    1992

  88. [96]

    R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation.Advances in neural informa- tion processing systems, 12, 1999

    1999

  89. [97]

    Tampuu, T

    A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente. Multiagent cooperation and competition with deep reinforcement learning.PloS one, 12(4):e0172395, 2017

    2017

  90. [98]

    M. Tan. Multi-agent reinforcement learning: Indepen- dent vs. cooperative agents. InProceedings of the tenth international conference on machine learning, pages 330–337, 1993

    1993

  91. [99]

    C. Tang, B. Abbatematteo, J. Hu, R. Chandra, R. Mart ´ ın-Mart ´ ın, and P. Stone. Deep reinforcement learning for robotics: A survey of real-world successes. Annual Review of Control, Robotics, and Autonomous Systems, 8(Volume 8, 2025):153–188, 2025

    2025

  92. [100]

    Van Hasselt, A

    H. Van Hasselt, A. Guez, and D. Silver. Deep reinforce- ment learning with double q-learning. InProceedings of the AAAI conference on artificial intelligence, vol- ume 30, 2016

    2016

  93. [101]

    Varela, P

    P. Varela, P. Su´ arez, F. Alc´ antara- ´Avila, A. Mir´ o, J. Rabault, B. Font, L. M. Garc ´ ıa-Cuevas, O. Lehmkuhl, and R. Vinuesa. Deep reinforcement learning for flow control exploits different physics for increasing reynolds number regimes.Actuators, 11(12), 2022

    2022

  94. [102]

    Vasanth, J

    J. Vasanth, J. Rabault, F. Alc´ antara- ´Avila, M. Mortensen, and R. Vinuesa. Multi-agent rein- forcement learning for the control of three-dimensional rayleigh-b´ enard convection. arXiv:2407.21565, 2024

    2024 arXiv

  95. [103]

    Verma, G

    S. Verma, G. Novati, and P. Koumoutsakos. Efficient collective swimming by harnessing vortices through deep reinforcement learning.Proceedings of the National Academy of Sciences of the United States of America, 115(23):5849–5854, 2018

    2018

  96. [104]

    Vignon, J

    C. Vignon, J. Rabault, J. Vasanth, F. Alc´ antara-´Avila, M. Mortensen, and R. Vinuesa. Effective control of two-dimensional Rayleigh–B´ enard convection: Invari- ant multi-agent reinforcement learning is all you need. Physics of Fluids, 35(6), 2023

    2023

  97. [105]

    Vignon, J

    C. Vignon, J. Rabault, and R. Vinuesa. Recent advances in applying deep reinforcement learning for flow control: Perspectives and future directions.Physics of Fluids, 35(3), 2023

    2023

  98. [106]

    Vinuesa, O

    R. Vinuesa, O. Lehmkuhl, A. Lozano-Dur´ an, and J. Rabault. Flow control in wings and discovery of novel approaches via deep reinforcement learning.Flu- 18 ids, 7(2), 2022

    2022

  99. [107]

    Weinan, J

    E. Weinan, J. Han, and J. Long. Empowering op- timal control with machine learning: A perspective from model predictive control.IF AC-PapersOnLine, 55(30):121–126, 2022. 25th International Symposium on Mathematical Theory of Networks and Systems MTNS 2022

    2022

  100. [108]

    P. J. Werbos. Backpropagation through time: what it does and how to do it.Proceedings of the IEEE, 78(10):1550–1560, 2002

    2002

  101. [109]

    P. J. Werbos. Backwards differentiation in ad and neural nets: Past links and new opportunities. In M. B¨ ucker, G. Corliss, U. Naumann, P. Hovland, and B. Norris, ed- itors,Automatic Differentiation: Applications, Theory, and Implementations, pages 15–34, Berlin, Heidelberg,

  102. [110]

    Wiedemann, V

    N. Wiedemann, V. W¨ uest, A. Loquercio, M. M¨ uller, D. Floreano, and D. Scaramuzza. Training efficient con- trollers via analytic policy gradient, 2023

    2023

  103. [111]

    R. J. Williams. Simple statistical gradient-following al- gorithms for connectionist reinforcement learning.Ma- chine learning, 8(3-4):229–256, 1992

    1992

  104. [112]

    C. Wu, A. Rajeswaran, Y. Duan, V. Kumar, A. M. Bayen, S. Kakade, I. Mordatch, and P. Abbeel. Variance reduction for policy gradient with action-dependent fac- torized baselines, 2018

    2018

  105. [113]

    C. Xia, J. Zhang, E. C. Kerrigan, and G. Rigas. Active flow control for bluff body drag reduction using rein- forcement learning with partial measurements.Journal of Fluid Mechanics, 981:A17, 2024

    2024

  106. [114]

    J. Xu, V. Makoviychuk, Y. Narang, F. Ramos, W. Ma- tusik, A. Garg, and M. Macklin. Accelerated policy learning with parallel differentiable simulation, 2022

    2022

  107. [115]

    J. Xu, V. Makoviychuk, Y. Narang, F. Ramos, W. Ma- tusik, A. Garg, and M. Macklin. Accelerated policy learning with parallel differentiable simulation.arXiv preprint arXiv:2204.07137, 2022

    2022 arXiv

  108. [116]

    D. Ye, Z. Liu, M. Sun, B. Shi, P. Zhao, H. Wu, H. Yu, S. Yang, X. Wu, Q. Guo, et al. Mastering complex con- trol in moba games with deep reinforcement learning. InProceedings of the AAAI Conference on Artificial In- telligence, volume 34, pages 6672–6679, 2020

    2020

  109. [117]

    Zhang, J

    F. Zhang, J. Leitner, M. Milford, B. Upcroft, and P. Corke. Towards vision-based deep reinforcement learning for robotic motion control. InAustralasian Conference on Robotics and Automation 2015. Aus- tralian Robotics and Automation Association (ARAA), 2015

    2015

  110. [118]

    W. Zhao, J. P. Queralta, and T. Westerlund. Sim-to- real transfer in deep reinforcement learning for robotics: a survey. In2020 IEEE symposium series on computa- tional intelligence (SSCI), pages 737–744. IEEE, 2020

    2020

  111. [119]

    Zolman, C

    N. Zolman, C. Lagemann, U. Fasel, J. N. Kutz, and S. L. Brunton. Sindy-rl for interpretable and efficient model- based reinforcement learning.Nature Communications, 16(1), 2025. Appendix A: Sample complexity of inexact gradient algorithms In this section, we derive the sample ...

    2025

  112. [2006]

    Springer Berlin Heidelberg

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.