Pith. sign in

REVIEW 4 major objections 5 minor 66 references

HypEMBER claims that hypernetwork-generated policy and value networks, with the ensemble critics' disagreement as an uncertainty estimate, keep reinforcement-learned controllers for parametrized dynamical systems effective under measurement

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 12:10 UTC pith:T6AMR7M4

load-bearing objection A plausible integration of hypernetwork-generated policies and ensemble uncertainty weighting; the robustness gains are real enough to warrant peer review, but the missing deployment-time argmax and sign inconsistency stand between the manuscript and its claims. the 4 major comments →

arxiv 2607.19628 v1 pith:T6AMR7M4 submitted 2026-07-21 cs.LG

HypEMBER: Hypernetwork-based Ensemble for Robust Policy Learning of Parametrized Dynamical Systems

Nicol\`o Botteghi , Gabriele Pascali , Urban Fasel , Andrea Manzoni This is my paper
classification cs.LG
keywords reinforcement learninghypernetworksensemble learninguncertainty quantificationparametrized dynamical systemsrobust controlKuramoto-Sivashinsky equationdouble-gyre flow
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a single reinforcement-learning framework can generalize across the physical parameters of a dynamical system and stay reliable when the sensors and the assumed parameter values are noisy. It argues that the right combination is parameter-conditioned weight generation via hypernetworks, so the policy knows which dynamical regime it is in, together with an ensemble of critics whose disagreement is treated as a measurable epistemic uncertainty. That uncertainty is used both during training, to down-weight unreliable transitions and guide exploration, and at deployment, to penalize actions the critics disagree about. The claim is supported by training all agents in idealized, noise-free simulators and then testing under Gaussian measurement noise, parameter perturbation, and both; across two continuous-control problems, stabilizing a Kuramoto-Sivashinsky PDE and navigating a particle in a double-gyre flow, HypEMBER reports higher rewards and gentler degradation than the comparison methods at the highest noise levels. A sympathetic reader would care because this is a concrete route toward closing the sim-to-real gap for PDE and flow control, where models are expensive and always somewhat wrong.

Core claim

HypEMBER's central claim is that the brittleness of reinforcement-learned controllers for parametrized physical systems can be reduced by making every function in the actor-critic loop conditional on the system parameter µ and by maintaining an ensemble of such functions. Hypernetworks generate the weights of each policy and each critic from the augmented state z_t = [s_t, µ]; the spread of the critics' Q-predictions is interpreted as epistemic uncertainty. That uncertainty is used twice: to down-weight uncertain transitions in the critic's Huber loss, and to replace the mean-actor deployment rule with Qmean − λ Qstd, an action-selection rule that avoids actions about which the critics disag

What carries the argument

The mechanism that carries the argument is the ensemble of N hypernetwork-generated actor-critic pairs, all conditioned on the augmented context z_t = [s_t, µ]. For each ensemble member, a hypernetwork produces the weights of a stochastic policy and another hypernetwork produces the weights of a critic, so the physical parameter is an explicit input to the function-generating process. The engine of the method is the empirical standard deviation Qstd of the critics' Q-value predictions: during training it re-weights the Bellman error through a sigmoid temperature schedule and drives exploration through an upper-confidence-bound action rule; at deployment it enters the action-selection criteri

Load-bearing premise

The load-bearing premise is that the spread of the ensemble's Q-value estimates is a trustworthy measure of how much the critic's judgment can be trusted; if that spread does not track real error under noise, both the uncertainty-weighted training updates and the deployment-time penalty lose their justification.

What would settle it

Run the noisy evaluation of HypEMBER with the deployment penalty set to λ = 0 and with Qstd replaced by a random or constant value of the same magnitude; if reward degradation under measurement noise is unchanged, the ensemble spread is not actually supplying the robustness. A calibration check on the double-gyre task, comparing the ensemble's Q-spread with the realized variance of returns over repeated noise samples, would settle whether the uncertainty estimate tracks true error.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • On both benchmark problems, agents trained without noise maintain usable performance when evaluated with up to 40% measurement noise and parameter perturbations; the drop is gradual rather than abrupt, and the ensemble-hypernetwork agent keeps the highest rewards at the largest perturbations.
  • A single trained ensemble generalizes over the parameter range: in the Kuramoto-Sivashinsky task it is tested on unseen µ values sampled from the full interval, and in the gyre task on random target positions and flow parameters.
  • The uncertainty-aware deployment rule (Qmean − λQstd) measurably improves reward level and reduces reward variance under measurement and combined uncertainties on both tasks; on pure parameter misspecification the paper reports no improvement on the KS task.
  • Training trajectories show lower variance across seeds for HypEMBER than for the plain ensemble method, supporting the claim of improved training stability and sample efficiency.
  • Hypernetwork conditioning and ensemble uncertainty appear complementary: moderate noise favors parameter conditioning, while high noise favors the ensemble's uncertainty-aware mechanisms.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not isolate which component carries the robustness; a natural test would ablate the deployment penalty (λ = 0) and the uncertainty-weighted Bellman update separately to see whether the deployment-time Qstd penalty alone explains the gains under measurement noise.
  • Because µ is part of the conditioning input, the robustness studied here is to noise around a known parameter value, not to a wholly unknown or wrongly identified parameter; feeding a posterior over µ into the same hypernetworks would be a direct extension.
  • The observation that uncertainty-aware action selection helps under measurement noise but not under pure parameter misspecification suggests the ensemble spread tracks sensor-induced value error better than parameter-shift error; a calibration study of Qstd against realized return variance by noise source would test that.
  • The same Qstd signal could be reused for online tasks the paper leaves untouched, such as deciding when to re-estimate parameters, where to place sensors, or when to flag low-confidence actions to a human supervisor.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HypEMBER, an actor-critic RL algorithm for parametrized dynamical systems. It maintains an ensemble of policies and critics whose weights are generated by hypernetworks conditioned on the augmented state z_t = [s_t, µ], and it combines SUNRISE-style uncertainty-weighted Bellman updates with an upper-confidence-bound training exploration rule and an uncertainty-aware deployment action-selection rule. The method is evaluated on two benchmarks: stabilization of a parametrized Kuramoto–Sivashinsky equation and particle navigation in a double-gyre flow, under measurement noise and parameter misspecification not seen during training. The authors report that HypEMBER improves training stability, sample efficiency, and robustness relative to TD3, PolyL0-TD3, SUNRISE, and HypeRL.

Significance. If the empirical claims are correct, the combination of hypernetwork-based parameter conditioning and ensemble uncertainty quantification is a useful contribution for sim-to-real control of PDE/flow systems. The paper has notable strengths: the evaluation protocol (train in clean settings, test under noise and misspecification) is well motivated; the comparison includes four baselines; and the authors point to released code. However, the central robustness mechanism is not yet established. The critic target is internally inconsistent between Section 3 and Algorithm 1; the deployment-time argmax over a continuous action space is underspecified; and the uncertainty penalty weight λ is tuned on the same evaluation benchmarks while Qstd calibration is only qualitatively illustrated. These issues block a clean assessment of the claimed robustness gains.

major comments (4)
  1. [Section 3, Eq. (15); Algorithm 1, Appendix A] The critic target in Section 3 is written as y_t = r_t − γ Q̄_{θ̄_i}(z_{t+1}, a_{t+1}), while Algorithm A in Appendix A defines y_t = r_t + γ(Q̄(·) − α_i log π(·)). These are different objects: the former is a TD-error-like quantity, the latter is the standard entropy-regularized SAC target. Since every critic update and hence the learned policies depend on y_t, the manuscript as written is internally inconsistent. The authors must correct the sign in Equation (15) (or in Algorithm 1) and confirm which form was actually implemented, ideally by pointing to the specific line in the released code.
  2. [Section 3, deployment action-selection rule] The deployment rule a_t^* = argmax_a (Q_mean(z_t,a) − λ Q_std(z_t,a)) is defined over a continuous action space, but no candidate set or discretization is specified. Algorithm 1 only implements the training-time UCB with +λ; the deployment-time procedure is absent from both the pseudocode and Appendix A. SUNRISE avoids this problem by selecting among ensemble actor proposals, but the text explicitly departs from SUNRISE's mean-action rule. The authors need to specify how the argmax is computed at deployment (e.g., sampled actions from ensemble actors, policy means, or a finite candidate set); otherwise the reported robustness improvements cannot be attributed to this rule.
  3. [Appendix D; Figures 9 and 14] The uncertainty-aware mechanism assumes that Q_std is a calibrated estimate of action-value uncertainty, but the evidence is only that Q_std rises when noise is injected (Figures 9 and 14). No correlation is shown between Q_std and Bellman error, return degradation, or action-selection mistakes. Moreover, Appendix D selects λ = 0.25 using the same evaluation benchmarks (Figures 15–20), which is a form of tuning on the test set. To support the claim of a principled robustness mechanism, the authors should validate Q_std against a ground-truth error measure and either fix λ a priori or evaluate on held-out noise levels, parameters, and initial conditions not used in the λ selection.
  4. [Section 4.1, Table 1 and training comparisons] The paper states that HypEMBER 'consistently improves training stability and sample efficiency,' but Table 1 shows HypeRL achieving the best training reward on the KS benchmark and HypEMBER having a lower mean reward with a comparable standard deviation. The evidence for improved training stability is qualitative ('smoother learning trajectories'), and the learning curves are not shown with the same quantitative detail as the evaluation curves. Please include the full training curves per seed and a quantitative stability metric (e.g., inter-quartile range of episode returns across the last X episodes) to substantiate the stability claim.
minor comments (5)
  1. [Section 2.1] The return definition G_t = Σ_{k=0}^{N_t} γ^{t+k} r_{t+k} appears to have the wrong index; it should be γ^k r_{t+k} (or equivalently Σ γ^t r_t with appropriate indexing).
  2. [Section 4.1, after Eq. (2)] The text says 'we set α = 0.1' when referring to the action-cost coefficient in the reward; the reward in Eq. (2) uses β. Since α is also used for the SAC entropy coefficient, this ambiguous notation should be fixed.
  3. [Section 4] The paragraph beginning 'In our numerical experiments, we consider three different scenarios...' appears twice, once before the KS results and once after Table 1. Remove the duplicate.
  4. [Figures 10–12 and 15–20] Figure numbering is garbled: Section 4.1 references Figures 10, 11, and 12, but those same numbers are reused in Section 4.2, and several earlier figures (3–5) appear to show the same content as later ones. Please renumber all figures and fix the in-text references.
  5. [References] Reference [35] has an incomplete arXiv identifier '2401.XXXX'. Update the reference if a version is available.

Circularity Check

0 steps flagged

No significant circularity: the claimed robustness comes from an empirical combination of two published mechanisms, with the building blocks re-tested as baselines in the same paper.

full rationale

The paper does not claim to derive robustness from first principles; HypEMBER is an algorithmic combination of hypernetwork-generated policies (from the authors' HypeRL) and SUNRISE-style ensemble uncertainty weighting, evaluated on KS and double-gyre benchmarks. The central comparison is self-contained: HypeRL and SUNRISE are both re-trained and compared as baselines under the same protocol, so the value of those building blocks is tested here rather than assumed from the cited prior work. The self-citations ([25], [26], [39]) are architectural or baseline references and are not used as a uniqueness theorem or to forbid alternatives. The deployment-time rule a* = argmax_a(Qmean − λ Qstd) and the weighted Bellman loss (Eq. 15) do not reduce by construction to fitted constants; the robustness claim rests on empirical reward measurements. The genuine concerns are non-circularity issues: λ = 0.25 is selected via ablation on the same evaluation benchmarks (Appendix D), Qstd is shown to increase under noise but is not calibrated to Bellman error, and the critic target sign is inconsistent between Section 3 (r_t − γ Qbar) and Algorithm 1 (r_t + γ(...)). These are correctness and evaluation-leakage concerns, not cases where the output is equivalent to the input by definition. Accordingly, no circular step is established; the score of 2 reflects only the presence of minor, non-load-bearing self-citations.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The method is empirical and imports its main mechanisms from SUNRISE and HypeRL. The free parameters are standard RL hyperparameters plus lambda selected by ablation. No new physical entities are introduced; the 'ensemble of hypernetworks' is an architectural choice, not a new entity with independent falsifiable handles.

free parameters (6)
  • Deployment and UCB weight lambda = 0.25
    Chosen after ablation in Appendix D and used for both training UCB and uncertainty-aware deployment; main results use this selected value.
  • Weighted Bellman temperature T = 20.0
    Taken from SUNRISE; governs the sigmoid weighting of ensemble critic standard deviation; not fitted here.
  • Action-cost weight beta (text sometimes says alpha) = 0.1
    Chosen to prioritize state cost; directly shapes the reward function and all reported rewards.
  • Initial entropy coefficient alpha = 0.1
    SAC entropy regularization weight; set in the hyperparameter table with no sensitivity analysis.
  • Ensemble size N = 5
    Number of actors and critics per ensemble; chosen without reported sensitivity analysis.
  • HypEMBER actor and critic learning rates = 1e-6 and 5e-6
    Much smaller than the baselines; likely important for hypernetwork training stability but no sensitivity study is reported.
axioms (6)
  • domain assumption The environment is an MDP with Markov state transitions; observations consist of state plus parameter vector
    Section 2.1; all actor-critic updates and Bellman equations rely on this.
  • domain assumption The true system dynamics and reward are known to the simulator; only the agent's observations and parameters are perturbed at evaluation
    Section 4; defines the robustness test; results may not transfer to unmodeled dynamics.
  • domain assumption Additive Gaussian noise on states and parameters captures the relevant uncertainty or sim-to-real gap
    Equations (16)-(17); all robustness conclusions are conditioned on this perturbation model.
  • domain assumption Ensemble standard deviation of Q predictions is a valid proxy for epistemic uncertainty about action values
    Sections 2.3 and 3; underlies the weighted Bellman updates and deployment penalty without validation on the PDE benchmarks.
  • domain assumption The numerical discretizations approximate the target PDEs well enough for RL training
    Section 4.1 uses Fourier pseudo-spectral KS; Section 4.2 uses forward Euler; the discretization is assumed adequate.
  • domain assumption Hypernetwork parameter generation is smooth enough in mu to interpolate between training parameter values
    Section 2.4; if the hypernetwork cannot represent sharp regime changes, parametric generalization fails.

pith-pipeline@v1.3.0-alltime-deepseek · 20209 in / 14148 out tokens · 150094 ms · 2026-08-01T12:10:36.661717+00:00 · methodology

0 comments
read the original abstract

In this work we investigate reinforcement learning (RL) as a framework for the robust control of parametrized dynamical systems in presence of measurements and model uncertainties. High-dimensional state spaces, expensive numerical solvers, the partial knowledge of the governing equations, and the dependence on physical parameters that may be uncertain or difficult to estimate accurately, make the use of standard RL approaches computationally unfeasible. Indeed, lack of robustness and poor generalization across parameter variations are further amplified in presence of noisy or incomplete measurements, ultimately hampering control performance. To address these challenges, we introduce HypEMBER, a novel RL framework based on the combination of hypernetworks and ensemble learning. In the proposed approach, both the policy and value functions are represented through hypernetworks that generate the weights of the underlying models conditioned on the physical parameters of the system, thereby enabling parametric generalization across different dynamical regimes. In addition, an ensemble of policy and value approximators is employed to quantify epistemic uncertainty, leading to improved exploration strategies and enhanced robustness during and after training. The performance of the proposed framework is assessed on two representative parametrized control problems: (i) the one-dimensional Kuramoto-Sivashinsky equation and (ii) a particle-navigation task in a two-dimensional time-dependent gyre flow, focusing on robustness with respect to measurement noise and parameter misspecification. Numerical results demonstrate that HypEMBER consistently improves training stability and sample efficiency, while achieving superior robustness to uncertainties affecting both the system dynamics and the available observations, in comparison with state-of-the-art RL methods.

Figures

Figures reproduced from arXiv: 2607.19628 by Andrea Manzoni, Gabriele Pascali, Nicol\`o Botteghi, Urban Fasel.

Figure 1
Figure 1. Figure 1: HypEMBER takes advantage of an ensemble of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Uncertainty-aware action selection strategy. The ensemble of actors and critics is exploited to select optimal [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance of the different agents when evalu [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Performance of the different agents when eval [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Controlled solutions for the HypEMBER (uncertainty-aware action selection with [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Controlled solutions for the sunrise case. Left column: ideal settings. Right column: noisy setting. From top [PITH_FULL_IMAGE:figures/full_fig_p014_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Controlled solutions for the hypeRL case. Left column: clean setting. Right column: noisy setting. From top [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Mean and standard deviation of the Q-values predicted by the ensemble for different levels of measurement [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Performance of the different agents when eval [PITH_FULL_IMAGE:figures/full_fig_p016_10.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance of the different agents when eval [PITH_FULL_IMAGE:figures/full_fig_p016_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Controlled trajectories obtained by HypEMBER, SUNRISE, and HypeRL in the double gyre flow starting [PITH_FULL_IMAGE:figures/full_fig_p017_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Mean and standard deviation of the Q-values predicted by the ensemble for different levels of measurement [PITH_FULL_IMAGE:figures/full_fig_p018_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Performance of the different agents when eval [PITH_FULL_IMAGE:figures/full_fig_p024_15.png] view at source ↗
Figure 17
Figure 17. Figure 17: Performance of the different agents when eval [PITH_FULL_IMAGE:figures/full_fig_p024_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Performance of the different agents when eval [PITH_FULL_IMAGE:figures/full_fig_p025_18.png] view at source ↗
Figure 20
Figure 20. Figure 20: Performance of the different agents when eval [PITH_FULL_IMAGE:figures/full_fig_p025_20.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

66 extracted references · 17 linked inside Pith

  1. [1]

    Optimal Control of Partial Differential Equations

    Andrea Manzoni, Alfio Quarteroni, and Sandro Salsa. Optimal Control of Partial Differential Equations. Springer, 2021

    2021

  2. [2]

    Donald E. Kirk. Optimal Control Theory: An Introduction. Courier Corporation, 2004

    2004

  3. [3]

    Fleming and H

    Wendell H. Fleming and H. Mete Soner. Controlled Markov Processes and Viscosity Solutions. Springer, 2006

    2006

  4. [4]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, second edition, 2018

    2018

  5. [5]

    Bertsekas and John N

    Dimitri P. Bertsekas and John N. Tsitsiklis. Neuro-Dynamic Programming. Athena Scientific, 1996

    1996

  6. [6]

    Warren B. Powell. Approximate Dynamic Programming. Wiley, 2011

    2011

  7. [7]

    Deep reinforcement learning: A brief survey

    Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26–38, 2017

    2017

  8. [8]

    Bellemare, and Joelle Pineau

    Vincent François-Lavet, Peter Henderson, Riashat Islam, Marc G. Bellemare, and Joelle Pineau. An introduction to deep reinforcement learning. Foundations and Trends in Machine Learning, 11(3–4):219–354, 2018

    2018

  9. [9]

    Human-level control through deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015

    2015

  10. [10]

    Mastering complex control in moba games with deep reinforcement learning

    Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6672–6679, 2020

    2020

  11. [11]

    Playing atari with deep reinforcement learning

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    Pith/arXiv arXiv 2013

  12. [12]

    Deep reinforcement learning with double q-learning

    Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  13. [13]

    Reinforcement learning for robots using neural networks

    Long-Ji Lin. Reinforcement learning for robots using neural networks. Carnegie Mellon University, 1992

    1992

  14. [14]

    Andrew Bagnell, and Jan Peters

    Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):1238–1274, 2013

    2013

  15. [15]

    Towards vision-based deep reinforcement learning for robotic motion control

    Fangyi Zhang, Jürgen Leitner, Michael Milford, Ben Upcroft, and Peter Corke. Towards vision-based deep reinforcement learning for robotic motion control. In Australasian Conference on Robotics and Automation 2015. Australian Robotics and Automation Association (ARAA), 2015

    2015

  16. [16]

    Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates

    Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3389–3396. IEEE, 2017

    2017

  17. [17]

    Sim-to-real transfer in deep reinforcement learning for robotics: a survey

    Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI) , pages 737–744. IEEE, 2020. 19 HypEMBER

    2020

  18. [18]

    Low dimensional state representation learning with robotics priors in continuous action spaces

    Nicolò Botteghi, Khaled Alaa, Mannes Poel, Beril Sirmacek, Christoph Brune, Abeje Mersha, and Stefano Stramigioli. Low dimensional state representation learning with robotics priors in continuous action spaces. International Conference on Intelligent Robots and Systems, 2021

    2021

  19. [19]

    Control of chaotic systems by deep reinforcement learning

    Michele Alessandro Bucci, Onofrio Semeraro, Alexandre Allauzen, Guillaume Wisniewski, Laurent Cordier, and Lionel Mathelin. Control of chaotic systems by deep reinforcement learning. Proceedings of the Royal Society A, 475(2231):20190351, 2019

    2019

  20. [20]

    Reinforcement learning for bluff body active flow control in experiments and simulations

    Dixia Fan et al. Reinforcement learning for bluff body active flow control in experiments and simulations. Proceedings of the National Academy of Sciences, 117(42):26091–26098, 2020

    2020

  21. [21]

    Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach

    Jean Rabault and Alexander Kuhnle. Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Physics of Fluids, 31(9), 2019

    2019

  22. [22]

    Active flow control for bluff body drag reduction using reinforcement learning with partial measurements

    Chengwei Xia, Junjie Zhang, Eric C Kerrigan, and Georgios Rigas. Active flow control for bluff body drag reduction using reinforcement learning with partial measurements. Journal of Fluid Mechanics, 981:A17, 2024

    2024

  23. [23]

    Brunton, and Kunihiko Taira

    Sebastian Peitz, Jan Stenner, Vikas Chidananda, Oliver Wallscheid, Steven L. Brunton, and Kunihiko Taira. Distributed control of partial differential equations using convolutional reinforcement learning. arXiv preprint arXiv:2301.10737, 2023

    Pith/arXiv arXiv 2023

  24. [24]

    Sindy-rl: Interpretable and efficient model-based reinforcement learning

    Nicholas Zolman, Urban Fasel, J Nathan Kutz, and Steven L Brunton. Sindy-rl: Interpretable and efficient model-based reinforcement learning. arXiv preprint arXiv:2403.09110, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Parametric PDE Control with Deep Reinforcement Learning and L <sub>0</sub> Sparse Polynomial Policies

    Nicolò Botteghi and Urban Fasel. Parametric PDE Control with Deep Reinforcement Learning and L <sub>0</sub> Sparse Polynomial Policies. In 2024 IEEE 63rd Conference on Decision and Control (CDC), pages 6108–6115. IEEE, 12 2024

    2024

  26. [26]

    HypeMARL: Multi- Agent Reinforcement Learning For High-Dimensional, Parametric, and Distributed Systems

    Nicolò Botteghi, Matteo Tomasetto, Urban Fasel, Francesco Braghin, and Andrea Manzoni. HypeMARL: Multi- Agent Reinforcement Learning For High-Dimensional, Parametric, and Distributed Systems. arXiv preprint arXiv:2509.16709, 9 2025

    arXiv 2025

  27. [27]

    Implementation matters in deep rl

    Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Implementation matters in deep rl. ICLR, 2020

    2020

  28. [28]

    Epopt: Learning robust neural network policies using model ensembles

    Aravind Rajeswaran, Sarvjeet Ghotra, Sergey Levine, and Balaraman Ravindran. Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283, 2017

    Pith/arXiv arXiv 2017

  29. [29]

    Challenges of real-world reinforcement learning

    Gabriel Dulac-Arnold, Nir Levine, Daniel Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv:1904.12901, 2019

    Pith/arXiv arXiv 1904

  30. [30]

    Deep reinforcement learning at the edge of the statistical precipice

    Rishabh Agarwal et al. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Information Processing Systems, 34:29304–29320, 2021

    2021

  31. [31]

    Assessing generaliza- tion in deep reinforcement learning

    Charles Packer, Katelyn Gao, Jernej Kos, Jonathan Krause, Ofir Nachum, and Dawn Song. Assessing generaliza- tion in deep reinforcement learning. arXiv preprint arXiv:1810.12282, 2018

    Pith/arXiv arXiv 2018

  32. [32]

    Crossing the reality gap: A survey on sim-to-real transferability of robot controllers in reinforcement learning

    Erica Salvato, Gianfranco Fenu, Eric Medvet, and Felice Andrea Pellegrino. Crossing the reality gap: A survey on sim-to-real transferability of robot controllers in reinforcement learning. IEEE Access, 9:153171–153187, 2021

    2021

  33. [33]

    Mankowitz, and Todd Hester

    Gabriel Dulac-Arnold, Daniel J. Mankowitz, and Todd Hester. An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint arXiv:2003.11881, 2020

    Pith/arXiv arXiv 2003

  34. [34]

    Aleatoric and epistemic uncertainty in machine learning: An introduc- tion to concepts and methods

    Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduc- tion to concepts and methods. Machine Learning, 110:457–506, 2021

    2021

  35. [35]

    Uncertainty-aware reinforcement learning agents for noisy environments

    Akash Singh, Thomas Beckers, Ann Nowé, and Damien Ernst. Uncertainty-aware reinforcement learning agents for noisy environments. arXiv preprint arXiv:2401.XXXX, 2025

    2025

  36. [36]

    Masksembles for uncertainty estimation.arXiv preprint arXiv:2012.08334, 2020

    Nikita Durasov, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Masksembles for uncertainty estimation.arXiv preprint arXiv:2012.08334, 2020

    Pith/arXiv arXiv 2012

  37. [37]

    Ted Moskovitz, Jack Parker-Holder, Rui Wang, Sergey Levine, and Michael I. Jordan. Inverse variance reinforce- ment learning. In International Conference on Learning Representations (ICLR), 2022

    2022

  38. [38]

    Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning

    Sungjin Lee, Youngwoon Lee, Byung-Jun Kim, and Byoung-Tak Lee. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. NeurIPS, 2021

    2021

  39. [39]

    Hyperl: Hypernetwork-based reinforcement learning for control of parametrized dynamical systems

    Nicolò Botteghi, Stefania Fresca, Mengwu Guo, and Andrea Manzoni. Hyperl: Hypernetwork-based reinforcement learning for control of parametrized dynamical systems. arXiv preprint arXiv:2501.04538, 2026

    arXiv 2026

  40. [40]

    David Ha, Andrew Dai, and Quoc V . Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016. 20 HypEMBER

    Pith/arXiv arXiv 2016

  41. [41]

    A brief review of hypernetworks in deep learning

    Vinod Kumar Chauhan, Jiandong Zhou, Ping Lu, Soheila Molaei, and David A Clifton. A brief review of hypernetworks in deep learning. arXiv preprint arXiv:2306.06955, 2023

    Pith/arXiv arXiv 2023

  42. [42]

    Johannes Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In International Conference on Learning Representations (ICLR), 2020

    2020

  43. [43]

    Hypernetworks in meta- reinforcement learning

    Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta- reinforcement learning. In Conference on Robot Learning, pages 1478–1487. PMLR, 2023

    2023

  44. [44]

    Bayesian hypernetworks

    David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv:1710.04759, 2017

    Pith/arXiv arXiv 2017

  45. [45]

    Puterman

    Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 1994

    1994

  46. [46]

    Bertsekas

    Dimitri P. Bertsekas. Dynamic Programming and Optimal Control. Athena Scientific, 2012

    2012

  47. [47]

    Mastering atari, go, chess and shogi by planning with a learned model

    Julian Schrittwieser et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604–609, 2020

    2020

  48. [48]

    Model-based reinforcement learning for atari

    Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:1903.00374, 2019

    Pith/arXiv arXiv 1903

  49. [49]

    Moerland, Joost Broekens, Aske Plaat, and Catholijn M

    Thomas M. Moerland, Joost Broekens, Aske Plaat, and Catholijn M. Jonker. Model-based reinforcement learning: A survey. Foundations and Trends in Machine Learning, 16(1):1–118, 2023

    2023

  50. [50]

    Christopher J. C. H. Watkins and Peter Dayan. Q-learning. Machine Learning, 8(3–4):279–292, 1992

    1992

  51. [51]

    Konda and John N

    Vijay R. Konda and John N. Tsitsiklis. Actor-critic algorithms. Advances in Neural Information Processing Systems, 12, 2000

    2000

  52. [52]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning (ICML), pages 1587–1596, 2018

    2018

  53. [53]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems

    Sergey Levine et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020

    Pith/arXiv arXiv 2005

  54. [54]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01290, 2018

    Pith/arXiv arXiv 2018

  55. [55]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International Conference on Machine Learning, pages 1587–1596. PMLR, 2018

    2018

  56. [56]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement learning: an introduction. Adaptive computation and machine learning series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018

    2018

  57. [57]

    Continuous control with deep reinforcement learning

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    Pith/arXiv arXiv 2015

  58. [58]

    Deterministic policy gradient algorithms

    David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387–395. PMLR, 2014

    2014

  59. [59]

    Generalization in reinforcement learning with hypernetworks

    Rui Wang et al. Generalization in reinforcement learning with hypernetworks. arXiv preprint arXiv:2006.13307, 2020

    Pith/arXiv arXiv 2006

  60. [60]

    Randomized prior functions for deep reinforcement learning

    Ian Osband et al. Randomized prior functions for deep reinforcement learning. arXiv preprint arXiv:1806.03335, 2018

    Pith/arXiv arXiv 2018

  61. [61]

    For sale: State-action representation learning for deep reinforcement learning

    Scott Fujimoto, Wei-Di Chang, Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in neural information processing systems, 36:61573–61624, 2023

    2023

  62. [62]

    Kudryashov

    Nikolai A. Kudryashov. Exact solutions of the generalized kuramoto–sivashinsky equation. Physics Letters A, 147(5–6):287–291, 1990

    1990

  63. [63]

    Evidence on the Regularisation Properties of Maximum-Entropy Reinforcement Learning

    Rémy Hosseinkhan Boucher, Onofrio Semeraro, and Lionel Mathelin. Evidence on the Regularisation Properties of Maximum-Entropy Reinforcement Learning. 1 2025

    2025

  64. [64]

    Learning efficient navigation in vortical flow fields

    Peter Gunnarson, Ioannis Mandralis, Guido Novati, Petros Koumoutsakos, and John O Dabiri. Learning efficient navigation in vortical flow fields. Nature communications, 12(1):7143, 2021

    2021

  65. [65]

    Finite time lyapunov exponent analysis of model predictive control and reinforcement learning

    Kartik Krishna, Steven L Brunton, and Zhuoyuan Song. Finite time lyapunov exponent analysis of model predictive control and reinforcement learning. IEEE Access, 11:118916–118930, 2023

    2023

  66. [66]

    A novel mechanism for mechanosensory-based rheotaxis in larval zebrafish

    Pablo Oteiza, Iris Odstrcil, George Lauder, Ruben Portugues, and Florian Engert. A novel mechanism for mechanosensory-based rheotaxis in larval zebrafish. Nature, 547(7664):445–448, 2017. 21 HypEMBER A Pseudo-code HypEMBER Algorithm 1 HypEMBER Initialize actor hypernetworks {gi(·; ϕgi )}N i=1 and actors {π(·; ϕi)}N i=1 ▷ Algorithm initialization Initializ...

    2017