Pith. sign in

REVIEW 4 major objections 5 minor 57 references

Umbrella Reinforcement Learning -- computationally efficient tool for hard non-linear problems

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Umbrella RL replaces trajectory-based exploration with a continuous agent ensemble and solves hard sparse-reward RL problems.

desk verdict A genuinely novel ensemble-entropy RL idea with an elegant derivation, but the practical updates drop terms the theory needs and the experiments are too thin to support the 'outperforms all SOTA' claim. read the letter →

arxiv 2411.14117 v1 pith:7OA2WI2W submitted 2024-11-21 cs.LG cs.AI

Egor E. Nuzhin , Nikolai V. Brilliantov This is my paper · ORCID
classification cs.LGcs.AI
keywords umbrellareinforcementlearningsamplingpolicygradientsparserewardstatetrapsensembleentropyexploration-exploitationbalancecontinuous-timeMarkovdecisionprocess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Umbrella RL, a reinforcement-learning method that replaces the usual single agent or small batch of agents with a continuous ensemble of agents spread over state space, and adds the ensemble's joint state-action entropy to the reward. The authors argue that this combination, borrowed from umbrella sampling in computational chemistry, lets a policy-gradient algorithm solve 'hard' RL problems that defeat standard trajectory-based methods: rewards given only at rare goal states, traps that stall agents, and environments with no clear terminal state. On two benchmark tasks, a multi-valley Mountain Car and a two-link StandUp arm, they report that their method reaches the optimal policy while PPO, RND, iLQR, and a version without the entropy term fail, and that it matches or beats value iteration using roughly 1 GB of memory instead of 16 GB and without sensitivity to the time-discretization step. The claim, if true, means a single parameter-light algorithm could replace problem-specific exploration tricks for a whole class of RL failures.

What carries the argument

The central object is the time-averaged agent density $p(s)=\int_0^\infty \gamma^t p(s,t)\,dt$ together with the modified advantage function $A_u(s,a)=r-\tilde{\alpha}\log(p\cdot\pi)+v\cdot\nabla_s V_u-|\log\gamma|V_u$. The density replaces many discrete trajectories and makes the policy gradient depend on global state coverage, while the log-density term is the entropy force that widens exploration. The algorithm then solves two steady-state equations, a Bellman-type equation $\mathbb{E}_{a\sim\pi}A_u(s,a)=0$ and a continuity-type equation $\nabla_s\cdot(p\langle v\rangle_\pi)-\log\gamma(p-p_0)=0$, with three neural networks for policy, value, and density, using the gradient updates of the paper. This is what carries the argument: the ensemble density and value function are computed as steady states of PDEs rather than accumulated from episodes.

What would settle it

Compute finite-difference solutions of the two steady-state equations, $\mathbb{E}_{a\sim\pi}A_u(s,a)=0$ and $G(p,s)=0$, on a coarse grid covering the Multi-Valley Mountain Car domain and compare them with the neural-network outputs at the end of training; if the residuals are significantly nonzero, the learned value function and density are not the steady-state solutions the policy update assumes.

Watch

Extended reading notes

Core claim

Umbrella RL's central claim is that a continuous, effectively infinite ensemble of agents, described by a probability density $p(s,t)$, can be driven toward optimal behavior by maximizing a modified expected return $J_{\mathrm{URL}}(\pi,p)=J(\pi)+\alpha H[p\cdot\pi]$, where $H$ is the discounted joint entropy of the state distribution and policy. The entropy term acts as an automatic exploration-exploitation switch: while no reward has been found it encourages the density to spread over the state space, and once reward states are reached the ordinary return dominates and the ensemble condenses around them. The resulting optimality conditions are a differential advantage equation $\mathbb{E}_{a\sim\pi}A_u(s,a)=0$ and a steady-state density equation $G(p,s)=0$, solved by three neural networks with policy-gradient-style updates. In the two reported environments the method converges to a sensible, near-deterministic policy where trajectory-based baselines produce random or zero-reward behavior.

Load-bearing premise

The neural-network updates are derived from simplified versions of the steady-state equations, and the method assumes that the dropped correction terms, including second-order gradient terms and the divergence term in the density loss, are negligible; if they are not, the learned value function and density are not the ones the theory requires.

Editorial extensions

If this is right

  • Sparse-reward tasks with no terminal state become solvable without hand-designed intrinsic rewards, reward shaping, or expert demonstrations.
  • Computational cost stays tied to the state distribution rather than to the time discretization, so finer simulation steps do not degrade performance as they do for value iteration.
  • The entropy coefficient $\alpha$ sets the exploration-exploitation balance globally, and removing the entropy term makes the algorithm fail on the tested hard problems.
  • Because the formulation is continuous-time and general, the same scheme transfers to discrete-time RL and, the authors state, to a model-free version.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the authors do not develop is annealing $\alpha$ over training: starting large to force coverage and reducing it later should sharpen the final policy, but the theory presented here gives no guidance on how $\alpha$ should be scheduled.
  • The method's reliance on estimating $\log p(s)$ suggests the entropy bonus will become harder to estimate in high-dimensional state spaces, so the curse of dimensionality may reappear for the density network even though the policy computation itself is mesh-free.
  • Since $p(s)$ is time-averaged with a discount factor $\gamma$, the method effectively prioritizes states visited earlier in time; tuning $\gamma$ may act as a horizon control for exploration, a connection the paper leaves implicit.
  • The comparison with value iteration suggests Umbrella RL could serve as a differentiable replacement for value iteration in continuous state spaces, where grid-based methods are memory-bound; testing on a higher-dimensional control task with a known optimal cost would directly probe that promise.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Umbrella RL, a method that replaces trajectory sampling with a continuous ensemble of agents. The state distribution p(s) and value function V(s) are defined as solutions of coupled steady-state PDEs, Eqs. (14)-(15), derived from a discounted objective J_URL that adds an ensemble-entropy term to the expected return. Three neural networks parameterize the policy, the value function, and the density, and the paper proposes coupled gradient updates, Eqs. (19)-(21). The method is tested on two hard-exploration tasks, Multi-Valley Mountain Car and StandUp, and compared with PPO, RND, iLQR, and value iteration. The authors conclude that Umbrella RL outperforms all available state-of-the-art algorithms on hard RL problems with sparse rewards, state traps, and lack of terminal states.

Significance. The conceptual idea is genuinely interdisciplinary and potentially interesting: instead of simulating trajectories, solve a discounted MDP through a steady-state PDE system for the state density and value function, using NNs as function approximators. If the derivation were rigorous and the empirical evidence robust, this could offer a memory-efficient alternative to value iteration and a principled exploration mechanism for sparse-reward problems. The paper also makes its code available on GitHub. However, the central claim is much stronger than the evidence, and the derivation of the practical NN update rules contains a load-bearing gap that, as it stands, invalidates the claimed convergence to the optimum of J_URL. The contribution is therefore currently unsupported.

major comments (4)
  1. [Appendix A.5, Eqs. (A.39)-(A.45); main text Eqs. (19)-(20)] The update rules used in all experiments are not gradient steps for the residual losses (A.37) and (A.42). For the value function, Eq. (A.39) drops the term involving ∇_phi[v·∇_s V], i.e. the second-order derivative of V; for the density, Eq. (A.43) drops the entire divergence term ∇_s·(p v) from the residual. The manuscript states these are approximations ('we assume that the second-order gradients may be neglected'; 'we omit the divergence term') but provides no scaling argument or numerical check. If these dropped terms are not negligible, the learned V and p do not solve Eqs. (14)-(15), so the advantage A_u in Eq. (11)-which is used in the policy update (21)-is not the function defined by the theory. The claim that the algorithm converges to the optimum of J_URL is therefore unsupported.
  2. [Appendix A.8] The 'general approach' does not repair the gap. It defines a natural-gradient-like update P(θ) = (∇_θ y · diag(p) · (∇_θ y)^T)^{-1} ∇_θ y · diag(p) · f and then assumes, without proof, that P(θ) is a gradient of some function Π(θ) (Eq. A.74). Even accepting that assumption, the final update (A.80) uses ∇_θ y · diag(p) · f, whereas the gradient of a squared-residual objective E[||f||^2] would require (∇_θ f)·diag(p)·f. For the value equation, ∇_θ f contains the same second-order terms that were dropped; for the density equation, f itself contains the divergence, so ∇_θ f ≠ ∇_θ y. No identity equates these quantities. Thus the general approach still does not provide a convergence guarantee for the practical algorithm.
  3. [Sec. 4.3 and Figs. 2-4] The empirical support for the abstract's global claim is limited. Only two low-dimensional, deterministic toy tasks are considered. The proposed method is tuned per task, with learning rates and weight decays listed in Sec. 4.3, whereas PPO is used with 'default parameters' from Stable Baselines3. No error bars or confidence intervals appear in Figs. 2-4, despite the text stating that results were averaged and variance estimated. The memory comparison (UR approximately 1 GB vs VI approximately 16 GB) is presented as a figure-caption statement without a measurement protocol. These issues preclude the conclusion that Umbrella RL 'outperforms ... all available state-of-the-art algorithms'.
  4. [Sec. 4.3] The baseline descriptions are incomplete. RND is said to be applied 'for the problems where the observation was a sequence of images', yet the two environments provide low-dimensional state vectors; it is unclear how RND was adapted or whether it received the same information. iLQR is applied with smoothing of discontinuous functions, but the smoothing procedure and the exact treatment of the discrete action set are not specified. Without this detail, the failure of these baselines is not a controlled comparison.
minor comments (5)
  1. [Sec. 4.1] The domain of position is given as 'x ∈ [−0, 99, 099]', which appears to be a typo for [−0.99, 0.99].
  2. [Introduction] The sentence 'iLQR [10], MBMF [11], iLQR [10] and MBMF [11]' duplicates the citations; the listing should be cleaned up.
  3. [Footnote 6] The GitHub URL contains 'SatndUP', which is a typo for 'StandUp'.
  4. [Eq. (3)] The notation pt is used both for the time-dependent density and for the integration variable; the subscript t in dpt is undefined.
  5. [Sec. 6] The statement that the approach is 'free from the curse of dimensionality' is at odds with the use of NNs; the paper provides no complexity analysis controlling the network size as the state dimension grows.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation chain is self-contained, and the noted approximations are correctness risks rather than circular reductions.

full rationale

The paper defines a single objective J_URL = J(pi) + alpha H[p·pi] and derives the policy gradient and modified advantage Au from that objective by a standard likelihood-ratio argument. The value function and average density are then characterized by the Bellman residual (Eq. 14) and the steady-state continuity equation (Eq. 15), which are not restatements of the objective or of the benchmark results. The neural-network updates in Eqs. (19)-(21) are presented as approximate gradient steps for those residuals; the approximations in Appendix A.5 (dropping second-order gradients and the divergence term) are explicit heuristic assumptions and could invalidate the claim that the learned V and p solve Eqs. (14)-(15), but they do not make the derivation circular: the terms omitted are not fitted targets or renamed outputs of the algorithm. The entropy coefficient alpha is a fixed scalar hyperparameter, not a parameter fitted to reproduce the benchmark curves; the UR-NE ablation is an independent experiment, not a constructional identity. Self-citations are limited to background examples (swarm defensive behavior in Ref. [39]) and to the conceptual umbrella-sampling antecedents (Refs. [44,45]); none of these carries the optimality or superiority claim. The empirical comparison against PPO, RND, iLQR, and VI is made against external algorithms and external implementations, so the central claim is not forced by the paper's own definitions. The load-bearing weaknesses of the paper are mathematical and empirical (unsupported neglect of second-order and divergence terms, small benchmarks), but they are not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The ledger is dominated by hand-set hyperparameters and unproved approximations rather than new entities. No new physical or ontological entities are introduced. The central method rests on the entropy weight, the surrogate steady-state PDEs, and the dropped-term approximations in Appendix A.5.

free parameters (3)
  • Entropy coefficient alpha_tilde = 0.01
    Hand-set in Section 4.3. It controls the strength of the ensemble entropy bonus relative to the reward and is central to the exploration-exploitation balance, yet no sensitivity analysis is reported.
  • Per-task learning rates and weight decays = Different values for each of the three networks and for each of the two tasks
    Section 4.3 specifies different values for Multi-Valley Mountain Car and StandUp. This hand-tuning means the comparison against default-parameter baselines is not a fair test of universality.
  • Discount factor gamma = 0.95
    Chosen as a standard RL hyperparameter. It enters the discounted return, the averaged density, and the steady-state equations.
assumptions (5)
  • domain assumption The environment is a continuous-time Markov decision process with known deterministic dynamics v(s,a) and a stochastic policy pi(a|s).
    Eqs. (1)-(4) and Appendix A.1. All theory and simulations assume this model, including the continuity equation for the ensemble density.
  • ad hoc to paper The coupled steady-state PDEs (14) and (15) characterize the optimum of the entropy-augmented objective.
    Derived from a Bellman-like argument in continuous time, but existence, uniqueness, and equivalence to the original optimization under neural-network approximation are not proven.
  • ad hoc to paper The dropped second-order gradients and density-divergence terms in the NN update rules are negligible.
    Appendix A.5, Eqs. (A.39)-(A.45). This is a load-bearing approximation with no error bound or numerical check.
  • domain assumption A uniform sampling distribution rho over the observation domain provides sufficient coverage for all gradient estimates.
    Algorithm 1 step 2 and Appendix A.5 rely on replacing averages over p with averages over rho; no coverage guarantee is given for higher-dimensional state spaces.
  • ad hoc to paper Hand-crafted state representations h(x) can encode the required reflective boundary conditions for each environment.
    Appendix A.7 builds custom representations for the two tasks. This is environment-specific engineering, so the claimed implementation universality is conditional on this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Umbrella Reinforcement Learning -- computationally efficient tool for hard non-linear problems." pith.science (2026). https://pith.science/paper/7OA2WI2W

@misc{pith2026241114117,
  author       = {Pith},
  title        = {Pith review of: Umbrella Reinforcement Learning -- computationally efficient tool for hard non-linear problems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7OA2WI2W}},
  note         = {Machine review of arXiv:2411.14117}
}
read the original abstract

We report a novel, computationally efficient approach for solving hard nonlinear problems of reinforcement learning (RL). Here we combine umbrella sampling, from computational physics/chemistry, with optimal control methods. The approach is realized on the basis of neural networks, with the use of policy gradient. It outperforms, by computational efficiency and implementation universality, all available state-of-the-art algorithms, in application to hard RL problems with sparse reward, state traps and lack of terminal states. The proposed approach uses an ensemble of simultaneously acting agents, with a modified reward which includes the ensemble entropy, yielding an optimal exploration-exploitation balance.

Figures

Figures reproduced from arXiv: 2411.14117 by the authors.

Figure 1
Figure 1. Illustration of typical hard RL problems. Mountain Car (a) and Acrobot (b) [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Average expected return for Multi-Valley Mountain Car problem: For UR, UR [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figure 3
Figure 3. The color map of the final action policy for different algorithms for Multi-Valley [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Average expected return for StandUp problem: For UR, UR-NE, PPO, RND, [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

57 extracted references · 38 canonical work pages

  1. [1]

    H. Dong, H. Dong, Z. Ding, S. Zhang, Chang, Deep Reinforcement Learning, Springer, 2020

    work page 2020

  2. [2]

    T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois, S. Zhang, G. Zhang, P. Abbeel, J. Ba, Benchmarking model-based reinforcement learning, arXiv preprint arXiv:1907.02057 (2019)

    arXiv 2019

  3. [3]

    V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, K. Kavukcuoglu, Asynchronous methods for deep reinforce- ment learning, in: International conference on machine learning, PMLR, 2016, pp. 1928–1937

    work page 2016

  4. [4]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms, arXiv preprint arXiv:1707.06347 (2017)

    arXiv 2017

  5. [5]

    Y. F. Wu, W. Zhang, P. Xu, Q. Gu, A finite-time analysis of two time- scale actor-critic methods, Advances in Neural Information Processing Systems 33 (2020) 17617–17628

    work page 2020

  6. [6]

    C. J. Watkins, P. Dayan, Q-learning, Machine learning 8 (3) (1992) 279–292

    1992

  7. [7]

    V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wier- stra, M. Riedmiller, Playing atari with deep reinforcement learning, arXiv preprint arXiv:1312.5602 (2013)

    arXiv 2013

  8. [8]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic ac- tor, in: International conference on machine learning, PMLR, 2018, pp. 1861–1870

    2018

Show all 57 references
  1. [9]

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Sil- ver, D. Wierstra, Continuous control with deep reinforcement learning, arXiv preprint arXiv:1509.02971 (2015)

    2015 arXiv

  2. [10]

    W. Li, E. Todorov, Iterative linear quadratic regulator design for nonlin- ear biological movement systems, in: First International Conference on Informatics in Control, Automation and Robotics, Vol. 2, SciTePress, 2004, pp. 222–229

    2004

  3. [11]

    Nagabandi, G

    A. Nagabandi, G. Kahn, R. S. Fearing, S. Levine, Neural network dy- namics for model-based deep reinforcement learning with model-free fine-tuning, in: 2018 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2018, pp. 7559–7566

    2018

  4. [12]

    Burda, H

    Y. Burda, H. Edwards, A. Storkey, O. Klimov, Exploration by random network distillation, arXiv preprint arXiv:1810.12894 (2018)

    2018 arXiv

  5. [13]

    Pathak, P

    D. Pathak, P. Agrawal, A. A. Efros, T. Darrell, Curiosity-driven ex- ploration by self-supervised prediction, in: International conference on machine learning, PMLR, 2017, pp. 2778–2787

    2017

  6. [14]

    Ecoffet, J

    A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, J. Clune, First return, then explore, Nature 590 (7847) (2021) 580–586

    2021

  7. [15]

    Y. Li, Y. Wu, H. Xu, X. Wang, Y. Wu, Solving compositional reinforcement learning problems via task reduction, arXiv preprint arXiv:2103.07607 (2021)

    2021 arXiv

  8. [16]

    R. A. Howard, Dynamic programming and markov processes, MIT Press google schola 2 (1960) 39–47

    1960

  9. [17]

    J. H. Friedman, On bias, variance, 0/1—loss, and the curse-of- dimensionality, Data mining and knowledge discovery 1 (1997) 55–77

    1997

  10. [18]

    Bengio, O

    Y. Bengio, O. Delalleau, N. Le Roux, The curse of dimensionality for local kernel machines, Techn. Rep 1258 (12) (2005) 1

    2005

  11. [19]

    M. Bain, C. Sammut, A framework for behavioural cloning., in: Machine Intelligence 15, 1995, pp. 103–129

    1995

  12. [20]

    Arora, P

    S. Arora, P. Doshi, A survey of inverse reinforcement learning: Chal- lenges, methods and progress, Artificial Intelligence 297 (2021) 103500

    2021

  13. [21]

    Levine, A

    S. Levine, A. Kumar, G. Tucker, J. Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems, arXiv preprint arXiv:2005.01643 (2020)

    2020 arXiv

  14. [22]

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, S. Song, Diffusion policy: Visuomotor policy learning via action diffusion (2024). arXiv:2303.04137. URL https://arxiv.org/abs/2303.04137

    2024 arXiv

  15. [23]

    S. Tao, A. Shukla, T. kai Chan, H. Su, Reverse forward curriculum learning for extreme sample and demonstration efficiency in reinforce- ment learning (2024). arXiv:2405.03379. URL https://arxiv.org/abs/2405.03379

    2024 arXiv

  16. [24]

    Bellemare, S

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, R. Munos, Unifying count-based exploration and intrinsic motivation, Advances in neural information processing systems 29 (2016)

    2016

  17. [25]

    Dorigo, M

    M. Dorigo, M. Colombetti, Robot shaping: an experiment in behavior engineering, MIT press, 1998

    1998

  18. [26]

    A. Y. Ng, D. Harada, S. Russell, Policy invariance under reward trans- formations: Theory and application to reward shaping, in: Icml, Vol. 99, 1999, pp. 278–287

    1999

  19. [27]

    Jaderberg, V

    M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Sil- ver, K. Kavukcuoglu, Reinforcement learning with unsupervised auxil- iary tasks, arXiv preprint arXiv:1611.05397 (2016)

    2016 arXiv

  20. [28]

    Laskin, D

    M. Laskin, D. Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang, L. Pinto, P. Abbeel, Urlb: Unsupervised reinforcement learning benchmark, arXiv preprint arXiv:2110.15191 (2021)

    2021 arXiv

  21. [29]

    Hazan, S

    E. Hazan, S. Kakade, K. Singh, A. Van Soest, Provably efficient max- imum entropy exploration, in: International Conference on Machine Learning, PMLR, 2019, pp. 2681–2691

    2019

  22. [30]

    Andrychowicz, F

    M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welin- der, B. McGrew, J. Tobin, O. Pieter Abbeel, W. Zaremba, Hindsight experience replay, Advances in neural information processing systems 30 (2017)

    2017

  23. [31]

    Nachum, S

    O. Nachum, S. S. Gu, H. Lee, S. Levine, Data-efficient hierarchical rein- forcement learning, Advances in neural information processing systems 31 (2018)

    2018

  24. [32]

    Lakshminarayanan, A

    B. Lakshminarayanan, A. Pritzel, C. Blundell, Simple and scalable pre- dictive uncertainty estimation using deep ensembles, Advances in neural information processing systems 30 (2017)

    2017

  25. [33]

    Dimakopoulou, B

    M. Dimakopoulou, B. Van Roy, Coordinated exploration in concurrent reinforcement learning, in: International Conference on Machine Learn- ing, PMLR, 2018, pp. 1271–1279

    2018

  26. [34]

    A. W. Moore, Efficient memory-based learning for robot control, Tech. Rep. UCAM-CL-TR-209, University of Cambridge, Computer Laboratory (Nov. 1990). doi:10.48456/tr-209. URL https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-209. pdf

    1990 doi

  27. [35]

    R. S. Sutton, Generalization in reinforcement learning: Successful ex- amples using sparse coarse coding, Advances in neural information pro- cessing systems 8 (1995)

    1995

  28. [36]

    Bahamid, A

    A. Bahamid, A. M. Ibrahim, A. Ibrahim, I. Z. Zahurin, A. N. Wahid, Intelligent robot-assisted evacuation: A review, Journal of Physics: Con- ference Series 1706 (1) (2020) 012159. doi:10.1088/1742-6596/1706/ 1/012159. URL https://dx.doi.org/10.1088/1742-6596/1706/1/012159

    2020 doi

  29. [37]

    Kuindersma, R

    S. Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, F. Perme- nter, T. Koolen, P. Marion, R. Tedrake, Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot, Autonomous robots 40 (2016) 429–455

    2016

  30. [38]

    Haarnoja, S

    T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, S. Levine, Learning to walk via deep reinforcement learning, arXiv preprint arXiv:1812.11103 (2018)

    2018 arXiv

  31. [39]

    E. E. Nuzhin, M. E. Panov, N. V. Brilliantov, Why animals swirl and how they group, Scientific reports 11 (1) (2021) 1–15

    2021

  32. [40]

    G. M. Torrie, J. P. Valleau, Nonphysical sampling distributions in monte carlo free-energy estimation: Umbrella sampling”, J. Computational Physics 23 (2) (1977) 187–199

    1977

  33. [41]

    Frenkel, B

    D. Frenkel, B. Smit, Understanding Molecular Simulation: From Algo- rithms to Applications, Elsevier, London, 1996

    1996

  34. [42]

    J. P. Valleau, Thermodynamic-scaling methods in monte carlo and their application to phase equilibria, Advances in Chemical Physics 105 (2007) 369 – 404. doi:10.1002/9780470141649.ch12

    2007 doi

  35. [43]

    J. P. Valleau, Temperature-and-density-scaling monte carlo: Methodol- ogy and the canonical thermodynamics of lennard-jonesium, Molecular Simulation 31 (2005) 223 – 253. doi:10.1080/08927020500035937

    2005 doi

  36. [44]

    N. V. Brillantov, J. P. Valleau, Effective hamiltonian analysis of fluid criticality and application to the square–well fluid, The Journal of Chem- ical Physics 108 (1998) 1123–1130. doi:10.1063/1.475474

    1998 doi

  37. [45]

    N. V. Brillantov, J. P. Valleau, Thermodynamic scaling monte carlo study of the liquid–gas transition in the square–well fluid, The Journal of Chemical Physics 108 (1998) 1115–1122

    1998

  38. [46]

    Ahmed, N

    Z. Ahmed, N. Le Roux, M. Norouzi, D. Schuurmans, Understanding the impact of entropy on policy optimization, in: International conference on machine learning, PMLR, 2019, pp. 151–160

    2019

  39. [47]

    X. Guo, O. Hern´ andez-Lerma, Continuous-time markov decision pro- cesses, in: Continuous-Time Markov Decision Processes, Springer, 2009, pp. 9–18

    2009

  40. [48]

    Agarwal, N

    A. Agarwal, N. Jiang, S. M. Kakade, W. Sun, Reinforcement learning: Theory and algorithms, CS Dept., UW Seattle, Seattle, W A, USA, Tech. Rep 32 (2019) 96

    2019

  41. [49]

    R. S. Sutton, A. G. Barto, Reinforcement learning: An introduction, MIT press, 2018

    2018

  42. [50]

    Ekroot, T

    L. Ekroot, T. M. Cover, The entropy of markov trajectories, IEEE Transactions on Information Theory 39 (1993) 1418

    1993

  43. [51]

    G. Neu, A. Jonsson, V. G´ omez, A unified view of entropy-regularized markov decision processes, arXiv preprint arXiv:1705.07798 (2017)

    2017 arXiv

  44. [52]

    LeCun, L

    Y. LeCun, L. Bottou, G. B. Orr, K.-R. M¨ uller, Efficient backprop, in: Neural networks: Tricks of the trade, Springer, 2002, pp. 9–50

    2002

  45. [53]

    R. S. Sutton, D. McAllester, S. Singh, Y. Mansour, Policy gradient methods for reinforcement learning with function approximation, Ad- vances in neural information processing systems 12 (1999)

    1999

  46. [54]

    Bellman, A markovian decision process, Journal of mathematics and mechanics (1957) 679–684

    R. Bellman, A markovian decision process, Journal of mathematics and mechanics (1957) 679–684

    1957

  47. [55]

    K. M. Prasad, R. Bapat, The generalized moore-penrose inverse, Linear Algebra and its Applications 165 (1992) 59–69

    1992

  48. [56]

    Amari, Natural gradient works efficiently in learning, Neural com- putation 10 (2) (1998) 251–276

    S.-I. Amari, Natural gradient works efficiently in learning, Neural com- putation 10 (2) (1998) 251–276

    1998

  49. [57]

    ∞X i=1 γi−1r(si, ai) s0=s,a0=a # , (A.4) which is termed in RL as

    C. C. Pugh, C. Pugh, Real mathematical analysis, Vol. 2011, Springer, 2002. Appendix A. Appendix A.1. Derivation of the evolution equation for p(s, t) The derivation of the equation for p(s, t) is straightforward, as it is es- sentially a continuity equation. Let p(s, t) be th...

    2011

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.