Pith. sign in

REVIEW 3 major objections 6 minor 44 references

Practical Risk Measures in Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single risk critic can optimize policies under any differentiable risk measure, the paper argues.

desk verdict A genuinely useful risk-critic idea, undercut by a load-bearing gradient error in Eq. (10) and a muddled risk-shaping step; both need fixing before the plug-and-play claim can stand. read the letter →

arxiv 1908.08379 v1 pith:SQYHIOJF submitted 2019-08-22 cs.LG stat.ML

Dotan Di Castro , Joel Oren , Shie Mannor This is my paper · ORCID
classification cs.LGstat.ML MSC 68T0590C40
keywords riskmeasuresreinforcementlearningpolicygradientshapingcriticnon-coherentMonteCarloestimationactor-critic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a reinforcement learning agent can optimize a policy under a general, user-specified risk measure without designing a new algorithm for each one. It defines the risk at a state as $R(x)=\mathbb{E}[f(B-J(x))]$, where $B$ is the discounted reward-to-go and $f$ is any differentiable shaping function, and derives a policy-gradient update that works for every such $f$. The paper also introduces risk shaping: fitting $f$ from Monte Carlo samples so the risk measure reflects the problem's actual failure modes, which need not be coherent or convex. If the representation is sound, practitioners could choose or learn an arbitrary risk shape and reuse a single actor-critic architecture.

What carries the argument

The load-bearing object is the risk functional $R(x)=\mathbb{E}[f(B-J(x))]$ together with its policy gradient, given in the paper's Eq. (10), which augments the standard likelihood-ratio gradient with the term $f'(B-J(x))\nabla J(x)$. Around this, the paper builds the ARCVC architecture: an actor for the policy, a value network estimating $J$, and a risk network estimating $R$, trained on a finite-time buffer of Monte Carlo reward-to-go samples through the losses in Eqs. (11) and (12). Risk shaping fits $f$ to empirical pairs $(B-J, R(x))$ obtained from simulation, allowing $f$ to be non-convex and non-coherent.

What would settle it

Fit $f$ on the gambler's ruin problem for a fixed look-ahead $k$, then compute $\mathbb{E}[f(B-J(x))]$ under the same process and compare the result with the true bankruptcy probability $p_k(m)$; if they diverge materially, Eq. (3) does not represent the bankruptcy risk.

Watch

Extended reading notes

Core claim

The paper's central claim is that a general risk measure can be written as $R(x)=\mathbb{E}[f(B-J(x))]$, where $B$ is the reward-to-go, $J(x)$ is the expected reward-to-go from state $x$, and $f$ is any differentiable function such as the square, absolute value, or square root. For this family, the paper derives an explicit policy-gradient update that includes a correction term involving $f'(B-J(x))\nabla J(x)$, because the reference $J$ itself depends on the policy. It then shows how to extract $f$ from data through risk shaping, using a gambler's ruin example where the natural bankruptcy-risk probability is non-convex and where variance is a poor risk indicator. The proposed actor-risk-critic-value-critic architecture estimates the risk from finite-time Monte Carlo buffers and uses it in a soft constraint, with experiments showing that non-coherent one-sided risk functions can reduce risk violations while sometimes lowering success rate.

Load-bearing premise

The risk measure of interest can be reproduced by fitting a single function $f$ in $R(x)=\mathbb{E}[f(B-J(x))]$ from pointwise samples of $B-J$ versus $R(x)$; the paper does not prove that this representation is valid for a given risk measure.

Editorial extensions

If this is right

  • Any differentiable $f$ can be used in the same policy-gradient machinery, so variance, one-sided penalties, and quantile-like shapes become special cases of one update rule.
  • A risk constraint enforced during training acts as safe exploration; in the experiments the one-sided square-root risk cuts risk violations while lowering success rate.
  • Replacing the state-wise reference $J(x)$ with a global scalar $\bar{J}$ costs little accuracy as the discount factor $\gamma$ approaches 1, enabling a cheaper two-network or one-network agent.
  • Replacing the risk network with a single sample in the penalty function saves memory and runtime at the price of worse accumulated reward and higher violation rate.
  • The extracted $f$ in the gambler's ruin example is non-convex, so restricting to coherent risk measures would exclude a natural bankruptcy-risk shape.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Eq. (3) representation holds beyond the examples, the same critic could be trained against distributional estimates of $B-J$ and applied to continuous-control safety, where many risk shapes are currently handled one at a time.
  • The risk-shaping step as described treats $f$ as fixed once fitted; interleaving fitting and policy optimization, which the paper flags as future work, would turn risk shaping into an online meta-critic.
  • The closeness of global and state references at high $\gamma$ suggests long-horizon tasks may be the easiest targets for the compact architecture, because the reward-to-go distribution becomes dominated by the mean over states.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes a general risk-measure framework for reinforcement learning based on a risk critic that estimates R(x)=E[f(B-J(x))] for any differentiable f, with policy gradient updates under a soft constraint. It introduces 'risk shaping' to learn f from Monte Carlo data and presents an actor-critic architecture (ARCVC) with compact variants. Experiments on a grid world and a gambler's ruin example illustrate different one-sided risk measures and compare sample-based versus network-based penalties.

Significance. If the proposed gradient formula and risk-shaping procedure were correct, the plug-and-play property would be a useful step beyond coherent-risk-only methods. The paper gives a clean decomposition of the objective and a Monte-Carlo-based critic that avoids TD bootstrapping, and the compact architecture is a reasonable engineering idea. However, the central gradient derivation contains a mathematical error and the risk-shaping procedure conflates a conditional expectation with a pointwise function; these issues invalidate the main claims as stated.

major comments (3)
  1. [§3.1, Eq. (10)] The formula for ∇R(x) is mathematically incorrect. From Eq. (9), the gradient is Σ[f(B-J)∇Pθ + Pθ∇f(B-J)], and with ∇f(B-J)=-f'(B-J)∇J, the second term is -∇J Σ Pθ f'. Eq. (10) instead writes both terms under Σ∇Pθ, i.e., Σ∇Pθ[f - f'∇J], omitting the Pθ weight on the f' term. This expression is not the gradient of R(x) and is generally not an expectation under the trajectory distribution. Since Algorithm 1 states that the policy is optimized according to Eqs. (5), (7), and (10), the implemented update is a biased estimator of the gradient of the stated objective. This error is independent of how f is learned and affects all variants.
  2. [§5] The risk-shaping procedure fits f by plotting R(x)=pk(m) against individual samples of B-J and fitting 1/(1+b(x-c)^2). But Eq. (3) defines R(x) as the expectation of f(B-J), not as a pointwise function of B-J. The relation between a single realization (B-J) and the bankruptcy probability is not f. The paper provides no argument that such a function exists or that this fitting procedure recovers the f appearing in Eq. (3). Without a valid identification argument, the learned f and the resulting risk measure are not meaningful, which undermines the 'risk shaping' contribution.
  3. [§6.1, Eq. (13)] The compact objective is self-referential: η(x) is defined as E[B + α(f(B-η(x))-D) g{f(B-η(x))-D}], so η appears on both sides of the equation. The text claims 'it is easy to show' that this converges to J(·) when the constraint is satisfied, but no proof or conditions are given. This is not merely a notation issue; the behavior of the compact architecture depends on the well-posedness of this recursive definition.
minor comments (6)
  1. [Abstract] The phrase 'can be use' should be 'can be used'.
  2. [§1] The text says 'any risk function that one can think off', which should be 'think of'.
  3. [§3.1, Proposition 1] The notation ∇Pθ(xt+1|xt) is ambiguous; it should be ∇θ Pθ(xt+1|xt) to clarify that the gradient is with respect to the policy parameters.
  4. [§5] The fitting of 1/(1+b(x-c)^2) is introduced without justification or details on how b and c were estimated; the figure is not described in enough detail to assess the fit.
  5. [§6.1] The definitions of η(x) and ˆη(x) are inconsistent between the first display and Eq. (13); for example, the first display includes a factor g{E[f(B-ν(x))]-D} while Eq. (13) uses g{f(B-η(x))-D} without the expectation.
  6. [§7.2] The claim that 'it is easy to show' the constraint scaling D, √D, D² for different risk functions is not demonstrated and is not obvious from the definitions.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the risk-critic derivation is self-contained, and the risk-shaping step fits an external target rather than predicting its own input.

full rationale

The paper's central derivation starts from the MDP definition, the reward-to-go Bτ, the value function J(x)=E[B|x0=x], and the risk definition R(x)=E[f(B-J(x))] in Eq. (3), and then derives the policy-gradient update by differentiating this expectation. Proposition 1 (Eq. (8)) is proved in the paper from the likelihood-ratio identity, with the Baxter and Bartlett result used only as background. The soft-constraint objective Eq. (5) is introduced explicitly as analogous to Tamar et al. (2012), but it is a simple construction and is not used as an unverified external theorem, so the self-citations in Section 2 and Section 3 are not load-bearing. In the gambler's ruin risk-shaping experiment (Section 5), the target risk pk(m) is computed from the Markov reward process model, and the function f is fit to samples of (B-J, pk(m)); this is supervised model fitting to an external quantity, not a prediction that reduces to its own input. The risk critic is trained by regression on Monte-Carlo samples in Eq. (11)-(12), and the experimental comparisons use fixed risk functions f rather than a fitted f that is then evaluated on the same data. No step in the paper equates a derived quantity to an input by construction, and no central claim is justified only by a self-citation. The serious flaw highlighted by the skeptical reviewer is an algebraic error in the gradient derivation at Eq. (10), where the f' term is weighted by ∇P instead of P; that is a correctness defect, not a circularity. Under the circularity rubric, the paper should receive a non-finding.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central derivation rests on the expectation representation of risk and on a data-fitted f. The two fitted parameters b and c in the risk function are not derived from theory or measured independently; they are learned from the same simulation data that the risk shaping experiment displays. No new physical or mathematical entities are postulated.

free parameters (2)
  • b (bell-curve scale in fitted risk function) = not reported
    Fitted to Monte Carlo samples in Sec. 5, Fig. 2. The fitted f enters Eq. (3) and is then optimized by the policy.
  • c (bell-curve center in fitted risk function) = not reported
    Center of the fitted function f(x)=1/(1+b(x-c)^2); selected by data fit in risk shaping.
assumptions (5)
  • domain assumption The risk measure is representable as E[f(B-J(x))] for a single scalar function f that does not depend on the state
    This is Eq. (3). It is assumed for all risk measures in the paper, but the gambler's ruin example fits f directly to conditional probabilities, so the representation is not justified.
  • standard math The likelihood-ratio policy gradient identity in Eq. (8) is valid
    Used in Sec. 3.1 to derive the risk gradient. Standard when the policy is differentiable, but the subsequent algebra in Eq. (10) is wrong.
  • ad hoc to paper The soft-constrained objective in Eq. (6) has the same optimum as the constrained problem in the limit of large lambda
    Stated without proof in Sec. 3; true only for exact penalty methods, and not established for the chosen g.
  • ad hoc to paper The risk function in the gambler's ruin example has the form 1/(1+b(x-c)^2)
    Chosen after viewing Monte Carlo samples in Sec. 5, not derived from the ruin probability. The fitted b and c are free parameters.
  • standard math The reward, policy, and risk function satisfy differentiability and boundedness conditions
    Implicit in Sec. 3 (bounded rewards, differentiable f and policy) to justify gradient updates and neural approximation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Practical Risk Measures in Reinforcement Learning." pith.science (2026). https://pith.science/paper/SQYHIOJF

@misc{pith2026190808379,
  author       = {Pith},
  title        = {Pith review of: Practical Risk Measures in Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SQYHIOJF}},
  note         = {Machine review of arXiv:1908.08379}
}
read the original abstract

Practical application of Reinforcement Learning (RL) often involves risk considerations. We study a generalized approximation scheme for risk measures, based on Monte-Carlo simulations, where the risk measures need not necessarily be \emph{coherent}. We demonstrate that, even in simple problems, measures such as the variance of the reward-to-go do not capture the risk in a satisfactory manner. In addition, we show how a risk measure can be derived from model's realizations. We propose a neural architecture for estimating the risk and suggest the risk critic architecture that can be use to optimize a policy under general risk measures. We conclude our work with experiments that demonstrate the efficacy of our approach.

Figures

Figures reproduced from arXiv: 1908.08379 by the authors.

Figure 1
Figure 1. An example of the grid world. The starting point is located [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Samples of a Monte Carlo experiment as described in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. We can see that three networks are involved: one for [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: The risk critic architecture for mitigating risk in MDPs. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 4
Figure 4. Figure 4: Risk Violation Rate and Success Rate for 3 different risk [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Plot of ¯ vs. the discount factor γ. In red dots different repeats are presented. The blue line is the average of ¯ for each γ. This plot is based on 50 values of γ and 30 runs for each value. network signal in the penalty function of Eq. 5 with a single sample based…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 37 canonical work pages

  1. [1]

    Constrained policy optimization

    [Achiam et al., 2017] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. arXiv preprint arXiv:1705.10528,

    arXiv 2017

  2. [9]

    [Duft, 1974] K. D. Duft. Gambler’s ruin and investment anal- ysis. In Proceedings, Annual Meeting (Western Agricultural Economics Association),

    work page 1974

  3. [12]

    A comprehensive survey on safe reinforce- ment learning

    [Garcıa and Fern´andez, 2015] Javier Garcıa and Fernando Fern´andez. A comprehensive survey on safe reinforce- ment learning. Journal of Machine Learning Research , 16(1):1437–1480,

    work page 2015

  4. [13]

    Risk-sensitive reinforcement learning applied to control under constraints

    [Geibel and Wysotzki, 2005] Peter Geibel and Fritz Wysotzki. Risk-sensitive reinforcement learning applied to control under constraints. Journal of Artificial Intelligence Research, 24:81–108,

    work page 2005

  5. [14]

    Likelihood ratio gradient es- timation for stochastic systems

    [Glynn, 1990] Peter W Glynn. Likelihood ratio gradient es- timation for stochastic systems. Communications of the ACM, 33(10):75–84,

    work page 1990

  6. [16]

    Risk-sensitive markov decision processes

    [Howard and Matheson, 1972] Ronald A Howard and James E Matheson. Risk-sensitive markov decision processes. Management science, 18(7):356–369,

    work page 1972

  7. [22]

    Func- tional value iteration for decision-theoretic planning with general utility functions

    [Liu and Koenig, 2006] Yaxin Liu and Sven Koenig. Func- tional value iteration for decision-theoretic planning with general utility functions. In PROCEEDINGS OF THE NATIONAL CONFERENCE ON ARTIFICIAL INTELLI- GENCE, volume 21, page

    work page 2006

  8. [25]

    Mean-variance optimization in markov decision pro- cesses

    [Mannor and Tsitsiklis, 2011] Shie Mannor and John Tsitsik- lis. Mean-variance optimization in markov decision pro- cesses. arXiv preprint arXiv:1104.5601,

    arXiv 2011

Show all 44 references
  1. [28]

    Safe exploration in markov decision pro- cesses

    [Moldovan and Abbeel, 2012] Teodor Mihai Moldovan and Pieter Abbeel. Safe exploration in markov decision pro- cesses. arXiv preprint arXiv:1205.4810,

    2012 arXiv

  2. [29]

    Parametric return density estimation for reinforcement learning

    [Morimura et al., 2012] Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Parametric return density estimation for reinforcement learning. arXiv preprint arXiv:1203.3497,

    2012 arXiv

  3. [30]

    Policy invariance under reward transformations: Theory and application to reward shaping

    [Ng et al., 1999] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, vol- ume 99, pages 278–287,

    1999

  4. [31]

    Markov chains

    [Norris, 1998] James R Norris. Markov chains. Number

    1998

  5. [33]

    Evaluation: from preci- sion, recall and f-measure to roc, informedness, markedness and correlation

    [Powers, 2011] David Martin Powers. Evaluation: from preci- sion, recall and f-measure to roc, informedness, markedness and correlation

    2011

  6. [34]

    Actor-critic algorithms for risk- sensitive mdps

    [Prashanth and Ghavamzadeh, 2013] LA Prashanth and Mo- hammad Ghavamzadeh. Actor-critic algorithms for risk- sensitive mdps. In Advances in neural information process- ing systems, pages 252–260,

    2013

  7. [35]

    Markov decision pro- cesses

    [Puterman, 1994] Martin L Puterman. Markov decision pro- cesses. j. Wiley and Sons,

    1994

  8. [37]

    Td algorithm for the variance of return and mean-variance reinforcement learning

    [Sato et al., 2001] Makoto Sato, Hajime Kimura, and Shibenobu Kobayashi. Td algorithm for the variance of return and mean-variance reinforcement learning. Trans- actions of the Japanese Society for Artificial Intelligence, 16(3):353–362,

    2001

  9. [38]

    Lectures on stochastic program- ming: modeling and theory

    [Shapiro et al., 2009] Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczy´nski. Lectures on stochastic program- ming: modeling and theory. SIAM,

    2009

  10. [43]

    Temporal difference methods for the variance of the reward to go

    [Tamar et al., 2013] Aviv Tamar, Dotan Di Castro, and Shie Mannor. Temporal difference methods for the variance of the reward to go. In International Conference on Machine Learning, pages 495–503,

    2013

  11. [44]

    The gambler’s ruin ap- proach to business risk

    [Wilcox, 1976] Jarrod W Wilcox. The gambler’s ruin ap- proach to business risk. Sloan Management Review (pre- 1986), 18(1):33, 1976

    1976

  12. [1186]

    Menlo Park, CA; Cam- bridge, MA; London; AAAI Press; MIT Press; 1999,

    1999

  13. [1972]

    Dynamic probabilistic systems: Markov models, volume

    [Howard, 2012] Ronald A Howard. Dynamic probabilistic systems: Markov models, volume

    2012

  14. [1974]

    Percentile performance criteria for limit- ing average markov decision processes

    [Filar et al., 1995] Jerzy A Filar, Dmitry Krass, and Kirsten W Ross. Percentile performance criteria for limit- ing average markov decision processes. IEEE Transactions on Automatic Control, 40(1):2–10,

    1995

  15. [1982]

    Reinforcement learning: An introduction

    [Sutton and Barto, 2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press,

    2018

  16. [1988]

    Policy gradients with variance related risk criteria

    [Tamar et al., 2012] Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the twenty-ninth international conference on machine learning, pages 387–396,

    2012

  17. [1990]

    Deep learning, vol- ume

    [Goodfellow et al., 2016] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, vol- ume

    2016

  18. [1994]

    Optimization of conditional value-at-risk

    [Rockafellar et al., 2000] R Tyrrell Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21–42,

    2000

  19. [1995]

    Monte Carlo: concepts, algorithms, and applications

    [Fishman, 2013] George Fishman. Monte Carlo: concepts, algorithms, and applications. Springer Science & Business Media,

    2013

  20. [1998]

    Safe policy iteration

    [Pirotta et al., 2013] Matteo Pirotta, Marcello Restelli, Alessio Pecorino, and Daniele Calandriello. Safe policy iteration. In International Conference on Machine Learning, pages 307–315,

    2013

  21. [1999]

    Safe policy search for lifelong reinforce- ment learning with sublinear regret

    [Ammar et al., 2015] Haitham Bou Ammar, Rasul Tutunov, and Eric Eaton. Safe policy search for lifelong reinforce- ment learning with sublinear regret. In International Con- ference on Machine Learning, pages 2361–2369,

    2015

  22. [2000]

    Stochastic approximation and recursive algorithms and applications, volume

    [Kushner and Yin, 2003] Harold Kushner and G George Yin. Stochastic approximation and recursive algorithms and applications, volume

    2003

  23. [2001]

    Dy- namic programming and optimal control, volume

    [Bertsekas et al., 2005] Dimitri P Bertsekas, Dimitri P Bert- sekas, Dimitri P Bertsekas, and Dimitri P Bertsekas. Dy- namic programming and optimal control, volume

    2005

  24. [2002]

    Human-level control through deep reinforcement learning

    [Mnih et al., 2015] V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidje- land, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529,

    2015

  25. [2003]

    Deep learning

    [LeCun et al., 2015] Yann LeCun, Yoshua Bengio, and Geof- frey Hinton. Deep learning. nature, 521(7553):436,

    2015

  26. [2005]

    Model predictive control

    [Camacho and Alba, 2013] Eduardo F Camacho and Car- los Bordons Alba. Model predictive control . Springer Science & Business Media,

    2013

  27. [2006]

    Exis- tence and finiteness conditions for risk-sensitive planning: Results and conjectures

    [Liu and Koenig, 2012] Yaxin Liu and Sven Koenig. Exis- tence and finiteness conditions for risk-sensitive planning: Results and conjectures. arXiv preprint arXiv:1207.1391,

    2012 arXiv

  28. [2009]

    The variance of discounted markov decision processes

    [Sobel, 1982] Matthew J Sobel. The variance of discounted markov decision processes. Journal of Applied Probability, 19(4):794–802,

    1982

  29. [2011]

    Risk-sensitive reinforcement learning

    [Mihatsch and Neuneier, 2002] Oliver Mihatsch and Ralph Neuneier. Risk-sensitive reinforcement learning. Machine learning, 49(2-3):267–290,

    2002

  30. [2012]

    Adam: A method for stochastic optimization

    [Kingma and Ba, 2014] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980,

    2014 arXiv

  31. [2013]

    Risk-constrained reinforcement learning with percentile risk criteria

    [Chow et al., 2017] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research , 18:167–1,

    2017

  32. [2014]

    Actor-critic algorithms

    [Konda and Tsitsiklis, 2000] Vijay R Konda and John N Tsit- siklis. Actor-critic algorithms. In Advances in neural information processing systems, pages 1008–1014,

    2000

  33. [2015]

    Concrete problems in ai safety

    [Amodei et al., 2016] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Man´e. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565,

    2016 arXiv

  34. [2016]

    Infinite-horizon policy-gradient estimation

    [Baxter and Bartlett, 2001] Jonathan Baxter and Peter L Bartlett. Infinite-horizon policy-gradient estimation. Jour- nal of Artificial Intelligence Research, 15:319–350,

    2001

  35. [2017]

    Constrained Markov decision processes, volume

    [Altman, 1999] Eitan Altman. Constrained Markov decision processes, volume

    1999

  36. [2018]

    Learning to predict by the methods of temporal differences

    [Sutton, 1988] Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9– 44,

    1988

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.