Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Adding the negative part of the budget to the value function makes the static-CVaR Bellman operator a contraction on the full space of bounded functions, with dense per-step rewards.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:51 UTC pith:M2DBXXGP

load-bearing objection A useful reformulation of static CVaR that gives a contracting bounded Bellman operator, with discretization guarantees that are solid only under non-positive rewards. the 3 major comments →

arxiv 2602.03778 v2 pith:M2DBXXGP submitted 2026-02-03 cs.LG cs.AI

Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity

Aneri Muni , Vincent Taboga , Esther Derman , Pierre-Luc Bacon , Erick Delage This is my paper
classification cs.LG cs.AI MSC 90C4093E20
keywords static CVaRconditional value-at-riskMarkov decision processesstate augmentationBellman operatorcontraction mappingdiscretization errorrisk-averse Q-learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Static conditional value-at-risk in infinite-horizon MDPs has resisted dynamic programming: the standard augmented-state operator has zero as its only fixed point on bounded functions and pays rewards only at the end of the horizon. This paper argues these failures are not intrinsic to CVaR but artifacts of the chosen value-function parameterization. Adding the negative part of the accumulated-return variable to the value function, with per-step reward z^- minus (r+z)^-, yields an equivalent fixed point whose Bellman operator is a gamma-contraction on all bounded functions and gives informative rewards at every step. Discretizing the budget by rounding down or up produces lower and upper operators with certified performance gaps of order gamma(1-gamma)^{-1} Delta / alpha, and the lower-bound policy provably attains its stated CVaR. The payoff is that value iteration and Q-learning on a finite grid become provably convergent tools for a genuinely tail-sensitive objective.

Core claim

The paper's central claim: the value function \bar v^*(s,z)=max_pi E[-(R+z)^-+z^-] (z^- = max(0,-z)) is the right object for static CVaR. It is bounded, 1-Lipschitz in z, and flat outside [-r_gamma, r_gamma] with r_gamma = rmax/(1-gamma), unlike the classical v^*(s,z)=E[-(R+z)^-], which is unbounded and whose only bounded fixed point is zero. The induced operator \bar T, with per-step reward \tilde r(s,z,a)=z^- -(r(s,a)+z)^-, is a gamma-contraction on L^infinity with unique fixed point \bar v^*, so value iteration converges from any bounded start. Discretizing z by floor/ceil envelopes yields two contracting operators bracketing \bar v^* within gamma(1-gamma)^{-1}Delta; CVaR performance is w

What carries the argument

The central object is the shifted augmented value function \bar v^*(s,z)=E[-(R+z)^-+z^-] and its Bellman operator \bar T with augmented reward \tilde r(s,z,a)=z^--(r(s,a)+z)^-, where z^- = max(0,-z) is the negative part of the budget. The shift by z^- converts the terminal CVaR signal into a dense per-step signal, makes the operator a gamma-contraction on L^infinity, and forces the fixed point to be constant outside the bounded interval [-(rmax/(1-gamma)), rmax/(1-gamma)], so the continuous budget can be projected onto a compact interval and discretized by lower/upper rounding functions l,u. The rounding operators \bar T^l, \bar T^u sandwich the exact operator whenever rewards are non-positi

Load-bearing premise

The whole sandwich argument rests on Assumption 1, that every reward is non-positive; the order preservation and error bounds are proved only under this sign condition.

What would settle it

Compute the true static-CVaR optimum by exhaustive policy enumeration in a tiny MDP (e.g., two states, one positive reward) and compare it to the upper and lower discretized fixed points on a fine grid; if the true optimum escapes the predicted sandwich, or if a non-decreasing value function stops being non-decreasing after one \bar T^l update under positive rewards, Theorems 6-7 fail.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Value iteration on the finite grid converges to the fixed point of the rounded operator and returns optimal Q-values for every discrete budget level in a single run (Proposition 9).
  • The upper and lower CVaR estimates bracket the true optimum, and the gap shrinks linearly with grid granularity at rate gamma/(1-gamma) times Delta/alpha (Theorems 6-7).
  • The policy extracted from the lower-bound operator is certified: its true CVaR is at least Psi_l, so it cannot overstate safety (Theorem 7).
  • Q-learning converges with probability one under Robbins-Monro step sizes and infinite exploration of every state-action pair (Theorem 10), with no restrictions on initialization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same z^- shift should apply to other static risk measures whose dual representation uses the hinge function (eta-X)^+, such as expectiles or spectral risk measures; the proof ingredients are measure-agnostic.
  • The 1/alpha factor in the error bound implies that strongly risk-averse regimes (small alpha) need disproportionately finer grids; one could test this by measuring the empirical bound gap at alpha = 0.05 versus alpha = 0.5.
  • The claim that the analysis adapts to non-negative rewards is not proven; a useful check is to run the bounding operators on a reward-shifted MDP and see whether order preservation and the Delta/alpha bound survive mixed signs.
  • The block update that reuses one transition to update all budget levels is portable: any augmented MDP with deterministic budget dynamics can use the same data-relabeling trick, which is likely where the sample-efficiency gain comes from.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an augmented-state Bellman operator \bar T for infinite-horizon static CVaR maximization. It defines the bounded value function \bar v*(s,z) = max_π E[-(R(τ)+z)^- + z^-] and proves that \bar T is a γ-contraction on L∞(\bar S) with \bar v* as its unique fixed point (Theorem 4), before developing lower/upper discretized operators based on rounding maps. Under Assumption 1 (non-positive rewards), it derives monotonicity, fixed-point bounds, and performance guarantees (Propositions 5, Theorems 6--7), which are then used to design tabular Q-value iteration and Q-learning algorithms. A stochastic gridworld with only negative rewards validates the methods and shows that the discretization gap closes as the grid is refined.

Significance. If the results hold, the paper makes a useful contribution to static CVaR RL. The core Bellman operator operates on the full space of bounded functions, avoiding the restricted admissible classes of Bäuerle--Ott, and it provides a dense per-step reward signal. The proof of Theorem 4 is detailed and self-contained, and the explicit O(γ(1−γ)^{-1}Δ/α) discretization gap is a concrete, falsifiable prediction. The empirical section confirms the ordering and convergence of the upper/lower approximations. The main limitation is that the approximation guarantees are proved only for non-positive rewards; the claimed adaptation to non-negative rewards is not proved and is load-bearing for the paper's generality.

major comments (3)
  1. [Section 5, footnote 3, Theorems 6-7] The discretization analysis is conditional on Assumption 1. Proposition 5 and Lemma 15 use Lemma 14 with Δ = r(s,a) ≤ 0 to show that h(z)=min(0,z+r)−min(0,z) is non-decreasing. For r ≥ 0 the map is non-increasing; for mixed-sign rewards it is not monotone. The footnote asserting that the results 'can also be adapted to non-negative rewards' gives no proof, and the displayed complement notation is not specified. Since the empirical evaluation is cost-only, it does not exercise the claimed extension. The paper should either provide a full adaptation proof or explicitly restrict the approximation-error claims to Assumption 1.
  2. [Theorem 7 / Appendix B.8, Eq. (8)] The lower-bound policy guarantee in Theorem 7 also depends essentially on Assumption 1. In the proof of the operator \hat T^†, Lemma 14 is applied with Δ = Σ_{t=1}^k γ^{t-1} r(s_t,a_t), and the inequality invoked requires this sum to be non-positive. Thus the performance guarantee for the policy induced by the lower-discretized value function is established only in the non-positive reward regime. The text should be explicit that this is part of the scope of Theorem 7.
  3. [Abstract / Contributions] The paper's title and abstract promise general CVaR MDPs, while the algorithmic approximation results in Sections 5-6 are proved only for r ≤ 0. The core fixed-point theorem is sign-agnostic, but the discretization bounds, the lower/upper envelope ordering, and the policy guarantee are not. This mismatch between advertised scope and proved scope should be corrected, either by proving the extension or by stating the reward-sign restriction as a formal limitation of the approximation results.
minor comments (5)
  1. [Footnote 3] The notation 'lc := u = [lc]c' is unclear and under-specified. Define the complement operation explicitly so the reader can see how lower/upper rounding would be transformed for non-negative rewards.
  2. [Appendix B.8, proof of Theorem 7] In the chain of inequalities, the final expression '≤ Ψ* + α \barΔ_u/α' contains a redundant α; it should be '≤ Ψ* + \barΔ_u/α'.
  3. [Algorithm 2] The block update over all \tilde z ∈ Z_Δ is a useful data-relabeling idea, but the pseudo-code mixes two roles of z: the environmental budget z_k updated in Line 16 and the auxiliary \tilde z used only for the update. Add a sentence distinguishing these roles to avoid confusion.
  4. [Figure 4] The caption calls the 10k-bin result the 'true value'; since this is still a finite discretization, it should be described as a high-resolution numerical estimate rather than the exact continuous optimal value.
  5. [Section 7] Minor grammatical issue: 'the corresponding the optimal starting budget' should read 'the corresponding optimal starting budget'.

Circularity Check

0 steps flagged

No circular derivation: the Bellman operator fixed point is independently defined and verified; discretization bounds are proved from stated assumptions, with only a non-circular missing-proof limitation under Assumption 1.

full rationale

The central chain is self-contained rather than circular. The paper defines the value function \bar v*(s,z) := max_{\pi} E[-(R(\tau)+z)^- + z^-] directly from the static CVaR objective, after the convex dual reformulation (Proposition 1, Eq. 3), and only then defines the operator \bar T with reward \tilde r(s,z,a)=z^- - (r(s,a)+z)^- (Eq. 4). Theorem 4 verifies, rather than assumes, that \bar T \bar v* = \bar v*, using the telescoping identity min(0,R(\tau)+z)-min(0,z), and proves \gamma-contraction on L^\infty(\bar S) from the discount factor. Thus the fixed point is not installed by construction; it is a checked consequence of the reward-redistribution identity. The discretization results (Propositions 5, 8; Theorems 6, 7) are likewise proved from contraction, 1-Lipschitzness of \bar v*, and the projection/rounding maps; no fitted parameter is later renamed as a prediction. The external citations [36] and [6] supply the standard CVaR dual form and the augmented-state setup, but the paper's load-bearing contraction and error-bound arguments do not reduce to those citations. The self-references in the reference list ([20], [30]) are not used as load-bearing evidence. One limitation is explicitly present and should be weighed as a correctness/completeness concern, not as circularity: Assumption 1 restricts rewards to r <= 0, and footnote 3 asserts without proof that the discretization results adapt to non-negative rewards ('Specifically, Propositions 5 to 7 extend to the non-negative rewards using \bar T^e \bar v(s,z) := ...'). The proof of Proposition 5 in Appendix B.6 indeed relies on the concavity of min and on r <= 0 to make z -> min(0,r+z)-min(0,z) non-decreasing, and the empirical section uses a cost-only gridworld, so the claimed extension is not exercised. However, the stated theorems are explicitly conditional on Assumption 1, and the unproved extension does not feed the conclusion back into the hypothesis. Consequently, no circular step is established under the quoted-evidence standard.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces the transformed reward \tilde r=z^- - (r+z)^- and the operator \bar T, but these are functions, not new physical entities. No fitted parameters appear; hyperparameters in experiments are standard. The main extra assumption is non-positive rewards, which is explicitly used in the discretization bounds.

axioms (4)
  • standard math The convex dual representation CVaR_alpha[X]=sup_eta{eta - (1/alpha)E[(eta-X)_+]} (Eq. 1, from Rockafellar-Uryasev [36]) holds for bounded returns.
    Background theorem used in Proposition 1; not proved in the paper.
  • domain assumption The augmented state (s,z) with deterministic update z'=(r+z)/gamma is a sufficient statistic for the inner CVaR maximization, and the max over history-dependent policies can be solved by stationary policies in the augmented MDP (Bauerle & Ott [6]).
    Central setup is inherited from [6].
  • ad hoc to paper Assumption 1: r(s,a) <= 0 for all (s,a).
    Used in Proposition 5 and Theorems 6-7 for monotonicity and bound order; not required for Theorem 4 contraction.
  • domain assumption For Q-learning convergence: Robbins-Monro step sizes and infinite visitation of every nominal (s,a) (Theorem 10, conditions (1)-(2)).
    Standard stochastic approximation conditions; not verified in the practical beta schedule.

pith-pipeline@v1.3.0-alltime-deepseek · 29537 in / 23959 out tokens · 240213 ms · 2026-08-03T04:51:26.936415+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity." pith.science (2026). https://pith.science/paper/M2DBXXGP

@misc{pith2026260203778,
  author       = {Pith},
  title        = {Pith review of: Reward Redistribution for CVaR MDPs using a Bellman Operator on L-infinity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M2DBXXGP}},
  note         = {Machine review of arXiv:2602.03778}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Tail-end risk measures such as static conditional value-at-risk (CVaR) are used in safety-critical applications to prevent rare, yet catastrophic events. Unlike risk-neutral objectives, the static CVaR of the return depends on entire trajectories without admitting a recursive Bellman decomposition in the underlying Markov decision process. A classical resolution relies on state augmentation with a continuous variable. However, unless restricted to a specialized class of admissible value functions, this formulation induces sparse rewards and degenerate fixed points. In this work, we propose a novel formulation of the static CVaR objective based on augmentation. Our alternative approach leads to a Bellman operator with: (1) dense per-step rewards; (2) contracting properties on the full space of bounded value functions. Building on this theoretical foundation, we develop risk-averse value iteration and model-free Q-learning algorithms that rely on discretized augmented states. We further provide convergence guarantees and approximation error bounds due to discretization. Empirical results demonstrate that our algorithms successfully learn CVaR-sensitive policies and achieve effective performance-safety trade-offs.

Figures

Figures reproduced from arXiv: 2602.03778 by Aneri Muni, Erick Delage, Esther Derman, Pierre-Luc Bacon, Vincent Taboga.

Figure 1
Figure 1. Figure 1: Illustration of the reward schemes produced along a trajectory under the augmented MDP of [6] [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Stochastic Gridworld Environment. We validate the proposed static CVaR algorithms on a stochastic gridworld, where a robot has to traverse a 2D terrain map ( [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance comparison between the proposed static CVaR Q-learning and Q-VI algorithms for [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparing the effect of augmented state discretization resolution on the accuracy of CVaR perfor [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of the reward schemes produced along a trajectory under the augmented MDP of [6] [PITH_FULL_IMAGE:figures/full_fig_p017_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Sample Complexity for Markov Decision Processes and Stochastic Optimal Control with Static Risk Measures

    math.OC 2026-04 unverdicted novelty 6.0

    State augmentation converts static risk measures on total cost into dynamic programs, yielding sample-complexity bounds for risk-averse MDPs and stochastic optimal control under φ-divergence robustness.

  2. Sample Complexity for Markov Decision Processes and Stochastic Optimal Control with Static Risk Measures

    math.OC 2026-04 unverdicted novelty 4.0

    State augmentation allows dynamic programming and sample complexity bounds for MDPs and optimal control under static risk measures including CVaR.

Reference graph

Works this paper leans on

49 extracted references · 9 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Andrychowicz, F

    M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay.Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Predictive CVar q-learning

    Anonymous. Predictive CVar q-learning. InSubmitted to The Fourteenth International Conference on Learning Representations, 2025. under review

    2025

  3. [3]

    Artzner, F

    P. Artzner, F. Delbaen, J.-M. Eber, and D. Heath. Coherent measures of risk.Mathematical Finance, 9(3):203–228, 1999

    1999

  4. [4]

    Bastani, J

    O. Bastani, J. Y. Ma, E. Shen, and W. Xu. Regret bounds for risk-sensitive reinforcement learning.Advances in Neural Information Processing Systems, 35:36259–36269, 2022

    2022

  5. [5]

    B¨ auerle, T

    N. B¨ auerle, T. G¨ oll, and A. Ja´ skiewicz. Yet another distributional bellman equation.arXiv preprint arXiv:2505.21098, 2025

    arXiv 2025

  6. [6]

    B¨ auerle and J

    N. B¨ auerle and J. Ott. Markov decision processes with average-value-at-risk criteria.Math- ematical Methods of Operations Research, 74(3):361–379, 2011. 13

    2011

  7. [7]

    M. G. Bellemare, W. Dabney, and M. Rowland.Distributional Reinforcement Learning. The MIT Press, 05 2023

    2023

  8. [8]

    D. P. Bertsekas.Dynamic Programming and Optimal Control. Athena Scientific, 1st edition, 1995

    1995

  9. [9]

    D. P. Bertsekas and J. N. Tsitsiklis.Neuro-Dynamic Programming. Athena Scientific, 1996

    1996

  10. [10]

    L. Bisi, D. Santambrogio, F. Sandrelli, A. Tirinzoni, B. D. Ziebart, and M. Restelli. Risk-averse policy optimization via risk-neutral policy optimization.Artificial Intelligence, 311:103765, 2022

    2022

  11. [11]

    Bodnar, A

    C. Bodnar, A. Li, K. Hausman, P. Pastor, and M. Kalakrishnan. Quantile qt-opt for risk-aware vision-based robotic grasping.arXiv preprint arXiv:1910.02787, 2019

    Pith/arXiv arXiv 1910

  12. [12]

    Y. Chow, A. Tamar, S. Mannor, and M. Pavone. Risk-sensitive and robust decision-making: a cvar optimization approach.Advances in neural information processing systems, 28, 2015

    2015

  13. [13]

    Coache and S

    A. Coache and S. Jaimungal. Reinforcement learning with dynamic convex risk measures. Mathematical Finance, 34(2):557–587, 2024

    2024

  14. [14]

    Dabney, G

    W. Dabney, G. Ostrovski, D. Silver, and R. Munos. Implicit quantile networks for distri- butional reinforcement learning. InInternational conference on machine learning, pages 1096–1105. PMLR, 2018

    2018

  15. [15]

    Dabney, M

    W. Dabney, M. Rowland, M. Bellemare, and R. Munos. Distributional reinforcement learn- ing with quantile regression. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  16. [16]

    Eysenbach, X

    B. Eysenbach, X. Geng, S. Levine, and R. R. Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement.Advances in neural information processing systems, 33:14783–14795, 2020

    2020

  17. [17]

    Godbout and A

    M. Godbout and A. Durand. On the fundamental limitations of dual static cvar decompo- sitions in markov decision processes.arXiv preprint arXiv:2507.14005, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    Greenberg, Y

    I. Greenberg, Y. Chow, M. Ghavamzadeh, and S. Mannor. Efficient risk-averse reinforce- ment learning. InProceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc

    2022

  19. [19]

    W. B. Haskell and R. Jain. A convex analytic approach to risk-aware markov decision processes.SIAM J. Control. Optim., 53:1569–1598, 2015

    2015

  20. [20]

    J. L. Hau, E. Delage, E. Derman, M. Ghavamzadeh, and M. Petrik. Q-learning for Quantile MDPs: A Decomposition, Performance, and Convergence Analysis, Oct. 2024. arXiv:2410.24128 [cs]

    Pith/arXiv arXiv 2024

  21. [21]

    J. L. Hau, E. Delage, M. Ghavamzadeh, and M. Petrik. On dynamic programming de- compositions of static risk measures in markov decision processes.Advances in Neural Information Processing Systems, 36:51734–51757, 2023

    2023

  22. [22]

    R. L. Karandikar and M. Vidyasagar. Convergence of batch asynchronous stochastic ap- proximation with applications to reinforcement learning.arXiv preprint arXiv:2109.03445, 2021. 14

    Pith/arXiv arXiv 2021

  23. [23]

    Keramati, C

    R. Keramati, C. Dann, A. Tamkin, and E. Brunskill. Being optimistic to be conserva- tive: Quickly learning a cvar policy. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 4436–4443, 2020

    2020

  24. [24]

    Kim and S

    J.-H. Kim and S. Min. Risk-sensitive policy optimization via predictive cvar policy gradient. InForty-first International Conference on Machine Learning, 2024

    2024

  25. [25]

    S. H. Lim and I. Malik. Distributional reinforcement learning for risk-sensitive policies. Advances in Neural Information Processing Systems, 35:30977–30989, 2022

    2022

  26. [26]

    Majumdar and M

    A. Majumdar and M. Pavone. How should a robot assess risk? towards an axiomatic theory of risk in robotics. In N. M. Amato, G. Hager, S. Thomas, and M. Torres-Torriti, editors, Robotics Research, pages 75–84, Cham, 2020. Springer International Publishing

    2020

  27. [27]

    H. Mead, C. Costen, B. Lacerda, and N. Hawes. Return capping: Sample-efficient cvar policy gradient optimisation.arXiv preprint arXiv:2504.20887, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    M. Minsky. Steps toward artificial intelligence.Proceedings of the IRE, 49(1):8–30, 2007

    2007

  29. [29]

    Moghimi and H

    M. Moghimi and H. Ku. Beyond cvar: Leveraging static spectral risk measures for enhanced decision-making in distributional reinforcement learning, 2025

    2025

  30. [30]

    A. Muni, E. Derman, V. Taboga, P.-L. Bacon, and E. Delage. What matters when modeling human behavior using imitation learning? In2nd Workshop on Models of Human Feedback for AI Alignment, 2025

    2025

  31. [31]

    X. Ni, G. Liu, and L. Lai. Risk-sensitive reward-free reinforcement learning with cvar. In Forty-first International Conference on Machine Learning, 2024

    2024

  32. [32]

    G. C. Pflug and A. Pichler. Time-consistent decisions and temporal decomposition of coherent risk functionals.Math. Oper. Res., 41:682–699, 2016

    2016

  33. [33]

    Pignatelli, J

    E. Pignatelli, J. Ferret, M. Geist, T. Mesnard, H. van Hasselt, O. Pietquin, and L. Toni. A survey of temporal credit assignment in deep reinforcement learning.arXiv preprint arXiv:2312.01072, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    B. ´A. Pires, M. Rowland, D. Borsa, Z. D. Guo, K. Khetarpal, A. Barreto, D. Abel, R. Munos, and W. Dabney. Optimizing return distributions with distributional dynamic programming.arXiv preprint arXiv:2501.13028, 2025

    Pith/arXiv arXiv 2025

  35. [35]

    M. L. Puterman.Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994

    1994

  36. [36]

    R. T. Rockafellar, S. Uryasev, et al. Optimization of conditional value-at-risk.Journal of risk, 2:21–42, 2000

    2000

  37. [37]

    Ruszczy´ nski

    A. Ruszczy´ nski. Risk-averse dynamic programming for markov decision processes.Mathe- matical programming, 125(2):235–261, 2010

    2010

  38. [38]

    Schneider, J

    L. Schneider, J. Frey, T. Miki, and M. Hutter. Learning risk-aware quadrupedal locomotion using distributional reinforcement learning. In2024 IEEE International Conference on Robotics and Automation (ICRA), pages 11451–11458. IEEE, 2024

    2024

  39. [39]

    A. Shapiro. On a time consistency concept in risk averse multistage stochastic program- ming.Operations Research Letters, 37(3):143–147, 2009. 15

    2009

  40. [40]

    Stanko and K

    S. Stanko and K. Macek. Risk-averse distributional reinforcement learning: A cvar opti- mization approach. InIJCCI, pages 412–423, 2019

    2019

  41. [41]

    Tamar, Y

    A. Tamar, Y. Chow, M. Ghavamzadeh, and S. Mannor. Policy gradient for coherent risk measures.Advances in neural information processing systems, 28, 2015

    2015

  42. [42]

    Tamar, Y

    A. Tamar, Y. Glassner, and S. Mannor. Optimizing the cvar via sampling. InProceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015

    2015

  43. [43]

    Y. C. Tang, J. Zhang, and R. Salakhutdinov. Worst cases policy gradients.arXiv preprint arXiv:1911.03618, 2019

    Pith/arXiv arXiv 1911

  44. [44]

    J. N. Tsitsiklis. Asynchronous stochastic approximation and q-learning.Mach. Learn., 16(3):185–202, Sept. 1994

    1994

  45. [45]

    K. Wang, N. Kallus, and W. Sun. Near-minimax-optimal risk-sensitive reinforcement learn- ing with cvar. InInternational Conference on Machine Learning, pages 35864–35907. PMLR, 2023

    2023

  46. [46]

    K. Wang, D. Liang, N. Kallus, and W. Sun. A reductions approach to risk-sensitive rein- forcement learning with optimized certainty equivalents.arXiv preprint arXiv:2403.06323, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Wang and M

    Y. Wang and M. P. Chapman. Risk-averse autonomous systems: A brief history and recent developments from the perspective of optimal control.Artificial Intelligence, 311:103743, 2022

    2022

  48. [48]

    − 0−1X t=0 γtr(st, at) +z ! − +z − # =−z − +z − = ˆv† 0(s, z). 29 We follow with atk+ 1: ˆv† k+1(s, z) =E τ∼P ¯π(·,z∗ l ) s

    C. J. C. H. Watkins and P. Dayan. Technical note: q -learning.Mach. Learn., 8(3–4):279–292, May 1992. 16 A Illustration of Various Reward Schemes Nominal MDP s1 s2 s3 s4 s4+1 −3 +1 0 0 Augmented MDP (B¨ auerle & Ott, 2011) s1 0 s2 2 s3 −2 s4 −2 s4 . . . 0 0 0 0 0 Transformed Augmented MDP (Ours) s1 0 s2 2 s3 −2 s4 −2 s4 . . . 0 −1 +1 0 0 ˜rt = (zt)−−(zt +...

    1992

  49. [49]

    These are used to calculate the empirical return distribution, the empirical CVaR performance as well as analyzing the number of times the robot enters the crater per episode

    We run the policy for eachαand collect 10k trajectories. These are used to calculate the empirical return distribution, the empirical CVaR performance as well as analyzing the number of times the robot enters the crater per episode. D.4 Static CVaR Q-learning The static CVaR Q-learning algorithm (Algorithm 2) is also run 10 times, with different seeds. Th...