Pith. sign in

REVIEW 4 major objections 5 minor 70 references

Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A dual-loop actor-critic optimizes static spectral risk measures directly, in both online and offline reinforcement learning, with convergence guarantees in finite MDPs.

desk verdict A real algorithmic extension of static-SRM RL to actor-critic and offline settings, but the headline convergence theorem does not cover the implemented approximate algorithm and the abstract oversells the experiments. read the letter →

arxiv 2507.03900 v1 pith:J5K4MKQK submitted 2025-07-05 cs.LG stat.ML

Mehrdad Moghimi , Hyejin Ku This is my paper · ORCID
classification cs.LGstat.ML MSC 90C4068T05
keywords risk-sensitivereinforcementlearningspectralriskmeasuresstaticactor-criticofflinedistributionalCVaRconvergenceguarantees
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper addresses a known flaw in risk-sensitive reinforcement learning: applying a risk measure at every time step, as distributional RL usually does, does not optimize any fixed risk measure of the whole trajectory. It proposes instead to directly maximize a static Spectral Risk Measure (SRM) of the total return, using a bi-level actor-critic method in which an outer loop updates a concave risk function based on the return distribution and an inner loop maximizes the policy under that function. The framework is instantiated for stochastic and deterministic policies in both online and offline settings, named AC-SRM, OAC-SRM, TD3-SRM, and TD3BC-SRM. The paper proves convergence of the inner policy search and monotonic improvement of the SRM objective in the finite state-action tabular setting, and reports strong empirical performance against iterative-risk baselines in finance, healthcare, and robotics benchmarks.

What carries the argument

The load-bearing object is the supremum representation of a Spectral Risk Measure: $\mathrm{SRM}_\varphi(Z) = \sup_{h \in H}\left(\mathbb{E}[h(Z)] + \int_0^1 \hat{h}(\varphi(u))\,du\right)$, with the supremum attained at $h_{\varphi,Z}(z) = \int_0^1 \left(F_Z^{-1}(\alpha) + \frac{1}{\alpha}(z - F_Z^{-1}(\alpha))^-\right)\mu(d\alpha)$. Proposition 4.1 approximates this $h$ by a piecewise-linear function built from $N$ quantiles with weights $w_i = \hat{\tau}_i(\varphi(\tau_{i-1}) - \varphi(\tau_i))$, so the risk preference is re-expressed as a deterministic, action-independent function of the return. This $h$ is then used to define $Q$-values on an extended state space (accumulated reward $s$ and discount $c$), which is what lets the same global risk spectrum act on returns from different time steps. The risk-sensitive Performance Difference Lemma converts the advantage function into policy improvement and is the engine behind both the inner convergence theorem and the offline advantage-weighted update.

What would settle it

Run AC-SRM or TD3-SRM on a small finite MDP where $J(\pi) = \mathrm{SRM}_\varphi(G^\pi)$ can be computed exactly by enumerating all trajectories, and record $J(\pi_k)$ at each outer iteration. A single decrease $J(\pi_{k+1}) < J(\pi_k)$, or a computed gradient sequence whose norm does not approach zero, would falsify Theorem 4.3 as stated. The toy setting needs only two states, two actions, a small horizon, and a fixed spectrum such as CVaR with $\alpha = 0.2$.

Watch

Extended reading notes

Core claim

The central claim is that $J(\pi) = \mathrm{SRM}_\varphi(G^\pi)$, the spectral risk measure of the return distribution from the initial state, can be optimized by alternating two steps instead of embedding the risk measure into every Bellman update. For a fixed concave function $h$, the inner loop maximizes $J(\pi, h) = \mathbb{E}[h(G^\pi)] + \int_0^1 \hat{h}(\varphi(u))\,du$ via a risk-sensitive policy gradient with $Q_h^\pi(\bar{x}, a) = \mathbb{E}[h(s + c G^\pi(\bar{x}, a))]/c$; the outer loop sets $h_{k+1} = \tilde{h}_{\varphi, G^{\pi_k}}$, a piecewise-linear quantile approximation of the exact maximizer from Proposition 4.1. Theorem 4.2 establishes that the inner natural-policy-gradient updates converge to the optimal softmax policy for fixed $h$, and Theorem 4.3 establishes that the alternating scheme monotonically improves $J(\pi)$, converges, and has $\liminf_k \|\nabla_\theta J(\pi_{\theta_k})\| = 0$. The authors also derive an advantage-weighted update for the offline constraint setting. If correct, this gives an actor-critic family that optimizes static SRM directly, covering CVaR and Mean-CVaR as special cases, and avoids the time-inconsistency of iterative risk measures.

Load-bearing premise

The convergence proofs assume the exact outer maximizer $h_{\varphi, G^{\pi_k}}$ and a compact parameter sublevel set, while the implemented algorithm uses a piecewise-linear approximation $\tilde{h}$ and finitely many inner policy updates; the monotonic-improvement guarantee does not strictly follow for the implemented approximation unless its error is controlled.

Editorial extensions

If this is right

  • Policies can be matched to a user's true risk preference: the risk measure is applied once to the whole return, not step by step, so CVaR- or Mean-CVaR-optimizing policies are no longer distorted by local risk decisions.
  • For offline RL, one fixed dataset yields a whole family of risk-tuned policies: changing the spectrum parameter (e.g., $\omega$ in Mean-CVaR) gives different risk-return profiles without further environment interaction.
  • The same bi-level scheme works for stochastic actors (AC-SRM, OAC-SRM) and deterministic actors (TD3-SRM, TD3BC-SRM), so discrete and continuous control can share the static-SRM objective.
  • In the finite tabular case, the inner policy search converges to the optimal softmax policy for the fixed $h$, and the overall loop monotonically improves the static SRM objective.
  • Iterative risk baselines such as DSAC-iCVaR and CODAC-iMC are consistently outperformed in the reported trading, portfolio, HIV, and MuJoCo experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: the framework should extend to any coherent distortion risk measure with an absolutely continuous distortion function, since the SRM representation covers those; experiments with Wang or proportional-hazard spectra would test this directly.
  • Our inference: the time-inconsistency advantage should grow with horizon length and tail severity, so longer-horizon stochastic MuJoCo tasks or rare-disaster reward models should show larger margins over iterative-risk baselines.
  • Our inference: the paper's observation about entropy-based critics suggests a decoupled risk-sensitive SAC where entropy is not risk-adjusted; this modification is not implemented in the paper but is a natural testable extension.
  • Our inference: if the approximate $\tilde{h}$ is replaced by an exact or better-quantized $h$ in the outer loop, Theorem 4.3's monotonicity could be brought closer to the implemented algorithm; comparing the two on a finite MDP would quantify the approximation gap.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes actor-critic algorithms (AC-SRM, OAC-SRM, TD3-SRM, TD3BC-SRM) for directly optimizing static spectral risk measures (SRM) of the return distribution in both online and offline reinforcement learning. The method is based on a bi-level reformulation of the SRM via its supremum representation, alternating an inner policy optimization for a fixed risk function h and an outer update of h using quantiles of the current return distribution. The authors prove convergence of the inner policy optimization in the finite tabular setting (Theorem 4.2), monotonic improvement and stationarity of the overall procedure (Theorem 4.3), and derive a risk-sensitive advantage-weighted update for the offline setting (Theorem 4.4). They report empirical results on trading, portfolio allocation, HIV treatment, and stochastic MuJoCo environments, comparing against risk-neutral and iterative-risk baselines.

Significance. If the convergence claims were fully supported, this would be a useful contribution: static SRM objectives are known to be poorly served by per-step iterative risk measures, and an actor-critic method that directly optimizes a static SRM with guarantees would fill a real gap in risk-sensitive RL, especially for offline policy tailoring. The supremum-representation reformulation is conceptually clean, the empirical study is broad and includes a compelling portfolio-allocation demonstration, and the authors provide code and detailed hyperparameters. However, the main theoretical results currently rest on a performance-difference lemma whose proof has a dimensional inconsistency, and the implemented algorithms use an approximate risk function and finite inner-loop updates that are not covered by the stated monotonicity theorem. These issues are load-bearing for the paper's central claim, so the contribution is not yet established as written.

major comments (4)
  1. [Appendix C, Lemma 1 (Eq. 16)]
  2. [Theorem 4.3 and Algorithm 1 (also Algorithms 2–4)]
  3. [Appendix D, compactness assumption in Theorem 4.3]
  4. [Equation (9) and Lemma 1 interplay]
minor comments (5)
  1. [Equation (15)]
  2. [Algorithms 2–4]
  3. [Section 3.3]
  4. [Section 5.5 / Figure 1]
  5. [Appendix C, Eq. (17)]

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the SRM bi-level formulation is a true variational identity, and the self-citations are attribution rather than load-bearing.

full rationale

The derivation is self-contained. The central reformulation max_pi SRM_phi(G_pi) = max_pi max_h J(pi,h) is the exact supremum (convex-conjugate) representation of a spectral risk measure (Eq. 3, via Pichler), so the inner h-update is a legitimate dual block, not a parameter fitted to the quantity being predicted. The closed-form h_phi,G and its piecewise-linear approximation tilde h are derived in Eqs. (4)-(5) and Proposition 4.1 with proof in Appendix B, so the citation to the authors' earlier ICML paper is attribution rather than load-bearing; no uniqueness or convergence fact is imported from that self-citation. Theorem 4.2's policy-gradient convergence is proved with the risk-sensitive performance-difference lemma proved in Appendix C, and Theorem 4.3 is a block-coordinate-ascent argument using standard external results (Tseng 2001, Wright 2015, Zhang et al. 2021). The genuine weakness, that Algorithm 1 uses approximate tilde h and finite T_inner while Theorem 4.3 assumes exact maximization in both blocks, is a rigor/coverage gap rather than a circular reduction: the theorem's conclusion is not identical to its assumptions. Empirical evaluation is against external baselines and held-out data, so no fitted input is relabeled as a prediction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rely on the exactness of the variational SRM representation, on treating the approximate quantile-based h as exact in the algorithm, and on compactness of the parameter sublevel set. The offline theory also treats the Lagrange multiplier as a tuned hyperparameter rather than a properly optimized dual variable. No new physical entities are introduced; the extended state (s,c) is a state augmentation from prior work.

free parameters (3)
  • Lambda (Lagrange multiplier) = 1.0 for OAC-SRM; 2.5 for TD3BC-SRM
    Hyperparameter in Equations 10 and 12 that controls the strength of the behavior-policy constraint or behavior-cloning term; tuned per experiment, not derived from the optimization.
  • Risk spectrum parameters (alpha, omega) = alpha=0.2 or 0.1, omega in [0,1] in experiments
    Set by the user to select the risk preference; they define the objective and affect all empirical comparisons.
  • Number of quantiles N = 50
    Discretization level for the quantile distributional critic and the piecewise-linear approximation of h; introduces an uncontrolled approximation error in Proposition 4.1.
assumptions (4)
  • standard math The supremum representation of SRM (Equation 3) holds and is attained by h_{phi,Z} (Equation 4).
    Invoked in Section 4 to derive the bi-level objective; attributed to Pichler 2015 and not proved in the paper.
  • ad hoc to paper The quantile-based tilde h in Proposition 4.1 approximates the exact maximizer h closely enough for the convergence guarantees.
    Algorithm 1 Step 1 uses tilde h as if it were the exact argmax; Proposition 4.1 only gives an approximation ('approx') and the convergence proof in Appendix D assumes the exact h.
  • ad hoc to paper The sublevel set Theta_bar = {theta,q | J(pi_theta, h_q) >= J(pi_theta0)} is compact.
    Stated as a standard assumption in the proof of Theorem 4.3 (Appendix D) and used to extract a convergent subsequence; not derived from the parameterization.
  • domain assumption Rewards are bounded with Rmin >= 0 and phi(0) is finite.
    Used in Appendix C to bound the advantage function and in Lemma 3; stated in Section 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/J5K4MKQK

@misc{pith2026250703900,
  author       = {Pith},
  title        = {Pith review of: Risk-sensitive Actor-Critic with Static Spectral Risk Measures for Online and Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J5K4MKQK}},
  note         = {Machine review of arXiv:2507.03900}
}
read the original abstract

The development of Distributional Reinforcement Learning (DRL) has introduced a natural way to incorporate risk sensitivity into value-based and actor-critic methods by employing risk measures other than expectation in the value function. While this approach is widely adopted in many online and offline RL algorithms due to its simplicity, the naive integration of risk measures often results in suboptimal policies. This limitation can be particularly harmful in scenarios where the need for effective risk-sensitive policies is critical and worst-case outcomes carry severe consequences. To address this challenge, we propose a novel framework for optimizing static Spectral Risk Measures (SRM), a flexible family of risk measures that generalizes objectives such as CVaR and Mean-CVaR, and enables the tailoring of risk preferences. Our method is applicable to both online and offline RL algorithms. We establish theoretical guarantees by proving convergence in the finite state-action setting. Moreover, through extensive empirical evaluations, we demonstrate that our algorithms consistently outperform existing risk-sensitive methods in both online and offline environments across diverse domains.

Figures

Figures reproduced from arXiv: 2507.03900 by the authors.

Figure 1
Figure 1. The risk-sensitivity changes implicitly with Static risk measures (Left). However, Iterative risk measures have the same risk-sensitivity regardless of the observed path (Right). Optimizing static risk measures offers a more interpretable solution by focusing on policies that maximize worst-case returns. Unlike dynamic or iterative risk measures, which assume that worst-case scenarios can occur at each step, static … view at source ↗
Figure 2
Figure 2. Illustration of vari￾ous risk spectrums. where Φ −1 (u) denotes the inverse CDF of a standard Normal distribution [Wang, 1995, 2000]. Our goal in this work is to find a policy that maximizes its static risk-adjusted value. In the next section, we show that achieving this requires using the supremum representation of SRM, which applies to any SRM with a bounded spectrum: SRMϕ(Z) = sup h∈H  E [h(Z)] + Z 1 0 hˆ(ϕ(u))d… view at source ↗
Figure 3
Figure 3. (Left) Diagram of our framework. The blue connections indicating the interaction of the new policy with the environment and storing the new interaction in the dataset are not present in the offline setting. Finding the solution of the outer optimization can be interpreted as updating the actor’s perception of the initial state’s return distribution (Gπk ), which is then combined with the actor’s risk preference ϕ to… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Risk-sensitive performance comparison of online and offline RL algorithms with the [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Risk-sensitive performance comparison of TD3-Exp and TD3-iExp. [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Figures 6a and 6b display the asset returns during the training and testing period. Figure 6c [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: The value of the portfolio in the training and testing period for different policies. The [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

70 extracted references · 56 canonical work pages

  1. [1]

    Spectral measures of risk: A coherent representation of subjective risk aversion

    Carlo Acerbi. Spectral measures of risk: A coherent representation of subjective risk aversion. Journal of Banking and Finance, 26 0 (7): 0 1505--1518, 2002. doi:10.1016/S0378-4266(02)00281-9

    doi:10.1016/s0378-4266(02)00281-9 2002

  2. [2]

    Kakade, Jason D

    Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the Theory of Policy Gradient Methods : Optimality , Approximation , and Distribution Shift . Journal of Machine Learning Research, 22 0 (98): 0 1--76, 2021. URL http://jmlr.org/papers/v22/19-736.html

    work page 2021

  3. [3]

    An Optimistic Perspective on Offline Reinforcement Learning

    Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An Optimistic Perspective on Offline Reinforcement Learning . In Proceedings of the 37th International Conference on Machine Learning , pages 104--114. PMLR, 2020. URL https://proceedings.mlr.press/v119/agarwal20c.html

    work page 2020

  4. [4]

    Saud Almahdi and Steve Y. Yang. An adaptive portfolio trading system: A risk-return portfolio optimization using recurrent reinforcement learning with expected maximum drawdown. Expert Systems with Applications, 87: 0 267--279, 2017. doi:10.1016/j.eswa.2017.06.023

    work page doi:10.1016/j.eswa.2017.06.023 2017

  5. [5]

    Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning

    Chenjia Bai, Ting Xiao, Zhoufan Zhu, Lingxiao Wang, Fan Zhou, Animesh Garg, Bin He, Peng Liu, and Zhaoran Wang. Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning . IEEE Transactions on Neural Networks and Learning Systems, pages 1--15, 2022. doi:10.1109/TNNLS.2022.3217189

    work page arXiv 2022

  6. [6]

    Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap

    Gabriel Barth-Maron , Matthew W. Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed Distributional Deterministic Policy Gradients . In International Conference on Learning Representations , 2018. URL https://openreview.net/forum?id=SyZipzbCb

    work page 2018

  7. [7]

    Minimizing spectral risk measures applied to Markov decision processes

    Nicole B \"a uerle and Alexander Glauner. Minimizing spectral risk measures applied to Markov decision processes. Mathematical Methods of Operations Research, 94 0 (1): 0 35--69, 2021. doi:10.1007/s00186-021-00746-w

    work page doi:10.1007/s00186-021-00746-w 2021

  8. [8]

    Markov Decision Processes with Average-Value-at-Risk criteria

    Nicole B \"a uerle and Jonathan Ott. Markov Decision Processes with Average-Value-at-Risk criteria. Mathematical Methods of Operations Research, 74 0 (3): 0 361--379, 2011. doi:10.1007/s00186-011-0367-0

    work page doi:10.1007/s00186-011-0367-0 2011

Show all 70 references
  1. [9]

    More Risk-Sensitive Markov Decision Processes

    Nicole B \"a uerle and Ulrich Rieder. More Risk-Sensitive Markov Decision Processes . Mathematics of Operations Research, 39 0 (1): 0 105--120, 2014. doi:10.1287/moor.2013.0601

    2014

  2. [10]

    Bellemare, Will Dabney, and R \'e mi Munos

    Marc G. Bellemare, Will Dabney, and R \'e mi Munos. A Distributional Perspective on Reinforcement Learning . In Proceedings of the 34th International Conference on Machine Learning , pages 449--458. PMLR, 2017. URL https://proceedings.mlr.press/v70/bellemare17a.html

    2017

  3. [11]

    Bellemare, Will Dabney, and Mark Rowland

    Marc G. Bellemare, Will Dabney, and Mark Rowland. Distributional Reinforcement Learning . The MIT Press, 2023. ISBN 978-0-262-37402-6. doi:10.7551/mitpress/14207.001.0001

    2023 doi

  4. [12]

    Ziebart, and Marcello Restelli

    Lorenzo Bisi, Davide Santambrogio, Federico Sandrelli, Andrea Tirinzoni, Brian D. Ziebart, and Marcello Restelli. Risk-averse policy optimization via risk-neutral policy optimization. Artificial Intelligence, 311 0 (Complete), 2022. doi:10.1016/j.artint.2022.103765

    2022

  5. [13]

    JAX : Composable transformations of Python + NumPy programs, 2018

    James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne , and Qiao Zhang. JAX : Composable transformations of Python + NumPy programs, 2018. URL https://github.com/jax-ml/jax

    2018

  6. [14]

    Stochastic optimal control with dynamic, time-consistent risk constraints

    Yinlam Chow and Marco Pavone. Stochastic optimal control with dynamic, time-consistent risk constraints. In 2013 American Control Conference , pages 390--395, 2013. doi:10.1109/ACC.2013.6579868

    2013

  7. [15]

    Risk- Sensitive and Robust Decision-Making : A CVaR Optimization Approach

    Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk- Sensitive and Robust Decision-Making : A CVaR Optimization Approach . In Advances in Neural Information Processing Systems , volume 28. Curran Associates, Inc., 2015. URL https://proceedings.neurips.cc/paper/2015/ha...

    2015

  8. [16]

    Risk- Constrained Reinforcement Learning with Percentile Risk Criteria

    Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk- Constrained Reinforcement Learning with Percentile Risk Criteria . Journal of Machine Learning Research, 18 0 (167): 0 1--51, 2018. URL http://jmlr.org/papers/v18/15-636.html

    2018

  9. [17]

    Reinforcement learning with dynamic convex risk measures

    Anthony Coache and Sebastian Jaimungal. Reinforcement learning with dynamic convex risk measures. Mathematical Finance, 2023. doi:10.1111/mafi.12388

    2023 doi

  10. [18]

    Implicit Quantile Networks for Distributional Reinforcement Learning

    Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit Quantile Networks for Distributional Reinforcement Learning . In Proceedings of the 35th International Conference on Machine Learning , pages 1096--1105. PMLR, 2018 a . URL https://proceedings.mlr.press/v80/d...

    2018

  11. [19]

    Distributional Reinforcement Learning With Quantile Regression

    Will Dabney, Mark Rowland, Marc Bellemare, and R \'e mi Munos. Distributional Reinforcement Learning With Quantile Regression . Proceedings of the AAAI Conference on Artificial Intelligence, 32 0 (1), 2018 b . doi:10.1609/aaai.v32i1.11791

    2018 doi

  12. [20]

    Clinical data based optimal STI strategies for HIV : A reinforcement learning approach

    Damien Ernst, Guy-Bart Stan, Jorge Goncalves, and Louis Wehenkel. Clinical data based optimal STI strategies for HIV : A reinforcement learning approach. In Proceedings of the 45th IEEE Conference on Decision and Control , pages 667--672, 2006. doi:10.1109/CDC.2006.377527

    2006

  13. [21]

    D4RL : Datasets for Deep Data-Driven Reinforcement Learning , 2021

    Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL : Datasets for Deep Data-Driven Reinforcement Learning , 2021. Preprint

    2021

  14. [22]

    A Minimalist Approach to Offline Reinforcement Learning

    Scott Fujimoto and Shixiang Gu. A Minimalist Approach to Offline Reinforcement Learning . In Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=Q32U7dzWXpc

    2021

  15. [23]

    Addressing Function Approximation Error in Actor-Critic Methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing Function Approximation Error in Actor-Critic Methods . In Proceedings of the 35th International Conference on Machine Learning , pages 1587--1596. PMLR, 2018. URL https://proceedings.mlr.press/v80/fujimoto18a.html

    2018

  16. [24]

    Off- Policy Deep Reinforcement Learning without Exploration

    Scott Fujimoto, David Meger, and Doina Precup. Off- Policy Deep Reinforcement Learning without Exploration . In Proceedings of the 36th International Conference on Machine Learning , pages 2052--2062. PMLR, 2019. URL https://proceedings.mlr.press/v97/fujimoto19a.html

    2019

  17. [25]

    Klein, William Dabney, and Jonathan P

    Alborz Geramifard, Christoph Dann, Robert H. Klein, William Dabney, and Jonathan P. How. RLPy : A Value-Function-Based Reinforcement Learning Framework for Education and Research . Journal of Machine Learning Research, 16 0 (46): 0 1573--1578, 2015. URL http://jmlr.org/papers/...

    2015

  18. [26]

    Efficient Risk-Averse Reinforcement Learning

    Ido Greenberg, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Efficient Risk-Averse Reinforcement Learning . In Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=LdAxczs3m0

    2022

  19. [27]

    Soft Actor-Critic : Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic : Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor . In Proceedings of the 35th International Conference on Machine Learning , pages 1861--1870. PMLR, 2018. URL http...

    2018

  20. [28]

    Double Q-learning

    Hado Hasselt. Double Q-learning . In Advances in Neural Information Processing Systems , volume 23. Curran Associates, Inc., 2010. URL https://papers.nips.cc/paper_files/paper/2010/hash/091d584fced301b442654dd8c23b3fc9-Abstract.html

    2010

  21. [29]

    On a relationship between distorted and spectral risk measures

    Gzyl Henryk and Mayoral Silvia. On a relationship between distorted and spectral risk measures. 2006. URL https://mpra.ub.uni-muenchen.de/1940/

    2006

  22. [30]

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and Jo \ a o G. M. Ara \'u jo. CleanRL : High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms . Journal of Machine Learning Research, 23 0 (274): 0...

    2022

  23. [31]

    Peter J. Huber. Robust Estimation of a Location Parameter . In Samuel Kotz and Norman L. Johnson, editors, Breakthroughs in Statistics : Methodology and Distribution , pages 492--518. Springer, New York, NY, 1992. ISBN 978-1-4612-4380-9. doi:10.1007/978-1-4612-4380-9_35

    1992 doi

  24. [32]

    Approximately Optimal Approximate Reinforcement Learning

    Sham Kakade and John Langford. Approximately Optimal Approximate Reinforcement Learning . In Proceedings of the Nineteenth International Conference on Machine Learning , ICML '02, pages 267--274, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc. ISBN 978-1-55860-873-3

    2002

  25. [33]

    A Natural Policy Gradient

    Sham M Kakade. A Natural Policy Gradient . In Advances in Neural Information Processing Systems , volume 14. MIT Press, 2001. URL https://proceedings.neurips.cc/paper_files/paper/2001/hash/4b86abe48d358ecf194c56c69108433e-Abstract.html

    2001

  26. [34]

    Being Optimistic to Be Conservative : Quickly Learning a CVaR Policy

    Ramtin Keramati, Christoph Dann, Alex Tamkin, and Emma Brunskill. Being Optimistic to Be Conservative : Quickly Learning a CVaR Policy . Proceedings of the AAAI Conference on Artificial Intelligence, 34 0 (04): 0 4436--4443, 2020. doi:10.1609/aaai.v34i04.5870

    2020 doi

  27. [35]

    Spectral- Risk Safe Reinforcement Learning with Convergence Guarantees

    Dohyeong Kim, Taehyun Cho, Seungyub Han, Hojun Chung, Kyungjae Lee, and Songhwai Oh. Spectral- Risk Safe Reinforcement Learning with Convergence Guarantees . In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/foru...

    2024

  28. [36]

    Offline Reinforcement Learning with Implicit Q-Learning

    Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Learning . In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=68n2s9ZJWF8

    2021

  29. [37]

    Conservative Q-Learning for Offline Reinforcement Learning

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Offline Reinforcement Learning . In Advances in Neural Information Processing Systems , volume 33, pages 1179--1191. Curran Associates, Inc., 2020. URL https://papers.neurips.cc/paper_file...

    2020

  30. [38]

    Offline Reinforcement Learning : Tutorial , Review , and Perspectives on Open Problems , 2020

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning : Tutorial , Review , and Perspectives on Open Problems , 2020. Preprint

    2020

  31. [39]

    Distributional Reinforcement Learning for Risk-Sensitive Policies

    Shiau Hong Lim and Ilyas Malik. Distributional Reinforcement Learning for Risk-Sensitive Policies . In Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=wSVEd3Ta42m

    2022

  32. [40]

    DSAC : Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning , 2020

    Xiaoteng Ma, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. DSAC : Distributional Soft Actor Critic for Risk-Sensitive Reinforcement Learning , 2020. Preprint

    2020

  33. [41]

    Conservative Offline Distributional Reinforcement Learning

    Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative Offline Distributional Reinforcement Learning . In Advances in Neural Information Processing Systems , volume 34, pages 19235--19247. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/has...

    2021

  34. [42]

    On the Global Convergence Rates of Softmax Policy Gradient Methods

    Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the Global Convergence Rates of Softmax Policy Gradient Methods . In Proceedings of the 37th International Conference on Machine Learning , pages 6820--6829. PMLR, 2020. URL https://proceedings.mlr.press/v11...

    2020

  35. [43]

    Rusu, Joel Veness, Marc G

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, ...

    2015 doi

  36. [44]

    Beyond CVaR : Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning

    Mehrdad Moghimi and Hyejin Ku. Beyond CVaR : Leveraging Static Spectral Risk Measures for Enhanced Decision-Making in Distributional Reinforcement Learning . In Forty-Second International Conference on Machine Learning , 2025. URL https://openreview.net/forum?id=WeMpvGxXMn

    2025

  37. [45]

    Nonparametric return distribution approximation for reinforcement learning

    Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, and Toshiyuki Tanaka. Nonparametric return distribution approximation for reinforcement learning. In Proceedings of the 27th International Conference on International Conference on Machine Learning , ICML '...

    2010

  38. [46]

    AWAC : Accelerating Online Reinforcement Learning with Offline Datasets , 2021

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. AWAC : Accelerating Online Reinforcement Learning with Offline Datasets , 2021. Preprint

    2021

  39. [47]

    An intelligent financial portfolio trading strategy using deep Q-learning

    Hyungjun Park, Min Kyu Sim, and Dong Gu Choi. An intelligent financial portfolio trading strategy using deep Q-learning . Expert Systems with Applications, 158: 0 113573, 2020. doi:10.1016/j.eswa.2020.113573

    2020

  40. [48]

    Pendharkar and Patrick Cusatis

    Parag C. Pendharkar and Patrick Cusatis. Trading financial indices with reinforcement learning agents. Expert Systems with Applications, 103: 0 1--13, 2018. doi:10.1016/j.eswa.2018.02.032

    2018 doi

  41. [49]

    Advantage- Weighted Regression : Simple and Scalable Off-Policy Reinforcement Learning , 2019

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- Weighted Regression : Simple and Scalable Off-Policy Reinforcement Learning , 2019. Preprint

    2019

  42. [50]

    Pflug and Alois Pichler

    Georg Ch. Pflug and Alois Pichler. Time- Consistent Decisions and Temporal Decomposition of Coherent Risk Functional . Mathematics of Operations Research, 41 0 (2): 0 682--699, 2016. doi:10.1287/moor.2015.0747

    2016

  43. [51]

    Premiums and reserves, adjusted by distortions

    Alois Pichler. Premiums and reserves, adjusted by distortions. Scandinavian Actuarial Journal, 2015 0 (4): 0 332--351, 2015. doi:10.1080/03461238.2013.830228

    2015

  44. [52]

    Optimizing Return Distributions with Distributional Dynamic Programming , 2025

    Bernardo \'A vila Pires, Mark Rowland, Diana Borsa, Zhaohan Daniel Guo, Khimya Khetarpal, Andr \'e Barreto, David Abel, R \'e mi Munos, and Will Dabney. Optimizing Return Distributions with Distributional Dynamic Programming , 2025. Preprint

    2025

  45. [53]

    Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, and Esther Luna Colombini. A Survey on Offline Reinforcement Learning : Taxonomy , Review , and Open Problems . IEEE Transactions on Neural Networks and Learning Systems, pages 1--0, 2023. doi:10.1109/TNNLS.2023.3250269

    2023

  46. [54]

    One Risk to Rule Them All : A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning

    Marc Rigter, Bruno Lacerda, and Nick Hawes. One Risk to Rule Them All : A Risk-Sensitive Perspective on Model-Based Offline Reinforcement Learning . Advances in Neural Information Processing Systems, 36: 0 77520--77545, 2023. URL https://proceedings.neurips.cc/paper_files/pape...

    2023

  47. [55]

    Trust Region Policy Optimization

    John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust Region Policy Optimization . In Proceedings of the 32nd International Conference on Machine Learning , pages 1889--1897. PMLR, 2015. URL https://proceedings.mlr.press/v37/schulman15.html

    2015

  48. [56]

    Ruszczy \'n ski

    Alexander Shapiro, Darinka Dentcheva, and Andrzej P. Ruszczy \'n ski. Lectures on Stochastic Programming: Modeling and Theory. MOS-SIAM Series on Optimization. Society for Industrial and Applied Mathematics : Mathematical Optimization Society , Philadelphia, Pennsylvania, seco...

    2014

  49. [57]

    Deterministic Policy Gradient Algorithms

    David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic Policy Gradient Algorithms . In Proceedings of the 31st International Conference on Machine Learning , pages 387--395. PMLR, 2014. URL https://proceedings.mlr.press/v32/s...

    2014

  50. [58]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning Series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-0-262-03924-6

    2018

  51. [59]

    Policy Gradient Methods for Reinforcement Learning with Function Approximation

    Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation . In Advances in Neural Information Processing Systems , volume 12. MIT Press, 1999. URL https://papers.nips.cc/paper_files/pa...

    1999

  52. [60]

    Policy gradients with variance related risk criteria

    Aviv Tamar, Dotan Di Castro, and Shie Mannor. Policy gradients with variance related risk criteria. In Proceedings of the 29th International Coference on International Conference on Machine Learning , ICML '12, pages 1651--1658, Madison, WI, USA, 2012. Omnipress. ISBN 978-1-45...

    2012

  53. [61]

    Optimizing the CVaR via Sampling

    Aviv Tamar, Yonatan Glassner, and Shie Mannor. Optimizing the CVaR via Sampling . Proceedings of the AAAI Conference on Artificial Intelligence, 29 0 (1), 2015. doi:10.1609/aaai.v29i1.9561

    2015 doi

  54. [62]

    Sequential Decision Making With Coherent Risk

    Aviv Tamar, Yinlam Chow, Mohammad Ghavamzadeh, and Shie Mannor. Sequential Decision Making With Coherent Risk . IEEE Transactions on Automatic Control, 62 0 (7): 0 3323--3338, 2017. doi:10.1109/TAC.2016.2644871

    2017

  55. [63]

    MuJoCo : A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo : A physics engine for model-based control. In 2012 IEEE / RSJ International Conference on Intelligent Robots and Systems , pages 5026--5033, 2012. doi:10.1109/IROS.2012.6386109

    2012

  56. [64]

    P. Tseng. Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization . Journal of Optimization Theory and Applications, 109 0 (3): 0 475--494, 2001. doi:10.1023/A:1017501703105

    2001 doi

  57. [65]

    Risk- Averse Offline Reinforcement Learning

    N \'u ria Armengol Urp \'i , Sebastian Curi, and Andreas Krause. Risk- Averse Offline Reinforcement Learning . In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=TBIzh9b5eaz

    2021

  58. [66]

    Risk-sensitive policies for portfolio management

    Mingfu Wang and Hyejin Ku. Risk-sensitive policies for portfolio management. Expert Systems with Applications, 198: 0 116807, 2022. doi:10.1016/j.eswa.2022.116807

    2022

  59. [67]

    Insurance pricing and increased limits ratemaking by proportional hazards transforms

    Shaun Wang. Insurance pricing and increased limits ratemaking by proportional hazards transforms. Insurance: Mathematics and Economics, 17 0 (1): 0 43--54, 1995. doi:10.1016/0167-6687(95)00010-P

    1995 doi

  60. [68]

    Shaun S. Wang. A Class of Distortion Operators for Pricing Financial and Insurance Risks . The Journal of Risk and Insurance, 67 0 (1): 0 15--36, 2000. doi:10.2307/253675

    doi

  61. [69]

    Stephen J. Wright. Coordinate descent algorithms. Mathematical Programming, 151 0 (1): 0 3--34, 2015. doi:10.1007/s10107-015-0892-3

    2015 doi

  62. [70]

    Mean- Variance Policy Iteration for Risk-Averse Reinforcement Learning

    Shangtong Zhang, Bo Liu, and Shimon Whiteson. Mean- Variance Policy Iteration for Risk-Averse Reinforcement Learning . Proceedings of the AAAI Conference on Artificial Intelligence, 35 0 (12): 0 10905--10913, 2021. doi:10.1609/aaai.v35i12.17302

    2021 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.