Pith. sign in

REVIEW 3 major objections 4 minor 65 references

Risk-Averse Reinforcement Learning with Itakura-Saito Loss

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that minimizing the Itakura-Saito divergence loss recovers the exponential-utility Bellman equation for risk-averse value learning, giving a numerically stable alternative to existing losses.

desk verdict The IS loss is a real, clean contribution to risk-averse RL, but the proof's target-network assumption and two speculative appendices need work before this is publishable. read the letter →

arxiv 2505.16925 v2 pith:CI26NMO4 submitted 2025-05-22 cs.LG

Igor Udovichenko , Olivier Croissant , Anita Toleutaeva , Evgeny Burnaev , Alexander Korotin This is my paper · ORCID
classification cs.LG MSC 90C4068T07
keywords risk-aversereinforcementlearningexponentialutilityItakura-SaitodivergenceBregmanentropicBellmanequationtemporaldifferencevaluefunctionapproximationrisk-sensitivecontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Risk-averse agents that maximize exponential-utility certainty equivalents need value functions satisfying the entropic Bellman equation rather than the usual one. This paper proposes learning those value functions by minimizing the Itakura-Saito divergence between the exponentiated predicted value and the exponentiated bootstrapped target. The central result, Proposition 1, proves that under mild integrability conditions and a target-network copy assumption, the minimizer of this loss is exactly the correct entropic value function. Because the loss depends on the temporal-difference error rather than on the absolute level of the value, it avoids the numerical blow-ups that plague the exponential mean-squared-error loss and does not require the Gaussianity that the softplus loss needs. If true, this gives a drop-in, scale-invariant loss for risk-averse RL algorithms in finance, hedging, and other high-stakes sequential decision problems.

What carries the argument

The machinery is the Itakura-Saito divergence $d_{IS}(x,y)=x/y-\log(x/y)-1$, the Bregman divergence generated by the convex function $\phi(z)=-\log z$. Plugging the exponentiated prediction $\exp(-\alpha V_\theta(s))$ and the exponentiated target $\exp(-\alpha r-\alpha V_{\theta^-}(s'))$ into $d_{IS}$ and scaling by $\alpha^{-2}$ gives the loss $L_{IS}=\mathbb{E}[\exp(\alpha\delta)-\alpha\delta-1]$, which is convex in $\delta$ and depends on the value only through the TD error $\delta$. Its power comes from the Bregman property that the conditional expectation minimizes expected divergence, so the minimizer's exponent agrees with the conditional expectation of the exponentiated target, and the log of that equality is precisely the entropic Bellman equation.

What would settle it

In a tabular MDP with a known entropic value function, run the stochastic approximation update (13) with target values updated every N steps instead of every step; if for some finite refresh period the iterates fail to converge to the analytical solution, the online algorithm does not inherit the paper's fixed-point guarantee.

Watch

Extended reading notes

Core claim

The central claim is Proposition 1: for a value network with target network frozen as a stop-gradient copy of the main network, the minimizer of $L_{IS}(\theta)=\alpha^{-2}\mathbb{E}[\exp(\alpha\delta_\theta)-\alpha\delta_\theta-1]$, where $\delta_\theta$ is the TD error, satisfies the entropic Bellman equation (EVV). The proof writes the loss as a conditional Bregman divergence with generator $\phi(z)=-\log z$, applies the known theorem that conditional expectation is the optimal Bregman predictor, and takes logarithms to recover the certainty-equivalent update. In the tabular setting this yields the stochastic approximation update (13). The paper additionally shows empirically, on portfolio optimization, deep hedging, and a risk-averse soft actor-critic benchmark, that the IS loss converges more reliably than the exponential MSE loss and matches or beats the softplus loss, including in non-Gaussian cases where softplus is not justified.

Load-bearing premise

The proof assumes the target network is an exact copy of the current network treated as a stop-gradient, whereas practical TD learning updates a lagging target, so the online minimizer may not be the Bellman fixed point and the learned values could be biased; the exponential integrability condition must also hold.

Editorial extensions

If this is right

  • Risk-averse algorithms can replace the exponential MSE loss with the IS loss without changing the Bellman target, obtaining the correct entropic value function under the fixed-point assumption.
  • The IS loss removes the dependence on the absolute scale of the learned value, so large positive value estimates no longer cause the loss to vanish or explode the way they do for EMSE.
  • For small TD errors the IS loss behaves like the MSE loss, so it remains compatible with risk-neutral behavior in low-risk regimes.
  • The derived update rule (13) gives a simple stochastic-approximation scheme for tabular risk-averse value iteration.
  • In actor-critic settings such as RSSAC, the IS loss preserves the distribution-shift objective while eliminating the training instability caused by EMSE.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper proves only a fixed-point property; proving convergence of the online algorithm with a lagging target network is the natural open step and would determine whether the bias noted in the assumptions matters in practice.
  • The same Bregman construction could generate numerically stable losses for other dynamic risk measures whose certainty-equivalent operator has an exponential/log form, not just exponential utility.
  • Because the IS loss is scale-invariant, it may also help offline or distributional RL where value magnitudes vary widely across states and are poorly conditioned.
  • A broader empirical test would apply the loss to high-dimensional continuous-control tasks with risk aversion, going beyond the financial and grid-world benchmarks reported.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes the Itakura-Saito (IS) divergence as a loss function for learning state-value and action-value functions in risk-averse MDPs with exponential utility. Section 3 derives the loss from Bregman divergence, shows that its population minimizer satisfies the entropic Bellman equation (Proposition 1, Appendix C), and gives the corresponding stochastic approximation rule (Eq. 13). Section 4 compares IS against exponential MSE (EMSE) and softplus (SP) losses on portfolio optimization, deep hedging, and a risk-averse soft actor-critic (RSSAC) combinatorial optimization benchmark, reporting improved stability and convergence for IS. The paper positions IS as a drop-in replacement for existing risk-averse RL losses.

Significance. If the theoretical claim is taken at face value, the IS loss is a clean and elegant contribution: it is derived from a known Bregman divergence, has a mathematically transparent minimizer, and is scale-invariant. The analytical benchmarks in Section 4.1 and Appendix A are useful and the experimental protocol is reasonably detailed. The authors also explicitly build on the prior work of Murray et al. [34] and make the relationship to the entropic Bellman equation explicit. However, the central claim that the loss makes the learned value function correct is only established for an idealized synchronized target-network setting; the actual algorithms executed in Section 4 use lagging or softly updated targets, and no convergence or bias analysis is provided for that setting. The paper would be significantly strengthened by either proving convergence for the practical update or carefully limiting the correctness claim to the population fixed point.

major comments (3)
  1. [Appendix C] Proposition 1 in Appendix C assumes that the target network weights are an exact copy of the main network weights with stop-gradient (Assumption 1). The experiments in Section 4 use TD(0) with target networks and the RSSAC setup of [16] with its periodic or soft target-update schedule. In the practical setting, the loss minimized at each step uses a lagging target θ- ≠ θ, so the stationary points solve V_θ(s) = CE[r + V_θ-(s')] rather than the entropic Bellman equation (EVV). The paper does not prove convergence of the coupled system (V_θ, θ-) to the synchronized fixed point, nor does it bound the bias from finite-sample replay-buffer updates. Since the paper's central claim is that IS loss yields a correct risk-averse value function, this gap is load-bearing. I recommend either proving a convergence result for the actual online algorithm or explicitly restricting the correctness claim to the population loss and treating the experiments as evidence of stability rather than value correctness.
  2. [§4.2] The RSSAC experiments use a discounted setting (γ = 0.99), whereas Proposition 1 is stated and proved only for the undiscounted finite-horizon case with γ = 1. The Bellman equation in Eq. (19) relies on the approximation E[X^γ] ≈ E[X]^γ from [16], which is not derived or justified in the present paper. As a result, even if the IS loss were shown to converge to the fixed point of the synchronized target, that fixed point would not necessarily be the entropic value function in the discounted experiments. The empirical demonstration in Figure 4 therefore does not establish that IS recovers the correct value function in that setting; it only shows relative stability and policy performance.
  3. [Appendix D] Appendix D and E contain speculative field-theoretic and holographic reinterpretations of the IS loss, including a theorem in Appendix E whose proof is only a sketch and relies on unspecified 'known results from random matrix theory and information geometry.' These appendices are not needed for the central contribution and are presented in a way that may be misleading: the conformal symmetry discussion is appended to a paper about reinforcement learning without a rigorous connection to the main result. I recommend removing these appendices or rewriting them as clearly non-technical discussion, and removing the theorem environment unless a complete proof is supplied.
minor comments (4)
  1. [Title page] The affiliation 'Vega Institue Foundation' contains a typo; it should read 'Vega Institute Foundation.'
  2. [§3.1, Eq. (12)] The statement that 'IS loss casts the risk-sensitive Bellman criterion into a form suitable for stochastic gradient descent—circumventing the bias issues identified in past risk-sensitive Q-learning attempts [32]' is stronger than what is shown. The loss is suitable for SGD, but the bias issues in [32] concern the stochastic approximation dynamics, and the present paper does not analyze SGD bias in the practical setting. Consider softening this sentence.
  3. [Appendix A] The derivation of the quadratic-penalty solution contains a duplicated sentence fragment ('The recursion is well-defined only if ασ2 < 1' followed by 'Provided ασ2 < 1 ...'). This should be cleaned up.
  4. [§4.1, Figure 2] The caption says 'Error in learning the obtained approximation of V*' which is unclear; it should be 'Error between the learned approximation and the analytical V*.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Proposition 1 derives the entropic Bellman equation from the external Bregman/conditional-expectation theorem, and the empirical claims are benchmarked against analytical or code-reproduced baselines.

full rationale

The paper's central theoretical claim (Proposition 1, Appendix C) is not circular. The Itakura–Saito loss (IS) is defined in Eq. (IS) as an expected divergence, and the proof shows that the minimizer condition exp(-alpha V_theta*(s)) = E[exp(-alpha r - alpha V_theta*(s'))] is exactly the entropic Bellman equation (EVV). This uses Theorem 1 of Tan and Touzi [41], an external mathematical result about Bregman minimizers being conditional expectations; the assumptions (target-copy with stop-gradient, integrability, existence of the minimizer) are stated explicitly in Appendix C. The derivation does not assume the Bellman equation; it derives it from the Bregman divergence property. No load-bearing step reduces to a fitted parameter or to a self-citation: the IS loss is attributed to Murray et al. [34], an external prior work, and the present authors do not cite themselves as the source of the central premise. The empirical sections compare against analytically known solutions (Appendix A) and the code of Enders et al. [16], so the reported improvements are not manufactured by construction. The reviewer-flagged gaps—the lagging target network in actual TD/RSSAC implementations and the unstated conditions of the external theorem—are correctness/robustness limitations, not circularity; Appendix C openly assumes the synchronized target, and Appendix E.1 admits that a formal proof of the conformal-conditioning claim 'is still open.' These caveats lower evidential strength but do not make any derivation equivalent to its inputs by definition.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central derivation has no fitted constants; it relies on standard properties of Bregman divergences and an external theorem. The only hand-chosen quantities are experiment risk-aversion values and optimization hyperparameters. No new physical or mathematical entities are postulated. The main unstated load is the target-network copy assumption, which the proof explicitly assumes but practical TD violates.

free parameters (1)
  • Risk aversion coefficient alpha = Gaussian: 1; Quadratic: 100; Deep Hedging: {0.1, 0.3, 1, 3, 10}; RSSAC: 0.1
    Exogenous preference parameter of the exponential utility, chosen by the experimenter for each benchmark, not fit to data. It is not part of the loss derivation, but it is a hand-chosen value that shapes all empirical results.
assumptions (6)
  • standard math For any Bregman divergence d_phi, E[X] = argmin_y E[d_phi(X, y)]
    Invoked in Section 3.1 and used in the proof of Proposition 1 in Appendix C to assert that the conditional expectation minimizes the expected IS divergence.
  • standard math Theorem 1 of Tan and Touzi [41] applies to the IS loss and gives the conditional expectation as the minimizer
    External theorem used as the load-bearing step in the proof of Proposition 1; its conditions are not restated in the paper.
  • domain assumption Finite horizon T < infinity and discount factor gamma = 1
    Section 2.1 sets these; extension to gamma < 1 is asserted to be straightforward but not proven.
  • domain assumption Existence of exponential moments E[exp(-alpha r - alpha V)] and first moment E[r + V]
    Assumption 2 in Appendix C; required for the Bregman minimizer property to hold.
  • ad hoc to paper Target network is a copy of the main network with stop-gradient
    Assumption 1 in Appendix C; needed to turn the loss minimizer into the Bellman fixed point, but not how target networks are updated in practice.
  • domain assumption Bachelier dynamics: price increments are iid Gaussian, and in the quadratic case alpha sigma^2 < 1
    Used in Section 4.1 and Appendix A to derive closed-form optimal value functions; restricts the scope of the analytical benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Risk-Averse Reinforcement Learning with Itakura-Saito Loss." pith.science (2026). https://pith.science/paper/CI26NMO4

@misc{pith2026250516925,
  author       = {Pith},
  title        = {Pith review of: Risk-Averse Reinforcement Learning with Itakura-Saito Loss},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CI26NMO4}},
  note         = {Machine review of arXiv:2505.16925}
}
read the original abstract

Risk-averse reinforcement learning finds application in various high-stakes fields. Unlike classical reinforcement learning, which aims to maximize expected returns, risk-averse agents choose policies that minimize risk, occasionally sacrificing expected value. These preferences can be framed through utility theory. We focus on the specific case of the exponential utility function, where one can derive the Bellman equations and employ various reinforcement learning algorithms with few modifications. To address this, we introduce to the broad machine learning community a numerically stable and mathematically sound loss function based on the Itakura-Saito divergence for learning state-value and action-value functions. We evaluate the Itakura-Saito loss function against established alternatives, both theoretically and empirically. In the experimental section, we explore multiple scenarios, some with known analytical solutions, and show that the considered loss function outperforms the alternatives.

Figures

Figures reproduced from arXiv: 2505.16925 by the authors.

Figure 1
Figure 1. Comparison of loss penalties for a one-step value prediction error [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Error in learning the obtained approximation of [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Loss performance on the Deep Hedging problem [10]. Loss [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Loss performance on the RSSAC problem [16]. Learning curves depict the mean validation [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 55 canonical work pages

  1. [34]

    Deep hedging: Continuous reinforcement learning for hedging of general portfolios across multiple risk aversions

    Phillip Murray et al. “Deep hedging: Continuous reinforcement learning for hedging of general portfolios across multiple risk aversions”. In: Proceedings of the Third ACM International Conference on AI in Finance. 2022, pp. 361–368

    work page 2022

  2. [16]

    Risk-sensitive soft actor-critic for robust deep reinforcement learning under distribution shifts

    Tobias Enders, James Harrison, and Maximilian Schiffer. “Risk-sensitive soft actor-critic for robust deep reinforcement learning under distribution shifts”. In: arXiv preprint arXiv:2402.09992 (2024)

    arXiv 2024

  3. [1]

    Théorie de la spéculation

    Louis Bachelier. “Théorie de la spéculation”. In: Annales scientifiques de l’École normale supérieure. V ol. 17. 1900, pp. 21–86

    work page 1900

  4. [2]

    On the optimality of conditional expectation as a Bregman predictor

    Arindam Banerjee, Xin Guo, and Hui Wang. “On the optimality of conditional expectation as a Bregman predictor”. In: IEEE Transactions on Information Theory 51.7 (2005), pp. 2664– 2669

    work page 2005

  5. [3]

    Clustering with Bregman divergences

    Arindam Banerjee et al. “Clustering with Bregman divergences”. In: Journal of machine learning research 6.Oct (2005), pp. 1705–1749

    work page 2005

  6. [4]

    Commentarii academiae scientiarum imperialis petropolitanae

    Daniel Bernoulli. “Commentarii academiae scientiarum imperialis petropolitanae”. In: Petropoli. Chap. De vibrationibus et sono laminarum elasticarum27 (1751), p. 28

    work page

  7. [5]

    A sensitivity formula for risk-sensitive cost and the actor–critic algorithm

    Vivek S Borkar. “A sensitivity formula for risk-sensitive cost and the actor–critic algorithm”. In: Systems & Control Letters 44.5 (2001), pp. 339–346

    work page 2001

  8. [6]

    Q-learning for risk-sensitive control

    Vivek S Borkar. “Q-learning for risk-sensitive control”. In:Mathematics of operations research 27.2 (2002), pp. 294–311

    work page 2002

Show all 65 references
  1. [7]

    Risk-sensitive optimal control for Markov decision processes with monotone cost

    Vivek S Borkar and Sean P Meyn. “Risk-sensitive optimal control for Markov decision processes with monotone cost”. In: Mathematics of Operations Research 27.1 (2002), pp. 192– 209

    2002

  2. [8]

    The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming

    Lev M Bregman. “The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming”. In: USSR computational mathematics and mathematical physics 7.3 (1967), pp. 200–217

    1967

  3. [9]

    Deep bellman hedging

    Hans Buehler, Phillip Murray, and Ben Wood. “Deep bellman hedging”. In: arXiv preprint arXiv:2207.00932 (2022)

    2022 arXiv

  4. [10]

    Deep hedging

    Hans Buehler et al. “Deep hedging”. In: Quantitative Finance 19.8 (2019), pp. 1271–1291

    2019

  5. [11]

    Deep hedging: learning to remove the drift under trading frictions with minimal equivalent near-martingale measures

    Hans Buehler et al. “Deep hedging: learning to remove the drift under trading frictions with minimal equivalent near-martingale measures”. In: arXiv preprint arXiv:2111.07844 (2021)

    2021 arXiv

  6. [12]

    Deep Hedging of Derivatives Using Reinforcement Learning

    Jay Cao et al. “Deep Hedging of Derivatives Using Reinforcement Learning”. In: The Journal of Financial Data Science 3.1 (2021), pp. 10–27

    2021

  7. [13]

    Advances in industrial engineering and operations research

    Alan HS Chan. Advances in industrial engineering and operations research. V ol. 5. Springer Science & Business Media, 2008

    2008

  8. [14]

    A Black–Scholes user’s guide to the Bachelier model

    Jaehyuk Choi et al. “A Black–Scholes user’s guide to the Bachelier model”. In: Journal of Futures Markets 42.5 (2022), pp. 959–980

    2022

  9. [15]

    Model-free risk-sensitive reinforcement learning

    Grégoire Delétang et al. “Model-free risk-sensitive reinforcement learning”. In: arXiv preprint arXiv:2111.02907 (2021)

    2021 arXiv

  10. [17]

    Cascaded gaps: Towards logarithmic regret for risk-sensitive reinforcement learning

    Yingjie Fei and Ruitu Xu. “Cascaded gaps: Towards logarithmic regret for risk-sensitive reinforcement learning”. In: International Conference on Machine Learning. PMLR. 2022, pp. 6392–6417

    2022

  11. [18]

    Risk-sensitive reinforcement learning with function approximation: A debiasing approach

    Yingjie Fei, Zhuoran Yang, and Zhaoran Wang. “Risk-sensitive reinforcement learning with function approximation: A debiasing approach”. In: International Conference on Machine Learning. PMLR. 2021, pp. 3198–3207

    2021

  12. [19]

    Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning

    Yingjie Fei et al. “Exponential bellman equation and improved regret bounds for risk-sensitive reinforcement learning”. In: Advances in neural information processing systems 34 (2021), pp. 20436–20446

    2021

  13. [20]

    Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret

    Yingjie Fei et al. “Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret”. In: Advances in Neural Information Processing Systems 33 (2020), pp. 22384–22395

    2020

  14. [21]

    Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis

    Cédric Févotte, Nancy Bertin, and Jean-Louis Durrieu. “Nonnegative matrix factorization with the Itakura-Saito divergence: With application to music analysis”. In:Neural computation 21.3 (2009), pp. 793–830

    2009

  15. [22]

    Stochastic finance: an introduction in discrete time

    Hans Föllmer and Alexander Schied. Stochastic finance: an introduction in discrete time . Walter de Gruyter, 2011

    2011

  16. [23]

    Risk-Sensitive Variational Actor- Critic: A Model-Based Approach

    Alonso Granados, Reza Ebrahimi, and Jason Pacheco. “Risk-Sensitive Variational Actor- Critic: A Model-Based Approach”. In: The Thirteenth International Conference on Learning Representations. 2025. 9

    2025

  17. [24]

    QLBS: Q-Learner in the Black-Scholes (-Merton) Worlds

    Igor Halperin. “QLBS: Q-Learner in the Black-Scholes (-Merton) Worlds”. In: Journal of Derivatives 28.1 (2020), pp. 99–122

    2020

  18. [25]

    Recent advances in reinforcement learning in finance

    Ben Hambly, Renyuan Xu, and Huining Yang. “Recent advances in reinforcement learning in finance”. In: Mathematical Finance 33.3 (2023), pp. 437–503

    2023

  19. [26]

    Entropic risk optimization in discounted MDPs

    Jia Lin Hau, Marek Petrik, and Mohammad Ghavamzadeh. “Entropic risk optimization in discounted MDPs”. In: International Conference on Artificial Intelligence and Statistics . PMLR. 2023, pp. 47–76

    2023

  20. [27]

    Risk-sensitive Markov decision processes

    Ronald A Howard and James E Matheson. “Risk-sensitive Markov decision processes”. In: Management science 18.7 (1972), pp. 356–369

    1972

  21. [28]

    Analysis synthesis telephony based on the maximum likelihood method

    Fumitada Itakura. “Analysis synthesis telephony based on the maximum likelihood method”. In: Reports of the 6th Int. Cong. Acoust., 1968 (1968)

    1968

  22. [29]

    Dynamic replication and hedging: A reinforcement learning approach

    Petter N Kolm and Gordon Ritter. “Dynamic replication and hedging: A reinforcement learning approach”. In: The Journal of Financial Data Science 1.1 (2019), pp. 159–171

    2019

  23. [30]

    Modern perspectives on reinforcement learning in finance

    Petter N Kolm and Gordon Ritter. “Modern perspectives on reinforcement learning in finance”. In: Modern Perspectives on Reinforcement Learning in Finance (September 6, 2019)(2019)

    2019

  24. [31]

    Deep Reinforcement Learning

    Yuxi Li. Deep Reinforcement Learning. 2018. arXiv: 1810.06339 [cs.LG] . URL: https: //arxiv.org/abs/1810.06339

    2018 arXiv

  25. [32]

    Risk-sensitive reinforcement learning

    Oliver Mihatsch and Ralph Neuneier. “Risk-sensitive reinforcement learning”. In:Machine learning 49 (2002), pp. 267–290

    2002

  26. [33]

    A policy gradient algorithm for the risk-sensitive exponential cost mdp

    Mehrdad Moharrami et al. “A policy gradient algorithm for the risk-sensitive exponential cost mdp”. In: Mathematics of Operations Research 50.1 (2025), pp. 431–458

    2025

  27. [35]

    Entropic risk measure in policy search

    David Nass, Boris Belousov, and Jan Peters. “Entropic risk measure in policy search”. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE. 2019, pp. 1101–1106

    2019

  28. [36]

    Exponential TD Learning: A Risk-Sensitive Actor-Critic Reinforcement Learning Algorithm

    Erfaun Noorani, Christos N Mavridis, and John S Baras. “Exponential TD Learning: A Risk-Sensitive Actor-Critic Reinforcement Learning Algorithm”. In: 2023 American Control Conference (ACC). IEEE. 2023, pp. 4104–4109

    2023

  29. [37]

    Dynamic convex risk measures: time consistency, prudence, and sustainability

    Irina Penner. “Dynamic convex risk measures: time consistency, prudence, and sustainability”. In: Humboldt-Universität zu Berlin (2007)

    2007

  30. [38]

    A stochastic approximation method

    Herbert Robbins and Sutton Monro. “A stochastic approximation method”. In: The annals of mathematical statistics (1951), pp. 400–407

    1951

  31. [39]

    Applying Reinforcement Learning to Option Pricing and Hedging

    Zoran Stoiljkovic. “Applying Reinforcement Learning to Option Pricing and Hedging”. In: arXiv preprint arXiv:2310.04336 (2023)

    2023 arXiv

  32. [40]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018

    2018

  33. [41]

    Optimal transportation under controlled stochastic dynamics

    Xiaolu Tan and Nizar Touzi. “Optimal transportation under controlled stochastic dynamics”. In: The annals of probability (2013), pp. 3201–3240

    2013

  34. [42]

    Theory of games and economic behavior, 2nd rev

    John V on Neumann and Oskar Morgenstern. “Theory of games and economic behavior, 2nd rev”. In: (1947). 10 A Analytical Solutions We provide here the closed-form derivations for the ground-truth benchmarks reported in the exper- iments (cf. Section 4 in the main text). All resu...

    1947

  35. [43]

    We took β = −0.1, which is α = 0.1 in our notations

  36. [44]

    The discount factor is γ ∈ {0.99, 1}

  37. [45]

    Every RSSAC run (500 k environment steps) takes ≈ 2 h on a single A100 GPU

    The loss function is simpler than in [16] and do not involve computing the expectation over the next actions. Every RSSAC run (500 k environment steps) takes ≈ 2 h on a single A100 GPU. No concurrent jobs are scheduled on the same device. 1https://github.com/tumBAIS/RiskSensit...

  38. [46]

    The target network weights θ− are a copy of the main network weights θ, V π θ− (s′) = stop gradient V π θ (s′) in (21)

  39. [47]

    Both E h exp −αr(s, a, s′) − α ˜V π θ− (s′) i and E h r(s, a, s′) + ˜V π θ− (s′) i exist

  40. [48]

    bulk-like

    LIS attains its minimum at θ⋆. Then, ˜V π θ∗ satisfies the risk–averse Bellman equation (EVV): ˜V π(s) = ˜Eα a,s′ h r(s, a, s′) + ˜V π(s′) i . (22) Proof. First, note that LIS(θ) = α−2Es,a,s′ h dIS exp n −αr(s, a, s′) − α ˜V π θ (s′) o , exp n −α ˜V π θ (s) o i , where α−2 is ...

  41. [49]

    Natural gradient works efficiently in learning

    Shun-Ichi Amari. “Natural gradient works efficiently in learning”. In: Neural computation 10.2 (1998), pp. 251–276. 17

    1998

  42. [50]

    Reconciling modern machine learning and the bias-variance trade-off

    Mikhail Belkin et al. “Reconciling modern machine learning and the bias-variance trade-off”. In: Proceedings of the National Academy of Sciences 116.32 (2019), pp. 15849–15854

    2019

  43. [51]

    Scaling and renormalization in statistical physics

    John Cardy. Scaling and renormalization in statistical physics. V ol. 5. Cambridge university press, 1996

    1996

  44. [52]

    Conformal field theory

    Philippe Francesco, Pierre Mathieu, and David Sénéchal. Conformal field theory. Springer Science & Business Media, 2012

    2012

  45. [53]

    Conformal Invariance and Critical Phenomena

    Malte Henkel. Conformal Invariance and Critical Phenomena. Springer, 1999. DOI: 10.1007/ 978-3-662-03937-3

    1999

  46. [54]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clément Hongler. “Neural tangent kernel: Convergence and generalization in neural networks”. In: NeurIPS (2018)

    2018

  47. [55]

    A random matrix approach to neural networks

    Camille Louart, Zhenyu Liao, and Romain Couillet. “A random matrix approach to neural networks”. In: Annals of Applied Probability 28.2 (2018), pp. 1190–1248

    2018

  48. [56]

    The large-N limit of superconformal field theories and supergravity

    Juan Maldacena. “The large-N limit of superconformal field theories and supergravity”. In: International journal of theoretical physics 38.4 (1999), pp. 1113–1133

    1999

  49. [57]

    Elements of Phase Transitions and Critical Phenom- ena

    Hidetoshi Nishimori and Gerardo Ortiz. Elements of Phase Transitions and Critical Phenom- ena. Oxford University Press, 2010. ISBN : 9780199577224

    2010

  50. [58]

    The emergence of spectral universality in deep net- works

    Jeffrey Pennington and Pratik Worah. “The emergence of spectral universality in deep net- works”. In: Proceedings of the 21st International Conference on Artificial Intelligence and Statistics (AISTATS). 2018, pp. 1924–1932

    2018

  51. [59]

    An Introduction to quantum field theory

    Michael E Peskin. An Introduction to quantum field theory. CRC press, 2018

    2018

  52. [60]

    The Conformal Bootstrap: Theory, Numerical Techniques, and Applications

    David Poland, Slava Rychkov, and Alessandro Vichi. The Conformal Bootstrap: Theory, Numerical Techniques, and Applications . arXiv preprint arXiv:1805.04405. 2018. arXiv: 1805.04405 [hep-th]

    2018 arXiv

  53. [61]

    Eigenvalues of the hessian in deep learning: Singularity and beyond

    Levent Sagun, Leon Bottou, and Yann LeCun. “Eigenvalues of the hessian in deep learning: Singularity and beyond”. In: arXiv preprint arXiv:1611.07476 (2016)

    2016 arXiv

  54. [62]

    Hessian-based analysis of large batch training and robustness to adversaries

    Zhewei Yao et al. “Hessian-based analysis of large batch training and robustness to adversaries”. In: Advances in Neural Information Processing Systems 31 (2018)

    2018

  55. [63]

    Emergent Symmetry in Quantum Phase Transitions

    You Zhou et al. “Emergent Symmetry in Quantum Phase Transitions”. In: Fundamental Research 3.4 (2023), pp. 27–38. DOI: 10.1016/j.fmre.2023.02.003

    2023 doi

  56. [64]

    Uncovering conformal symmetry in the 3D Ising transition: state-operator correspondence from a quantum fuzzy sphere regularization

    Wei Zhu et al. “Uncovering conformal symmetry in the 3D Ising transition: state-operator correspondence from a quantum fuzzy sphere regularization”. In: Physical Review X 13.2 (2023), p. 021009

    2023

  57. [65]

    Quantum field theory and critical phenomena

    Jean Zinn-Justin. Quantum field theory and critical phenomena. V ol. 171. Oxford university press, 2021. 18

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.