Pith. sign in

REVIEW 3 major objections 6 minor 96 references

Robust General Utility for Reinforcement Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes robust general-utility RL, a minimax formulation that trains policies against misspecification of the utility functional, and proves convergence of two stochastic algorithms in concave and nonconcave regimes.

desk verdict A useful minimax reframing of robust utility RL with plausible convergence theory, but the sign error in Proposition 1 and an unverified Minty VI requirement for the nonconcave regime need fixing before I'd rely on the unification claims. read the letter →

arxiv 2608.03562 v1 pith:CU6S6HZH submitted 2026-08-04 cs.LG

Zixuan Liu , Fangzheng Wu , Brian Summa , Zizhan Zheng This is my paper · ORCID
classification cs.LG
keywords robustreinforcementlearninggeneralutilityminimaxoptimizationmisspecificationnonconvex-nonconcaveproximalpointmethodstochasticextragradientMintyvariationalinequality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces robust general-utility reinforcement learning, framed as the minimax problem $\min_{\theta\in\Theta}\max_{\xi\in\Xi} f_\xi(\lambda_\theta)$, where $\lambda_\theta$ is the occupancy measure of policy $\theta$ and $\Xi$ is a prescribed uncertainty set of utility parameters. The motivation is that utility functions used at deployment often deviate from those used in training, a gap prior general-utility RL did not address. The paper shows this formulation strictly generalizes standard general-utility RL (recovered when $\Xi$ is a singleton) and unifies reward-robust RL and constrained RL as linear special cases. It then provides two provably convergent stochastic algorithms: a projected stochastic gradient descent-ascent method for strongly concave utilities, and a stochastic prox-extragradient method for nonconcave utilities, with stationarity guarantees at rates $O(1/\sqrt{K})$ and $O(\log K/K)$ respectively. Experiments on LLM safety alignment and exploration maximization corroborate the convergence behavior.

What carries the argument

The load-bearing object in the concave regime is the robust envelope $\Gamma(\theta):=\max_{\xi\in\Xi} f_\xi(\lambda_\theta)$, together with its differentiability and smoothness, obtained by Danskin's theorem when $f$ is strongly concave in $\xi$. In the nonconcave regime the load-bearing object is the prox-regularized subproblem $\Phi_k(\theta,\xi)=f_\xi(\lambda_\theta)+\frac{\sigma}{2}\|\theta-\theta_k\|^2-\frac{\sigma}{2}\|\xi-\xi_k\|^2$, whose saddle point is characterized by a variational inequality with a $(\sigma-L_F)$-strongly monotone operator $F_k(z)=F(z)+\sigma(z-z_k)$. The gap function $\mathrm{Gap}_k(z)=\max_{z'}\langle F_k(z),z-z'\rangle$ measures subproblem accuracy, and the Minty variational inequality (Assumption 8) keeps successive anchors from cycling. The stochastic extragradient update (prediction at a lookahead point, then correction) damps the rotational behavior induced by nonconcavity.

What would settle it

Run Algorithm 2 on the paper's exploration-maximization example and compute the residual $R_{\mathrm{GM}}(\theta,\xi)$ at the returned iterate across many seeds; if the residual does not converge to zero, or if the iterates cycle, the example would be a counterexample to the applicability of Theorem 3, indicating that Assumption 8 fails for that instance. More directly, numerically check Assumption 8 by testing whether $\langle\nabla_\theta f_\xi(\lambda_\theta),\theta-\theta_\star\rangle - \langle\nabla_\xi f_\xi(\lambda_\theta),\xi-\xi_\star\rangle \ge 0$ holds at every feasible point of the single feature-drift instance; if it fails at even one point, the condition is violated.

Watch

Extended reading notes

Core claim

The central claim is that robustness to utility misspecification is naturally modeled by the minimax objective $\min_{\theta\in\Theta}\max_{\xi\in\Xi} f_\xi(\lambda_\theta)$, and that this objective can be solved provably in two regimes. Under strong concavity in $\xi$, the robust envelope $\Gamma(\theta)=\max_{\xi\in\Xi} f_\xi(\lambda_\theta)$ is differentiable with Lipschitz gradient, so the projected stochastic gradient descent-ascent method (Algorithm 1) converges to a first-order stationary point of the envelope at rate $O(1/\sqrt{K})$ (Theorem 2). In the nonconcave regime, where the envelope may lose differentiability and plain ascent can cycle, the paper solves a sequence of prox-regularized subproblems $f_\xi(\lambda_\theta)+\frac{\sigma}{2}\|\theta-\theta_k\|^2-\frac{\sigma}{2}\|\xi-\xi_k\|^2$, using stochastic extragradient updates; under a Minty variational inequality condition (Assumption 8), Algorithm 2 converges to a first-order stationary point of the original problem at rate $O(\log K/K)$ (Theorem 3). The framework also provides a unified view of reward-robust RL and constrained RL as linear special cases, and exhibits a nonconcave exploration-maximization example motivated by feature drift.

Load-bearing premise

In the nonconcave regime, the proof assumes the existence of a global Minty variational-inequality solution — a single point $z_\star$ toward which the entire joint gradient field aligns — and the paper does not verify this condition for its own exploration-maximization example.

Editorial extensions

If this is right

  • When $\Xi$ is a singleton, the objective reduces to standard general-utility RL, so the framework is a strict generalization of that setting.
  • With $f_\xi(\lambda)=-\langle\xi,\lambda\rangle$ the reward-robust RL objective is recovered; with $\xi$ as Lagrange multipliers and $\Xi$ containing an optimal multiplier, constrained RL is recovered, so both prior paradigms become instances of the same minimax formulation.
  • In the concave regime, Theorem 2 implies that a single-loop stochastic descent-ascent update on $\theta$ and $\xi$ drives the projected gradient mapping of the robust envelope to zero at rate $O(1/\sqrt{K})$.
  • In the nonconcave regime, Theorem 3 implies that a uniformly sampled iterate of Algorithm 2 has expected gradient-mapping residual going to zero at rate $O(\log K/K)$, provided the Minty variational inequality holds.
  • The exploration-maximization example shows that robustness to feature drift (a quadratic map $W_\xi M(\lambda)W_\xi^\top$ composed with $-\sigma_{\min}$) is generally nonconcave in $\xi$, so the nonconcave algorithm addresses a practically motivated case rather than a purely theoretical one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not analyze the combined case where both transition dynamics and utility parameters are uncertain; a natural extension is to define the uncertainty set over $(p,\xi)$ jointly, though the current theory would need new Lipschitz and strong-concavity conditions.
  • Because Assumption 8 is not verified for the feature-drift example, a numerical check of the Minty inequality on that example would settle whether Theorem 3 actually applies to the paper's own motivating nonconcave instance; if it fails, the prox-extragradient guarantee would need a weaker condition.
  • The framework's view of uncertainty sets suggests a practical recipe for reward hacking: choose $\Xi$ as a ball of plausible evaluation utilities around the training utility; then the worst-case training objective is a direct defense against correlated proxies, and benchmark tasks with known proxy shifts could test the benefit.
  • The gap-based proof structure for Algorithm 2 leaves room for accelerating the inner variational-inequality solver; the paper does not claim an optimal rate, so an improved inner iteration count could potentially lower the $O(\log K/K)$ outer rate.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes robust general-utility reinforcement learning, formulated as min_theta max_{xi in Xi} f_xi(lambda_theta), where lambda_theta is the occupancy measure of policy theta and Xi is an ambiguity set over utility parameters. It argues that this strictly generalizes general-utility RL and recovers reward-robust RL and constrained RL as special cases. For utilities that are convex in lambda and strongly concave in xi, the paper develops a projected stochastic gradient descent-ascent method (Algorithm 1) and proves convergence to a first-order stationary point of the robust envelope Gamma(theta) at rate O(1/sqrt(K)). For nonconcave utilities, it develops a stochastic prox-extragradient method (Algorithm 2) and proves convergence to a first-order stationary point of the original problem at rate O(log K / K) under a Minty variational inequality assumption. Experiments on an LLM safety-alignment task and a tabular exploration-maximization task are reported as corroborating evidence.

Significance. If the results hold as stated, the framework is a natural and useful extension of general-utility RL, and the algorithmic contributions for the nonconvex-concave and nonconvex-nonconcave regimes are valuable. The paper's strengths include a detailed, mostly self-contained appendix: the policy gradient theorem, finite-sample stochastic gradient estimates, envelope smoothness, and the prox-extragradient variational-inequality analysis are all developed with explicit constants. The theorems are conditional on clearly stated assumptions and do not appear to fit constants to data. The main weaknesses are (i) a false sign identity in the formal claim that reward-robust RL is a special case, (ii) an unverified Minty variational inequality for the paper's own motivating nonconcave exploration-maximization example, and (iii) a mismatch between the stated 'concave utility' regime (strong concavity in xi) and the linear-in-xi examples used to advertise the unified view. These issues are fixable, but they need to be addressed before publication.

major comments (3)
  1. [Section 3.2, Proposition 1; Appendix B.1] Appendix B.1 contains the identity min_theta max_R -<R, lambda_theta> = max_theta min_R <R, lambda_theta>. This is false; the correct identity is min_theta max_R -<R, lambda_theta> = -max_theta min_R <R, lambda_theta>. With f_xi(lambda_theta) = -<xi, lambda_theta>, problem (2) is the negative of the reward-robust objective in (3), not the objective itself. The optimal policy sets coincide, so the unification claim can be repaired by stating equivalence up to negation, but Proposition 1 and the printed equality in Appendix B.1 are wrong as written.
  2. [Sections 5.1-5.2, Theorem 3] Theorem 3's nonconcave convergence guarantee is conditional on Assumption 8 (the Minty variational inequality), but the exploration-maximization utility in Section 5.1, f_xi(lambda) = -sigma_min(W_xi M(lambda) W_xi^T), is never shown to satisfy that assumption. Because -sigma_min is convex in the positive semidefinite argument while W_xi M(lambda) W_xi^T is quadratic in W_xi, the saddle operator F is not generally monotone, and the global alignment condition <F(z), z - z*> >= 0 for all z is a stringent requirement that will typically fail for a generic feature map. Without either a proof that a structured subclass of this example satisfies Assumption 8 or a numerical verification of the condition on the actual instance used in Section 6, the claim that Algorithm 2 provides provable convergence for the paper's motivating nonconcave example is unsupported.
  3. [Section 4, Assumption 6; Section 3.2] Assumption 6 requires f_xi(lambda) to be strongly concave in xi, but the unifying examples in Section 3.2 (reward-robust RL and the Lagrangian view of constrained RL) are linear in xi and therefore are not covered by Theorem 2 as stated. The text refers to this as the 'concave utility' regime and says it subsumes the examples in Section 3.2, but strong concavity is strictly stronger than concavity. The paper should either explicitly state that the unifying examples are outside the analyzed algorithmic regime, or provide a regularization or limiting argument explaining how the linear case can be recovered.
minor comments (6)
  1. [Abstract and Introduction] There are typographical errors such as 'proposestochastic' and 'hyperparemters' that should be corrected in a copyedit pass.
  2. [Theorem 2, Remark] In the remark after Theorem 2, the expression E||nabla_theta G_Gamma(theta_k)||^2 should be E||G_Gamma(theta_k)||^2; the projected gradient mapping is already a first-order stationarity measure.
  3. [Theorem 4, Appendix B.3] The final bound in Theorem 4 appears to contain a duplicated factor L_{theta,xi}^2; please check whether the intended coefficient is L_{theta,xi}^2 L_xi / (mu_xi^3), consistent with the bias bound from Lemma 5, rather than L_{theta,xi}^4 / mu_xi^3.
  4. [Section 6, Algorithm 2 experiment] The experiments for Algorithm 2 report the objective f_xi(lambda_theta) and the proximal subproblem gradient norms ||G^k_{Theta,sigma}|| and ||G^k_{Xi,sigma}||, but Theorem 3 bounds the original gradient-mapping residual R_GM at a uniformly sampled iterate. These proximal residuals can decrease even if the anchor drift cycles, so the reported metrics do not directly corroborate the theorem's conclusion.
  5. [Appendix C, Figure 1] The text states that results are averaged over five random seeds, but Figure 1 appears to show only mean curves; per-seed curves or error bars should be included so the reader can assess variability.
  6. [Theorem 3, statement and proof] Theorem 3 should explicitly list the choices of T_k and the Monte-Carlo budgets that appear in Proposition 6 and Corollary 1, rather than referring only to Appendix B.16, so that the stated O(log K / K) rate is self-contained.

Circularity Check

0 steps flagged · score 1.0 of 10

Convergence proofs are self-contained under explicitly stated assumptions; the only self-citation is incidental and not load-bearing.

full rationale

The central derivation chain is the minimax formulation min_theta max_xi f_xi(lambda_theta), the projected gradient descent-ascent analysis for the concave regime, and the prox-extragradient analysis for the nonconcave regime. The formulation itself is defined independently of any result it claims to predict, and the claimed special cases are actual reductions: Proposition 1 shows reward-robust RL is the linear special case f_xi(lambda_theta) = -<xi, lambda_theta>, and Proposition 2 shows constrained RL is recovered through a standard Lagrangian equivalence under Slater's condition. Neither reduces a prediction to a fitted input. Algorithm 1's convergence proof builds Proposition 5 (differentiability and smoothness of the envelope Gamma(theta) = max_xi f_xi(lambda_theta)) from Assumptions 1-6 via Danskin's theorem and Lipschitz estimates, and Theorem 2 derives the O(1/sqrt(K)) stationarity bound for Gamma rather than assuming it. Algorithm 2's convergence proof explicitly relies on Assumption 8 (Minty variational inequality) to control the anchor-drift term in Eq. (55); this is a stated premise, not a hidden equivalent of the conclusion. The O(log K/K) bound in Theorem 3 follows from that premise and the strong monotonicity of the prox-regularized operator, and no constant is fitted to data. The paper does cite its own prior work, Liu, Sun, and Zheng 2025, but only as one entry in a list of reward-robust RL references in the introduction; it is not used to justify the convergence theorems, the envelope regularity, or the MVI assumption. The fact that the exploration-maximization example is not verified to satisfy Assumption 8 is a legitimate applicability concern, but it is not circularity: the theorem openly states the condition under which its rate holds. Overall, the derivations are self-contained given standard compactness, smoothness, strong-concavity, and Minty-VI assumptions, and no load-bearing reduction to the paper's own inputs was found.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No constants are fitted to data in the theoretical results. The user-specified quantities such as uncertainty radius d0, proximal weight σ, and step sizes are design choices, not free parameters tuned to make proofs work. No new physical or mathematical entities are postulated; the uncertainty parameter ξ is an existing parameterization of the utility function.

assumptions (6)
  • domain assumption Θ and Ξ are convex and compact (Assumption 1)
    Used throughout to ensure projections, maxima, and diameters exist; standard in robust RL.
  • domain assumption f_ξ is µ_ξ-strongly concave in ξ and convex in λ (Assumption 6) for Algorithm 1
    Assumption 6 is needed for Proposition 5's envelope differentiability and Theorem 2's rate; it excludes nonconcave utility misspecification.
  • domain assumption Minty variational inequality holds (Assumption 8) for Algorithm 2
    Theorem 3's O(log K/K) convergence relies on the existence of z⋆ with nonnegative alignment of the saddle operator; the paper does not prove it for its own exploration example.
  • domain assumption Slater's condition for constrained RL equivalence (Proposition 2)
    Used to bound optimal Lagrange multipliers; without Slater, the Lagrangian equivalence can fail.
  • standard math Danskin's theorem for differentiating the envelope
    Invoked in Proposition 5 to get ∇Γ(θ) = ∇θ f_{ξ*(θ)}(λθ); requires the max set to be nonempty and the function continuous.
  • domain assumption Lipschitz and smoothness of f_ξ and policy parameterization (Assumptions 2-5)
    These boundedness conditions underlie all Lipschitz constants and stochastic gradient error bounds in Propositions 3, 4, and 8.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Robust General Utility for Reinforcement Learning." pith.science (2026). https://pith.science/paper/CU6S6HZH

@misc{pith2026260803562,
  author       = {Pith},
  title        = {Pith review of: Robust General Utility for Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CU6S6HZH}},
  note         = {Machine review of arXiv:2608.03562}
}
read the original abstract

Reinforcement learning (RL) with general utility extends classic RL by optimizing an arbitrary utility functional of the policy-induced occupancy measure, thereby enabling a broader range of applications. However, previous work on general utility RL typically assumes the evaluation utility is fixed and correctly specified. In practice, the utility used at deployment can deviate from the training one, creating a robustness gap that prior work does not address. Motivated by this, we propose robust general-utility RL, a minimax learning framework that trains policies against utility misspecification within a prescribed uncertainty set. Our framework strictly generalizes standard general-utility RL while also providing a unified view of many existing RL frameworks, including reward-robust RL and constrained RL, through appropriate choices of the utility uncertainty set. We further develop provably convergent stochastic algorithms for two regimes. For concave utilities, we develop a projected stochastic gradient descent-ascent method and establish stationarity guarantees. For the more challenging nonconcave regime, we propose a stochastic prox-extragradient algorithm that mitigates ill-posed behavior induced by nonconcavity, with convergence guarantees to approximate first-order stationarity. Experiments on LLM safety alignment and exploration maximization tasks further corroborate the convergence behavior consistent with our theory.

Figures

Figures reproduced from arXiv: 2608.03562 by the authors.

Figure 1
Figure 1. (a) Convergence of Algorithm 1. (b)–(d) Results of Algorithm 2. LLMs. To evaluate the convergence behavior of Algorithm 2 on the nonconcave utilities, we follow the synthetic tabu￾lar RL setting of (Chen et al. 2024) but optimize the utility function defined in Section 5.1. Additional implementation details are in Appendix C. Results. Figure 1a reports the experimental results of Algo￾rithm 1 on the LLM safety align… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

96 extracted references · 43 canonical work pages

  1. [2]

    International Conference on Machine Learning , pages=

    Reinforcement learning with general utilities: Simpler variance reduction and large state-action space , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  2. [3]

    Advances in Neural Information Processing Systems , volume=

    Robust reinforcement learning with general utility , author=. Advances in Neural Information Processing Systems , volume=

  3. [4]

    Advances in Neural Information Processing Systems , volume=

    Variational policy gradient method for reinforcement learning with general utilities , author=. Advances in Neural Information Processing Systems , volume=

  4. [5]

    Advances in Neural Information Processing Systems , volume=

    On the convergence and sample efficiency of variance-reduced policy gradient method , author=. Advances in Neural Information Processing Systems , volume=

  5. [6]

    Policy Gradient for Reinforcement Learning with General Utilities

    Policy gradient for reinforcement learning with general utilities , author=. arXiv preprint arXiv:2210.00991 , year=

    work page Pith review arXiv

  6. [7]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Solving non-rectangular reward-robust MDPs via frequency regularization , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  7. [8]

    IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

    A review of safe reinforcement learning: Methods, theories and applications , author=. IEEE Transactions on Pattern Analysis and Machine Intelligence , year=

  8. [9]

    2021 , publisher=

    Constrained Markov decision processes , author=. 2021 , publisher=

    2021

Show all 96 references
  1. [10]

    Advances in neural information processing systems , volume=

    Policy gradient for coherent risk measures , author=. Advances in neural information processing systems , volume=

  2. [11]

    2008 , publisher=

    Portfolio selection: efficient diversification of investments , author=. 2008 , publisher=

    2008

  3. [12]

    International conference on machine learning , pages=

    Constrained policy optimization , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  4. [13]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Optimizing the CVaR via sampling , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  5. [14]

    2021 , publisher=

    Lectures on stochastic programming: modeling and theory , author=. 2021 , publisher=

    2021

  6. [15]

    International Conference on Artificial Intelligence and Statistics , pages=

    A general sample complexity analysis of vanilla policy gradient , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2022 , organization=

    2022

  7. [16]

    International conference on machine learning , pages=

    On gradient descent ascent for nonconvex-concave minimax problems , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  8. [17]

    SIAM Journal on Applied Mathematics , volume=

    The theory of max-min, with applications , author=. SIAM Journal on Applied Mathematics , volume=. 1966 , publisher=

    1966

  9. [18]

    Operations Research Letters , volume=

    Robust linear optimization under general norms , author=. Operations Research Letters , volume=. 2004 , publisher=

    2004

  10. [19]

    Lecture notes

    Lectures on robust convex optimization , author=. Lecture notes. Georgia Institute of Technology , year=

  11. [20]

    International Conference on Machine Learning , pages=

    On the occupancy measure of non-Markovian policies in continuous MDPs , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  12. [21]

    arXiv preprint arXiv:1901.01992 , year=

    Large-scale Markov decision problems via the linear programming dual , author=. arXiv preprint arXiv:1901.01992 , year=

    1901 arXiv

  13. [22]

    2004 , publisher=

    Convex optimization , author=. 2004 , publisher=

    2004

  14. [23]

    SIAM Journal on Optimization , volume=

    Prox-method with rate of convergence O (1/t) for variational inequalities with Lipschitz continuous monotone operators and smooth convex-concave saddle point problems , author=. SIAM Journal on Optimization , volume=. 2004 , publisher=

    2004

  15. [24]

    Advances in Neural Information Processing Systems , volume=

    Natural policy gradient primal-dual method for constrained markov decision processes , author=. Advances in Neural Information Processing Systems , volume=

  16. [25]

    Proceedings of Thirty Third Conference on Learning Theory , pages =

    Optimality and Approximation with Policy Gradient Methods in Markov Decision Processes , author =. Proceedings of Thirty Third Conference on Learning Theory , pages =. 2020 , editor =

    2020

  17. [26]

    Management Science , volume=

    Linear programming and sequential decisions , author=. Management Science , volume=. 1960 , publisher=

    1960

  18. [27]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Multi-agent reinforcement learning with general utilities via decentralized shadow reward actor-critic , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  19. [28]

    International conference on machine learning , pages=

    Provably efficient maximum entropy exploration , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  20. [29]

    arXiv preprint arXiv:2106.03787 , year=

    Concave utility reinforcement learning: The mean-field game viewpoint , author=. arXiv preprint arXiv:2106.03787 , year=

    arXiv

  21. [30]

    Advances in neural information processing systems , volume=

    Generative adversarial imitation learning , author=. Advances in neural information processing systems , volume=

  22. [31]

    Part I: Theory , author=

    Survey of linear programming for standard and nonstandard Markovian control problems. Part I: Theory , author=. Zeitschrift f. 1994 , publisher=

    1994

  23. [32]

    Mathematics of Operations Research , volume=

    Risk-sensitive optimal control for Markov decision processes with monotone cost , author=. Mathematics of Operations Research , volume=. 2002 , publisher=

    2002

  24. [33]

    IEEE Journal on Selected Areas in Information Theory , volume=

    Cautious reinforcement learning via distributional risk in the dual domain , author=. IEEE Journal on Selected Areas in Information Theory , volume=. 2021 , publisher=

    2021

  25. [34]

    Journal of Machine Learning Research , volume=

    A comprehensive survey on safe reinforcement learning , author=. Journal of Machine Learning Research , volume=

  26. [35]

    2018 IEEE international conference on robotics and automation (ICRA) , pages=

    Sim-to-real transfer of robotic control with dynamics randomization , author=. 2018 IEEE international conference on robotics and automation (ICRA) , pages=. 2018 , organization=

    2018

  27. [36]

    Management Science , volume=

    Bias and variance approximation in value function estimates , author=. Management Science , volume=. 2007 , publisher=

    2007

  28. [37]

    Advances in Neural Information Processing Systems , year=

    Reward-rational (implicit) choice: A unifying formalism for reward learning , author=. Advances in Neural Information Processing Systems , year=

  29. [38]

    arXiv preprint arXiv:2402.09992 , year=

    Risk-sensitive soft actor-critic for robust deep reinforcement learning under distribution shifts , author=. arXiv preprint arXiv:2402.09992 , year=

    arXiv

  30. [39]

    Neural computation , volume=

    Robust reinforcement learning , author=. Neural computation , volume=. 2005 , publisher=

    2005

  31. [40]

    Advances in Neural Information Processing Systems , volume=

    Reinforcement learning in robust markov decision processes , author=. Advances in Neural Information Processing Systems , volume=

  32. [41]

    International conference on machine learning , pages=

    Robust adversarial reinforcement learning , author=. International conference on machine learning , pages=. 2017 , organization=

    2017

  33. [42]

    2025 , url=

    Robust Optimization for Mitigating Reward Hacking with Correlated Proxies , author=. 2025 , url=

    2025

  34. [43]

    arXiv preprint arXiv:2410.04108 , year=

    On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning , author=. arXiv preprint arXiv:2410.04108 , year=

  35. [44]

    Advances in Neural Information Processing Systems , volume=

    Scalable primal-dual actor-critic method for safe multi-agent rl with general utilities , author=. Advances in Neural Information Processing Systems , volume=

  36. [45]

    arXiv preprint arXiv:2311.13589 , year=

    Risk-sensitive markov decision process and learning under general utility functions , author=. arXiv preprint arXiv:2311.13589 , year=

    arXiv

  37. [46]

    2018 IEEE International Conference on Robotics and Automation (ICRA) , pages=

    Optlayer-practical constrained optimization for deep reinforcement learning in the real world , author=. 2018 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2018 , organization=

    2018

  38. [47]

    arXiv preprint arXiv:1610.03295 , year=

    Safe, multi-agent, reinforcement learning for autonomous driving , author=. arXiv preprint arXiv:1610.03295 , year=

    arXiv

  39. [48]

    2020 IEEE International conference on healthcare informatics (ICHI) , pages=

    Safe reinforcement learning for sepsis treatment , author=. 2020 IEEE International conference on healthcare informatics (ICHI) , pages=. 2020 , organization=

    2020

  40. [49]

    Mathematical finance , volume=

    Coherent measures of risk , author=. Mathematical finance , volume=. 1999 , publisher=

    1999

  41. [50]

    2024 , publisher=

    Risk-averse optimization and control , author=. 2024 , publisher=

    2024

  42. [51]

    Grimmer et al

    The landscape of the proximal point method for nonconvex--nonconcave minimax optimization: B. Grimmer et al. , author=. Mathematical Programming , volume=. 2023 , publisher=

    2023

  43. [52]

    INFORMS Journal on Optimization , volume=

    On the linear convergence of extragradient methods for nonconvex--nonconcave minimax problems , author=. INFORMS Journal on Optimization , volume=. 2024 , publisher=

    2024

  44. [53]

    Matecon , volume=

    The extragradient method for finding saddle points and other problems , author=. Matecon , volume=

  45. [54]

    Mathematics of Operations Research , volume=

    Robust dynamic programming , author=. Mathematics of Operations Research , volume=. 2005 , publisher=

    2005

  46. [55]

    Mathematics of Operations Research , volume=

    Robust Markov decision processes , author=. Mathematics of Operations Research , volume=. 2013 , publisher=

    2013

  47. [56]

    arXiv preprint arXiv:1206.4643 , year=

    Lightning does not strike twice: Robust MDPs with coupled uncertainty , author=. arXiv preprint arXiv:1206.4643 , year=

    arXiv

  48. [57]

    arXiv preprint arXiv:2409.15360 , year=

    Reward-robust rlhf in llms , author=. arXiv preprint arXiv:2409.15360 , year=

    arXiv

  49. [58]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Reinforcement learning with perturbed rewards , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  50. [59]

    IEEE Transactions on Artificial Intelligence , volume=

    Robotic control in adversarial and sparse reward environments: A robust goal-conditioned reinforcement learning approach , author=. IEEE Transactions on Artificial Intelligence , volume=. 2023 , publisher=

    2023

  51. [60]

    International Conference on Machine Learning , pages=

    Action robust reinforcement learning and applications in continuous control , author=. International Conference on Machine Learning , pages=. 2019 , organization=

    2019

  52. [61]

    Mathematics of Operations Research , volume=

    Robust MDPs with k-rectangular uncertainty , author=. Mathematics of Operations Research , volume=. 2016 , publisher=

    2016

  53. [62]

    Mathematics of Operations Research , volume=

    Robust markov decision processes: Beyond rectangularity , author=. Mathematics of Operations Research , volume=. 2023 , publisher=

    2023

  54. [63]

    Advances in Neural Information Processing Systems , volume=

    Fast Algorithms for L-constrained S-rectangular Robust MDPs , author=. Advances in Neural Information Processing Systems , volume=

  55. [64]

    Journal of Machine Learning Research , volume=

    Partial policy iteration for l1-robust markov decision processes , author=. Journal of Machine Learning Research , volume=

  56. [65]

    2001 , publisher=

    Solving uncertain Markov decision processes , author=. 2001 , publisher=

    2001

  57. [66]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Scalable first-order methods for robust mdps , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  58. [67]

    arXiv preprint arXiv:2505.20359 , year=

    Risk-aware Direct Preference Optimization under Nested Risk Measure , author=. arXiv preprint arXiv:2505.20359 , year=

    arXiv

  59. [68]

    arXiv preprint arXiv:2511.20726 , year=

    Learning from Risk: LLM-Guided Generation of Safety-Critical Scenarios with Prior Knowledge , author=. arXiv preprint arXiv:2511.20726 , year=

  60. [69]

    arXiv preprint arXiv:1206.6404 , year=

    Policy gradients with variance related risk criteria , author=. arXiv preprint arXiv:1206.6404 , year=

    arXiv

  61. [70]

    Journal of risk , volume=

    Optimization of conditional value-at-risk , author=. Journal of risk , volume=

  62. [71]

    Advances in neural information processing systems , volume=

    Risk-sensitive and robust decision-making: a cvar optimization approach , author=. Advances in neural information processing systems , volume=

  63. [72]

    International conference on machine learning , pages=

    Policy gradient in robust mdps with global convergence guarantee , author=. International conference on machine learning , pages=. 2023 , organization=

    2023

  64. [73]

    Advances in Neural Information Processing Systems , volume=

    Solving a class of non-convex min-max games using iterative first order methods , author=. Advances in Neural Information Processing Systems , volume=

  65. [74]

    Journal of Machine Learning Research , volume=

    Convergence and sample complexity of natural policy gradient primal-dual methods for constrained mdps , author=. Journal of Machine Learning Research , volume=

  66. [75]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume=

    Achieving zero constraint violation for constrained reinforcement learning via conservative natural policy gradient primal-dual algorithm , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume=

  67. [76]

    Advances in Neural Information Processing Systems , volume=

    Risk-averse fine-tuning of large language models , author=. Advances in Neural Information Processing Systems , volume=

  68. [77]

    arXiv preprint arXiv:2310.12773 , year=

    Safe rlhf: Safe reinforcement learning from human feedback , author=. arXiv preprint arXiv:2310.12773 , year=

    arXiv

  69. [78]

    International Conference on Machine Learning , pages=

    Pythia: A suite for analyzing large language models across training and scaling , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  70. [79]

    1997 , publisher=

    Numerical analysis , author=. 1997 , publisher=

    1997

  71. [80]

    arXiv preprint arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv preprint arXiv:1707.06347 , year=

    arXiv

  72. [81]

    Advances in neural information processing systems , volume=

    Deep reinforcement learning from human preferences , author=. Advances in neural information processing systems , volume=

  73. [82]

    International Conference on Learning Representations , year=

    The effects of reward misspecification: Mapping and mitigating misaligned models , author=. International Conference on Learning Representations , year=

  74. [83]

    International Conference on Learning Representations , year=

    Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking , author=. International Conference on Learning Representations , year=

  75. [84]

    Leike, Jan and Martic, Miljan and Krakovna, Victoria and Ortega, Pedro and Everitt, Tom and Lefrancq, Ryan and Orseau, Laurent and Legg, Shane , journal=

  76. [85]

    Machine learning , volume=

    Simple statistical gradient-following algorithms for connectionist reinforcement learning , author=. Machine learning , volume=. 1992 , publisher=

    1992

  77. [86]

    Advances in neural information processing systems , volume=

    A simple proximal stochastic gradient method for nonsmooth nonconvex optimization , author=. Advances in neural information processing systems , volume=

  78. [87]

    Sbornik: Mathematics , volume=

    The gradient projection algorithm for a proximally smooth set and a function with Lipschitz continuous gradient , author=. Sbornik: Mathematics , volume=. 2020 , publisher=

    2020

  79. [88]

    Journal of Mathematical Analysis and Applications , volume=

    On Young's inequality , author=. Journal of Mathematical Analysis and Applications , volume=. 2019 , publisher=

    2019

  80. [89]

    Journal of banking & finance , volume=

    Spectral measures of risk: A coherent representation of subjective risk aversion , author=. Journal of banking & finance , volume=. 2002 , publisher=

    2002

  81. [90]

    arXiv preprint arXiv:2503.03064 , year=

    Improving llm-as-a-judge inference with the judgment distribution , author=. arXiv preprint arXiv:2503.03064 , year=

  82. [91]

    arXiv preprint arXiv:2210.03096 , year=

    Accelerated single-call methods for constrained min-max optimization , author=. arXiv preprint arXiv:2210.03096 , year=

    arXiv

  83. [92]

    International Conference on Artificial Intelligence and Statistics , pages=

    Efficient methods for structured nonconvex-nonconcave min-max optimization , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2021 , organization=

    2021

  84. [93]

    Journal of Machine Learning Research , volume=

    First-order convergence theory for weakly-convex-weakly-concave min-max problems , author=. Journal of Machine Learning Research , volume=

  85. [94]

    Stochastic Systems , volume=

    Solving variational inequalities with stochastic mirror-prox algorithm , author=. Stochastic Systems , volume=. 2011 , publisher=

    2011

  86. [95]

    Mathematical programming , volume=

    Equivalent differentiable optimization problems and descent methods for asymmetric variational inequality problems , author=. Mathematical programming , volume=. 1992 , publisher=

    1992

  87. [96]

    Mathematical Programming , volume=

    A class of gap functions for variational inequalities , author=. Mathematical Programming , volume=. 1994 , publisher=

    1994

  88. [97]

    Convex analysis and monotone operator theory in Hilbert spaces , pages=

    Correction to: convex analysis and monotone operator theory in Hilbert spaces , author=. Convex analysis and monotone operator theory in Hilbert spaces , pages=. 2020 , publisher=

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.