Pith. sign in

REVIEW 4 major objections 4 minor 32 references

This paper shows that peak-cost constrained MDPs may have a strictly positive duality gap, so Lagrangian or primal-dual methods can miss the safe optimum.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Peak-cost constrained MDPs can have a nonzero duality gap, and a robust surrogate RL method enforces worst-case peak-cost constraints in perturbed simulations.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection The two-state duality-gap counterexample is correct and genuinely new; the surrogate guarantee's proof is wrong as printed, but the paper is worth refereeing if the proof is repairable. the 4 major comments →

arxiv 2607.15457 v1 pith:DY6SHWTI submitted 2026-07-16 cs.LG

Robust Peak-cost Constrained Reinforcement Learning

Shilpa Mukhopadhyay , Sourav Ganguly , Santosh Mohan Rajkumar , Honghao Wei , Debdipta Goswami , Arnob Ghosh This is my paper
classification cs.LG MSC 90C4090C46
keywords peak-cost constraintsconstrained Markov decision processduality gaprobust reinforcement learningLagrangian methodsreachability constraintsintegral probability metricssurrogate optimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper establishes that when a constrained Markov decision process measures safety by the maximum per-step cost along a trajectory rather than the cumulative cost, the optimization problem can lose the zero-duality-gap property that standard CMDPs enjoy. The central evidence is a two-state counterexample with a strict gap between the primal optimum and the Lagrangian dual value. To restore a practical method, the paper proposes a robust surrogate objective and an actor-critic algorithm with IPM-based robust critics, and claims that with a sufficiently large penalty the surrogate achieves the same robust reward as the original problem while violating the peak constraint by at most epsilon. This matters because existing reachability-constrained RL methods rely on Lagrangian updates without such a guarantee.

Core claim

The paper's key discovery is a counterexample: a discounted two-state, two-action MDP with discount factor 0.9 and cost budget 0.4 where the optimal constrained reward is 180/7 ≈ 25.71 but the Lagrangian dual value is 64.8, giving a strictly positive duality gap. The cause is that the peak-cost constraint becomes nonconvex in the state-action occupancy measure. The paper further claims that a surrogate problem, which replaces the hard peak constraint with a max of normalized reward and a soft violation penalty, achieves the same robust reward as the original robust problem and violates the peak constraint by at most epsilon when the penalty parameter satisfies beta ≥ C_max/epsilon.

What carries the argument

The peak-cost value recursion V_c,peak(s,a) = (1−γ)c(s) + γ max{ c(s), E[V_c,peak(s′)] }, which replaces additive accumulation with a trajectory-wise maximum. The surrogate objective max{ max_P J_r/β, max_{P,s} V_c,peak(s) − b }, which turns constraint satisfaction into a soft penalty. The IPM-based robust Bellman operator L^P V = (1−γ)c + γ V + ρ‖w_{2:d}‖₂, which handles transition uncertainty under linear function approximation.

Load-bearing premise

The epsilon-violation guarantee for the surrogate relies on the discounted cumulative reward J_r being bounded by the same constant C_max that bounds per-step costs; if C_max is only a per-step bound, the required beta scales like C_max/((1−γ)ε), and the proof's inequality directions in the appendix must be corrected to justify the claim.

What would settle it

Reproduce the two-state MDP and compute the Lagrangian dual value for several budgets b in (0,0.5); a zero duality gap for any budget other than the one shown would weaken the claim. Separately, run the proposed surrogate with beta set to C_max/ε on a simple bandit-like MDP where the peak-cost constraint is active, and check whether the actual peak violation ever exceeds ε by a nontrivial margin.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the counterexample is correct, Lagrangian or primal-dual methods applied directly to peak-cost constraints can converge to strictly suboptimal policies even when Slater's condition holds.
  • If the surrogate guarantee holds, a single penalty parameter is enough to recover near-optimal robust reward while keeping peak violation within epsilon.
  • The IPM-based robust critics provide a tractable way to estimate worst-case peak-cost values under rectangular transition uncertainty.
  • The results motivate direct policy-space optimization or surrogate-based methods instead of relying on primal-dual updates for reachability-style safety objectives.
  • The epsilon-violation bound supports practical schemes that adjust the penalty beta during training, as long as beta meets the stated threshold.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The nonconvexity mechanism behind the two-state gap likely extends to finite-horizon and continuous-state problems, suggesting that any peak-cost constraint needs specialized treatment, not just a Lagrangian patch.
  • The surrogate reward bound in Proposition 1 may require scaling beta by 1/(1−γ) if the reward-critic bound is not a discounted-cumulative bound; if so, the beta values used in the experiments could be smaller than the stated guarantee needs.
  • The IPM-based peak-cost critic could be adapted to other distributionally robust safety constraints, such as worst-case conditional value at risk or constraints over perturbed state-visitation frequencies, with similar closed-form robust Bellman operators.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper studies robust reinforcement learning with peak-cost constraints, in which safety is measured by the maximum (or a discounted recursive proxy of the maximum) cost encountered along a trajectory. The authors first claim that, unlike standard CMDPs, a peak-cost constrained MDP can fail to admit zero duality gap even under Slater's condition; Theorem 1 proves this with an explicit two-state discounted MDP. They then formulate a robust counterpart under an IPM-based uncertainty set and propose a surrogate objective (6), for which Proposition 1 claims the surrogate solution has at least the optimal robust reward and violates the peak constraint by at most ε when β ≥ C_max/ε. An actor–critic algorithm with robust TD errors is developed and evaluated on CartPole and MuJoCo environments.

Significance. The duality-gap counterexample, if correctly scoped, is a valuable negative result: it shows that Lagrangian/primal-dual methods cannot be applied to peak-style constraints without additional justification, in contrast to standard CMDPs. The surrogate-optimization direction is a reasonable response, and the paper is honest that convergence analysis is left for future work. The explicit, self-contained computation in Theorem 1 — P*=180/7, D*=64.8, gap=1368/35 — is a reproducible strength. However, the proof of Proposition 1, which is the theoretical basis for Algorithm 1, contains a reversed inequality, an invalid inference in Eq. (24), and an inconsistent reward-sign convention. As printed, the claimed ε-violation and reward-equivalence guarantees do not follow. The experiments are illustrative but heuristic, with no code or detailed ablations. The central theoretical contribution is therefore only partially established.

major comments (4)
  1. [Appendix A, Proposition 1 proof, Eq. (21)] Since π̂* is a minimizer of the surrogate objective (6), the correct inequality is S(π̂*) ≤ S(π*) for every feasible π*, not the reverse. Equation (21) asserts S(π̂*) ≥ max_P J_r^{π*,P}/β, which is the opposite of minimization. The subsequent inequalities (22)–(23) inherit this wrong direction, and the conclusion that the constraint term alone is at least max_P J_r^{π*}/β does not follow. This is load-bearing: Proposition 1 is the stated theoretical justification for the surrogate approach used by Algorithm 1. The proof must be rewritten with the correct ordering and the consequences re-derived.
  2. [Appendix A, Eq. (24)] Even after correcting the inequality direction, Eq. (24) is a non sequitur. The boundedness of V_{c,peak} and J_r by C_max does not imply max_{P,s} V_{c,peak} - b ≤ C_max/β; the preceding equation gives a lower bound on the violation, not an upper bound. Moreover, C_max is introduced as a per-step cost bound; a discounted cumulative reward can be as large as C_max/(1−γ), so J_r ≤ C_max is not valid if C_max is per-step. The assumptions and constants in Proposition 1 need to be restated precisely, and the ε-violation claim re-proved.
  3. [Section II-A vs. Theorem 1 and Proposition 1] The manuscript defines V_r and J_r as discounted negative reward and formulates the objective as min_π J_r. In Theorem 1, however, J_r is treated as the positive discounted reward and is maximized, and Proposition 1 states 'Ĵr ≥ Jπ*r' in a way that is ambiguous under the negative-reward convention. This inconsistency propagates into the surrogate objective (6) and Algorithm 1, where A=(1/|D_t|)Σ V_r/β is compared with a cost violation. Please choose a single convention — e.g., define positive reward-to-go R = −V_r and maximize, or keep negative costs and adjust all inequalities — and restate Proposition 1 with unambiguous inequalities.
  4. [Theorem 1, formal statement] The formal statement claims that the recursive peak value in Eq. (16) is 'the same' as the peak cost over the trajectory for this MDP. This is not true for γ=0.9. For example, under π_{0.2}, the trajectory almost surely reaches the self-loop action a1 and incurs cost 0.5 at least once, so the literal trajectory maximum is 0.5, while the computed V is 0.4. Thus Theorem 1 as stated is a duality-gap result for the recursive peak-value constraint in Eq. (2)/(16), not for the literal maximum-cost constraint described in Section II-B. The authors should state the theorem specifically for the recursive value, or discuss the γ→1 limit in which the recursive value approaches the peak. This affects the scope of the abstract's claim that 'peak-cost constrained MDPs may not admit zero duality gap.'
minor comments (4)
  1. [Proposition 1 statement] The desired bound should be written as (max_{P,s} V_{c,peak}^{π,P}(s) − b)_+ ≤ ε, with an explicit positive-part operator. There is also a typo in the condition: 'β≥C_max/ε' appears as 'β≥C max/ϵ' or similar in places.
  2. [Equation (5)] The notation L_P in Eq. (5) conflicts with the uncertainty set P, and the term ∥w_{c,2:d}∥_2 is not defined in the text. Please clarify how w_{c,2:d} relates to the linear parametrization V_{c,peak}^{π}(s)=Ψ(s)^T w^{π,c}.
  3. [Algorithms 1 and 2] The selector F = arg max{A,B} does not specify what value is returned (0 or 1), and Algorithm 2's branches 'if F=0' and 'else if F=1' are consequently unclear. The pseudocode also appears to update the reward critic inside the F=1 branch; please reconcile.
  4. [Throughout] There are several typos and inconsistencies: 'copnsider' (Section IV-A), 'peak-constained' (Section I-A), and inconsistent 'Mujoco'/'MuJoCo' capitalization. Please proofread.

Circularity Check

0 steps flagged

No circular reduction: duality-gap counterexample is self-contained; the surrogate proposition is adapted from a self-citation but not by construction.

full rationale

The central claim, Theorem 1, is a two-state numerical counterexample. Its proof uses only the Bellman equation, the fixed-point expression for V_{c,peak}, and direct evaluation of the Lagrangian dual; no author-overlapping result or fitted input is used. The surrogate problem (6) is introduced as 'Inspired from [17]' and Proposition 1 is 'adapted from Proposition 1 in [17]', where [17] shares authors with this paper. That is a self-citation, but it is not a definitional circularity: the paper attempts an independent proof in Appendix A rather than merely asserting equivalence, and the surrogate objective is not constructed to be equal to the original problem. The Appendix proof has genuine validity gaps (Eq. (21) uses the wrong inequality for a minimizer; Eq. (24) does not follow from the preceding bounds), but those are correctness defects, not reductions of outputs to inputs. The IPM robust operator is credited to external work [28], and the experiments are RL training comparisons, not fitted predictions. The paper itself notes in the Conclusion that convergence guarantees are 'left for the future,' a stated limitation but not a circularity. Under the rule that self-citation only counts when load-bearing and unverified, this single adaptation is minor and non-load-bearing for the paper's headline result; hence score 2.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

No new physical or ontological entities are introduced. The free parameters that the central claims depend on are the surrogate weight β and the IPM radii ρ_r and ρ_c, neither of which is derived from the problem data or reported with a concrete value. The most fragile axiom is the implicit boundedness of the reward objective used in Proposition 1.

free parameters (4)
  • β (surrogate weight) = 25 in experiments
    Chosen by hand; the theory requires β ≥ C_max/ε, but C_max and ε are not specified or instantiated for the experiments.
  • ρ_r (IPM reward radius) = not reported
    Appears in the reward TD error Eq. (7); no value is given, yet it controls the size of the robust uncertainty set.
  • ρ_c (IPM peak-cost radius) = not reported
    Appears in the peak-cost TD error Eq. (8); no value is given, and it directly controls how aggressively constraints are robustified.
  • warm start episodes = 300
    Used for RP-CRL in experiments; chosen without sensitivity analysis or connection to theory.
axioms (4)
  • standard math The peak-cost Bellman operator in Eq. (1) is a γ-contraction for γ<1, so V_{c,peak} can be computed by fixed-point iteration.
    Relied on throughout the formulation; inherited from the cited prior work [23] and standard contraction arguments.
  • domain assumption There exists at least one feasible policy from the initial distribution (Slater-type condition).
    Assumed in Section II-B and used in Proposition 1 to ensure π* exists and the surrogate comparison is meaningful.
  • domain assumption The IPM uncertainty set combined with linear value-function approximation yields the closed-form robust Bellman update in Eq. (5) and the TD regularizers in Eqs. (7)-(8).
    This is a substantial modeling assumption imported from [28]; if the function class F or the linear approximation is misspecified, the regularized TD errors do not equal the robust Bellman operator.
  • ad hoc to paper Costs are bounded in [0, C_max] and, implicitly, the reward objective satisfies J^{π,P}_r ≤ C_max.
    Appendix A uses 'J^{π*,P}_r ≤ C_max' with C_max otherwise introduced as a bound on per-step costs. For a discounted infinite-horizon reward, the cumulative value can be C_max/(1−γ), so this assumption is not stated and is generally false as written.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Robust Peak-cost Constrained Reinforcement Learning." pith.science (2026). https://pith.science/paper/DY6SHWTI

@misc{pith2026260715457,
  author       = {Pith},
  title        = {Pith review of: Robust Peak-cost Constrained Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DY6SHWTI}},
  note         = {Machine review of arXiv:2607.15457}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

We study robust peak-cost constrained reinforcement learning (RP-CRL), where the objective is to maximize expected reward while controlling the maximum cost encountered along a trajectory. This setting is motivated by safety-critical applications in which a single large violation can be catastrophic and therefore cannot be adequately captured by the standard CMDP framework based on expected cumulative cost. Existing reachability-constrained RL methods adopt Lagrangian-based approaches, yet the underlying duality properties of peak-cost constrained MDPs remain unclear. We show that, unlike standard CMDPs, peak-cost constrained MDPs may not admit zero duality gap. We further consider a robust formulation to address simulator-to-real-world mismatch in the transition dynamics. To solve this problem, we develop a surrogate optimization framework and a robust value estimation method based on integral probability metrics. We prove that, with appropriate hyperparameter choices, the surrogate solution attains the same robust reward value as the original problem while violating the constraint by at most epsilon. Experiments show that the proposed method effectively enforces safety under dynamics perturbations while retaining strong reward performance.

Figures

Figures reproduced from arXiv: 2607.15457 by Arnob Ghosh, Debdipta Goswami, Honghao Wei, Santosh Mohan Rajkumar, Shilpa Mukhopadhyay, Sourav Ganguly.

Figure 1
Figure 1. Figure 1: Comparison of Primal Dual(PD) and Surrogate Ob [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of training curves of RP-CRL for differ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Learning RP-CRL on C2. The policy learns the peak [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training and inference performance across four MuJoCo environments. Rows show training returns, training maximum [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 4 linked inside Pith

  1. [1]

    Altman,Constrained Markov decision processes

    E. Altman,Constrained Markov decision processes. Routledge, 2021

    2021

  2. [2]

    Towards achieving sub-linear regret and hard constraint violation in model-free rl,

    A. Ghosh, X. Zhou, and N. Shroff, “Towards achieving sub-linear regret and hard constraint violation in model-free rl,” inInternational Conference on Artificial Intelligence and Statistics. PMLR, 2024, pp. 1054–1062

    2024

  3. [3]

    Achieving sub-linear regret in infinite horizon average reward constrained mdp with linear function approximation,

    ——, “Achieving sub-linear regret in infinite horizon average reward constrained mdp with linear function approximation,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  4. [4]

    Reward constrained policy optimization,

    C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,” inInternational Conference on Learning Repre- sentations

  5. [5]

    Reachability constrained rein- forcement learning,

    D. Yu, H. Ma, S. Li, and J. Chen, “Reachability constrained rein- forcement learning,” inInternational conference on machine learning. PMLR, 2022, pp. 25 636–25 655

    2022

  6. [6]

    Iterative reachability estimation for safe reinforcement learning,

    M. Ganai, Z. Gong, C. Yu, S. Herbert, and S. Gao, “Iterative reachability estimation for safe reinforcement learning,”Advances in Neural Information Processing Systems, vol. 36, pp. 69 764–69 797, 2023

    2023

  7. [7]

    Safe policies for reinforcement learning via primal-dual methods,

    S. Paternain, M. Calvo-Fullana, L. F. Chamon, and A. Ribeiro, “Safe policies for reinforcement learning via primal-dual methods,”IEEE Transactions on Automatic Control, vol. 68, no. 3, pp. 1321–1336, 2022

    2022

  8. [8]

    Responsive safety in reinforce- ment learning by pid lagrangian methods,

    A. Stooke, J. Achiam, and P. Abbeel, “Responsive safety in reinforce- ment learning by pid lagrangian methods,” inInternational conference on machine learning. PMLR, 2020, pp. 9133–9143

    2020

  9. [9]

    Constrained upper confidence reinforcement learning,

    L. Zheng and L. Ratliff, “Constrained upper confidence reinforcement learning,” inLearning for Dynamics and Control. PMLR, 2020, pp. 620–629

    2020

  10. [10]

    Natural policy gradient primal-dual method for constrained markov decision pro- cesses,

    D. Ding, K. Zhang, T. Basar, and M. Jovanovic, “Natural policy gradient primal-dual method for constrained markov decision pro- cesses,”Advances in Neural Information Processing Systems, vol. 33, pp. 8378–8390, 2020

    2020

  11. [11]

    Crpo: A new approach for safe reinforcement learning with convergence guarantee,

    T. Xu, Y . Liang, and G. Lan, “Crpo: A new approach for safe reinforcement learning with convergence guarantee,” inInternational Conference on Machine Learning. PMLR, 2021, pp. 11 480–11 491

    2021

  12. [12]

    Constrained policy optimization,

    J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” inInternational conference on machine learning. Pmlr, 2017, pp. 22–31

    2017

  13. [13]

    A lyapunov-based approach to safe reinforcement learning,

    Y . Chow, O. Nachum, E. Duenez-Guzman, and M. Ghavamzadeh, “A lyapunov-based approach to safe reinforcement learning,”Advances in neural information processing systems, vol. 31, 2018

    2018

  14. [14]

    Robust constrained reinforcement learning,

    Y . Wang, F. Miao, and S. Zou, “Robust constrained reinforcement learning,”arXiv preprint arXiv:2209.06866, 2022

    Pith/arXiv arXiv 2022

  15. [15]

    Dis- tributionally robust constrained reinforcement learning under strong duality,

    Z. Zhang, K. Panaganti, L. Shi, Y . Sui, A. Wierman, and Y . Yue, “Dis- tributionally robust constrained reinforcement learning under strong duality,”arXiv preprint arXiv:2406.15788, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Near-optimal policy identification in robust constrained markov decision processes via epigraph form,

    T. Kitamura, T. Kozuno, W. Kumagai, K. Hoshino, Y . Hosoe, K. Kasaura, M. Hamaya, P. Parmas, and Y . Matsuo, “Near-optimal policy identification in robust constrained markov decision processes via epigraph form,” inThe Thirteenth International Conference on Learning Representations

  17. [17]

    Efficient policy optimization in robust constrained mdps with iteration com- plexity guarantees,

    S. Ganguly, K. Panaganti, A. Ghosh, and A. Wierman, “Efficient policy optimization in robust constrained mdps with iteration com- plexity guarantees,” inThe Thirty-ninth Annual Conference on Neural Information Processing Systems

  18. [18]

    Safe learning in robotics: From learning-based control to safe reinforcement learning,

    L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig, “Safe learning in robotics: From learning-based control to safe reinforcement learning,”Annual Review of Control, Robotics, and Autonomous Systems, vol. 5, no. 1, pp. 411–444, 2022

    2022

  19. [19]

    Robust control barrier–value functions for safety-critical control,

    J. J. Choi, D. Lee, K. Sreenath, C. J. Tomlin, and S. L. Herbert, “Robust control barrier–value functions for safety-critical control,” in 2021 60th IEEE Conference on Decision and Control (CDC). IEEE, 2021, pp. 6814–6821

    2021

  20. [20]

    Learning barrier certificates: Towards safe rein- forcement learning with zero training-time violations,

    Y . Luo and T. Ma, “Learning barrier certificates: Towards safe rein- forcement learning with zero training-time violations,”Advances in Neural Information Processing Systems, vol. 34, pp. 25 621–25 632, 2021

    2021

  21. [21]

    Joint synthesis of safety certificate and safe control policy using constrained reinforce- ment learning,

    H. Ma, C. Liu, S. E. Li, S. Zheng, and J. Chen, “Joint synthesis of safety certificate and safe control policy using constrained reinforce- ment learning,” inLearning for Dynamics and Control Conference. PMLR, 2022, pp. 97–109

    2022

  22. [22]

    Hamilton-jacobi reachability: A brief overview and recent advances,

    S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin, “Hamilton-jacobi reachability: A brief overview and recent advances,” in2017 IEEE 56th annual conference on decision and control (CDC). IEEE, 2017, pp. 2242–2253

    2017

  23. [23]

    A general safety framework for learning-based control in uncertain robotic systems,

    J. F. Fisac, A. K. Akametalu, M. N. Zeilinger, S. Kaynama, J. Gillula, and C. J. Tomlin, “A general safety framework for learning-based control in uncertain robotic systems,”IEEE Transactions on Automatic Control, vol. 64, no. 7, pp. 2737–2752, 2018

    2018

  24. [24]

    Bridging hamilton-jacobi safety analysis and reinforcement learning,

    J. F. Fisac, N. F. Lugovoy, V . Rubies-Royo, S. Ghosh, and C. J. Tomlin, “Bridging hamilton-jacobi safety analysis and reinforcement learning,” in2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019, pp. 8550–8556

    2019

  25. [25]

    Solving minimum-cost reach avoid using reinforcement learning,

    O. So, C. Ge, and C. Fan, “Solving minimum-cost reach avoid using reinforcement learning,”Advances in Neural Information Processing Systems, vol. 37, pp. 30 951–30 984, 2024

    2024

  26. [26]

    Con- strained reinforcement learning has zero duality gap,

    S. Paternain, L. Chamon, M. Calvo-Fullana, and A. Ribeiro, “Con- strained reinforcement learning has zero duality gap,”Advances in Neural Information Processing Systems, vol. 32, 2019

    2019

  27. [27]

    Robust dynamic programming,

    G. N. Iyengar, “Robust dynamic programming,”Mathematics of Op- erations Research, vol. 30, no. 2, pp. 257–280, 2005

    2005

  28. [28]

    Natural actor-critic for robust reinforcement learning with function approximation,

    R. Zhou, T. Liu, M. Cheng, D. Kalathil, P. Kumar, and C. Tian, “Natural actor-critic for robust reinforcement learning with function approximation,”Advances in neural information processing systems, vol. 36, pp. 97–133, 2023

    2023

  29. [29]

    Integral probability metrics and their generating classes of functions,

    A. M ¨uller, “Integral probability metrics and their generating classes of functions,”Advances in applied probability, vol. 29, no. 2, pp. 429– 443, 1997

    1997

  30. [30]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  31. [31]

    Openai gym,

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schul- man, J. Tang, and W. Zaremba, “Openai gym,”arXiv preprint arXiv:1606.01540, 2016. APPENDIX A. Proof of Theorem 1and Proposition 1 Formal Statement of Theorem 1: Consider a discounted MDP with discount factorγ= 0.9, one nonterminal states, and one absorbing terminal state⊥. At states, there ar...

    Pith/arXiv arXiv 2016

  32. [32]

    Since 1−p 2 ≤ 1 2 for allp∈[0,1], this is self-consistent

    Then max{ 1 2 , V(p)}= 1 2 , and thus V(p) = (1−p) 0.05 + 0.9· 1 2 = (1−p)· 1 2 = 1−p 2 . Since 1−p 2 ≤ 1 2 for allp∈[0,1], this is self-consistent. Therefore, V πp c (s) = 1−p 2 ,∀p∈[0,1]. We now solve the primal problem max p∈[0,1] Jr(πp)s.t. 1−p 2 ≤0.4. The constraint is equivalent to 1−p≤0.8⇐ ⇒p≥0.2. Moreover, Jr(πp) = 8−4p 0.1+0.9p , and differentiat...

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.