Pith. sign in

REVIEW 3 major objections 6 minor 62 references

Operator Splitting for Convex Constrained Markov Decision Processes

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Douglas-Rachford splitting decouples dynamics from constraints in convex MDPs, giving a first-order algorithm with last-iterate convergence and an infeasibility certificate.

desk verdict Solid contribution with a real gap between the ℓ=2 implementation and the exact-inner-loop convergence theorems; worth referee time. read the letter →

arxiv 2412.14002 v1 pith:XVWZDOG6 submitted 2024-12-18 math.OC cs.SYeess.SY

Panagiotis D. Grontas , Anastasios Tsiamis , John Lygeros This is my paper · ORCID
classification math.OCcs.SYeess.SY MSC 90C4090C2565K10
keywords convexconstrainedMarkovdecisionprocessDouglas-Rachfordsplittingoperatoroccupancymeasureinfeasibilitydetectionquadraticregularizationpolicyiterationfirst-orderoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a first-order splitting method can solve finite, known convex-constrained Markov decision processes without giving up convergence guarantees or scalability. The proposed OS-CMDP decomposes the problem into the occupancy-measure polytope $D$, which encodes the MDP dynamics, and the convex constraint set $C$, then alternates between solving a quadratically regularized MDP and projecting onto $C$. With exact inner solves, the iterates converge to a primal-dual solution of the convex CMDP under standard feasibility conditions. When $C$ and $D$ are disjoint, the difference $w_k - w_{k+1}$ converges to the minimal displacement vector $v$, and for polyhedral $C$ the occupancy iterates converge to the optimum of the shifted feasible problem $d \in D \cap (C - v)$. This matters because it gives a scalable first-order route to convex CMDPs and turns infeasible specifications into a certificate plus a minimally violating policy.

What carries the argument

The load-bearing object is the Douglas-Rachford splitting of the convex program into $f(d)=c^{\top}d + I_D(d)$, the unconstrained MDP part, and $g(d)=I_C(d)$, the constraint part. Its iteration alternates evaluating the proximal operator of $f$ (a quadratically regularized MDP, solved efficiently by the quadratically regularized policy iteration inner loop) with projection onto $C$. The infeasibility analysis centers on the minimal displacement vector $v=\arg\min_{\beta \in D-C}\|\beta\|$, which is the limit of $w_k - w_{k+1}$ and, when $C \cap D=\emptyset$, the normal vector of a strongly separating hyperplane between $D$ and $C$.

What would settle it

Take a feasible linearly constrained Garnet or grid-world CMDP whose optimal occupancy lies on a constraint boundary, run OS-CMDP with $\ell=2$ and the termination criterion $\|d_k-z_k\|_\infty \le \varepsilon_{\mathrm{opt}}$, and compare the returned objective with the LP optimum: if the criterion fires while $c^{\top} d_k$ is measurably above the optimum or while $d_k$ still violates a constraint, the inexact implementation is not converging to a primal-dual solution.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that Douglas-Rachford splitting with $f(d)=c^{\top}d + I_D(d)$ and $g(d)=I_C(d)$ lets an MDP's dynamics and its convex constraints be handled separately while still converging to a primal-dual solution. Each outer iteration performs two simple operations: a proximal step for $f$, which is a quadratically regularized MDP solved by the proposed QRPI scheme, and the projection of a reflected point onto $C$. Under Assumption 1 the generated $(d_k, z_k, \nu_k)$ converges to a primal-dual solution of (4). If $C \cap D = \emptyset$, the paper shows that $w_k - w_{k+1}$ tends to the minimal displacement vector $v=\arg\min_{\beta \in D-C}\|\beta\|$, and for polyhedral $C$ the occupancy measures converge to an optimal solution of the translated feasible problem $d \in D \cap (C-v)$.

Load-bearing premise

The convergence theorem assumes the inner regularized MDP is solved to optimality, whereas the implementation runs only two QRPI inner iterations; numerical evidence is given, but no proof that the resulting prox error is summable.

Editorial extensions

If this is right

  • Convex CMDPs can be solved by alternating an MDP solve with a projection, so any constraint set with an implementable projection — norm balls, entropy constraints, robust-cost constraints — fits the same algorithm.
  • The algorithm returns a last-iterate primal occupancy measure, not a Cesàro average, which is useful when the policy is represented by a nonlinear function approximator.
  • An infeasible specification is not a failure: the algorithm returns the minimal displacement vector as a certificate and, for polyhedral constraints, the optimal policy for the minimally relaxed feasible problem.
  • Because the inner QRPI loop converges R-linearly to the regularized MDP solution, the per-iteration subproblem can be solved by a policy-iteration-style scheme rather than by a generic quadratic program.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Proving that the inexact inner loop with a small fixed number of QRPI steps produces a summable error sequence $e_k=\|d_k - \mathrm{prox}_{\sigma f}(w_k)\|$ would turn Proposition 2 into a guarantee for the algorithm exactly as implemented; that error bound is the natural missing piece.
  • The same splitting could be adapted to model-free or estimated-dynamics settings by replacing QRPI with an approximate policy evaluator, with the minimal displacement vector then serving as a certificate that the learned safety constraints are incompatible with the estimated model.
  • The minimal displacement vector also suggests a practical tuning rule: shift the right-hand sides of the constraints along $v$ just enough to make the problem feasible, then re-run the algorithm to obtain the best policy under the adjusted specification.
  • Because the constraint step is a generic projection, one could in principle plug in learned or data-defined constraint sets and retain the algorithm's structure, although the convergence theory would no longer apply.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies finite discounted MDPs with convex constraints on the occupancy measure and known dynamics. It reformulates the convex CMDP as the sum f(d)=c^T d + I_D(d) and g(d)=I_C(d), applies Douglas-Rachford splitting, and develops a double-loop algorithm: an inner QRPI method for the regularized MDP prox_{σf}(w) and an outer projection onto C. The authors prove convergence of QRPI, then, assuming the inner problem is solved exactly, prove DRA convergence under a constraint qualification, characterize the minimal displacement vector and shifted-problem convergence when C∩D is empty, and propose termination criteria. Numerical experiments on Garnet MDPs and a grid world compare OS-CMDP with Gurobi, SCS, and a primal-dual baseline.

Significance. If the exact-inner-loop theory holds, the paper contributes a modular first-order method for convex CMDPs that does not require explicit regularization of the original problem and provides an infeasibility certificate. The infeasibility analysis in Propositions 3 and 4 is a valuable extension over algorithms that merely return a policy when the problem is feasible. The paper is also transparent about the practical inexact variant, but this transparency does not remove the gap between theory and implementation. The use of standard operator-splitting results and the numerical comparisons give reasonable confidence that the exact claims are correct and the method is competitive at medium accuracy.

major comments (3)
  1. [Section V and Propositions 2–4] The convergence guarantees proved in Section IV apply to the variant of Algorithm 1 in which the inner QRPI loop is solved to optimality, i.e. ℓ=+∞. The implemented algorithm in the experiments uses ℓ=2, as stated in Sections V and VI. Section V acknowledges that the inexact variant needs ∑_k e_k < ∞, where e_k := ||d_k − prox_{σf}(w_k)||, and cites [47, Th. 7], but it does not prove or even estimate this condition. Since Propositions 3 and 4 also rely on the exact DRA iteration (7), the abstract's claims of last-iterate convergence and infeasibility detection, and the termination criterion (21), are not established for the algorithm actually run.
  2. [Section V, Proposition 1, and Appendix C] Proposition 1 gives an R-linear convergence rate for QRPI for a fixed outer point w_k. This does not imply that ℓ=2 inner steps at each outer iteration produce a prox error e_k satisfying the summability condition in Section V: the outer update changes w_k at every step, so the contraction factor may depend on k, and warm starting need not yield a uniform bound. The numerical evidence in Appendix C compares ℓ=2 against ℓ=100 on only two problem instances; it is suggestive but cannot substitute for a proof that the inexact iteration inherits the DRA convergence or infeasibility-certification properties.
  3. [Appendix B, proof of Proposition 1] The identification of QRPI with the coordinate descent analysis of [48] is too terse. In particular, the displayed quantities E := [γP−Ξ −I] and b are not explicitly reconciled with the dual objective κ in (9), and the conditions of [48, Th. 2.1] are only asserted. Since Proposition 1 is the basis for the inner-loop convergence used to compute prox_{σf}(w_k), please expand the mapping and verify all assumptions, including strong convexity of the relevant function and the exact block-coordinate update order.
minor comments (6)
  1. [Abstract] The name "Douglas-Rachord" should be "Douglas-Rachford".
  2. [Section III-A] The text contains typographical errors: "Largrange" should be "Lagrange", and "disadvantegeous" should be "disadvantageous".
  3. [Algorithm 1] The loop header "For ℓ = 0,...,ℓ−1" uses ℓ both as the total number of inner iterations and as the running index; rename the running index (e.g., m) to avoid confusion.
  4. [Section II-A] The notation "ρ ∈ ∆(RS)" is inconsistent with the earlier definition ∆(S) = {x ∈ R^{|S|}_{≥0} : 1^T x = 1}; please clarify whether ρ is a vector in R^S or a distribution in the simplex.
  5. [Appendix D] The phrase "uncostrained MDP" should be "unconstrained MDP".
  6. [Table II] In the S=3000, f_b=0.5 row, the SCS entry "194.85 ( -1.55 )" is missing the unit "s" used elsewhere in the table.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the convergence and infeasibility results derive from external DRA and coordinate-descent theory, and the inexact-inner-loop gap is a rigor gap rather than a circular one.

full rationale

The derivation chain is self-contained. The splitting f(d) = c^T d + I_D(d), g(d) = I_C(d) is a design choice made from the problem data, and the DRA iterates (7) are the standard Douglas-Rachford iteration, not a construction that presupposes a solution of (4). Proposition 2 is obtained by verifying the hypotheses of Bauschke-Combettes [32, Cor. 27.6 and 28.3], i.e., proper closed convex functions and a constraint qualification, and then invoking the external DRA convergence theorem; the conclusion is not assumed in the hypotheses. Proposition 1 reduces QRPI to block coordinate maximization of the explicit dual (9): (13a) is exactly the stationarity equation of the strongly concave quadratic kappa in V, (13b) is the exact maximizer in phi, and the result then invokes Luo-Tseng [48, Th. 2.1]. Propositions 3 and 4 are likewise imported from the external infeasible-DRA analysis [41], [42], with the minimal displacement vector v = argmin_{beta in D-C} ||beta|| defined purely from the problem data; no algorithm output is fitted and then renamed as a prediction. The numerical claims are benchmarked against Gurobi, SCS, and an independently implemented primal-dual method, so the performance statements are not calibrated to the paper's own outputs. The only substantive caveat is in Section V: the implemented algorithm fixes ell = 2 inner QRPI iterations, while the formal guarantee applies when the inner problem is solved exactly or when the prox error sequence e_k = ||d_k - prox_{sigma f}(w_k)|| is summable, and the paper supplies no proof of summability for the fixed ell = 2 scheme. That is an incompleteness or correctness gap, not a circularity: it does not redefine any theorem's conclusion as its own input. The only co-authored citation, [39], supports a non-load-bearing remark that policy iteration is a Newton method and does not carry the convergence analysis.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central derivation rests on standard convex analysis and monotone operator theory, all external to the authors. The only paper-specific assumptions are the standing convexity of C, the constraint qualification for the exact convergence theorem, and the ad hoc inexactness of the inner loop in the practical algorithm. Hyperparameters sigma, omega, ell, and tolerances are hand-chosen in experiments but do not affect the theoretical convergence, which holds for any sigma > 0 and omega in (0,2) in the exact case.

free parameters (4)
  • sigma = 2e-5 (grid-world infeasible cases use smaller values, unspecified)
    Scaling parameter of the proximal updates. The authors give intuitive guidance (large sigma prioritizes cost, small sigma prioritizes constraints) but the value is hand-chosen in experiments and affects the transient behavior and speed.
  • omega (relaxation parameter) = 1.5
    Relaxation parameter for the DRA update, standard range (0,2). Chosen by hand based on prior numerical studies; not required for convergence.
  • ell (inner QRPI iterations) = 2
    Number of QRPI inner iterations used in the implementation. The theoretical convergence claims require ell = infinity; the practical algorithm uses ell = 2 with only numerical justification.
  • Termination tolerances = epsilon_opt = 1e-5, epsilon_con = 1e-4, epsilon_inf = 1e-6
    Termination tolerances for optimality, constraint violation, and infeasibility detection. In infeasible grid-world cases the authors state smaller tolerances are used but do not specify exact values.
assumptions (6)
  • domain assumption The set C is non-empty, closed, and convex (Standing Assumption 1).
    This is the standing assumption on the constraint set throughout the paper, used to define the projection and the indicator function g = I_C.
  • domain assumption Assumption 1: either D ∩ ri C != empty, or C is polyhedral and D ∩ C != empty.
    Required for the convergence of the exact DRA iteration via the qualification condition 0 in ∂f + ∂g, invoked in Proposition 2.
  • domain assumption The MDP is finite, tabular, discounted (gamma in (0,1)), with known dynamics and initial distribution.
    The entire formulation relies on the occupancy polytope D being a compact polytope and on known transition probabilities P.
  • standard math External DRA and coordinate-descent convergence results are correct (Bauschke-Combettes Cor. 28.3, Luo-Tseng Th. 2.1, Banjac 2021, Bauschke-Moursi 2023).
    The proofs of Propositions 2, 3, and 4 directly invoke these external theorems.
  • standard math The Gram matrix (gamma P - Xi)^T (gamma P - Xi) is positive definite (Lemma 1).
    Used to define the QRPI linear solve (13a) and to ensure unicity of the V update; proved in Appendix A via standard dynamic programming arguments.
  • ad hoc to paper The inexact inner loop with ell = 2 behaves like the exact inner loop.
    The practical algorithm uses finite ell with no proof of the summability condition on inner errors; the paper provides only numerical evidence in Appendix C. This is an additional ad hoc assumption for the implemented variant.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Operator Splitting for Convex Constrained Markov Decision Processes." pith.science (2026). https://pith.science/paper/XVWZDOG6

@misc{pith2026241214002,
  author       = {Pith},
  title        = {Pith review of: Operator Splitting for Convex Constrained Markov Decision Processes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XVWZDOG6}},
  note         = {Machine review of arXiv:2412.14002}
}
read the original abstract

We consider finite Markov decision processes (MDPs) with convex constraints and known dynamics. In principle, this problem is amenable to off-the-shelf convex optimization solvers, but typically this approach suffers from poor scalability. In this work, we develop a first-order algorithm, based on the Douglas-Rachford splitting, that allows us to decompose the dynamics and constraints. Thanks to this decoupling, we can incorporate a wide variety of convex constraints. Our scheme consists of simple and easy-to-implement updates that alternate between solving a regularized MDP and a projection. The inherent presence of regularized updates ensures last-iterate convergence, numerical stability, and, contrary to existing approaches, does not require us to regularize the problem explicitly. If the constraints are not attainable, we exploit salient properties of the Douglas-Rachord algorithm to detect infeasibility and compute a policy that minimally violates the constraints. We demonstrate the performance of our algorithm on two benchmark problems and show that it compares favorably to competing approaches.

Figures

Figures reproduced from arXiv: 2412.14002 by the authors.

Figure 1
Figure 1. Decomposition of A(V ) onto d and φ. Informally, if [A(V in ℓ )](s,a) < 0 then playing action a at state s will improve performance, up to a slack of wk/σ, therefore, we use it to compute d in ℓ . Conversely, state-action pairs satisfying [A(V in ℓ )](s,a) > 0 are undesirable as they would deteriorate performance, hence are placed in the dual occupancy φ in ℓ . [32, Examples 3.18, 29.27], convex cones [32, Examples … view at source ↗
Figure 2
Figure 2. We stress that the results of this subsection only require Standing Assumption 1, and not Assumption 1. Propositions 3 and 4 characterize the asymptotic behavior of the DRA irrespective of feasibility. D. Termination Criteria To specify meaningful termination criteria, we derive the optimality conditions of (6). First, we introduce the auxiliary [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 2
Figure 2. Asymptotic behavior of Alg. 1 for infeasible problems. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: State marginal occupancy measure (in color) and policy (as arrows) for two feasible choices of [PITH_FULL_IMAGE:figures/full_fig_p009_3.png]
Figure 4
Figure 4. Figure 4: State marginal occupancy measure (in color) and policy (as arrows) for two infeasible choices of [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Performance comparison of Alg. 1 with exact (continuous line) and inexact (dotted line) QRPI in the inner loop. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 47 canonical work pages

  1. [28]

    Policy-based primal-dual methods for convex constrained Markov decision processes,

    D. Ying, M. A. Guo, Y . Ding, J. Lavaei, and Z.-J. Shen, “Policy-based primal-dual methods for convex constrained Markov decision processes,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 37, pp. 10963–10971, 2023

    work page 2023

  2. [30]

    Reinforcement learning with convex constraints,

    S. Miryoosefi, K. Brantley, H. Daume III, M. Dudik, and R. E. Schapire, “Reinforcement learning with convex constraints,” Advances in neural information processing systems , vol. 32, 2019

    work page 2019

  3. [48]

    On the convergence of the coordinate descent method for convex differentiable minimization,

    Z.-Q. T. Luo and P. Tseng, “On the convergence of the coordinate descent method for convex differentiable minimization,” Journal of Optimization Theory and Applications , vol. 72, pp. 7–35, 1992

    work page 1992

  4. [1]

    Mastering the game of go without human knowledge,

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. , “Mastering the game of go without human knowledge,” Nature, vol. 550, no. 7676, pp. 354–359, 2017

    work page 2017

  5. [2]

    Magnetic control of tokamak plasmas through deep reinforcement learning,

    J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmaleki, D. de Las Casas, et al., “Magnetic control of tokamak plasmas through deep reinforcement learning,” Nature, vol. 602, no. 7897, pp. 414–419, 2022

    work page 2022

  6. [3]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018

    2018

  7. [4]

    Altman, Constrained Markov decision processes

    E. Altman, Constrained Markov decision processes . Routledge, 2021

    2021

  8. [5]

    Policy gradients with variance related risk criteria,

    A. Tamar, D. Di Castro, and S. Mannor, “Policy gradients with variance related risk criteria,” in Proceedings of the twenty-ninth international conference on machine learning , pp. 387–396, 2012

    work page 2012

Show all 62 references
  1. [6]

    Risk-constrained reinforcement learning with percentile risk criteria,

    Y . Chow, M. Ghavamzadeh, L. Janson, and M. Pavone, “Risk-constrained reinforcement learning with percentile risk criteria,” Journal of Machine Learning Research, vol. 18, no. 167, pp. 1–51, 2018

    2018

  2. [7]

    Control and optimization meet the smart power grid: Scheduling of power demands for optimal energy management,

    I. Koutsopoulos and L. Tassiulas, “Control and optimization meet the smart power grid: Scheduling of power demands for optimal energy management,” in Proceedings of the 2nd International Conference on Energy-efficient Computing and Networking , pp. 41–50, 2011

    2011

  3. [8]

    Constrained policy optimization,

    J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in Proceedings of the 34th International Conference on Machine Learning (D. Precup and Y . W. Teh, eds.), vol. 70 ofProceedings of Machine Learning Research , pp. 22–31, PMLR, 06–11 Aug 2017

    2017

  4. [9]

    Dynamic programming equations for dis- counted constrained stochastic control,

    R. Chen and G. Blankenship, “Dynamic programming equations for dis- counted constrained stochastic control,” IEEE Transactions on Automatic Control, vol. 49, no. 5, pp. 699–709, 2004

    2004

  5. [10]

    Dynamic programming in constrained Markov decision processes,

    A. B. Piunovskiy, “Dynamic programming in constrained Markov decision processes,” Control and Cybernetics, vol. 35, no. 3, pp. 645–660, 2006

    2006

  6. [11]

    A gradient-aware search algorithm for constrained Markov decision processes,

    S. Khairy, P. Balaprakash, and L. X. Cai, “A gradient-aware search algorithm for constrained Markov decision processes,” arXiv preprint arXiv:2005.03718, 2020. 12

    2005 arXiv

  7. [12]

    Natural policy gradient primal-dual method for constrained Markov decision processes,

    D. Ding, K. Zhang, T. Basar, and M. Jovanovic, “Natural policy gradient primal-dual method for constrained Markov decision processes,” in Advances in Neural Information Processing Systems (H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, eds.), vol. 33, pp. 8378– ...

    2020

  8. [13]

    Learning policies with zero or bounded constraint violation for constrained MDPs,

    T. Liu, R. Zhou, D. Kalathil, P. Kumar, and C. Tian, “Learning policies with zero or bounded constraint violation for constrained MDPs,” Advances in Neural Information Processing Systems , vol. 34, pp. 17183– 17193, 2021

    2021

  9. [14]

    State augmented constrained reinforcement learning: Overcoming the limitations of learning with rewards,

    M. Calvo-Fullana, S. Paternain, L. F. O. Chamon, and A. Ribeiro, “State augmented constrained reinforcement learning: Overcoming the limitations of learning with rewards,” arXiv preprint arXiv:2102.11941 , 2023

    2023 arXiv

  10. [15]

    Constrained MDPs and the reward hypothesis

    C. Szepesvári, “Constrained MDPs and the reward hypothesis.” https://readingsml.blogspot.com/2020/03/ constrained-mdps-and-reward-hypothesis.html

    2020

  11. [16]

    Two “well-known

    K. M. Anstreicher and L. A. Wolsey, “Two “well-known” properties of subgradient optimization,” Mathematical Programming, vol. 120, pp. 213– 220, 2009

    2009

  12. [17]

    Algorithm for constrained Markov decision process with linear convergence,

    E. Gladin, M. Lavrik-Karmazin, K. Zainullina, V . Rudenko, A. Gasnikov, and M. Takac, “Algorithm for constrained Markov decision process with linear convergence,” in International Conference on Artificial Intelligence and Statistics, pp. 11506–11533, PMLR, 2023

    2023

  13. [18]

    Faster algorithm and sharper analysis for constrained Markov decision process,

    T. Li, Z. Guan, S. Zou, T. Xu, Y . Liang, and G. Lan, “Faster algorithm and sharper analysis for constrained Markov decision process,” arXiv preprint arXiv:2110.10351, 2021

    2021 arXiv

  14. [19]

    Cancellation-free regret bounds for lagrangian approaches in constrained Markov decision processes,

    A. Müller, P. Alatur, G. Ramponi, and N. He, “Cancellation-free regret bounds for lagrangian approaches in constrained Markov decision processes,” arXiv preprint arXiv:2306.07001 , 2023

    2023 arXiv

  15. [20]

    Last-iterate convergent policy gradient primal-dual methods for constrained MDPs,

    D. Ding, C.-Y . Wei, K. Zhang, and A. Ribeiro, “Last-iterate convergent policy gradient primal-dual methods for constrained MDPs,” arXiv preprint arXiv:2306.11700, 2024

    2024 arXiv

  16. [21]

    Reload: Reinforcement learning with optimistic ascent- descent for last-iterate convergence in constrained MDPs,

    T. Moskovitz, B. O’Donoghue, V . Veeriah, S. Flennerhag, S. Singh, and T. Zahavy, “Reload: Reinforcement learning with optimistic ascent- descent for last-iterate convergence in constrained MDPs,” in Interna- tional Conference on Machine Learning , pp. 25303–25336, PMLR, 2023

    2023

  17. [22]

    Ipo: Interior-point policy optimization under constraints,

    Y . Liu, J. Ding, and X. Liu, “Ipo: Interior-point policy optimization under constraints,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, pp. 4940–4947, 2020

    2020

  18. [23]

    Projection- based constrained policy optimization,

    T.-Y . Yang, J. Rosca, K. Narasimhan, and P. J. Ramadge, “Projection- based constrained policy optimization,” arXiv preprint arXiv:2010.03152, 2020

    2010 arXiv

  19. [24]

    Reward is enough for convex MDPs,

    T. Zahavy, B. O’Donoghue, G. Desjardins, and S. Singh, “Reward is enough for convex MDPs,” Advances in Neural Information Processing Systems, vol. 34, pp. 25746–25759, 2021

    2021

  20. [25]

    Apprenticeship learning via inverse reinforce- ment learning,

    P. Abbeel and A. Y . Ng, “Apprenticeship learning via inverse reinforce- ment learning,” in Proceedings of the twenty-first international conference on Machine learning , p. 1, 2004

    2004

  21. [26]

    Provably efficient maximum entropy exploration,

    E. Hazan, S. Kakade, K. Singh, and A. Van Soest, “Provably efficient maximum entropy exploration,” in International Conference on Machine Learning, pp. 2681–2691, PMLR, 2019

    2019

  22. [27]

    Diversity is all you need: Learning skills without a reward function,

    B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, “Diversity is all you need: Learning skills without a reward function,” arXiv preprint arXiv:1802.06070, 2018

    2018 arXiv

  23. [29]

    Variational policy gradient method for reinforcement learning with general utilities,

    J. Zhang, A. Koppel, A. S. Bedi, C. Szepesvari, and M. Wang, “Variational policy gradient method for reinforcement learning with general utilities,” Advances in Neural Information Processing Systems , vol. 33, pp. 4572– 4583, 2020

    2020

  24. [31]

    A simple reward-free approach to constrained reinforcement learning,

    S. Miryoosefi and C. Jin, “A simple reward-free approach to constrained reinforcement learning,” in International Conference on Machine Learn- ing, pp. 15666–15698, PMLR, 2022

    2022

  25. [32]

    Bauschke and P

    H. Bauschke and P. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces . CMS Books in Mathematics, Springer International Publishing, 2017

    2017

  26. [33]

    Distributed optimization and statistical learning via the alternating direction method of multipliers,

    S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, et al., “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends® in Machine learning , vol. 3, no. 1, pp. 1–122, 2011

    2011

  27. [34]

    A note on the equivalence of operator splitting methods,

    W. M. Moursi and Y . Zinchenko, “A note on the equivalence of operator splitting methods,” Splitting Algorithms, Modern Operator Theory, and Applications, pp. 331–349, 2019

    2019

  28. [35]

    Provably efficient algorithms for multi-objective competitive RL,

    T. Yu, Y . Tian, J. Zhang, and S. Sra, “Provably efficient algorithms for multi-objective competitive RL,” in International Conference on Machine Learning, pp. 12167–12176, PMLR, 2021

    2021

  29. [36]

    A splitting method for optimal control,

    B. O’Donoghue, G. Stathopoulos, and S. Boyd, “A splitting method for optimal control,” IEEE Transactions on Control Systems Technology , vol. 21, no. 6, pp. 2432–2442, 2013

    2013

  30. [37]

    A unified view of entropy-regularized Markov decision processes,

    G. Neu, A. Jonsson, and V . Gómez, “A unified view of entropy-regularized Markov decision processes,” arXiv preprint arXiv:1705.07798 , 2017

    2017 arXiv

  31. [38]

    A theory of regularized Markov decision processes,

    M. Geist, B. Scherrer, and O. Pietquin, “A theory of regularized Markov decision processes,” in Proceedings of the 36th International Conference on Machine Learning (K. Chaudhuri and R. Salakhutdinov, eds.), vol. 97 of Proceedings of Machine Learning Research , pp. 2160–2169, ...

    2019

  32. [39]

    Dynamic programming through the lens of semismooth Newton-type methods,

    M. Gargiani, A. Zanelli, D. Liao-McPherson, T. Summers, and J. Lygeros, “Dynamic programming through the lens of semismooth Newton-type methods,” IEEE Control Systems Letters , vol. 6, pp. 2996–3001, 2022

    2022

  33. [40]

    From optimization to control: quasi policy iteration,

    M. A. S. Kolarijani and P. M. Esfahani, “From optimization to control: quasi policy iteration,” arXiv preprint arXiv:2311.11166 , 2023

    2023

  34. [41]

    On the minimal displacement vector of the Douglas–Rachford operator,

    G. Banjac, “On the minimal displacement vector of the Douglas–Rachford operator,” Operations Research Letters, vol. 49, no. 2, pp. 197–200, 2021

    2021

  35. [42]

    On the Douglas–Rachford algorithm for solving possibly inconsistent optimization problems,

    H. H. Bauschke and W. M. Moursi, “On the Douglas–Rachford algorithm for solving possibly inconsistent optimization problems,” Mathematics of Operations Research , 2023

    2023

  36. [43]

    Infeasibility detection in alternating direction method of multipliers for convex quadratic programs,

    A. U. Raghunathan and S. Di Cairano, “Infeasibility detection in alternating direction method of multipliers for convex quadratic programs,” in 53rd IEEE Conference on Decision and Control, pp. 5819–5824, IEEE, 2014

    2014

  37. [44]

    A new use of Douglas-Rachford splitting and admm for identifying infeasible, unbounded, and pathological conic programs,

    Y . Liu, E. K. Ryu, and W. Yin, “A new use of Douglas-Rachford splitting and admm for identifying infeasible, unbounded, and pathological conic programs,” arXiv preprint arXiv:1706.02374 , 2017

    2017 arXiv

  38. [45]

    Infeasibility detection in the alternating direction method of multipliers for convex optimization,

    G. Banjac, P. Goulart, B. Stellato, and S. Boyd, “Infeasibility detection in the alternating direction method of multipliers for convex optimization,” Journal of Optimization Theory and Applications , vol. 183, pp. 490–519, 2019

    2019

  39. [46]

    R. T. Rockafellar and R. J.-B. Wets, Variational analysis, vol. 317. Springer Science & Business Media, 2009

    2009

  40. [47]

    On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators,

    J. Eckstein and D. P. Bertsekas, “On the Douglas—Rachford splitting method and the proximal point algorithm for maximal monotone operators,” Mathematical programming, vol. 55, pp. 293–318, 1992

    1992

  41. [49]

    Nocedal and S

    J. Nocedal and S. Wright, Numerical Optimization. Springer Series in Operations Research and Financial Engineering, Springer New York, 2006

    2006

  42. [50]

    Operator-splitting methods for monotone affine variational inequalities, with a parallel application to optimal control,

    J. Eckstein and M. C. Ferris, “Operator-splitting methods for monotone affine variational inequalities, with a parallel application to optimal control,” INFORMS Journal on Computing , vol. 10, no. 2, pp. 218–235, 1998

    1998

  43. [51]

    Parallel alternating direction multiplier decomposition of convex programs,

    J. Eckstein, “Parallel alternating direction multiplier decomposition of convex programs,” Journal of Optimization Theory and Applications , vol. 80, no. 1, pp. 39–62, 1994

    1994

  44. [52]

    Natural Actor- Critic Algorithms,

    S. Bhatnagar, R. Sutton, M. Ghavamzadeh, and M. Lee, “Natural Actor- Critic Algorithms,” Automatica, vol. 45, July 2009

    2009

  45. [53]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. , “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems , vol. 32, 2019

    2019

  46. [54]

    PID accelerated value iteration algorithm,

    A.-m. Farahmand and M. Ghavamzadeh, “PID accelerated value iteration algorithm,” in International Conference on Machine Learning , pp. 3143– 3153, PMLR, 2021

    2021

  47. [55]

    Scalable first-order methods for robust MDPs,

    J. Grand-Clément and C. Kroer, “Scalable first-order methods for robust MDPs,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, pp. 12086–12094, 2021

    2021

  48. [56]

    Integrating a partial model into model free reinforcement learning.,

    A. Tamar, D. Di Castro, R. Meir, and P. Dayan, “Integrating a partial model into model free reinforcement learning.,” Journal of Machine Learning Research, vol. 13, no. 6, 2012

    2012

  49. [57]

    Gurobi Optimizer Reference Manual,

    Gurobi Optimization, LLC, “Gurobi Optimizer Reference Manual,” 2023

    2023

  50. [58]

    Safe policies for reinforcement learning via primal-dual methods,

    S. Paternain, M. Calvo-Fullana, L. F. Chamon, and A. Ribeiro, “Safe policies for reinforcement learning via primal-dual methods,” IEEE Transactions on Automatic Control , vol. 68, no. 3, pp. 1321–1336, 2022

    2022

  51. [59]

    Conic optimization via operator splitting and homogeneous self-dual embedding,

    B. O’donoghue, E. Chu, N. Parikh, and S. Boyd, “Conic optimization via operator splitting and homogeneous self-dual embedding,” Journal of Optimization Theory and Applications , vol. 169, pp. 1042–1068, 2016. 13

    2016

  52. [60]

    Reward constrained policy optimization,

    C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,” arXiv preprint arXiv:1805.11074 , 2018

    2018 arXiv

  53. [61]

    Markov decision processes,

    M. L. Puterman, “Markov decision processes,” Handbooks in operations research and management science , vol. 2, pp. 331–434, 1990

    1990

  54. [62]

    S. P. Boyd and L. Vandenberghe, Convex optimization . Cambridge university press, 2004

    2004

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.