Pith. sign in

REVIEW 4 minor 142 references

On the Policy Convergence of Policy Mirror Descent Methods

T0 review · 0 major / 4 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Unregularized policy mirror descent with any constant step size converges to an optimal policy for a wide class of mirror maps, even when many optima exist.

desk verdict Clean unification of policy (not just value) convergence for constant-step unregularized PMD under general decomposable Legendre maps, with a useful four-case taxonomy. read the letter →

arxiv 2607.11626 v1 pith:TCMVQURW submitted 2026-07-13 math.OC

Wenye Li , Ke Wei This is my paper
classification math.OC MSC 90C4090C2568T05
keywords policymirrordescentconvergenceLegendremapsconstantstepsizefinite-timeterminationsoftmaxnaturalgradientprojectedQ-ascentMarkovdecisionprocess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Value convergence for policy mirror descent has been known for years, but value can approach the optimum without the policy itself settling when multiple optimal policies exist. This paper proves that, for any decomposable Legendre-type mirror map of the form sum of a scalar function psi, unregularized PMD with a fixed step size produces a policy sequence that actually converges to one limiting optimal policy. The same theory covers the squared Euclidean map behind projected Q-ascent, the negative entropy behind softmax natural policy gradient, Tsallis, Hellinger, and Fermi-Dirac maps. The qualitative behavior is completely determined by whether psi is differentiable at the endpoints 0 and 1: finite-time termination, pure asymptotic convergence, or an MDP-dependent split between the two. When the map also has positive finite curvature, the paper supplies matching local rates for both value error and policy distance.

What carries the argument

The summability criterion (Lemma 3.1): once the value errors are summable, Bregman distances to any optimal policy form a quasi-Fejer sequence whose only accumulation point is a single optimal policy; local dual-gap quantities G_k then supply the missing summability via the boundary behavior of psi'.

What would settle it

Construct a finite discounted MDP and a decomposable Legendre map satisfying the paper's assumptions for which the constant-step PMD sequence has two distinct accumulation points that are both optimal; any such counter-example would refute the global policy-convergence claim.

Watch

Extended reading notes

Core claim

For every decomposable mirror map h(p)=sum psi(p(a)) obeying the standard Legendre conditions, the unregularized PMD iteration with arbitrary constant step size generates a policy sequence that converges in the policy domain to a limiting optimal policy, even when the optimal set is not a singleton. Differentiability of psi at 0 and 1 partitions the maps into four regimes that respectively yield finite-time termination, asymptotic convergence, an MDP-dependent dichotomy, or asymptotic convergence with half-rate local bounds.

Load-bearing premise

The map must send gradients to infinity at every boundary point of its domain; without that essential-smoothness condition the dual-gap growth arguments and the four-case classification fail.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 4 minor

Summary. The paper establishes policy-domain convergence of unregularized policy mirror descent (PMD) with arbitrary constant step sizes for finite discounted MDPs, under decomposable Legendre-type mirror maps h(p)=\sum_a \psi(p(a)) satisfying Assumption 1.1. The central claim is that the generated policy sequence converges to a limiting optimal policy even when the optimal set is non-singleton. Convergence regimes are classified by the one-sided differentiability of \psi at 0 and 1 (finite-time termination, asymptotic convergence, or an MDP-dependent dichotomy), with local value-error bounds and policy rates under additional C^{2} curvature assumptions. The argument proceeds via a summability criterion (Lemma 3.1), dual-gap evolution (Lemma 3.2), and an integral test (Lemma 3.3), with complete proofs in Sections 4–5 and supporting grid-world experiments.

Significance. If correct, this supplies the first systematic, unified policy-convergence theory for unregularized constant-step PMD across the standard family of decomposable Legendre maps (squared Euclidean/PQA, Shannon entropy/softmax NPG, Tsallis, Hellinger, Fermi–Dirac). Prior guarantees were either instance-specific (relying on closed-form updates) or required regularization/homotopy. The four-case taxonomy, the reduction of policy convergence to value-error summability, and the local rates under positive finite curvature are technically substantial and immediately useful for understanding which mirror maps produce finite-time versus asymptotic behavior. Full proofs and matching numerical illustrations strengthen the contribution.

minor comments (4)
  1. Table 1 is dense; a short caption sentence clarifying that “/” means “not applicable / finite-time so no local rate” would help readers scan the four cases.
  2. In the numerical section the bisection accuracy 10^{-13} and fixed \eta=0.1 are stated, but a one-line remark that the qualitative regimes are insensitive to modest changes in these parameters would improve reproducibility claims.
  3. A few typographical slips remain (e.g., “Inthis paper”, missing spaces after periods in the introduction). A final proof-reading pass would clean them.
  4. The comparison with Lin & Zhang (2022) L-coercivity is clear in Remark 3.1; adding a one-sentence pointer in the abstract or introduction that the present result removes that assumption for the Tsallis q\in(1,2) case would make the novelty more immediately visible.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: self-contained Legendre/Bregman analysis with comparison-only self-citations

full rationale

This is a pure mathematical optimization paper. The central claim (policy convergence of unregularized constant-step PMD for decomposable Legendre maps under Assumption 1.1, even when Π* is non-singleton) is derived from first-order KKT conditions of the PMD update (Lemma 2.2), the three-point Bregman identity (Lemma 2.3), dual-gap evolution (Lemma 3.2), a general summability criterion reducing policy convergence to ∑∥V*−Vπk∥∞<∞ (Lemma 3.1), and case-by-case local bounds obtained by tracking ψ′-gaps and applying the integral inverse criterion (Lemma 3.3). All proofs are written out in Sections 4–5; no parameters are fitted to data, no uniqueness theorem is imported as an external black box that forces the result, and no ansatz is smuggled in. Self-citations (Li et al. on HPMD and softmax NPG; Liu et al. on PQA) appear only for comparison of prior special cases or regularized variants; the unregularized constant-step theory for general decomposable maps is developed independently and does not rest on those results as load-bearing lemmas. The four-case taxonomy is a classification by boundary differentiability of ψ, not a renaming of a known empirical pattern. Score 0 is therefore appropriate.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

Pure mathematical theory resting on standard convex-analysis and MDP facts; no free parameters are fitted to data. The only non-standard ingredients are the Legendre-type conditions on ψ and the finite discounted MDP setting, both stated explicitly.

assumptions (4)
  • domain assumption ψ satisfies Assumption 1.1 (proper closed convex, essentially smooth and strictly convex on int dom ψ, dom ψ ⊇ [0,1])
    Invoked from the first statement of the main theorems; essential for Bregman divergence properties and dual-gap monotonicity.
  • domain assumption Finite state-action discounted MDP with rewards in [0,1] and γ∈[0,1)
    Standard setting of Section 2; used for bounded values, existence of optimal policies, and the sub-optimality gap Δ.
  • standard math Three-point identity and first-order optimality conditions for Bregman projections onto the simplex (Lemmas 2.2–2.3)
    Classical convex-analysis facts (Rockafellar, Beck) used to derive the dual update and descent inequality.
  • standard math Global O(1/k) value convergence of PMD under Legendre maps (Xiao 2022, Lemma 2.4)
    Imported to guarantee that the local entrance time T(σ) is finite; the paper’s novelty begins after this global bound.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On the Policy Convergence of Policy Mirror Descent Methods." pith.science (2026). https://pith.science/paper/TCMVQURW

@misc{pith2026260711626,
  author       = {Pith},
  title        = {Pith review of: On the Policy Convergence of Policy Mirror Descent Methods},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TCMVQURW}},
  note         = {Machine review of arXiv:2607.11626}
}
abstract

We study the policy convergence of unregularized policy mirror descent (PMD) with arbitrary constant step sizes for finite discounted Markov decision processes. We focus on decomposable mirror maps of the form $h(p)=\sum_a \psi(p(a))$, where $\psi$ satisfies standard Legendre-type assumptions. Under these conditions, we prove that the policy sequence generated by PMD converges in the policy domain to a limiting optimal policy, even when the optimal policy set is not a singleton. This result covers a broad class of commonly used mirror maps, including the squared Euclidean mirror map underlying projected Q-ascent, the negative Shannon entropy underlying softmax natural policy gradient, Tsallis entropy, the Hellinger mapping, and the Fermi-Dirac entropy. Although policy convergence has been established previously for specific PMD instances or for regularized variants such as homotopic PMD, to the best of our knowledge, this is the first systematic and unified policy convergence theory for unregularized PMD under general decomposable mirror maps and arbitrary constant step sizes. Our analysis further reveals that the convergence behavior is governed by the differentiability of $\psi$ at $0$ and $1$, leading to different behaviors, including finite-time termination, asymptotic convergence, and an MDP-dependent dichotomy. When $\psi$ is twice continuously differentiable with strictly positive finite curvature, we further establish local policy convergence rates for the asymptotic convergence cases, covering the standard mirror maps mentioned above.

Figures

Figures reproduced from arXiv: 2607.11626 by the authors.

Figure 1
Figure 1. Two Grid World MDPs that used in our experiment setting. [PITH_FULL_IMAGE:figures/full_fig_p017_1.png] view at source ↗
Figure 2
Figure 2. The plot of ∥πk+1 − πk∥2 for Case 1, tested on Grid World A. Tsallis entropy with q = 1/2, and use πlast := πT as an approximation to the limiting policy π∞. For the two representative states s1 = 3 ∈ S=1 and s2 = 7 ∈ S>1, we plot the curves of log [PITH_FULL_IMAGE:figures/full_fig_p019_2.png] view at source ↗
Figure 3
Figure 3. The curves of log ∥πk(·|s) − πlast(·|s)∥2 for Case 2, tested on Grid World A (Figure 1a). The blue curve shows the result on state s1 = 3 ∈ S=1 and the orange one corresponds to state s2 = 7 ∈ S>1. Policy convergence dichotomy of Case 3. Since the policy convergence behavior in Case 3 differs depending on whether S>1 = S or S>1 ̸= S, we test PMD with the Hellinger mapping ψ(x) = − √ 1 − x 2 on both Grid Worlds. For … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Policy convergence results for PMD with Hellinger mapping on Grid Worlds A (left) and B (right). [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: Policy convergence result for PMD with Fermi-Dirac entropy on Grid World A. [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

142 extracted references · 15 linked inside Pith

  1. [1]

    Nature , volume=

    Human-level control through deep reinforcement learning , author=. Nature , volume=

  2. [2]

    Mastering the game of

    Silver, David and Huang, Aja and Maddison, Chris J and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and Dieleman, Sander and Grewe, Dominik and Nham, John and Kalchbrenner, Nal and Sutskever, Ilya and Lillicrap, Timothy and Leach, Madeleine and Ka...

  3. [3]

    2019 , journal=

    Dota 2 with Large Scale Deep Reinforcement Learning , author=. 2019 , journal=

    2019

  4. [4]

    arxiv:2404.03372 , year =

    Liu, Jiacai and Li, Wenye and Wei, Ke , title =. arxiv:2404.03372 , year =

    arXiv

  5. [5]

    Starcraft

    Vinyals, Oriol and others , journal=. Starcraft

  6. [6]

    Science Robotics , year=

    Learning agile and dynamic motor skills for legged robots , author=. Science Robotics , year=

  7. [7]

    Science Robotics , volume =

    Lee, Joonho and Hwangbo, Jemin and Wellhausen, Lorenz and Koltun, Vladlen and Hutter, Marco , title =. Science Robotics , volume =

  8. [8]

    Learning robust perceptive locomotion for quadrupedal robots in the wild , volume =

    Miki, Takahiro and Lee, Joonho and Hwangbo, Jemin and Wellhausen, Lorenz and Koltun, Vladlen and Hutter, Marco , year =. Learning robust perceptive locomotion for quadrupedal robots in the wild , volume =

Show all 142 references
  1. [9]

    arXiv:1606.03966 , year=

    Making Contextual Decisions with Low Technical Debt , author=. arXiv:1606.03966 , year=

    arXiv

  2. [10]

    , booktitle=

    Chen, Minmin and Beutel, Alex and Covington, Paul and Jain, Sagar and Belletti, Francois and Chi, Ed H. , booktitle=. Top-k off-policy correction for a

  3. [11]

    Nature , volume=

    A graph placement methodology for fast chip design , author=. Nature , volume=

  4. [12]

    Advances in Neural Information Processing Systems , year=

    Policy Gradient Methods for Reinforcement Learning with Function Approximation , author=. Advances in Neural Information Processing Systems , year=

  5. [13]

    2019 , pages=

    A Theory of Regularized Markov Decision Processes , author=. 2019 , pages=

    2019

  6. [14]

    Adaptive Trust Region Policy Optimization:

    Shani, Lior and Efroni, Yonathan and Mannor, Shie , year=. Adaptive Trust Region Policy Optimization:. Proceedings of the AAAI Conference on Artificial Intelligence , pages=

  7. [15]

    On the Convergence Rates of Policy Gradient Methods , journal=

    Xiao, Lin , year=. On the Convergence Rates of Policy Gradient Methods , journal=

  8. [16]

    Mathematical Programming , author=

    Policy Mirror Descent for Reinforcement Learning:. Mathematical Programming , author=. 2021 , pages=

    2021

  9. [17]

    Escaping the Gravitational Pull of Softmax , booktitle=

    Mei, Jincheng and Xiao, Chenjun and Dai, Bo and Li, Lihong and Szepesvári, Csaba and Schuurmans, Dale , year=. Escaping the Gravitational Pull of Softmax , booktitle=

  10. [18]

    2023 , booktitle=

    Optimal Convergence Rate for Exact Policy Mirror Descent in Discounted Markov Decision Processes , author=. 2023 , booktitle=

    2023

  11. [19]

    Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization , journal=

    Cen, Shicong and Cheng, Chen and Chen, Yuxin and Wei, Yuting and Chi, Yuejie , year=. Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization , journal=

  12. [20]

    Mathematical Programming , author=

    Homotopic policy mirror descent:. Mathematical Programming , author=

  13. [21]

    SIAM Journal on Optimization , author=

    Policy Mirror Descent for Regularized Reinforcement Learning:. SIAM Journal on Optimization , author=. 2023 , volume =

    2023

  14. [22]

    Journal of Machine Learning Research , author=

    On the Theory of Policy Gradient Methods:. Journal of Machine Learning Research , author=. 2021 , volume=

    2021

  15. [23]

    On the Global Convergence Rates of Softmax Policy Gradient Methods , booktitle=

    Mei, Jincheng and Xiao, Chenjun and Szepesvári, Csaba and Schuurmans, Dale , year=. On the Global Convergence Rates of Softmax Policy Gradient Methods , booktitle=

  16. [24]

    2023 , booktitle=

    Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies , author=. 2023 , booktitle=

    2023

  17. [25]

    2022 , journal=

    Linear Convergence for Natural Policy Gradient with Log-linear Policy Parametrization , author=. 2022 , journal=

    2022

  18. [26]

    On the Linear Convergence of Natural Policy Gradient Algorithm , booktitle=

    Khodadadian, Sajad and Jhunjhunwala, Prakirt Raj and Varma, Sushil Mahavir and Maguluri, Siva Theja , year=. On the Linear Convergence of Natural Policy Gradient Algorithm , booktitle=

  19. [27]

    2020 , pages =

    Advances in Neural Information Processing Systems , author=. 2020 , pages =

    2020

  20. [28]

    International Conference on Artificial Intelligence and Statistics , author=

    On the Linear Convergence of Policy Gradient Methods for Finite. International Conference on Artificial Intelligence and Statistics , author=. 2021 , pages=

    2021

  21. [29]

    , year =

    Wang, Weiran and Carreira-Perpiñán, Miguel A. , year =. Projection onto the probability simplex:

  22. [30]

    International Conference on Machine Learning , pages=

    Approximately Optimal Approximate Reinforcement Learning , author=. International Conference on Machine Learning , pages=

  23. [31]

    On the Linear Convergence of Policy Gradient under

    Liu, Jiacai and Chen, Jinchi and Wei, Ke , journal=. On the Linear Convergence of Policy Gradient under

  24. [32]

    , title =

    Bertsekas, Dimitri P. , title =

  25. [33]

    Improved and Generalized Upper Bounds on the Complexity of Policy Iteration , volume =

    Scherrer, Bruno , booktitle =. Improved and Generalized Upper Bounds on the Complexity of Policy Iteration , volume =

  26. [34]

    Journal of Machine Learning Research , year=

    On the Convergence of Projected Policy Gradient for Any Constant Step Sizes , author=. Journal of Machine Learning Research , year=

  27. [35]

    arXiv:1206.2459 , year=

    Renyi Divergence and Kullback-Leibler Divergence , author=. arXiv:1206.2459 , year=

    arXiv

  28. [36]

    International Conference on Machine Learning , year =

    Mei, Jincheng and Gao, Yue and Dai, Bo and Szepesvári, Csaba and Schuurmans, Dale , title=. International Conference on Machine Learning , year =

  29. [37]

    Machine Learning , year =

    Simple statistical gradient-following algorithms for connectionist reinforcement learning , author =. Machine Learning , year =

  30. [38]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto , title =

  31. [39]

    A natural policy gradient , year =

    Sham Kakade , booktitle =. A natural policy gradient , year =

  32. [40]

    International conference on machine learning , pages=

    Trust region policy optimization , author=. International conference on machine learning , pages=

  33. [41]

    arXiv:1707.06347 , year=

    Proximal policy optimization algorithms , author=. arXiv:1707.06347 , year=

    arXiv

  34. [42]

    Operations Research , year =

    Jalj Bhandari and Daniel Russo , title =. Operations Research , year =

  35. [43]

    AAAI Conference on Artifical Intelligence , year =

    Lior Shani and Yonathan Efroni and Shie Mannor , title =. AAAI Conference on Artifical Intelligence , year =

  36. [44]

    Advances in Neural Information Processing Systems , year =

    Carlo Alfano and Rui Yuan and Patrick Rebeschini , title =. Advances in Neural Information Processing Systems , year =

  37. [45]

    Mathematical Programming , year =

    Gen Li and Yuting Wei and Yuejie Chi and Yuxin Chen , title =. Mathematical Programming , year =

  38. [46]

    Puterman and Shelby L

    Martin L. Puterman and Shelby L. Brumelle , title =. Mathematics of Operations Research , year =

  39. [47]

    Advances in Neural Information Processing Systems , year =

    Ofir Nachum and Mohammad Norouzi and Kelvin Xu and Dale Schuurmans , title =. Advances in Neural Information Processing Systems , year =

  40. [48]

    Nature , author=

    Discovering faster matrix multiplication algorithms with reinforcement learning , volume=. Nature , author=. 2022 , pages=

    2022

  41. [49]

    Nature , year=

    Mastering the game of Go with deep neural networks and tree search , author=. Nature , year=

  42. [50]

    Nature , year=

    Highly accurate protein structure prediction with AlphaFold , volume=. Nature , year=

  43. [51]

    Puterman, Martin L , title=

  44. [52]

    Bellman, Richard , title =

  45. [53]

    Operations Research Letters , volume=

    Mirror descent and nonlinear projected subgradient methods for convex optimization , author=. Operations Research Letters , volume=. 2003 , publisher=

    2003

  46. [54]

    arXiv:1509.02971 , year=

    Continuous control with deep reinforcement learning , author=. arXiv:1509.02971 , year=

    arXiv

  47. [55]

    Advances in neural information processing systems , year=

    Policy gradient methods for reinforcement learning with function approximation , author=. Advances in neural information processing systems , year=

  48. [56]

    arxiv:2005.09814 , year =

    Manan Tomar and Lior Shani and Yonathan Efroni and Mohammad Ghavamzadeh , title =. arxiv:2005.09814 , year =

    2005 arXiv

  49. [57]

    arxiv:1506.02438 , year =

    John Schulman and Philipp Moritz and Sergey Levine and Michael Jordan and Pieter Abbeel , title =. arxiv:1506.02438 , year =

    arXiv

  50. [58]

    International Conference on Machine Learning , pages=

    Stochastic gradient succeeds for bandits , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  51. [59]

    arXiv preprint arXiv:2405.13136 , year=

    Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPs , author=. arXiv preprint arXiv:2405.13136 , year=

    arXiv

  52. [60]

    2019 IEEE 58th Conference on Decision and Control (CDC) , pages=

    Convergence and iteration complexity of policy gradient method for infinite-horizon reinforcement learning , author=. 2019 IEEE 58th Conference on Decision and Control (CDC) , pages=. 2019 , organization=

    2019

  53. [61]

    SIAM Journal on Control and Optimization , volume=

    Global convergence of policy gradient methods to (almost) locally optimal policies , author=. SIAM Journal on Control and Optimization , volume=. 2020 , publisher=

    2020

  54. [62]

    arXiv preprint arXiv:2411.04913 , year=

    Structure Matters: Dynamic Policy Gradient , author=. arXiv preprint arXiv:2411.04913 , year=

    arXiv

  55. [63]

    Advances in neural information processing systems , volume=

    Policy search by dynamic programming , author=. Advances in neural information processing systems , volume=

  56. [64]

    2003 , publisher=

    On the sample complexity of reinforcement learning , author=. 2003 , publisher=

    2003

  57. [65]

    SIAM Journal on Optimization , volume=

    Convergence analysis of a proximal-like minimization algorithm using Bregman functions , author=. SIAM Journal on Optimization , volume=. 1993 , publisher=

    1993

  58. [66]

    International Conference on Machine Learning , pages=

    Safe reinforcement learning using advantage-based intervention , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  59. [67]

    Advances in Neural Information Processing Systems , volume=

    Policy improvement via imitation of multiple oracles , author=. Advances in Neural Information Processing Systems , volume=

  60. [68]

    arXiv preprint arXiv:2108.05828 , year=

    A functional mirror ascent view of policy gradient methods with function approximation , author=. arXiv preprint arXiv:2108.05828 , year=

    arXiv

  61. [69]

    Advances in Neural Information Processing Systems , volume=

    An operator view of policy gradient methods , author=. Advances in Neural Information Processing Systems , volume=

  62. [70]

    International Conference on Learning Representations , year=

    Linear convergence of natural policy gradient methods with log-linear policies , author=. International Conference on Learning Representations , year=

  63. [71]

    International Conference on Learning Representations , year=

    Mirror descent policy optimization , author=. International Conference on Learning Representations , year=

  64. [72]

    International Conference on Artificial Intelligence and Statistics , pages=

    A general sample complexity analysis of vanilla policy gradient , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2022 , organization=

    2022

  65. [73]

    arXiv preprint arXiv:2407.16602 , year=

    Functional Acceleration for Policy Mirror Descent , author=. arXiv preprint arXiv:2407.16602 , year=

    arXiv

  66. [74]

    1998 , publisher=

    Introduction to reinforcement learning , author=. 1998 , publisher=

    1998

  67. [75]

    International Conference on Machine Learning , pages=

    Stochastic policy gradient methods: Improved sample complexity for fisher-non-degenerate policies , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  68. [76]

    International Conference on Artificial Intelligence and Statistics , pages=

    Improved sample complexity analysis of natural policy gradient algorithm with general parameterization for infinite horizon discounted reward markov decision processes , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2024 , organization=

    2024

  69. [77]

    Journal of Scientific Computing , volume=

    Global Convergence of Natural Policy Gradient with Hessian-Aided Momentum Variance Reduction , author=. Journal of Scientific Computing , volume=. 2024 , publisher=

    2024

  70. [78]

    Advances in Neural Information Processing Systems , volume=

    An improved analysis of (variance-reduced) policy gradient and natural policy gradient methods , author=. Advances in Neural Information Processing Systems , volume=

  71. [79]

    Mathematics of Operations Research , year=

    Stochastic first-order methods for average-reward markov decision processes , author=. Mathematics of Operations Research , year=

  72. [80]

    Advances in Neural Information Processing Systems , volume=

    Policy Mirror Descent with Lookahead , author=. Advances in Neural Information Processing Systems , volume=

  73. [81]

    Journal of Machine Learning Research , volume=

    On the convergence of optimistic policy iteration , author=. Journal of Machine Learning Research , volume=

  74. [82]

    International Conference on Artificial Intelligence and Statistics , pages=

    On the convergence of policy iteration-based reinforcement learning with monte carlo policy evaluation , author=. International Conference on Artificial Intelligence and Statistics , pages=. 2023 , organization=

    2023

  75. [83]

    Advances in Neural Information Processing Systems , volume=

    Leverage the average: an analysis of kl regularization in reinforcement learning , author=. Advances in Neural Information Processing Systems , volume=

  76. [84]

    Advances in Neural Information Processing Systems , volume=

    Performance bounds for policy-based average reward reinforcement learning algorithms , author=. Advances in Neural Information Processing Systems , volume=

  77. [85]

    2025 , booktitle=

    -update: A class of policy update methods with policy convergence guarantee , author=. 2025 , booktitle=

    2025

  78. [86]

    IEEE Transactions on Automatic Control , volume=

    Finite-sample analysis of two-time-scale natural actor--critic algorithm , author=. IEEE Transactions on Automatic Control , volume=. 2022 , publisher=

    2022

  79. [87]

    SIAM Journal on Control and Optimization , volume=

    A small gain analysis of single timescale actor critic , author=. SIAM Journal on Control and Optimization , volume=. 2023 , publisher=

    2023

  80. [88]

    Advances in Neural Information Processing Systems , volume=

    Improving sample complexity bounds for (natural) actor--critic algorithms , author=. Advances in Neural Information Processing Systems , volume=

  81. [89]

    Advances in neural information processing systems , volume=

    Provably global convergence of actor--critic: A case for linear quadratic regulator with ergodic cost , author=. Advances in neural information processing systems , volume=

  82. [90]

    SIAM Journal on Optimization , volume=

    A two-timescale stochastic algorithm framework for bilevel optimization: Complexity analysis and application to actor--critic , author=. SIAM Journal on Optimization , volume=. 2023 , publisher=

    2023

  83. [91]

    International Conference on Machine Learning , pages=

    Finite-sample analysis of off-policy natural actor--critic algorithm , author=. International Conference on Machine Learning , pages=. 2021 , organization=

    2021

  84. [92]

    International Conference on Learning Representations , year=

    Neural policy gradient methods: Global optimality and rates of convergence , author=. International Conference on Learning Representations , year=

  85. [93]

    arXiv preprint arXiv:2005.03557 , year=

    Non-asymptotic convergence analysis of two time-scale (natural) actor--critic algorithms , author=. arXiv preprint arXiv:2005.03557 , year=

    2005 arXiv

  86. [94]

    International Conference on Machine Learning , year=

    Non-asymptotic analysis for single-loop (natural) actor--critic with compatible function approximation , author=. International Conference on Machine Learning , year=

  87. [95]

    Neurocomputing , volume=

    Natural actor--critic , author=. Neurocomputing , volume=

  88. [96]

    Transactions on Machine Learning Research , year=

    Finite-time analysis of entropy-regularized neural natural actor--critic algorithm , author=. Transactions on Machine Learning Research , year=

  89. [97]

    Finite-sample analysis for

    Zou, Shaofeng and Xu, Tengyu and Liang, Yingbin , journal=. Finite-sample analysis for

  90. [98]

    2021 , journal=

    Statistical estimation of ergodic Markov chain kernel over discrete state space , author=. 2021 , journal=

    2021

  91. [99]

    Bernstein-type inequalities for

    Huang, De and Li, Xiangyuan , journal=. Bernstein-type inequalities for

  92. [100]

    Conference on Learning Theory , year=

    Estimating the mixing time of ergodic markov chains , author=. Conference on Learning Theory , year=

  93. [101]

    2015 , journal=

    Concentration inequalities for Markov chains by Marton couplings and spectral methods , author=. 2015 , journal=

    2015

  94. [102]

    Advances in Neural Information Processing Systems , year=

    A Finite-time analysis of two time-scale actor--critic methods , author=. Advances in Neural Information Processing Systems , year=

  95. [103]

    Journal of Applied Probability , author=

    Sensitivity and convergence of uniformly ergodic Markov chains , volume=. Journal of Applied Probability , author=. 2005 , pages=

    2005

  96. [104]

    SIAM Journal on Optimization , volume =

    Li, Yan and Lan, Guanghui , title =. SIAM Journal on Optimization , volume =

  97. [105]

    Closing the gap:

    Gaur, Mudit and Singh Bedi, Amrit and Wang, Di and Aggarwal, Vaneet , journal=. Closing the gap:

  98. [106]

    arXiv preprint arXiv:2509.18822 , year=

    On the convergence of policy mirror descent with temporal difference evaluation , author=. arXiv preprint arXiv:2509.18822 , year=

  99. [107]

    International Conference on Learning Representations , year=

    Single-timescale actor--critic provably finds globally optimal policy , author=. International Conference on Learning Representations , year=

  100. [108]

    Advances in Neural Information Processing Systems , year=

    Finite-time analysis of single-timescale actor--critic , author=. Advances in Neural Information Processing Systems , year=

  101. [109]

    Simple and optimal methods for stochastic variational inequalities, II: Markovian noise and policy evaluation in reinforcement learning , volume =

    Kotsalis, Georgios and Lan, Guanghui and Li, Tianjiao , year =. Simple and optimal methods for stochastic variational inequalities, II: Markovian noise and policy evaluation in reinforcement learning , volume =

  102. [110]

    2021 , booktitle =

    Chen, Tianyi and Sun, Yuejiao and Yin, Wotao , title =. 2021 , booktitle =

    2021

  103. [111]

    Machine Learning , year =

    Kumar, Harshat and Koppel, Alec and Ribeiro, Alejandro , title =. Machine Learning , year =

  104. [112]

    2017 , publisher=

    Markov chains and mixing times , author=. 2017 , publisher=

    2017

  105. [113]

    The American Mathematical Monthly , volume=

    Markov chains as random input automata , author=. The American Mathematical Monthly , volume=. 1961 , publisher=

    1961

  106. [114]

    2025 , journal=

    Reinforcing action policies by prophesying , author=. 2025 , journal=

    2025

  107. [115]

    arxiv preprint arXiv:2509.08827 , year=

    A survey of reinforcement learning for large reasoning models , author=. arxiv preprint arXiv:2509.08827 , year=

    arXiv

  108. [116]

    2021 , booktitle =

    Xu, Tengyu and Yang, Zhuoran and Wang, Zhaoran and LIANG, Yingbin , title =. 2021 , booktitle =

    2021

  109. [117]

    On Finite-Time Convergence of Actor-Critic Algorithm , year=

    Qiu, Shuang and Yang, Zhuoran and Ye, Jieping and Wang, Zhaoran , journal=. On Finite-Time Convergence of Actor-Critic Algorithm , year=

  110. [118]

    and Bedi, Amrit Singh and Patel, Bhrij and Sadler, Brian M

    Suttle, Wesley A. and Bedi, Amrit Singh and Patel, Bhrij and Sadler, Brian M. and Koppel, Alec and Manocha, Dinesh , title =. 2023 , booktitle =

    2023

  111. [119]

    International Conference on Artificial Intelligence and Statistics , year=

    Analysis of a target-based actor-critic algorithm with linear function approximation , author=. International Conference on Artificial Intelligence and Statistics , year=

  112. [120]

    Convergence of actor-critic with multi-layer neural networks , year =

    Tian, Haoxing and Olshevsky, Alex and Paschalidis, Yannis , booktitle =. Convergence of actor-critic with multi-layer neural networks , year =

  113. [121]

    2017 , publisher=

    First-order methods in optimization , author=. 2017 , publisher=

    2017

  114. [122]

    Operations Research , year=

    Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysis , author=. Operations Research , year=

  115. [123]

    Reinforcement Learning: Theory and Algorithms (Draft) , author=

  116. [124]

    , title =

    Cayci, Semih and He, Niao and Srikant, R. , title =. SIAM Journal on Optimization , volume =

  117. [125]

    AAAI Conference on Artificial Intelligence , year=

    Policy Optimization with Stochastic Mirror Descent , author=. AAAI Conference on Artificial Intelligence , year=

  118. [126]

    International Conference on Learning Representations , year=

    Huang, Feihu and Gao, Shangqian and Huang, Heng , title=. International Conference on Learning Representations , year=

  119. [127]

    and Tsitsiklis, John N

    Konda, Vijay R. and Tsitsiklis, John N. , title=. Advances in Neural Information Processing Systems , year=

  120. [128]

    arxiv preprint arXiv:2303.08774 , year=

    OpenAI , title=. arxiv preprint arXiv:2303.08774 , year=

    arXiv

  121. [129]

    Stochastic Policy Gradient Methods: Improved Sample Complexity for

    Fatkhullin, Ilyas and Barakat, Anas and Kireeva, Anastasia and He, Niao , booktitle =. Stochastic Policy Gradient Methods: Improved Sample Complexity for. 2023 , editor =

    2023

  122. [130]

    and Yudin, D.B

    Nemirovski, A. and Yudin, D.B. , title =

  123. [131]

    Jin, Qinian , title =

  124. [132]

    Annales de l'institut Fourier , year =

    Kurdyka, Krzysztof , title =. Annales de l'institut Fourier , year =

  125. [133]

    Advances in Neural Information Processing Systems , year =

    Mei, Jincheng and Dai, Bo and Agarwal, Alekh and Vaswani, Sharan and Raj, Anant and Szepesvári, Csaba and Schuurmans, Dale , title =. Advances in Neural Information Processing Systems , year =

  126. [134]

    arxiv preprint arXiv:2505.03155 , year=

    Lin, Max Qiushi and Mei, Jincheng and Aghaei, Matin and Lu, Michael and Dai, Bo and Agarwal, Alekh and Schuurmans, Dale and Szepesvári, Csaba and Vaswani, Sharan , title=. arxiv preprint arXiv:2505.03155 , year=

    arXiv

  127. [135]

    2018 , booktitle =

    Dalal, Gal and Szorenyi, Balazs and Thoppe, Gugan and Mannor, Shie , title =. 2018 , booktitle =

    2018

  128. [136]

    International Conference on Learning Representations , year =

    Hu, Yuzheng and Ji, Ziwei and Telgarsky, Matus , title =. International Conference on Learning Representations , year =

  129. [137]

    2025 , journal=

    Li, Wenye and Chen, Hongxu and Liu, Jiacai and Wei, Ke , title=. 2025 , journal=

    2025

  130. [138]

    2022 , journal=

    Lin, Dachao and Zhang, Zhihua , title=. 2022 , journal=

    2022

  131. [139]

    Journal of Convex Analysis , volume =

    Legendre Functions and the Method of Random Bregman Projections , author =. Journal of Convex Analysis , volume =. 1997 , publisher =

    1997

  132. [140]

    Rockafellar, Ralph Tyrell , title =

  133. [141]

    1983 , publisher=

    Problem Complexity and Method Efficiency in Optimization , author=. 1983 , publisher=

    1983

  134. [142]

    Communications in Contemporary Mathematics , volume=

    Essential smoothness, essential strict convexity, and Legendre functions in Banach spaces , author=. Communications in Contemporary Mathematics , volume=. 2001 , publisher=

    2001

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.