Pith. sign in

REVIEW 2 major objections 4 minor 24 references

Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs

T0 review · 2 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proves that TRPO with adaptive scaling converges globally, at O(1/√N) in the unregularized case and O(1/N) for regularized MDPs.

desk verdict Read this for the planning/exact analysis, not for the sample-based guarantee: the convergence proof for Sample-Based TRPO has a real gap in Lemma 20's unbiasedness step. read the letter →

arxiv 1909.02769 v2 pith:3DNUS24C submitted 2019-09-06 cs.LG math.OCstat.ML

Lior Shani , Yonathan Efroni , Shie Mannor This is my paper · ORCID
classification cs.LGmath.OCstat.ML MSC 90C4090C25
keywords reinforcementlearningtrustregionpolicyoptimizationmirrordescentregularizedMarkovdecisionprocessesglobalconvergenceratessamplecomplexityconcentrabilitycoefficientgradient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Trust Region Policy Optimization is widely used in reinforcement learning but has been treated as a heuristic inspired by Conservative Policy Iteration. This paper argues that the adaptive proximity term in TRPO is the natural RL counterpart of the Bregman proximity term in mirror descent, not an ad hoc approximation. It proves that a general adaptive TRPO framework, containing NE-TRPO and Projected Policy Gradient as instances, converges to the global optimum in tabular discounted MDPs, at $O(1/\sqrt{N})$ in the unregularized case and at $O(1/N)$ when a strongly convex regularizer is added to the cost. The paper thereby positions these as the first global non-asymptotic convergence guarantees for NE-TRPO and Projected Policy Gradient, and the first demonstration of faster convergence rates from regularization in MDPs.

What carries the argument

The load-bearing object is the adaptive scaling of the Bregman proximity term by the discounted visitation operator, together with the directional-derivative identity that makes the scaling natural. The identity reads $\langle \nabla_\pi \mu v_\lambda^\pi, \pi'-\pi\rangle = \frac{1}{1-\gamma} d_{\mu,\pi}(T_\lambda^{\pi'} v_\lambda^\pi - v_\lambda^\pi - \lambda B_\omega(\pi',\pi))$, where $B_\omega$ is the Bregman distance of a strongly convex $\omega$ (squared Euclidean norm for Projected Policy Gradient, negative entropy for NE-TRPO). Because the same discounted visitation factor multiplies both the gradient term and the proximity term in the TRPO subproblem, the optimization decouples across states: in each visited state the algorithm solves a small mirror-descent problem of the form $\min_{\pi(\cdot|s)} t_k \langle \hat q_\lambda^{\pi_k}(s,\cdot)+\lambda\nabla\omega(s;\pi_k), \pi\rangle + B_\omega(s;\pi,\pi_k)$. The convergence proofs then transplant the standard Bregman proximal machinery of mirror descent, including the three-point identity and the fundamental proximal inequality, to the non-convex policy-search problem by replacing the convex gradient inequality with this directional-derivative identity.

What would settle it

Run Sample-Based TRPO on a small tabular MDP with a known optimal value, choosing a restart distribution that gives zero probability to a state the optimal policy visits; if the value gap still tends to zero, Assumption 1 is not load-bearing. Separately, with exact gradient information, record $\|v^{\pi_N}_\lambda - v^{*}_\lambda\|_\infty$ under the regularized step sizes; the paper's central fast-rate claim predicts $O(\log N / N)$, while a $1/\sqrt{N}$ decay would falsify it.

Watch

Extended reading notes

Core claim

The central discovery is that the adaptive scaling in TRPO is load-bearing rather than a heuristic. In the planning setting, Uniform TRPO updates by $\pi_{k+1} \in \arg\min_\pi \{ \langle \nabla v_\lambda^{\pi_k}, \pi-\pi_k \rangle + t_k^{-1}(I-\gamma P^{\pi_k})^{-1} B_\omega(\pi,\pi_k) \}$. Using the closed-form directional derivative $\langle \nabla v_\lambda^{\pi}, \pi'-\pi \rangle = (I-\gamma P^\pi)^{-1}(T_\lambda^{\pi'} v_\lambda^\pi - v_\lambda^\pi - \lambda B_\omega(\pi',\pi))$, this update separates into independent per-state mirror-descent steps. The paper proves Uniform TRPO converges at rate $O(C_{\omega,1}C_{\max}/((1-\gamma)^2\sqrt{N}))$ in the unregularized case and $O(C_{\omega,1}^2 C_{\max,\lambda}^2/(\lambda(1-\gamma)^3 N))$ in the regularized case, in max norm. For Sample-Based TRPO with a $\nu$-restart model and finite concentrability $C_{\pi^*}<\infty$, the best policy in hindsight satisfies, with high probability, $O(C_{\omega,1}C_{\max}/((1-\gamma)^2\sqrt{N}) + C_{\pi^*}\epsilon/(1-\gamma)^2)$ in the unregularized case and $O(C_{\omega,1}^2 C_{\omega,2} C_{\max,\lambda}^2/(\lambda(1-\gamma)^3 N) + C_{\pi^*}\epsilon/(1-\gamma)^2)$ in the regularized case.

Load-bearing premise

The sample-based results stand on the assumption that the restart distribution can reach every state the optimal policy might visit; if some such state is never sampled, the global convergence argument collapses.

Editorial extensions

If this is right

  • NE-TRPO and Projected Policy Gradient, with the paper's step sizes, provably converge to a globally optimal policy in tabular discounted MDPs, removing the need to interpret them as heuristic approximations to CPI.
  • Adding a strongly convex regularizer to the instantaneous cost accelerates the guaranteed optimization rate from $\tilde O(1/\sqrt{N})$ to $\tilde O(1/N)$, matching the speed-up known for strongly convex objectives in convex optimization.
  • Sample-Based TRPO attains the same asymptotic performance as CPI, with better polynomial dependence on $1-\gamma$, and it does so without an improvement assumption or early stopping.
  • The analysis supplies explicit learning-rate schedules: $t_k = (1-\gamma)/(C_{\omega,1}C_{\max}\sqrt{k+1})$ for the unregularized case and $t_k = 1/(\lambda(k+2))$ for the regularized case.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same decoupling identity suggests that other policy-update rules built from a policy-dependent metric should inherit mirror-descent rates whenever the metric is the discounted visitation operator, so the result may extend to natural-policy-gradient-style updates.
  • Beyond the paper: the $\lambda$ in the denominator of the fast rate means the speed-up is largest for moderate regularization; a bound on the bias $\|v^{\pi^*_\lambda}-v^*\|_\infty$ as a function of $\lambda$ would convert the optimization improvement into a practical recommendation for choosing $\lambda$.
  • Beyond the paper: since the sample-based result only needs $C_{\pi^*}$ finite, the coverage assumption could in principle be relaxed by letting the restart distribution adapt to the current policy; the paper does not explore this, but the proof structure would only require replacing $\nu$ by $\nu_k$ with a matching bound.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes viewing adaptive TRPO as mirror descent with an adaptive Bregman proximity term, and studies tabular discounted MDPs with a regularized cost. It derives a closed-form directional derivative (Proposition 1), defines Uniform TRPO for the planning setting and establishes O(1/sqrt(N)) and O(1/N) rates (Theorem 2). It then introduces Exact TRPO and Sample-Based TRPO under a nu-restart model, claiming high-probability convergence rates of O(1/sqrt(N)) and O(1/N) plus an additive approximation error (Theorem 5), and claims these are the first global non-asymptotic guarantees for NE-TRPO and Projected Policy Gradient and the first faster rates for regularized MDPs.

Significance. The adaptive-scaling interpretation of TRPO and the planning analysis (Theorem 2) are clean and appear correct, and they constitute a useful conceptual contribution. If the sample-based results were valid, the paper would be highly significant: it would supply the first finite-sample global convergence guarantees for NE-TRPO and PPG and the first accelerated rates due to regularization in RL. However, the sample-based proof contains a load-bearing flaw in the concentration argument for the approximation error, so the main advertised sample-based results are not established as written. The planning results remain valuable, but the paper's central claim about sample-based convergence and faster regularized rates is unsupported.

major comments (2)
  1. [Appendix E.5, Lemma 20, equations (53)-(59)] The unbiasedness claim in Lemma 20 is invalid because pi_{k+1} is data-dependent. In equation (54), the quantity \hat X_k(s_m,.,m) contains the term \nabla\omega(s_m;\pi_{k+1}) - \nabla\omega(s_m;\pi_k), and \pi_{k+1} is the argmin of the sampled objective (49), hence a nonlinear function of the entire batch of samples in episode k. Equation (58) asserts E[\hat X_k(s_m,.,m)|s_m] = X_k(s_m,.), but conditioning on s_m alone does not fix \pi_{k+1}, and the conditional expectation of \nabla\omega(s_m;\pi_{k+1}) is not equal to the gradient at the realized (random) \pi_{k+1}. Therefore the empirical mean in (59) is not an unbiased estimator of a fixed quantity, and the subsequent Hoeffding/union-bound argument does not control the actual data-dependent error. Since Lemma 19 and Theorem 5 rely directly on this bound, the sample-based convergence guarantee is not proven.
  2. [Appendix E.5, Lemma 21, equations (62)-(65)] Lemma 21 suffers from the same data-dependence issue. The quantity X_k^{trunc}(s,a) = E[\hat X_k(s,a,m)] is treated as a per-trajectory expectation with \pi_{k+1} held fixed, but \pi_{k+1} is a deterministic function of the entire batch of M trajectories and is therefore random when the expectation is taken. Consequently the decomposition of the bias into a sampling term (1) and a truncation term (2) in equation (65) is not justified, and the bound on the truncation error is not established. This further undermines Theorem 5, which depends on Lemma 22 and hence on Lemma 21.
minor comments (4)
  1. [Theorem 5] The definition of v_N_best is malformed: "\mu v_N_best := arg min_{k=0,...,N} \mu v^{\pi_k}_\lambda - \mu v^*_\lambda" is not a quantity being minimized. It should define v_N_best as the policy among \pi_0,...,\pi_N with minimal \mu v^{\pi_k}_\lambda, and the bound should be stated on \min_k \mu v^{\pi_k}_\lambda - \mu v^*_\lambda (or analogous).
  2. [Assumption 1 / Theorem 5] The constant C_{\pi^*} is defined as \|d_{\mu,\pi^*}/\nu\|_\infty in Assumption 1, but Theorem 5 states the additive error as C_{\pi^*}\epsilon/(1-\gamma)^2 without restating this definition; please make the notation consistent throughout.
  3. [Throughout] There are several typos and OCR artifacts: "Aproximate TRPO" in Lemmas 17-19, "we we will now show" in the proof of Lemma 20, "arg min" written as "min" in the definition of the orthogonal projection in Section 2, and "Table 6.2" referring to Table 1.
  4. [Abstract / Section 7] The claim of being the "first result in RL of better rates when regularizing" should be qualified to the specific setting (tabular MDPs, cost formulation, nu-restart model), given that faster rates for entropy-regularized MDPs have been studied in related work.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the convergence proof is self-contained and the regularizer choice is an explicit modeling assumption, not a circular prediction.

full rationale

I walked the derivation chain from the linear-approximation Proposition 1, through Uniform TRPO, Exact TRPO, and Sample-Based TRPO, looking for steps where a claimed prediction is equivalent by construction to an input or where a load-bearing premise is imported only from the authors' own prior work. The adaptive scaling in (11) is derived, not assumed: Proposition 1 supplies the closed-form directional derivative, and the scaling (I-γP_{π_k})^{-1} cancels exactly to produce the state-wise update (13). The regularized MDP is defined by adding λω to the instantaneous cost, and the Bregman distance B_ω is built from the same ω; the paper states explicitly that c^π_λ is then λ-strongly convex w.r.t. B_ω 'by definition.' This is a transparent modeling assumption that matches standard mirror-descent analysis, not a fitted parameter renamed as a prediction. The O(1/√N) and O(1/N) rates follow from standard telescoping Bregman inequalities (Lemma 10/19) plus, for the sample-based case, a concentration argument. The finite concentrability assumption C_{π*} is a stated condition from the CPI literature, and while it is load-bearing, it is not a circular input: it is an assumption about the MDP, not an encoding of the theorem. The possible measurability issue in Lemma 20 concerning the data-dependent policy π_{k+1} is a soundness concern about the concentration proof, not a circularity: the theorem's conclusion is not defined as its own input, and the gap, if real, would make the proof incomplete rather than circular. Self-citations to Beck, Sutton, Kakade, and Schulman are normal scholarship and none is used to import the paper's own conclusion. No uniqueness theorem is imported from the authors, and no known empirical pattern is merely renamed. I therefore find no circular step under the requested criteria.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central proof relies on standard mirror descent assumptions and on the strong convexity of the chosen regularizer. There are no fitted parameters. The concentrability and restart-model assumptions are inherited from the CPI literature and are explicitly flagged.

assumptions (5)
  • ad hoc to paper The regularizer omega is 1-strongly convex and is the same function that generates the Bregman distance B_omega.
    Section 3 states this choice makes c^pi_lambda strongly convex by definition. It is central to the fast regularized rates, but it is a modeling assumption rather than a property of the original MDP.
  • domain assumption Assumption 1: finite concentrability coefficient C_pi* = || d_mu,pi* / nu ||_infty < infinity.
    Section 6 introduces this standard assumption for sample-based policy search. It is needed for the global guarantees of Exact and Sample-Based TRPO.
  • domain assumption Access to a nu-restart model for Sample-Based TRPO.
    Section 6 states the algorithm interacts with the MDP through a restart distribution nu. This is stronger than purely online access and is a standard but restrictive assumption.
  • standard math Standard mirror descent assumptions (Assumptions 2 and 3): omega is proper closed convex and differentiable, the domain is convex and compact, and the optimal set is nonempty.
    Appendix A lists these assumptions. They hold for the two choices of omega considered in the paper.
  • domain assumption The MDP is tabular, discounted, and has finite state and action spaces.
    The problem setup in Section 3 assumes finite S and A and a discount factor gamma in (0,1). Constraining to tabular representation is necessary for the closed-form updates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs." pith.science (2026). https://pith.science/paper/3DNUS24C

@misc{pith2026190902769,
  author       = {Pith},
  title        = {Pith review of: Adaptive Trust Region Policy Optimization: Global Convergence and Faster Rates for Regularized MDPs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3DNUS24C}},
  note         = {Machine review of arXiv:1909.02769}
}
abstract

Trust region policy optimization (TRPO) is a popular and empirically successful policy search algorithm in Reinforcement Learning (RL) in which a surrogate problem, that restricts consecutive policies to be 'close' to one another, is iteratively solved. Nevertheless, TRPO has been considered a heuristic algorithm inspired by Conservative Policy Iteration (CPI). We show that the adaptive scaling mechanism used in TRPO is in fact the natural "RL version" of traditional trust-region methods from convex analysis. We first analyze TRPO in the planning setting, in which we have access to the model and the entire state space. Then, we consider sample-based TRPO and establish $\tilde O(1/\sqrt{N})$ convergence rate to the global optimum. Importantly, the adaptive scaling mechanism allows us to analyze TRPO in regularized MDPs for which we prove fast rates of $\tilde O(1/N)$, much like results in convex optimization. This is the first result in RL of better rates when regularizing the instantaneous cost or reward.

Figures

Figures reproduced from arXiv: 1909.02769 by the authors.

Figure 1
Figure 1. The adaptive TRPO: a solid line implies a formal [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 22 canonical work pages

  1. [1]

    Ty is a contraction operator in the max norm

    work page

  2. [2]

    Its fixed-point is v(y) and satisfies vs(y) = (Tyv(y))s. Proof. We start by proving the first claim. Unlike in classical resul ts on MDPs, y is not a policy. However, since it is not ‘too far’ from being a policy we get the usual contraction propert y by standard proof techniques. Letv′,v∈ RS, and assume (Tyv′)s≥ (Tyv)s. (Tyv′)s− (Tyv)s =γ ∑ a y(a|s) ∑ s′ p(...

    work page 2000

  3. [3]

    The only difference is the approximation ter m which we bound in E.5

    In Appendix E.6 we use a similar analysis to the one used for the rates guarantees of Exact TRPO (Appendix D.4), using the above results. The only difference is the approximation ter m which we bound in E.5. There, we make use of the fact that the approximation term is proportional to the step size tk and thus decreasing with the number of iterations, to ...

    work page

  4. [4]

    non- euclidean, unregularized vs

    Lastly, in Appendix E.7, we calculate the overall sample c omplexity – previously we bounded the number of needed itera tions and the number of samples needed in every iteartion – for each of the four cases of Sample-Based TRPO (euclidean vs. non- euclidean, unregularized vs. regularized). E.4 Fundamental Inequality of Sample-Based TRPO Lemma 17 (sample-b...

    work page 1997

  5. [6]

    (Unregularized) Letλ = 0,tk = (1−γ) Cω, 1Cmax √k+1 then ‖vπN−v∗‖∞≤O (Cω,1Cmax(Cω,3 + logN ) (1−γ)2 √ N )

    work page

  6. [7]

    WhereCω,1 = √ A,Cω,3 = 1 for the euclidean case, and Cω,1 = 1,Cω,3 = logA for the non-euclidean case

    (Regularized) Letλ> 0,tk = 1 λ(k+2) then ‖vπN λ −v∗ λ‖∞≤O ( C2 ω,1 Cmax,λ 2 logN λ(1−γ)3N ) . WhereCω,1 = √ A,Cω,3 = 1 for the euclidean case, and Cω,1 = 1,Cω,3 = logA for the non-euclidean case. We are now ready to prove Theorem 2, while following argument s from (Beck 2017, Theorem 9.18). The Unregularized case Proof. Applying Lemma 10 with π = π∗ and λ...

    work page 2017

  7. [8]

    (Unregularized) Letλ = 0,tk = (1−γ) Cω, 1Cmax √k+1 then µvπN−µv∗≤O (Cω,1Cmax(Cω,3 + logN ) (1−γ)2√ N )

    work page

  8. [9]

    WhereCω,1 = √ A,Cω,3 = 1 for the euclidean case, and Cω,1 = 1,Cω,3 = logA for the non-euclidean case

    (Regularized) Letλ> 0,tk = 1 λ(k+2) then µvπN λ −µv∗ λ≤O ( C2 ω,1 Cmax,λ 2 logN λ(1−γ)3N ) . WhereCω,1 = √ A,Cω,3 = 1 for the euclidean case, and Cω,1 = 1,Cω,3 = logA for the non-euclidean case. The Unregularized case Proof. Applying Lemma 14 and λ = 0 (the unregularized case), tk(1−γ)(µvπk−µv∗) ≤dµ,π∗ (Bω (π∗,πk)−Bω (π∗,πk+1)) + t2 kh2 ω 2 . 28 Summing t...

    work page 2017

Show all 24 references
  1. [10]

    We use the Sample-Based TRPO optimization problem descri bed in E.2, to derive a fundamental inequality in Lemma 19 for the sample-based case (in Appendix E.4): (a) We derive a state-wise inequality by applying similar an alysis to Exact TRPO, but for the Sample-Based TRPO opt...

  2. [11]

    (a) In Lemma 20 we deal with the sampling error

    In Appendix E.5, we show that the approximation error term is made of two sources of errors: (a) a sampling error due to the finite number of trajectories in each iteration; (b) a tru ncation error due to the finite length of each trajectory, eve n in the infinite-horizon case. (...

    2003

  3. [14]

    In the next lemma we combine the results of Lemmas 20 and 21 to b ound the overall approximation error due to both sampling and truncation

    (64) Finally, combining (62) and (64) concludes the results. In the next lemma we combine the results of Lemmas 20 and 21 to b ound the overall approximation error due to both sampling and truncation. Lemma 22 (Approximation error bound using truncated biased samplin g). F or ...

  4. [15]

    (Unregularized) Letλ = 0,tk = (1−γ)2 Cω, 1 Cmax √k+1 then µvN best−µv∗ ≤O (Cω,1 Cmax(Cω,3 + logN ) (1−γ) √ N + Cπ∗ ǫ (1−γ)2 )

  5. [16]

    (Regularized) Letλ> 0,tk = 1 λ(k+2) then µvN best−µv∗ λ≤O ( C2 ω,1Cω,2 Cmax,λ 2 logN λ(1−γ)3N + Cπ∗ ǫ (1−γ)2 ) . WhereCω,1 = √ A,Cω,2 = 1,Cω,3 = 1,rω(k,λ ) = 4A Cmax,λ 1−γ for the euclidean case, and Cω,1 = 1,Cω,2 = A2,Cω,3 = logA,rω(k,λ ) = 4A Cmax,λ 1−γ (1 + /BD {λ⁄= 0} logk...

    2017

  6. [17]

    The statement holds by the properties of 1 2‖·‖2 2 and thus holds for both the uniform and exact versions

    Thus, for every state s, ‖∇ω(π(·|s))‖2 =‖π(·|s)‖2≤‖π(·|s)‖1 = 1, where the inequality is due to the fact that ‖·‖2≤‖·‖ 1. The statement holds by the properties of 1 2‖·‖2 2 and thus holds for both the uniform and exact versions. For the non-euclidean case,ω(·) = H(·) + logA. N...

  7. [18]

    For the non-euclidean case,ω(·) = H(·) + logA

    Thus, for every state s, ‖∇ω(π(·|s))‖2 =‖π(·|s)‖2≤‖π(·|s)‖1 = 1, where the inequality is due to the fact that ‖·‖2≤‖·‖ 1. For the non-euclidean case,ω(·) = H(·) + logA. The bound for the sample-based version for the non-euclide an choice of ω follows similar reasoning with mil...

  8. [19]

    (74) By the first order optimality condition, for any state s and policyπ, ⟨∇ω (s;πk+1)−∇ω (s;πk),πk+1(·| s)−π⟩≤ tk⟨ˆqπk λ (s,·) +λ∇ω (s;πk),π−πk+1(·| s)⟩

    Thus, the derivative of ω at a state s is, ∇ω (s;π) = π(·| s). (74) By the first order optimality condition, for any state s and policyπ, ⟨∇ω (s;πk+1)−∇ω (s;πk),πk+1(·| s)−π⟩≤ tk⟨ˆqπk λ (s,·) +λ∇ω (s;πk),π−πk+1(·| s)⟩. Plugging inπ :=πk(·| s), we get ⟨∇ω (s;πk+1)−∇ω (s;πk),πk+1...

  9. [20]

    F orω(·) = 1 2‖·‖2 2,Dω = 1

  10. [21]

    F orω(·) = H(·),Dω = logA. Proof. For brevity, without loss of generality we omit the dependen cy on the states. We start by proving the first claim. For the euclidean case, Bω (π,π 0) = 1 2‖π−π0‖2 2 = 1 2 ∑ a (π(a)− 1 A )2 ≤ 1 2 ∑ a π2(a) + ∑ a 1 A2 = 1 2A + 1 2 ∑ a π2(a) ≤ 1 ...

    2014

  11. [22]

    vπ′ λ −vπ λ = (I−γPπ′ )−1(Tπ′ λ vπ λ−vπ λ )

  12. [23]

    Tπ′ λ vπ λ−vπ λ = (I−γPπ′ )(vπ′ λ −vπ λ )

  13. [24]

    µ ( vπ′ λ −vπ λ ) = 1 1−γdµ,π′ (Tπ′ λ vπ λ−vπ λ ). Proof. The first claim holds by the following relations. vπ′ λ −vπ λ = (I−γPπ′ )−1cπ′ λ − (I−γPπ′ )−1(I−γPπ′ )vπ λ = (I−γPπ′ )−1(cπ′ λ +γPπ′ vπ λ−vπ λ ) = (I−γPπ′ )−1(Tπ′ λ vπ λ−vπ λ ). The second claim follows by multiplying b...

    2017

  14. [2002]

    In ICML, volume 2, 267–274

    Approximately optimal approximate reinforcement learning. In ICML, volume 2, 267–274. [Kakade and others 2003] Kakade, S. M., et al. 2003. On the sample complexity of reinforcement learning . Ph.D. Disser- tation, University of London London, England. [Liu et al. 2019] Liu, B....

    2003 arXiv

  15. [2018]

    MIT press

    Reinforcement learning: An introduction. MIT press. [Sutton et al. 2000] Sutton, R. S.; McAllester, D. A.; Singh, S. P .; and Mansour, Y . 2000. Policy gradient methods for reinforcement learning with function approximation. I n Advances in neural information processing system...

    2000

  16. [2019]

    arXiv preprint arXiv:1906.01786

    Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786 . [Bhatnagar et al. 2009] Bhatnagar, S.; Sutton, R. S.; Ghavamzadeh, M.; and Lee, M. 2009. Natural actor–critic algorithms. Automatica 45(11):2471–2482. [Chow, Nachum, and Ghavamzadeh 201...

    1906 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.