Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Exploration-Enhanced POLITEX

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that one pre-existing exploration policy, interleaved with policy iteration, suffices to keep regret sublinear without requiring every learned policy to explore.

desk verdict EE-POLITEX is a promising modular idea, but the headline regret bound is not established because the value-estimation error is proved in the exploration policy's norm while the regret theorem needs it in the reference policy's norms. read the letter →

arxiv 1908.10479 v1 pith:2C5FSIRA submitted 2019-08-27 cs.LG stat.ML

Yasin Abbasi-Yadkori , Nevena Lazic , Csaba Szepesvari , Gellert Weisz This is my paper · ORCID
classification cs.LGstat.ML
keywords average-costreinforcementlearningregretboundspolicyiterationPOLITEXexplorationleast-squaresMonteCarlofunctionapproximationuniformlymixingMDPs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper modifies POLITEX, a policy-iteration algorithm whose policy is near-optimal in hindsight for past value estimates, so that only one pre-existing exploration policy needs to cover the state space; all other policies are free to be greedy. In uniformly mixing average-cost MDPs with linear value function approximation, the resulting EE-POLITEX has regret $\tilde{O}(T^{4/5}+\epsilon_0 T)$, where $\epsilon_0$ is the irreducible linear approximation error. This matters because prior guarantees required every learned policy to explore, which is unrealistic when policies control where they go. The proof works by alternating short exploration runs with target-policy rollouts — the exploration acts as soft resets — and by estimating value functions with least-squares Monte Carlo from those rollouts. Experiments on DeepSea and cartpole swing-up show the scheme succeeds where dithering-based exploration does not.

What carries the argument

The load-bearing object is the data-collection schedule of EE-POLITEX combined with a least-squares Monte Carlo (LSMC) estimator. Each phase alternates $s'$ exploration steps with $s$ on-policy steps, so initial states of the rollouts are approximately i.i.d. from $\mu_e$ and state-action pairs are distributed as $\nu=\mu_e\otimes u$. The estimator solves $\hat{w}=\arg\min_w \sum_j (Q^*_{\pi}(x_j,a_j)-w^{\top}\psi(x_j,a_j))^2$, with $Q^*_{\pi}$ formed from finite-rollout differential costs. The main identity is the error decomposition in Equation (10), which separates the estimation error into a statistical fluctuation term between empirical and true feature covariance and a rollout-bias term; the uniform excitation condition $\sigma$ on $\Psi^{\top} D_{\nu_e}\Psi$ keeps the inverse covariance bounded. This avoids the projected-Bellman contraction coefficient that can make TD/LSPE bounds vacuous.

What would settle it

Construct a unichain MDP with two nearly disjoint recurrent regions: the exploration policy mixes fast and excites all features in region A, while the reward-optimal baseline and all learned policies stay in region B. Then compute $\|Q_{\pi_i}-\Psi\hat{w}\|$ under $\mu_*\otimes\pi_i$ as $m$ grows. If the error fails to decay at the claimed $O(\sqrt{\log(1/\delta)/m})$ rate, the premise of Theorem 3.1 is violated.

Watch

Extended reading notes

Core claim

The central claim is that the exploration burden in POLITEX can be moved from all policies to a single fast-mixing exploration policy $\pi_e$. Given $\pi_e$ whose stationary distribution excites every feature direction (Assumption A3), EE-POLITEX runs $\pi_e$ for $s'=\log T$ steps, samples an initial state-action pair, then runs the current policy for $s=T^{1/5}$ steps to produce an on-policy trajectory; with $n=m=T^{2/5}$ phases, the regret against a reference policy $\pi_*$ is $\tilde{O}(T^{4/5}+\epsilon_0 T)$ under uniform mixing. The proof decomposes regret into pseudo-regret, exploration cost, and two mixing-noise terms, and shows that a least-squares Monte Carlo value estimator achieves $\|Q_{\pi}-\Psi\hat{w}\|_{\mu_e\otimes u}=O(\sqrt{\log(1/\delta)/m})$ plus a term controlled by $\epsilon_0$. The paper presents this as the first regret bound for function-approximation RL where exploration is nontrivial but only a single known policy needs to explore.

Load-bearing premise

The theorem's premise is that value-estimation error is small under the baseline policy's stationary distribution, but the proof bounds it only under the exploration policy's distribution, and no stated assumption forces one to imply the other.

Editorial extensions

If this is right

  • Up to the unavoidable $\epsilon_0 T$ approximation term, per-round regret decays as $T^{-1/5}$, so the method is no-regret when the linear value function error is negligible.
  • The feature-excitation assumption is needed only for the single exploration policy, not for the policies being learned, which is what improves on POLITEX.
  • The exploration cost and noise terms can be balanced at $n=m=T^{2/5}$, giving a concrete hyperparameter recipe for implementing the algorithm.
  • The value estimation is modular: any estimator satisfying the $\nu_e$-weighted error bound in Eq. (3) can be plugged into EE-POLITEX and inherit the regret guarantee.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The proof would be complete if an assumption linked the baseline distribution $\mu_*$ to the exploration distribution $\mu_e$, for instance absolute continuity with a bounded density; without that, the bound in Lemma 4.2 does not formally imply Theorem 3.1's premise.
  • The analysis treats rollout-start states as i.i.d. draws from $\mu_e$ even though Algorithm 1 generates them from one dependent path; a fully formal version would have to absorb the uniform-mixing penalty into the LSMC concentration argument.
  • The modular framing suggests a practical design rule: spend a no-reward phase learning a policy that spans the feature space, then plug it into EE-POLITEX for reward optimization. This could be tested on continuous-control benchmarks where dithering provably fails.
  • Extending Lemma 4.2 to nonlinear value function classes would turn the neural experiments into a theorem; this is the most natural next step but is not carried out in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes EE-POLITEX, a modification of the POLITEX algorithm for average-cost reinforcement learning with linear function approximation. The key idea is to replace the assumption that all policies sufficiently explore with the assumption that a single exploration policy is available, and to interleave short exploration segments with on-policy rollouts of the current target policy. The value function is estimated by least-squares Monte Carlo (LSMC) from data whose initial states are approximately drawn from the exploration policy's stationary distribution. The paper claims a regret bound of O~(T^{4/5} + ε0 T) under assumptions A1–A3, and presents experiments on DeepSea and a sparse Cartpole task to illustrate the benefits of explicit exploration.

Significance. The algorithmic idea is attractive: separating exploration from reward maximization and using a single exploratory policy to provide coverage is a useful conceptual step, and the modular 'soft reset' scheme could be of independent interest. If the regret bound were rigorously established, it would extend POLITEX-style guarantees to settings where uniform exploration by all policies is unrealistic. The LSMC estimation procedure with on-policy rollouts from exploration-sampled initial states is also a sensible construction. However, the central theoretical claim is not currently supported: the regret theorem requires value-estimation error in norms weighted by the reference policy, while the estimation lemma only provides a bound in the exploration policy's norm. The i.i.d. assumption in the estimation lemma also does not match the dependent trajectory produced by the algorithm. As a result, the main contribution is not established in the present form.

major comments (3)
  1. [§3.1, Eq. (3) and §4.1, Lemma 4.2] Theorem 3.1 requires, for every phase i and with probability 1−δ, that ‖Q_{π_i}−Q̂_i‖_{ν*} ≤ ε(δ,m) and ‖Q_{π_i}−Q̂_i‖_{μ*⊗π_i} ≤ ε(δ,m), where ν*=μ*⊗π* is the stationary state-action distribution of the reference policy. Lemma 4.2, however, establishes the bound only in the norm ‖·‖_{μ_e⊗u}, weighted by the exploration policy's stationary distribution. Assumptions A1–A3 do not relate μ_e to μ*: A2 is a mixing-rate condition, and A3 only lower-bounds λ_min(Ψ^T D_{ν_{π_e}} Ψ). Without an additional coverage or density-ratio condition, an error bound in the μ_e⊗u norm does not control the error in the ν* or μ*⊗π_i norms. The paper's own Section 2.2 states that the goal is to bound the error in the ν_{π_e}-weighted norm, which underscores the mismatch. This gap directly undermines the main regret guarantee.
  2. [§4.1, Lemma 4.2 and Algorithm 1 (COLLECT DATA)] Lemma 4.2 assumes that the m state-action pairs (x_j,a_j) are independent draws from μ_e⊗u; its proof explicitly refers to 'm i.i.d. samples.' Algorithm 1, however, obtains x_j by running a single trajectory with alternating exploration and target-policy segments, so successive x_j are neither exactly stationary nor independent. The choice s′=log T suggests approximate stationarity under A2, but the paper provides no argument quantifying the total-variation distance to stationarity or controlling the dependence among samples. As stated, the lemma does not apply to the data actually produced by Algorithm 1, so the estimation error bound used in the regret analysis is unsupported.
  3. [§4.1, Lemma 4.2, hypothesis] The lemma states 'Under the assumption that ‖Ψ^T DΨ‖ ≥ σ', but the proof uses the inverse of Ψ^T DΨ, so the required hypothesis is that the smallest eigenvalue of Ψ^T DΨ is at least σ (as in Assumption A3), not that the matrix norm is at least σ. As written, the hypothesis is both misstated and insufficient: the matrix norm is the largest singular value, and its being large does not guarantee invertibility with a controlled inverse.
minor comments (5)
  1. [§3.1, Theorem 3.1] The symbol μ* is used in the theorem and in the definition of S_δ(A,μ*) without being defined there; it should be stated explicitly that μ* is the stationary distribution of the reference policy π*.
  2. [Abstract] The claim that this is 'the first result that shows how to control the regret in the presence of function approximation errors on problems where exploration is nontrivial' is not substantiated and is difficult to verify; it would be safer to phrase the contribution more modestly.
  3. [Lemma 3.2 and Appendix A] There is a discrepancy between the lemma statement and its proof: the lemma states |V_T| ≤ nmκ + 4nκ√(4ms log(2ms/δ)), while the proof concludes |V_T| ≤ 2nmκ + 4nκ√(4ms log(2m(s+s′)/δ)). The extra factor of 2 must be reconciled.
  4. [§5.1, Figure 1] The y-axis of Figure 1 is labeled 'cost' but the text says 'all policies achieve the lowest cost' in a regime where plotted values are negative; the sign convention should be clarified so the reader knows whether lower or higher values are better.
  5. [§5.2] In the Cartpole experiment, the exploration policy is trained with a separate reward function, whereas the theory assumes a given exploration policy. The text should clarify whether the experiments are intended only as an illustration or whether the theory is claimed to extend to learned exploration policies.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the regret bound is an application of the authors' prior POLITEX theorem together with a new LSMC estimation analysis; no fitted quantity is renamed as a prediction.

full rationale

The paper's central derivation has two parts: a regret bound inherited from the authors' earlier POLITEX analysis, and a new least-squares Monte-Carlo estimation guarantee. The regret theorem (Theorem 3.1) is stated as a direct application of Theorem 4.1 of Abbasi-Yadkori et al. [2019a]. Although this is a self-citation, the cited theorem is an independently published result with its own hypotheses; the new paper adds a concrete exploration scheme and proves a new estimator bound in Lemma 4.2. The final bound does not reduce to its inputs by construction: ε(δ,m) is a sufficient condition on the estimator, and the regret is a function of that error, not identical to it. No parameter is fitted to data and then reported as a prediction. The apparent mismatch between the norms in Eq. (3), which are weighted by the reference policy π*, and the norm in Lemma 4.2, which is weighted by the exploration policy πe, is a genuine proof gap but not a circular step: the paper never defines ν* in terms of μe, nor does it derive the regret bound from its own conclusion. The self-citations to POLITEX are load-bearing as applications of a prior theorem, but they do not smuggle in the target result by definition. Therefore, no significant circularity is present.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

The central claim rests on the standard assumptions A1-A3 plus several unstated regularity conditions. No free parameters are fitted in the regret analysis. The exploration policy πe is an assumed input, not an invented entity. The most fragile items are the unstated norm transfer and the i.i.d. approximation.

assumptions (7)
  • domain assumption A1: Every policy's state process forms a single recurrent class (unichain).
    Section 2.1; ensures unique stationary distribution μπ and well-defined average costs.
  • domain assumption A2: Every policy mixes uniformly fast with constant κ: sup_π ||(μπ−μ')^T Pπ||_1 ≤ exp(−1/κ)||μπ−μ'||_1.
    Section 2.1; used in Lemma 3.2 to bound noise terms; assumed for all policies, not just the exploration policy.
  • domain assumption A3: The exploration policy πe uniformly excites all feature directions: λmin(Ψ^T D_{ν_πe} Ψ) ≥ σ > 0.
    Section 2.2; the only new exploration-related assumption the paper offers in place of all-policies-explore; not accompanied by a construction.
  • domain assumption Bounded costs: ||c||_∞ ≤ 1.
    Appendix A proof of Lemma 3.2 assumes this normalization; can be removed by scaling but is not stated as an assumption in the main text.
  • domain assumption Uniformly bounded best-linear approximation: sup_π min_w ||Qπ − Ψw||_{μe⊗u} ≤ ε0 and ||w~π|| ≤ Wmax.
    Section 4; needed to set Qmax = 2Wmax max_{x,a}||ψ(x,a)|| in Theorem 3.1; uniform over all policies, a strong regularity condition.
  • ad hoc to paper Norm transfer from μe⊗u to ν* and μ*⊗πi.
    Implicitly assumed when applying Theorem 3.1 after proving Lemma 4.2; no such inequality is stated or derived, and it is generally false in unichain MDPs.
  • ad hoc to paper Approximate i.i.d. sampling from μe after s'=log T mixing steps.
    Assumed in Lemma 4.2; Algorithm 1 generates states from a single dependent trajectory, so this is only an approximation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploration-Enhanced POLITEX." pith.science (2026). https://pith.science/paper/2C5FSIRA

@misc{pith2026190810479,
  author       = {Pith},
  title        = {Pith review of: Exploration-Enhanced POLITEX},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2C5FSIRA}},
  note         = {Machine review of arXiv:1908.10479}
}
read the original abstract

We study algorithms for average-cost reinforcement learning problems with value function approximation. Our starting point is the recently proposed POLITEX algorithm, a version of policy iteration where the policy produced in each iteration is near-optimal in hindsight for the sum of all past value function estimates. POLITEX has sublinear regret guarantees in uniformly-mixing MDPs when the value estimation error can be controlled, which can be satisfied if all policies sufficiently explore the environment. Unfortunately, this assumption is often unrealistic. Motivated by the rapid growth of interest in developing policies that learn to explore their environment in the lack of rewards (also known as no-reward learning), we replace the previous assumption that all policies explore the environment with that a single, sufficiently exploring policy is available beforehand. The main contribution of the paper is the modification of POLITEX to incorporate such an exploration policy in a way that allows us to obtain a regret guarantee similar to the previous one but without requiring that all policies explore environment. In addition to the novel theoretical guarantees, we demonstrate the benefits of our scheme on environments which are difficult to explore using simple schemes like dithering. While the solution we obtain may not achieve the best possible regret, it is the first result that shows how to control the regret in the presence of function approximation errors on problems where exploration is nontrivial. Our approach can also be seen as a way of reducing the problem of minimizing the regret to learning a good exploration policy. We believe that modular approaches like ours can be highly beneficial in tackling harder control problems.

Figures

Figures reproduced from arXiv: 1908.10479 by the authors.

Figure 1
Figure 1. Average cost of POLITEX with LSPE and LSMC, EE-POLITEX with LSMC, and RLSVI on the DeepSea environment. measure of contractiveness. Second, the TD fixed point solution is not the same as the best possible estimation in the linear span of the features, and this introduces an additional approximation error. Let ρ = k(I − ΠπHπ) −1ΠπHπ(I − Ππ)k 2 νπ . Theorem 2.2 of Yu and Bertsekas [2010] shows that we might lose a mul… view at source ↗
Figure 2
Figure 2. Left: Rewards obtained by the agents during the course of an episode. The plots are based [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Rewards obtained by the agents during the course of an episode. The plots are based on [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Rethinking the Global Convergence of Softmax Policy Gradient with Linear Function Approximation

    cs.LG 2025-05 conditional novelty 7.0 of 10

    Under explicit feature-ordering conditions, softmax policy gradient with linear function approximation converges to the optimal policy in stochastic bandits even with non-zero approximation error.

Reference graph

Works this paper leans on

35 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    POLITEX : Regret bounds for policy iteration using expert prediction

    Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazi\'c, Csaba Szepesv\'ari, and Gell\'ert Weisz. POLITEX : Regret bounds for policy iteration using expert prediction. In ICML, 2019 a

    work page 2019

  2. [2]

    Model-free linear quadratic control via reduction to expert prediction

    Yasin Abbasi-Yadkori, Nevena Lazi\'c, and Csaba Szepesv\'ari. Model-free linear quadratic control via reduction to expert prediction. In AISTATS, 2019 b

    work page 2019

  3. [3]

    Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path

    Andr \'a s Antos, Csaba Szepesv \'a ri, and R \'e mi Munos. Learning near-optimal policies with Bellman-residual minimization based fitted policy iteration and a single sample path . Machine Learning, 71 0 (1): 0 89--129, 2008

    work page 2008

  4. [4]

    Fernando Hernandez - Garcia, G

    Kristopher De Asis, J. Fernando Hernandez - Garcia, G. Zacharias Holland, and Richard S. Sutton. Multi-step reinforcement learning: A unifying algorithm. CoRR, abs/1703.01327, 2017. URL http://arxiv.org/abs/1703.01327

    arXiv 2017

  5. [5]

    Minimax regret bounds for reinforcement learning

    Mohammad Gheshlaghi Azar, Ian Osband, and R \' e mi Munos. Minimax regret bounds for reinforcement learning. In ICML, pages 263--272, 2017

    work page 2017

  6. [6]

    Approximate policy iteration: A survey and some new methods

    Dimitri P Bertsekas. Approximate policy iteration: A survey and some new methods. Journal of Control Theory and Applications, 9 0 (3): 0 310--335, 2011

    work page 2011

  7. [7]

    Temporal differences-based policy iteration and applications in neuro-dynamic programming

    Dimitri P Bertsekas and Sergey Ioffe. Temporal differences-based policy iteration and applications in neuro-dynamic programming. Lab. for Info. and Decision Systems Report LIDS-P-2349, MIT, Cambridge, MA, 14, 1996

    work page 1996

  8. [8]

    Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In ICLR, 2019

    work page 2019

Show all 35 references
  1. [9]

    Regularized policy iteration with nonparametric function spaces

    Amir-massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesv \'a ri, and Shie Mannor. Regularized policy iteration with nonparametric function spaces. Journal of Machine Learning Research, 17 0 (1): 0 4809--4874, 2016

    2016

  2. [10]

    Off-policy learning with eligibility traces: A survey

    Matthieu Geist and Bruno Scherrer. Off-policy learning with eligibility traces: A survey. Journal of Machine Learning Research, 15: 0 289--333, 2014

    2014

  3. [11]

    Provably efficient maximum entropy exploration

    Elad Hazan, Sham M Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In ICML, 2019

    2019

  4. [12]

    Distributed prioritized experience replay

    Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Van Hasselt, and David Silver. Distributed prioritized experience replay. arXiv preprint arXiv:1803.00933, 2018

    2018 arXiv

  5. [13]

    Chi Jin, Zeyuan Allen - Zhu, S \' e bastien Bubeck, and Michael I. Jordan. Is Q -learning provably efficient? In NeurIPS, pages 4868--4878, 2018

    2018

  6. [14]

    Finite-sample analysis of least-squares policy iteration

    Alessandro Lazaric, Mohammad Ghavamzadeh, and R \'e mi Munos. Finite-sample analysis of least-squares policy iteration. Journal of Machine Learning Research, 13 0 (Oct): 0 3041--3074, 2012

    2012

  7. [15]

    Regularized off-policy TD -learning

    Bo Liu, Sridhar Mahadevan, and Ji Liu. Regularized off-policy TD -learning. In NeurIPS, 2012

    2012

  8. [16]

    Finite-sample analysis of proximal gradient TD algorithms

    Bo Liu, Ji Liu, Mohammad Ghavamzadeh, Sridhar Mahadevan, and Marek Petrik. Finite-sample analysis of proximal gradient TD algorithms. In UAI, 2015

    2015

  9. [17]

    H. R. Maei, Cs. Szepesv\' a ri, S. Bhatnagar, and R. S. Sutton. Toward off-policy learning control with function approximation. In ICML, 2010

    2010

  10. [18]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518 0 (7540): 0 529, 2015

    2015

  11. [19]

    Scale-free algorithms for online linear optimization

    Francesco Orabona and D \'a vid P \'a l. Scale-free algorithms for online linear optimization. In ALT, pages 287--301, 2015

    2015

  12. [20]

    Generalization and exploration via randomized value functions

    Ian Osband, Zheng Wen, and Benjamin Van Roy. Generalization and exploration via randomized value functions. In ICML, 2016

    2016

  13. [21]

    Puterman

    M. Puterman. M arkov decision processes : Discrete stochastic dynamic programming . John Wiley & Sons, New York, 1994

    1994

  14. [22]

    Prioritized experience replay

    Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952, 2015

    2015 arXiv

  15. [23]

    Adaptive confidence and adaptive curiosity

    J \"u rgen Schmidhuber. Adaptive confidence and adaptive curiosity. Technical report, Institut fur Informatik, Technische Universitat Munchen, Arcisstr. 21, 800 Munchen 2, 1991

    1991

  16. [24]

    Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L

    Alexander L. Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L. Littman. PAC model-free reinforcement learning. In ICML, pages 881--888, 2006

    2006

  17. [25]

    R. S. Sutton, Cs. Szepesv\' a ri, and H. R. Maei. A convergent o(n) algorithm for off-policy temporal-difference learning with linear function approximation. In NeurIPS, 2009

    2009

  18. [26]

    Learning to predict by the methods of temporal differences

    Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, 3: 0 9--44, 1988

    1988

  19. [27]

    Deepmind control suite

    Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018

    2018 arXiv

  20. [28]

    Active exploration in dynamic environments

    Sebastian B Thrun and Knut M \"o ller. Active exploration in dynamic environments. In NIPS, pages 531--538, 1992

    1992

  21. [29]

    Tsitsiklis and Benjamin Van Roy

    John N. Tsitsiklis and Benjamin Van Roy . An analysis of temporal-difference learning with function approximation. IEEE Transactions on Automatic Control, 42: 0 674--690, 1997

    1997

  22. [30]

    Tsitsiklis and Benjamin Van Roy

    John N. Tsitsiklis and Benjamin Van Roy . Average cost temporal-difference learning. Automatica, 35: 0 1799--1808, 1999

    1999

  23. [31]

    Deep reinforcement learning with double q-learning

    Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. CoRR, abs/1509.06461, 2015. URL http://arxiv.org/abs/1509.06461

    2015 arXiv

  24. [32]

    Dueling network architectures for deep reinforcement learning

    Ziyu Wang, Nando de Freitas, and Marc Lanctot. Dueling network architectures for deep reinforcement learning. CoRR, abs/1511.06581, 2015. URL http://arxiv.org/abs/1511.06581

    2015 arXiv

  25. [33]

    Convergence of least squares temporal difference methods under general conditions

    Huizhen Yu. Convergence of least squares temporal difference methods under general conditions. In ICML, 2010

    2010

  26. [34]

    Convergence results for some temporal difference methods based on least squares

    Huizhen Yu and Dimitri P Bertsekas. Convergence results for some temporal difference methods based on least squares. IEEE Transactions on Automatic Control, 54 0 (7): 0 1515--1531, 2009

    2009

  27. [35]

    Error bounds for approximations from projected linear equations

    Huizhen Yu and Dimitri P Bertsekas. Error bounds for approximations from projected linear equations. Mathematics of Operations Research, 35 0 (2): 0 306--329, 2010

    2010

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.