Pith. sign in

REVIEW 2 major objections 4 minor 85 references

HBPI-UCRL proves the first PAC sample-complexity bound for parallel hierarchical reinforcement learning, with a factor-S gain over flat BPI-UCRL in sparse-reward, goal-directed SMDPs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:49 UTC pith:7N7K4G5N

load-bearing objection First explicit PAC bound for parallel HRL with a clear proof structure, but the sparse-reward factor-S improvement rests on a genuine proof gap in Appendix F, and the experiments are an approximate implementation. the 2 major comments →

arxiv 2607.29294 v1 pith:7N7K4G5N submitted 2026-07-31 cs.LG

Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification

Anders Jonsson , Emilie Kaufmann , Gianmarco Tedeschi , Lorenzo Steccanella This is my paper
classification cs.LG MSC 68Q3290C40
keywords hierarchical reinforcement learningbest policy identificationsample complexityPAC-MDPsemi-Markov decision processsparse rewardmodel-based RLoptimism
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper aims to establish that hierarchical reinforcement learning can be provably sample-efficient when the low-level dynamics satisfy two specific conditions. It introduces HBPI-UCRL, a model-based algorithm that learns high-level SMDP policies and low-level subproblem policies in parallel, and proves that under these conditions it is (ε, δ)-PAC. The general sample-complexity bound is polynomial in the subproblem state-action space and horizons, and independent of the number of high-level states and subproblems. In sparse-reward, goal-directed settings, the bound is a factor S smaller than that of the non-hierarchical BPI-UCRL, giving the first theoretical justification for why HRL can outperform flat RL. The paper also identifies conditions under which parallel HRL is learnable at all, arguing that without them the high-level learning problem is ill-posed.

Core claim

The paper's central claim is that parallel hierarchical RL becomes PAC-learnable if, for every subproblem, any policy that is nearly optimal under the local reward is also nearly optimal in two quantitative senses: its H-step terminal-state distribution is close in L1 distance to the optimal policy's distribution, and its value under the environment reward is close to the optimal value. Under these conditions, HBPI-UCRL stops and returns a hierarchical policy whose value is within ε of the optimal hierarchical value, with sample complexity Õ(SAH^4 H̄^6/ε^2) in general and Õ(SAH^2 H̄^2/ε^2) in sparse-reward SMDPs. The sparse-reward bound is a factor S smaller than flat BPI-UCRL's bound, suppo

What carries the argument

The load-bearing mechanism is a pair of composite-distribution error bounds. HBPI-UCRL estimates low-level transition kernels from data and propagates per-step confidence bonuses through the H-step horizon to build high-level confidence sets of radius proportional to 5H times a recursively defined error function bL^{t,k}. Assumption 3.1 links this low-level estimation error to the error in the induced SMDP transition kernel and reward, so that optimistic value iteration on the high level remains valid. The stopping rule compares a compounded SMDP error function L^t to a threshold derived from ε and the horizons, yielding a data-dependent stopping time that drives the PAC guarantee.

Load-bearing premise

The paper's guarantees rest on Assumption 3.1: for every subproblem, any policy that is close in local-reward value must also be close in its H-step terminal distribution and in environment-reward value—which in practice means optimal subproblem policies must terminate in a single state with probability 1.

What would settle it

Construct a two-room sparse-reward SMDP where the optimal subproblem policy reaches its terminal state with probability 0.8 and an alternative policy reaches it with probability 0.7, so Assumption 3.1's first inequality is violated; if HBPI-UCRL still stops quickly and returns an ε-optimal hierarchical policy on this instance, the assumption is sufficient but not necessary, whereas a failure or a stopping time violating the theorem's bound would confirm that the assumption is load-bearing.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Parallel HRL is PAC-learnable with polynomial sample complexity whenever Assumption 3.1 holds, and the bound does not scale with the number of high-level states or subproblems.
  • For sparse-reward, goal-directed SMDPs, HBPI-UCRL's sample complexity is a factor S smaller than flat BPI-UCRL's, providing a formal reason hierarchy can help in such settings.
  • The sparse-reward bound is only a factor H̄^2 larger than BPI-UCRL's bound on a single sparse-reward subproblem, suggesting the overhead of learning the hierarchy is modest.
  • The data-dependent stopping rule yields a PAC bound whose δ dependence improves on the regret-to-PAC conversion approach used in earlier parallel-HRL regret analyses.
  • The paper's variant of BPI-UCRL also corrects a claimed factor-H improvement from prior analysis, showing the corrected bound matches the general stopping-rule analysis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the result holds, hierarchy is primarily a way to replace dependence on the full flat state space |Σ| with the subproblem state-action space SA, at the cost of two horizon factors—so HRL pays off exactly when the flat state space is large and subproblem horizons are short.
  • Assumption 3.1 can be read as a design principle: to make HRL provably efficient, subproblem rewards should be aligned with the environment reward, and optimal subproblem policies should terminate in a single state with high probability; options or skills designed this way would inherit the guarantees.
  • The paper's Discussion notes that all known examples satisfying Assumption 3.1 have deterministic single-state termination; if that restriction is in fact necessary, then provable parallel HRL may be intrinsic to goal-directed problems, and new conditions would be needed for broadly stochastic settings.
  • The empirical observation that HBPI-UCRL's stopping time grows sublinearly in S in sparse deterministic grids suggests the proven bound may be loose in the number of high-level states; a tighter analysis might remove the S dependence entirely in such domains.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper proposes HBPI-UCRL, a model-based algorithm for episodic hierarchical reinforcement learning that learns high-level (SMDP) and low-level (subproblem) policies in parallel. The main contribution is a PAC-style analysis under two new conditions (Assumption 3.1) relating subproblem value gaps to SMDP transition/reward accuracy. Under these conditions, Theorem 4.1 gives an ε,δ-PAC guarantee with sample complexity Õ(SAH^4 Hbar^6/ε^2), independent of the number of high-level states S and the number of subproblems K. For sparse-reward SMDPs, a modified algorithm is claimed in Theorem 4.2 to achieve Õ(SAH^2 Hbar^2/ε^2), a factor S smaller than the flat BPI-UCRL bound. The paper also presents a revised stopping rule for flat BPI-UCRL (Theorem 2.1) and reports experiments on gridworld-style domains.

Significance. If the main results hold, this is a valuable step: it is one of the first explicit sample-complexity guarantees for the parallel HRL setting, and the two conditions in Assumption 3.1 are a useful attempt to identify when the non-stationary SMDP learning problem is well posed. The proof architecture is coherent at a first read: the low-level error function bL and the high-level error function L compose in a natural way, and the general Theorem 4.1 proof is detailed and internally consistent. The paper also honestly discusses the restrictiveness of its assumptions. However, the sparse-reward analysis in Appendix F contains a load-bearing gap: the pathwise reward bound for the true MDP is not transferred to the empirical confidence-set kernels used in the proof. This directly affects the claimed factor-S improvement in Theorem 4.2 and the sparse-reward variant of Theorem 2.1. The experimental section is illustrative rather than a validation of the exact theoretical bounds, because the implemented algorithm uses modified constants and infrequent policy updates.

major comments (2)
  1. [Appendix F, first displayed equation; also Appendix C, sparse-reward paragraph] The proof of Theorem 4.2 relies on the assertion that 'all value functions (both true as well as empirical) are upper bounded by 1'. This does not follow from the definition of sparse-reward in Section 2, which constrains only valid episodes under the true kernel P. The empirical rewards and transition kernels are constructed from bP^t, which is set to the uniform distribution for unsampled (s,a) (Section 2). bP^t can therefore assign positive probability to transitions with P(s'|s,a)=0 and to trajectories that are not valid under P; along such trajectories the flat reward sum can exceed 1. Hence the inequality sum_h bR^{t,π^{t+1}}(σ_h,π^{t+1}_h(σ_h)) ≤ 1 is not justified, and Lemma F.1's use of U^t_h(σ)≤2 collapses. The same gap appears in Appendix C's sparse-reward variant of BPI-UCRL, where the reward sum is bounded by 1 for trajectories under the optimistic kernel P^{τ,π^{τ+1}} rathe
  2. [Section 3, Assumption 3.1 and Section 7] Assumption 3.1 is the load-bearing assumption for the paper's main theorem, and the paper itself concedes in Section 7 that all known examples satisfying it require optimal subproblem policies that terminate in a single state with probability 1. The class of HRL problems covered by Theorem 4.1 is therefore much narrower than the general framework in Sections 2–3, and the abstract's phrase 'theoretical justification for the empirical success of HRL' is stronger than what is established. I view this as a scope caveat rather than a correctness error, but it should be foregrounded in the introduction and abstract.
minor comments (4)
  1. [Section 2] The claim that the proof of Kaufmann et al. (2021) is incorrect is stated without specifics. Since this is a strong claim about prior work, the authors should either identify the precise step that fails or soften the wording.
  2. [Section 3] Typo: 'consistute' should be 'constitute'.
  3. [Section 6 / Appendix I] The experiments use simplified constants and update policies only every Ψ episodes. This is reasonable for computational tractability, but the text should state explicitly that these deviations are not covered by the theoretical analysis.
  4. [References] The reference to Matthews et al. contains a typo ('Rocktäsche1' should be 'Rocktäschel').

Circularity Check

0 steps flagged

No circular derivation: the guarantees are conditional on Assumption 3.1 and follow from standard optimism/BPI arguments; the sparse-reward appendix has a correctness gap but not a circular one.

full rationale

I find no circular step under the six enumerated patterns. Theorem 4.1 is explicitly conditional on Assumption 3.1, and the proof in Appendices D-E derives the needed ingredients independently: Lemma D.1 shows P^{\pi*} lies in the SMDP confidence set, Lemma D.2 bounds the empirical reward bias, and Lemma E.1 controls composite-distribution distance through the recursively defined error function L^t. These are standard optimism arguments based on the empirical kernel, counts, and Hoeffding bonuses; the stopping rule is the same error function, not a fitted parameter relabeled as a prediction. The reuse of BPI-UCRL, Lemma H.1 from Kaufmann et al. (2021), and Proposition 1 from Jonsson et al. (2020) is external prior-work support with proofs, not a load-bearing self-citation or an imported uniqueness theorem. One genuine weakness should be flagged, but it is not circularity: the sparse-reward analysis in Appendix F applies the Section 2 definition of sparse-reward, which is stated for valid episodes under the true kernel P, to empirical/optimistic kernels \hat P^t and P^t. The displayed chain in Appendix F concludes 'all value functions (both true as well as empirical) are upper bounded by 1' after rewriting empirical SMDP rewards under a composite distribution \Gamma^t. Since \hat P^t assigns default uniform mass to unsampled (s,a) pairs and can create invalid cycles, pathwise reward sparsity under P does not by itself bound the empirical quantities; Lemma F.1's U^t_h \le 2 and the factor-S improvement in Theorem 4.2 therefore rest on an unstated stronger condition. This is a missing-support/correctness issue in the proof, not a reduction of the conclusion to its own definition or a fitted quantity renamed as a prediction. The central derivation remains self-contained modulo that gap.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The central claim rests on Assumption 3.1, an explicit but very restrictive domain condition, plus a hidden stronger reading of 'sparse-reward' in Appendix F. There are no fitted data points in the theory; constants 3/5/10 and the policy-update interval Ψ are algorithm/experiment choices, not free parameters in the sample-complexity exponents.

axioms (5)
  • domain assumption Assumption 3.1: for each subproblem k, every optimal policy π*,k, and every other policy πk, ||P^{πk}_{1:H+1}(·|s1) − P^{π*,k}_{1:H+1}(·|s1)||_1 ≤ 2|V^{πk}_1(s1;R^k) − V^{π*,k}_1(s1;R^k)|, and a similar bound for the environment rewards Y_s.
    Stated in Section 3 and used in the proof of every theorem. The paper's Discussion admits all known examples require optimal subproblem policies that terminate in a single state with probability 1, making this a strong, restrictive condition.
  • ad hoc to paper Sparse-reward implies total reward ≤ 1 along every state sequence, not merely every valid episode under the true transition kernel.
    Appendix F's proof of Theorem 4.2 bounds empirical value functions by 1 using 'for any state sequence' language. The formal definition in Section 2 only constrains valid episodes under P, so this is an implicit strengthening that is load-bearing for the sparse-reward sample complexity.
  • domain assumption All subproblems share the same low-level transition kernel P.
    Stated in Section 3 and used to pool counts N^t across subproblems and all high-level states. Without it the bound would need per-subproblem counts and the S-independence would fail.
  • domain assumption Reward functions Y and R^k are known; only the transition kernel P is unknown.
    The algorithm computes optimistic value functions from known rewards; this is the standard BPI setting and is used throughout Sections 3–4.
  • standard math Standard martingale/KL concentration inequalities (Jonsson et al. 2020, Dann et al. 2017) and the inversion Lemma H.1 (Kaufmann et al. 2021) are valid.
    Used in Appendices G and H to construct the event E, the pseudo-count event E_cnt, and to invert quasi-linear inequalities for the final sample complexity.

pith-pipeline@v1.3.0-daily-deepseek · 38402 in / 20547 out tokens · 214797 ms · 2026-08-03T09:49:01.769710+00:00 · methodology

0 comments
read the original abstract

We present HBPI-UCRL, a model-based algorithm for hierarchical reinforcement learning (HRL) that learns high-level and low-level policies in parallel. HBPI-UCRL exploits the fact that a high-level transition corresponds to a multi-step transition at the low level. We introduce two conditions on the low-level dynamics that are sufficient to make parallel HRL learnable. When these conditions hold, we prove that HBPI-UCRL has a polynomial sample complexity in the problem parameters. In the sparse-reward, goal-directed setting, our sample complexity upper bound for HBPI-UCRL is strictly lower than that of its non-hierarchical counterpart, providing theoretical justification for the empirical success of HRL.

Figures

Figures reproduced from arXiv: 2607.29294 by Anders Jonsson, Emilie Kaufmann, Gianmarco Tedeschi, Lorenzo Steccanella.

Figure 1
Figure 1. Figure 1: a) An MDP representing a gridworld with 4 rooms and treasures; b) five subproblems [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 1
Figure 1. Figure 1: Using the regret-to-PAC conversion [Jin et al., 2018], which runs their algorithm for a [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Stopping time of BPI-UCRL and HBPI-UCRL as functions of the number of high-level [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Stopping time and execution time as functions of the policy update interval [PITH_FULL_IMAGE:figures/full_fig_p031_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of observed reward between BPI-UCRL and HBPI-UCRL (dotted line for BPI [PITH_FULL_IMAGE:figures/full_fig_p032_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of observed reward between BPI-UCRL and HBPI-UCRL (dotted line for BPI [PITH_FULL_IMAGE:figures/full_fig_p032_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Comparison of observed reward between BPI-UCRL and HBPI-UCRL (dotted line for BPI [PITH_FULL_IMAGE:figures/full_fig_p033_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Heatmap of visited states in the domain with [PITH_FULL_IMAGE:figures/full_fig_p033_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Ablations varying S and HH while maintaining everything else constant. the horizon much too large. We conjecture that the stopping time of HBPI-UCRL depends on the effective subproblem horizon H, i.e. the number of time steps necessary to solve each subproblem in practice [PITH_FULL_IMAGE:figures/full_fig_p034_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Stopping time τ of BPI-UCRL and HBPI-UCRL as a function of S in stochastic domains. Results for Stochastic Domains. Even though Assumption 3.1 is not guaranteed to hold for stochastic domains in which the probability of reaching a terminal state is less than 1, we carry out experiments in a version of the domain in [PITH_FULL_IMAGE:figures/full_fig_p034_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Results for HBPI-UCRL with treasures. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

85 extracted references · 1 canonical work pages

  1. [1]

    2018 , journal=

    Provably Efficient Maximum Entropy Exploration , author=. 2018 , journal=

    2018

  2. [2]

    Advances in neural information processing systems , pages=

    \# exploration: A study of count-based exploration for deep reinforcement learning , author=. Advances in neural information processing systems , pages=

  3. [3]

    Proceedings of the 23rd international conference on Machine learning , pages=

    PAC model-free reinforcement learning , author=. Proceedings of the 23rd international conference on Machine learning , pages=

  4. [4]

    Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages=

    Count-based exploration with neural density models , author=. Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages=. 2017 , organization=

    2017

  5. [5]

    The True Sample Complexity of Identifying Good Arms , booktitle =

    Julian Katz. The True Sample Complexity of Identifying Good Arms , booktitle =

  6. [6]

    2019 , eprint=

    Autonomous exploration for navigating in non-stationary CMPs , author=. 2019 , eprint=

    2019

  7. [7]

    Conference on Learning Theory , pages=

    Autonomous exploration for navigating in mdps , author=. Conference on Learning Theory , pages=

  8. [8]

    arXiv preprint arXiv:2002.09869 , year=

    Near-optimal Regret Bounds for Stochastic Shortest Path , author=. arXiv preprint arXiv:2002.09869 , year=

    Pith/arXiv arXiv 2002

  9. [9]

    arXiv preprint arXiv:1912.03517 , year=

    No-Regret Exploration in Goal-Oriented Reinforcement Learning , author=. arXiv preprint arXiv:1912.03517 , year=

    Pith/arXiv arXiv 1912

  10. [10]

    A possibility for implementing curiosity and boredom in model-building neural controllers , author=. Proc. of the international conference on simulation of adaptive behavior: From animals to animats , pages=

  11. [11]

    2007 , booktitle =

    Coquelin, Pierre-Arnaud and Munos, Rémi , url =. 2007 , booktitle =

    2007

  12. [12]

    Proceedings of the 36th International Conference on Machine Learning, (ICML) , year =

    Andrea Zanette and Emma Brunskill , title =. Proceedings of the 36th International Conference on Machine Learning, (ICML) , year =

  13. [13]

    International Conference on Learning Representations , year=

    Combining Q-Learning and Search with Amortized Value Estimates , author=. International Conference on Learning Representations , year=

  14. [14]

    Kearns and Satinder P

    Michael J. Kearns and Satinder P. Singh , title =. Machine Learning , volume =

  15. [15]

    2012 , publisher=

    Elements of information theory , author=. 2012 , publisher=

    2012

  16. [16]

    Annals of probability , pages=

    Self-normalized processes: exponential inequalities, moment bounds and iterated logarithm laws , author=. Annals of probability , pages=. 2004 , publisher=

    2004

  17. [17]

    Proceedings of the 24th annual conference on learning theory , pages=

    The KL-UCB algorithm for bounded stochastic bandits and beyond , author=. Proceedings of the 24th annual conference on learning theory , pages=

  18. [18]

    Annals of Statistics , Year =

    Capp. Annals of Statistics , Year =

  19. [19]

    and Lattimore, T

    Dann, C. and Lattimore, T. and Brunskill, E. , booktitle=

  20. [20]

    arXiv:2002.02794 , year =

    Chi Jin and Akshay Krishnamurthy and Max Simchowitz and Tiancheng Yu , title =. arXiv:2002.02794 , year =

    Pith/arXiv arXiv 2002

  21. [21]

    Jin and Z

    C. Jin and Z. Allen. Advances in Neural Information Processing Systems , year =

  22. [22]

    arXiv preprint arXiv:1805.05071 , year=

    KL-UCB-switch: optimal regret bounds for stochastic bandits from both a distribution-dependent and a distribution-free viewpoints , author=. arXiv preprint arXiv:1805.05071 , year=

    Pith/arXiv arXiv

  23. [23]

    , Publisher =

    Puterman, M.L. , Publisher =

  24. [24]

    and Li, L

    Dann, C. and Li, L. and Wei, W. and Brunskill, E. , booktitle =

  25. [25]

    , Author =

    From bandits to Monte-Carlo Tree Search: The optimistic principle applied to optimization and planning. , Author =. 2014 , Number =

    2014

  26. [26]

    Planning in entropy-regularized Markov decision processes and games , booktitle =

    Jean. Planning in entropy-regularized Markov decision processes and games , booktitle =

  27. [27]

    Ajallooeian and Csaba Szepesv

    Ruitong Huang and Mohammad M. Ajallooeian and Csaba Szepesv. Structured Best Arm Identification with Fixed Confidence , booktitle =

  28. [28]

    Koolen , title =

    Emilie Kaufmann and Wouter M. Koolen , title =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  29. [29]

    Proceedings of the 17th European Conference on Machine Learning (ECML) , Year =

    Bandit Based Monte-carlo Planning , Author =. Proceedings of the 17th European Conference on Machine Learning (ECML) , Year =

  30. [30]

    and Mannor, S

    Even-Dar, E. and Mannor, S. and Mansour, Y. , Journal =. 2006 , Pages =

    2006

  31. [31]

    Lattimore, Tor and Szepesvari, Csaba , publisher =

  32. [32]

    and Cesa-Bianchi, N

    Auer, P. and Cesa-Bianchi, N. and Fischer, P. , Journal =. 2002 , Number =

    2002

  33. [33]

    Jaksch and R

    T. Jaksch and R. Ortner and P. Auer , title =. Journal of Machine Learning Research , volume =

  34. [34]

    Proceedings of the Twenty-Sixth

    David Tolpin and Solomon Eyal Shimony , title =. Proceedings of the Twenty-Sixth

  35. [35]

    Tom Pepels and Tristan Cazenave and Mark H. M. Winands and Marc Lanctot , title =. Third Workshop on Computer Games (CGW) , pages =

  36. [36]

    Journal of Artifial Intelligence Research , volume =

    Zohar Feldman and Carmel Domshlak , title =. Journal of Artifial Intelligence Research , volume =

  37. [37]

    Kearns and Yishay Mansour and Andrew Y

    Michael J. Kearns and Yishay Mansour and Andrew Y. Ng , title =. Machine Learning , volume =

  38. [38]

    Advances in Neural Information Processing Systems (NIPS) , Year =

    Optimistic Planning in Markov Decision Processes using a generative model , Author =. Advances in Neural Information Processing Systems (NIPS) , Year =

  39. [39]

    1998 , Owner =

    Reinforcement Learning: an Introduction , Author =. 1998 , Owner =

    1998

  40. [40]

    Lillicrap and Karen Simonyan and Demis Hassabis , title =

    David Silver and Thomas Hubert and Julian Schrittwieser and Ioannis Antonoglou and Matthew Lai and Arthur Guez and Marc Lanctot and Laurent Sifre and Dharshan Kumaran and Thore Graepel and Timothy P. Lillicrap and Karen Simonyan and Demis Hassabis , title =. Science , volume =

  41. [41]

    Advances in Neural Information Processing Systems (NIPS) , year =

    Christoph Dann and Emma Brunskill , title =. Advances in Neural Information Processing Systems (NIPS) , year =

  42. [42]

    Jonsson and E

    A. Jonsson and E. Kaufmann and P. M\'enard and O. Darwiche Domingues and E. Leurent and M. Valko , title =. Neural Information Processing Systems , year =

  43. [43]

    Lillicrap and David Silver , title =

    Julian Schrittwieser and Ioannis Antonoglou and Thomas Hubert and Karen Simonyan and Laurent Sifre and Simon Schmitt and Arthur Guez and Edward Lockhart and Demis Hassabis and Thore Graepel and Timothy P. Lillicrap and David Silver , title =. arXiv:1911.08265 , year =

    Pith/arXiv arXiv 1911

  44. [44]

    IEEE Transactions on Computational Intelligence and AI in games, , Year =

    A Survey of Monte Carlo Tree Search Methods , Author =. IEEE Transactions on Computational Intelligence and AI in games, , Year =

  45. [45]

    Neural Information Processing Systems (NIPS) , Year =

    Blazing the trails before beating the path: Sample-efficient Monte-Carlo planning , Author =. Neural Information Processing Systems (NIPS) , Year =

  46. [46]

    Advances in Neural Information Processing Systems , pages=

    Best arm identification: A unified approach to fixed budget and fixed confidence , author=. Advances in Neural Information Processing Systems , pages=

  47. [47]

    Gheshlaghi Azar and I

    M. Gheshlaghi Azar and I. Osband and R. Munos , title =. Proceedings of the 34th International Conference on Machine Learning, (ICML) 2017 , year =

    2017

  48. [48]

    On the Sample Complexity of Reinforcement Learning with a Generative Model , booktitle =

    Mohammad Gheshlaghi Azar and R. On the Sample Complexity of Reinforcement Learning with a Generative Model , booktitle =

  49. [49]

    Kearns and Satinder P

    Michael J. Kearns and Satinder P. Singh , title =. Advances in Neural Information Processing Systems (NIPS) , pages =

  50. [50]

    Efficient Reinforcement Learning , booktitle =

    Claude. Efficient Reinforcement Learning , booktitle =

  51. [51]

    Expected Mistake Bound Model for On-Line Reinforcement Learning , booktitle =

    Claude. Expected Mistake Bound Model for On-Line Reinforcement Learning , booktitle =

  52. [52]

    and Capp

    Filippi, S. and Capp

  53. [53]

    Brafman and Moshe Tennenholtz , title =

    Ronen I. Brafman and Moshe Tennenholtz , title =. Journal of Machine Learning Research , volume =

  54. [54]

    Strehl and Lihong Li and Eric Wiewiora and John Langford and Michael L

    Alexander L. Strehl and Lihong Li and Eric Wiewiora and John Langford and Michael L. Littman , title =. Proceedings of the Twenty-Third International Conference on Machine Learning (ICML , year =

  55. [55]

    Strehl and Michael L

    Alexander L. Strehl and Michael L. Littman , title =. Journal of Computer and System Sciences , volume =

  56. [56]

    Conference on Learning Theory , year=

    Open Loop Optimistic Planning , author=. Conference on Learning Theory , year=

  57. [57]

    2019 , booktitle=

    Practical Open-Loop Optimistic Planning , author=. 2019 , booktitle=

    2019

  58. [58]

    Artificial Intelligence and Statistics , pages=

    Optimistic planning for Markov decision processes , author=. Artificial Intelligence and Statistics , pages=

  59. [59]

    Advances in Neural Information Processing Systems 28 , editor =

    Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning , author =. Advances in Neural Information Processing Systems 28 , editor =. 2015 , publisher =

    2015

  60. [60]

    2016 , eprint=

    Information Theoretically Aided Reinforcement Learning for Embodied Agents , author=. 2016 , eprint=

    2016

  61. [61]

    2012 , Journal =

    An information-theoretic approach to curiosity-driven reinforcement learning , Author =. 2012 , Journal =. doi:10.1007/s12064-011-0142-z , Number =

    work page doi:10.1007/s12064-011-0142-z 2012

  62. [62]

    Advances in Neural Information Processing Systems 17 , editor =

    Intrinsically Motivated Reinforcement Learning , author =. Advances in Neural Information Processing Systems 17 , editor =. 2005 , publisher =

    2005

  63. [63]

    arXiv preprint arXiv:1901.00210 , year=

    Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds , author=. arXiv preprint arXiv:1901.00210 , year=

    Pith/arXiv arXiv 1901

  64. [64]

    Kaufmann and P

    E. Kaufmann and P. M\'enard and O. Darwiche Domingues and A. Jonsson and E. Leurent and M. Valko , booktitle=

  65. [65]

    M\'enard and O

    P. M\'enard and O. Darwiche Domingues and A. Jonsson and E. Kaufmann and E. Leurent and M. Valko , booktitle=

  66. [66]

    Infante and A

    G. Infante and A. Jonsson and V. G\'omez , booktitle=

  67. [67]

    Drappo and A

    G. Drappo and A. Robert and M. Restelli and A. A. Faisal and A. M. Metelli and C. Pike-Burke , booktitle=

  68. [68]

    Robert and C

    A. Robert and C. Pike-Burke and A. A. Faisal , booktitle=

  69. [69]

    Wen and D

    Z. Wen and D. Precup and M. Ibrahimi and A. Barreto and B. Van Roy and S. Singh , booktitle=

  70. [70]

    Nachum and S

    O. Nachum and S. Gu and H. Lee and S. Levine , booktitle=

  71. [71]

    Konidaris and A

    G. Konidaris and A. Barto , booktitle=

  72. [72]

    A. S. Vezhnevets and S. Osindero and T. Schaul and N. Heess and M. Jaderberg and D. Silver and K. Kavukcuoglu , booktitle=

  73. [73]

    Levy and G

    A. Levy and G. Konidaris and R. Platt and K. Saenko , booktitle=

  74. [74]

    Fruit and M

    R. Fruit and M. Pirotta and A. Lazaric and E. Brunskill , booktitle=

  75. [75]

    Drappo and A

    G. Drappo and A. M. Metelli and R. Marcello , journal=

  76. [76]

    Rafati and D

    J. Rafati and D. C. Noelle , booktitle=

  77. [77]

    Gopalan and M

    N. Gopalan and M. desJardins and M. Littman and J. MacGlashan and S. Squire and S. Tellex and J. Winder and L. Wong , booktitle=

  78. [78]

    M. G. Bellemare and S. Candido and P. S. Castro and J. Gong and M. C. Machado and S. Moitra and S. S. Ponda and Z. Wang , journal=

  79. [79]

    Ahn and A

    M. Ahn and A. Brohan and N. Brown and Y. Chebotar and O. Cortes and B. David and C. Finn and C. Fu and K. Gopalakrishnan and K. Hausman and A. Herzog and D. Ho and J. Hsu and J. Ibarz and B. Ichter and A. Irpan and E. Jang and R. J. Ruano and K. Jeffrey and S. Jesmonth and N. Joshi and R. Julian and D. Kalashnikov and Y. Kuang and K.-H. Lee and S. Levine ...

  80. [80]

    Brunskill and L

    E. Brunskill and L. Li , booktitle=

Showing first 80 references.