Pith. sign in

REVIEW 4 major objections 5 minor 43 references

Exploration by Running Away from the Past

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that maximizing the divergence between the agent's current state distribution and a discounted mixture of its past distributions, plus a policy-entropy bonus, is an effective proxy for maximizing the Shannon entropy of…

desk verdict A useful exploration heuristic with a solid classifier-based formulation; the entropy-maximization interpretation is only partially supported, but the empirical case for RAMPW is real and worth a serious look. read the letter →

arxiv 2411.14085 v1 pith:NJ524Y4J submitted 2024-11-21 cs.LG

Paul-Antoine Le Tolguenec , Yann Besse , Florent Teichteil-Koenigsbuch , Dennis G. Wilson , Emmanuel Rachelson This is my paper · ORCID
classification cs.LG
keywords reinforcementlearningintrinsicmotivationexplorationShannonentropystateoccupancymeasureKLdivergenceWassersteindistancetemporal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that an agent can explore by actively distancing itself from its own past, rather than by directly estimating the entropy of its state distribution. The proposed RAMP method rewards states that separate the current occupancy measure $\rho_{n+1}$ from the discounted past mixture $\mu_{n+1}=\beta\rho_{n+1}+(1-\beta)\mu_n$, together with a policy-entropy bonus. A theorem lower-bounds the per-epoch entropy increase by this divergence, and the divergence itself is estimated either by a contrastive classifier (RAMPKL) or by a temporal-distance Wasserstein critic (RAMPW). If the claim holds, exploration in continuous control reduces to training one classifier or critic against a replay buffer, and the paper's experiments report state coverage competitive with or better than several existing exploration methods on mazes, locomotion, and manipulation tasks.

What carries the argument

The load-bearing object is the discounted past mixture $\mu_{n+1}(s)=\beta\rho_{n+1}(s)+(1-\beta)\mu_n(s)$, which compresses all previous behavior into one distribution that a replay buffer can sample. Theorem 1 shows the entropy difference $H_{n+1}-H_n$ is bounded below by $\beta(D_{\mathrm{KL}}(\rho_{n+1}\|\mu_{n+1}) + H_{\rho_{n+1}}[S] - H_n)$, turning entropy maximization into a divergence-maximization problem. For RAMPKL, the density ratio inside the KL term is estimated by a contrastive classifier whose pre-sigmoid output is the state reward. For RAMPW, the repulsion is quantified by a Wasserstein distance whose dual potential $f_\phi$ is trained under a 1-Lipschitz constraint with respect to the temporal distance, making the reward grow with the minimum number of environment steps separating a current state from past states.

What would settle it

Run RAMPKL in a continuous environment where every random action keeps the agent near its start position but a single deterministic action reaches a distant never-visited state. If the intrinsic reward increases while discretized state-space coverage stays flat or falls, the policy-entropy proxy rather than true state diversity is carrying the objective.

Watch

Extended reading notes

Core claim

The central discovery is that the Shannon entropy of the agent's experience distribution can be increased monotonically by repeatedly maximizing a divergence between present and past, and this divergence can be learned with a single network. Concretely, Theorem 1 gives $H_{n+1}-H_n \ge \beta\big(D_{\mathrm{KL}}(\rho_{n+1}\|\mu_{n+1}) + H_{\rho_{n+1}}[S] - H_n\big)$, so the authors set the next policy to maximize the repulsive term with an entropy regularization. RAMPKL learns the density ratio inside the KL through binary classification between current and past states, while RAMPW maximizes the Wasserstein distance under a 1-Lipschitz constraint in the temporal distance. In the paper's experiments, RAMPW reaches the highest discretized state-space coverage on four of five locomotion tasks, including a score on Ant six times that of the second-best baseline, whereas RAMPKL remains competitive when an extrinsic reward guides exploration.

Load-bearing premise

The load-bearing premise is that rewarding the agent for high-action randomness (policy entropy) actually spreads the states it visits; if random actions do not widen the state distribution, the 'run from the past' reward can be satisfied by revisiting a few states that were never seen before, and coverage can fall instead of rise.

Editorial extensions

If this is right

  • If Theorem 1's proxy holds, any policy optimization method can add exploration by training a single classifier or critic against a replay buffer of past states, bypassing full-state density estimation.
  • On the paper's experiments, RAMPW achieves the highest state-space coverage on four of five locomotion environments, with an Ant score about six times the second-best baseline.
  • When an extrinsic reward is present, the paper reports that RAMPKL's aggregate score across five locomotion environments is slightly higher than RAMPW's, while RAMPW exceeds the second-best baseline on Ant by more than 40 percent.
  • In high-dimensional locomotion, the KL objective can be satisfied by changing joint configurations without translating the agent; the Wasserstein version with temporal distance is the authors' fix for this failure mode.
  • The authors suggest the same running-away-from-the-past objective can be combined with skill-based or hierarchical exploration strategies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the KL variant's failure mode is the one Proposition 4 predicts—optimal solutions place mass where the past mixture $\mu_n$ has zero mass—so RAMPKL should be expected to underperform whenever a novel joint configuration is easier to reach than a novel location.
  • Editorial inference: the comparison between RAMPKL and RAMPW hints that exploration gains decompose into a metric component and a divergence component; replacing the temporal distance with a Euclidean or task-specific latent metric would test which component drives each benchmark's improvement.
  • Editorial inference: because the objective compares occupancy measures rather than trajectories, it does not by itself reward diverse behaviors that visit identical states; conditioning the same repulsion on skill descriptors would be a natural extension for skill discovery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes RAMP, an exploration method for RL that maximizes a divergence between the agent's current state-occupancy measure and a discounted mixture of past occupancies, plus a policy-entropy bonus. Two instantiations are presented: RAMPKL, which uses a contrastive classifier to estimate a log-density ratio, and RAMPW, which uses a Wasserstein critic with a temporal-distance Lipschitz constraint. The authors prove a lower bound on the increment of the Shannon entropy of the past-mixture distribution, provide two monotonic-improvement theorems for the approximate reward models, and evaluate coverage and episodic returns on maze, MuJoCo locomotion, and Fetch tasks.

Significance. If the central claim held, RAMP would offer a simple, classifier- or critic-based way to increase state-occupancy entropy without expensive density estimation, and the empirical comparison is useful, especially RAMPW's strong coverage on Ant. The paper is also honest about identifying where KL-based exploration collapses. However, the theoretical bridge from divergence maximization to Shannon-entropy maximization is incomplete for both variants: the variant that works well, RAMPW, is explicitly not covered by Theorem 1, and the variant that is covered, RAMPKL, fails on high-dimensional locomotion. The paper therefore does not currently substantiate its abstract-level claim that it maximizes the Shannon entropy of the state occupation measure.

major comments (4)
  1. [Section 2, Eq. (2), and Theorem 1] Theorem 1's lower bound contains Hρπ[S], the Shannon entropy of the state-occupancy measure, but the proposed objective in Eq. (2) replaces this term with the policy-entropy bonus λA Es∼ρπ,a∼π[−log π(a|s)]. The paper states that this proxy 'may not universally apply across all environments,' and Appendix I's Proposition 4 shows that the KL repulsive term alone is maximized by concentrating mass where the past mixture µn has zero density, which can reduce rather than increase occupancy entropy. Table 1 shows the concrete failure mode: RAMPKL obtains only 1.2% relative coverage on Ant and 29.76% on HalfCheetah. The central claim that RAMP maximizes Shannon state entropy is therefore not supported without a direct measurement of Hρπ[S] over training; I recommend adding such a measurement and comparing with a policy-entropy-only baseline.
  2. [Section 2, Eq. (4), and Section 5.2, Table 1] The version that actually performs well, RAMPW, is explicitly not maximizing a lower bound on the entropy increment: Section 2 states 'this objective does not maximize a lower bound on ∆n per se.' Its strong coverage results (78.35% on Ant, 74.43% on Hopper) may reflect alignment between the temporal-distance Wasserstein reward and the xyz-coordinate coverage metric used in Table 1, rather than an increase in the Shannon entropy of the state-occupancy measure. To separate these explanations, the paper should report coverage on state features not directly predicted by the temporal-distance potential (e.g., joint configurations) and measure Hρπ[S] directly.
  3. [Appendix C, Theorem 2] The statement and proof of Theorem 2 are inconsistent. The theorem defines ε0 by ∥ρπ′/ρπ − 1∥∞ ≥ ε0 and calls ρπ′ 'close to ρπ', but the proof immediately assumes 1−ε0 ≤ ρπ′/ρπ ≤ 1+ε0, which requires the norm to be at most ε0. As written, the premise does not express closeness, and the conclusion does not follow. The proof also contains an unjustified step: from |ˆrDKL(s)−f(ρ,µn)(s)|≤ε1 it does not follow that ⟨ρ,f(ρ,µn)⟩≥−ε1. The theorem and proof need to be corrected before this result can serve as a guarantee for RAMPKL's reward model.
  4. [Section 3.2, Eq. (6), and Appendix G] The derivation of the estimated reward fϕ relies on equal prior probabilities P(L=1)=P(L=0). The algorithm and appendices never specify how the positive and negative samples are balanced; in Algorithm 1, Dρ and Dµn are updated in different ways (episode-based collection versus accept-reject), so the class priors are not equal in general. If the batches are not explicitly balanced, the learned fϕ is not the log-density ratio in Eq. (2), and Theorem 2's approximation-error assumption is not satisfied. Please state the exact sampling procedure used in the experiments.
minor comments (5)
  1. [Section 5.3, paragraph after Table 2] The aggregate mean scores are reported as '33, 527×10^3 and 32, 408×10^3', but summing the five entries in Table 2 gives 33,527 and 32,408; the ×10^3 factor appears erroneous and should be corrected.
  2. [Appendix D and Theorem 3] The notation W(ρπ, βρπ + µn(β − 1)) is confusing; it should be written as βρπ + (1 − β)µn to match Eq. (3), since µn(β − 1) is not defined as a distribution.
  3. [Equation (3)] The term '1-Lipshitz' should be '1-Lipschitz'.
  4. [Algorithm 1] Eq. (6) defines LDKL as a loss to be minimized, but Algorithm 1 writes 'ϕ = arg max_ϕ LDKL(ϕ)'; please clarify whether the displayed quantity is the negative loss or the classification objective in Appendix H, and align the notation.
  5. [Section 3.4 and Appendix F] The update of Dµn uses a Bernoulli accept-reject procedure, but the paper does not state the buffer capacity or the exact number of samples Neρ used; these details are needed for reproducibility.

Circularity Check

2 steps flagged · score 2.0 of 10

Theorems 2 and 3 are definitional consistency lemmas: each reward is, by construction, the integrand or dual potential of the divergence whose increase the theorem establishes. The central Shannon-entropy claim is not circular; it rests on an admitted policy-entropy proxy and on RAMPW being explicitly outside the Theorem 1 bound.

  1. self definitional [Section 3.1, Eq. (2) and Theorem 2 (Appendix C)]
    "Let us define rπ DKL(s) = log (ρπ(s)/(βρπ(s) + (1 − β)µn(s))) the reward model based on the first term in Equation 2. This term can thus be written ⟨ρπ, rπ DKL⟩. ... If ε2 ≥ 2ε1 − log(1 − ε0), then D KL (ρπ′||ρπ′β + (1 − β)µn) ≥ DKL (ρπ||ρπβ + (1 − β)µn)."

    The reward rDKL is defined as the pointwise log-density ratio whose expectation under ρπ is exactly the KL divergence DKL(ρπ||βρπ+(1−β)µn) appearing in both Objective (2) and Theorem 1. Theorem 2 then 'proves' that a policy improving this reward increases the same divergence. The condition ε2 is the increase in ⟨ρ′, r̂DKL⟩, i.e. the increase of the integrand's expectation; the conclusion is already contained in the definition of rDKL. This is a self-consistency statement, not an independent derivation of the entropy-increase objective.

  2. self definitional [Section 3.1, Eq. (4) and Theorem 3 (Appendix D)]
    "For the repulsive term in Objective 4, the reward model is defined as rW (s) = f ∗(s), where f ∗ belongs to the solutions of the problem defined in Equation 3. ... If ε2 ≥ 2ε1(1 + β), then W(ρπ′, βρπ′ + µn(β − 1)) > W(ρπ, βρπ + µn(β − 1))."

    By Kantorovich duality in Eq. (3), W(ρπ, βρπ+(1−β)µn) = max_{∥f∥≤1} E_{s+∼ρπ}[f(s+)] − E_{s−∼βρπ+(1−β)µn}[f(s−)]. The reward rW is deliberately chosen as a maximizer f* of this dual problem, so the policy objective E_{s∼ρπ}[rW(s)] is the positive part of the very Wasserstein dual being maximized. Theorem 3's ε2 condition is an increase in that dual term, so the conclusion that W increases is a direct consequence of the definition of rW as the dual potential. The theorem is a consistency lemma, not an independent prediction about state-entropy maximization.

full rationale

The paper's headline claim—maximizing a sequence of divergences from past behavior maximizes the Shannon entropy of the state occupation measure—is not circular in the strong sense. Theorem 1 is a genuine inequality linking the entropy increment to DKL + Hρ[S], and the paper explicitly identifies the step where it substitutes the policy-entropy proxy: 'A classic proxy in RL for obtaining a large state occupation ρπ ... is the maximization of the policy's entropy' and 'This hypothesis may not universally apply across all environments.' That is a load-bearing assumption and a correctness risk, but not a tautology. Similarly, the RAMPW objective is explicitly said not to maximize a lower bound on Δn: 'this objective does not maximize a lower bound on Δn per se,' so its strong empirical results validate a temporal-distance objective rather than the Shannon-entropy bound. The paper's own Appendix I (Proposition 4) further shows that the KL repulsive term alone favors supports on zero-density regions of µn, which is an admitted limitation rather than a circular reduction. The definitional circularity is confined to Theorems 2 and 3, where the reward is, by construction, the integrand or dual potential of the divergence whose increase is 'proved.' These are consistency lemmas and do not by themselves support the entropy claim. Additional limitations, such as coverage being measured on torso xyz-coordinates that the temporal-distance reward directly encourages, are metric-alignment concerns, not circularity. Because the central claim does not reduce to the definitions—it reduces to an admitted proxy—the circularity score is low (2).

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical entities. It does introduce two intrinsic reward models, rDKL and rW, but these are algorithmic constructs rather than postulated entities with independent falsifiable handles. The central claim rests on hand-chosen hyperparameters (β, λA, λ, ε) and on domain assumptions about density existence, the entropy proxy, and the approximate temporal-distance Lipschitz constraint.

free parameters (4)
  • beta = 7e-3 (mentioned in Appendix H as an example)
    Controls how much of the new occupancy measure replaces the past mixture µn. Chosen by hand, not derived; the algorithm's behavior depends on it.
  • lambda_A = not stated in the provided text
    Weights the policy entropy bonus in Objectives 2 and 4. Hand-chosen hyperparameter.
  • lambda = adaptive, not stated
    Lagrange multiplier for the Wasserstein 1-Lipschitz constraint in Equation 7. Adjusted by gradient descent.
  • epsilon = not stated
    Relaxation constant in the Wasserstein constraint loss in Equation 7. Hand-chosen.
assumptions (4)
  • domain assumption State occupation measures have well-defined densities, so Shannon entropy, KL divergence, and the Wasserstein dual are finite and computable.
    Section 2 defines ρn and µn as densities over S without regularity conditions; the theoretical results rely on these quantities existing.
  • ad hoc to paper Maximizing policy entropy across visited states is a valid proxy for maximizing occupancy entropy Hρπ[S].
    Section 2 states this as a classic proxy and admits it may not universally apply; the paper's entropy-increase claim depends on it.
  • domain assumption The temporal distance is a metric on the state space and the learned critic is close to 1-Lipschitz under it.
    Section 3.3 and Equation 7 maintain the Lipschitz constraint only approximately, with a relaxation epsilon; the Wasserstein reward interpretation depends on this.
  • domain assumption The classifier in Equation 6 is trained with balanced positive and negative samples, so Bayes' rule gives the claimed log-ratio.
    Section 3.2 states the label balance condition; if positive and negative sampling rates differ, fϕ estimates a shifted log-ratio.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploration by Running Away from the Past." pith.science (2026). https://pith.science/paper/NJ524Y4J

@misc{pith2026241114085,
  author       = {Pith},
  title        = {Pith review of: Exploration by Running Away from the Past},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NJ524Y4J}},
  note         = {Machine review of arXiv:2411.14085}
}
abstract

The ability to explore efficiently and effectively is a central challenge of reinforcement learning. In this work, we consider exploration through the lens of information theory. Specifically, we cast exploration as a problem of maximizing the Shannon entropy of the state occupation measure. This is done by maximizing a sequence of divergences between distributions representing an agent's past behavior and its current behavior. Intuitively, this encourages the agent to explore new behaviors that are distinct from past behaviors. Hence, we call our method RAMP, for ``$\textbf{R}$unning $\textbf{A}$way fro$\textbf{m}$ the $\textbf{P}$ast.'' A fundamental question of this method is the quantification of the distribution change over time. We consider both the Kullback-Leibler divergence and the Wasserstein distance to quantify divergence between successive state occupation measures, and explain why the former might lead to undesirable exploratory behaviors in some tasks. We demonstrate that by encouraging the agent to explore by actively distancing itself from past experiences, it can effectively explore mazes and a wide range of behaviors on robotic manipulation and locomotion tasks.

Figures

Figures reproduced from arXiv: 2411.14085 by the authors.

Figure 1
Figure 1. The four steps of the RAMP algorithm. 3.4 The RAMP Algorithm The full RAMP algorithm is described in Algorithm 1 and illustrated in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. (a) An illustration of the different experience distributions on the U-maze. (b) A compari [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. XY -coordinates of the Ant’s torso at different timesteps T of training. Color indicates the density used as reward model for RAMPW. 5.2 Quantifying the capacity to explore We now compare the two versions of RAMP (RAMPKL and RAMPW) with 10 baselines which represent different approaches to exploration. We aim to understand whether the objective of RAMP leads to effective exploration, compared to similar and contempor… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Euclidean coordinates of the states contained in the buffers used by RAMP with the color [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]
Figure 7
Figure 7. Figure 7: Set of tasks. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

43 extracted references · 11 canonical work pages

  1. [1]

    Understanding the impact of entropy on policy optimization

    Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pp.\ 151--160. PMLR, 2019

    2019

  2. [2]

    Logarithmic online regret bounds for undiscounted reinforcement learning

    Peter Auer and Ronald Ortner. Logarithmic online regret bounds for undiscounted reinforcement learning. Advances in neural information processing systems, 19, 2006

    2006

  3. [3]

    Never give up: Learning directed exploration strategies

    Adri \`a Puigdom \`e nech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Mart \' n Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprint arXiv:2002.06038, 2020

    arXiv 2002

  4. [4]

    Unifying count-based exploration and intrinsic motivation

    Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016

    2016

  5. [5]

    Tightening exploration in upper confidence reinforcement learning

    Hippolyte Bourel, Odalric Maillard, and Mohammad Sadegh Talebi. Tightening exploration in upper confidence reinforcement learning. In International Conference on Machine Learning, pp.\ 1056--1066. PMLR, 2020

    work page 2020

  6. [6]

    R-max-a general polynomial time algorithm for near-optimal reinforcement learning

    Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3 0 (Oct): 0 213--231, 2002

    work page 2002

  7. [7]

    Openai gym, 2016

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016

    2016

  8. [8]

    Exploration by random network distillation

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12894, 2018

    arXiv 2018

Show all 43 references
  1. [9]

    Explore, discover and learn: Unsupervised discovery of state-covering skills

    V \' ctor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Gir \'o -i Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pp.\ 1317--1327. PMLR, 2020

    2020

  2. [10]

    Gymnasium robotics, 2023

    Rodrigo de Lazcano, Kallinteris Andreas, Jun Jet Tai, Seungjae Ryan Lee, and Jordan Terry. Gymnasium robotics, 2023. URL http://github.com/Farama-Foundation/Gymnasium-Robotics

    2023

  3. [11]

    Adversarial intrinsic motivation for reinforcement learning

    Ishan Durugkar, Mauricio Tec, Scott Niekum, and Peter Stone. Adversarial intrinsic motivation for reinforcement learning. Advances in Neural Information Processing Systems, 34: 0 8622--8636, 2021

    2021

  4. [12]

    Diversity is all you need: Learning skills without a reward function

    Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018

    2018 arXiv

  5. [13]

    C-learning: Learning to achieve goals via recursive classification

    Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. C-learning: Learning to achieve goals via recursive classification. arXiv preprint arXiv:2011.08909, 2020

    2011 arXiv

  6. [14]

    A theory of regularized markov decision processes

    Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference on Machine Learning, pp.\ 2160--2169. PMLR, 2019

    2019

  7. [15]

    Variational intrinsic control

    Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.07507, 2016

    2016 arXiv

  8. [16]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp.\ 1861--1870. PMLR, 2018

    2018

  9. [17]

    Dynamical distance learning for semi-supervised and unsupervised skill discovery

    Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learning for semi-supervised and unsupervised skill discovery. arXiv preprint arXiv:1907.08225, 2019

    1907 arXiv

  10. [18]

    Provably efficient maximum entropy exploration

    Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, pp.\ 2681--2691. PMLR, 2019

    2019

  11. [19]

    Reinforcement learning with unsupervised auxiliary tasks

    Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397, 2016

    2016 arXiv

  12. [20]

    Near-optimal regret bounds for reinforcement learning

    Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 11: 0 1563--1600, 2010

    2010

  13. [21]

    Curiosity in hindsight

    Daniel Jarrett, Corentin Tallec, Florent Altch \'e , Thomas Mesnard, R \'e mi Munos, and Michal Valko. Curiosity in hindsight. arXiv preprint arXiv:2211.10515, 2022

    2022 arXiv

  14. [22]

    Learning to achieve goals

    Leslie Pack Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pp.\ 1094--8. Citeseer, 1993

    1993

  15. [23]

    Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching

    Pierre-Alexandre Kamienny, Jean Tarbouriech, Sylvain Lamprier, Alessandro Lazaric, and Ludovic Denoyer. Direct then diffuse: Incremental unsupervised skill discovery for state covering and goal reaching. arXiv preprint arXiv:2110.14457, 2021

    arXiv

  16. [24]

    Near-optimal reinforcement learning in polynomial time

    Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49: 0 209--232, 2002

    2002

  17. [25]

    Efficient exploration via state marginal matching

    Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXiv:1906.05274, 2019

    1906 arXiv

  18. [26]

    Behavior from the void: Unsupervised active pre-training

    Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34: 0 18459--18473, 2021

    2021

  19. [27]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    2013 arXiv

  20. [28]

    From bandits to monte-carlo tree search: The optimistic principle applied to optimization and planning

    R \'e mi Munos et al. From bandits to monte-carlo tree search: The optimistic principle applied to optimization and planning. Foundations and Trends in Machine Learning , 7 0 (1): 0 1--129, 2014

    2014

  21. [29]

    Lipschitz-constrained unsupervised skill discovery

    Seohong Park, Jongwook Choi, Jaekyeom Kim, Honglak Lee, and Gunhee Kim. Lipschitz-constrained unsupervised skill discovery. In International Conference on Learning Representations, 2022

    2022

  22. [30]

    Controllability-aware unsupervised skill discovery

    Seohong Park, Kimin Lee, Youngwoon Lee, and Pieter Abbeel. Controllability-aware unsupervised skill discovery. arXiv preprint arXiv:2302.05103, 2023 a

    2023 arXiv

  23. [31]

    Metra: Scalable unsupervised rl with metric-aware abstraction

    Seohong Park, Oleh Rybkin, and Sergey Levine. Metra: Scalable unsupervised rl with metric-aware abstraction. arXiv preprint arXiv:2310.08887, 2023 b

    2023 arXiv

  24. [32]

    Curiosity-driven exploration by self-supervised prediction

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp.\ 2778--2787. PMLR, 2017

    2017

  25. [33]

    Quality diversity: A new frontier for evolutionary computation

    Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3: 0 202845, 2016

    2016

  26. [34]

    Markov decision processes: discrete stochastic dynamic programming

    Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014

    2014

  27. [35]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  28. [36]

    Dynamics-aware unsupervised discovery of skills

    Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657, 2019

    1907 arXiv

  29. [37]

    Loss is its own reward: Self-supervision for reinforcement learning

    Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv:1612.07307, 2016

    2016 arXiv

  30. [38]

    Mujoco: A physics engine for model-based control

    Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.\ 5026--5033. IEEE, 2012

    2012

  31. [39]

    Optimal transport: old and new, volume 338

    C \'e dric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009

    2009

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  33. [41]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  34. [42]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  35. [43]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.