Pith. sign in

REVIEW 4 major objections 4 minor 40 references

ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ELEMENT maximizes state entropy at two time scales—within each episode and across the agent's whole history—and reports that this dual-scale intrinsic reward outperforms four state-of-the-art exploration methods in episodic entropy…

desk verdict A plausible multiscale exploration method whose episodic reward is theoretically under-supported and whose lifelong reward is essentially APT/RE3; worth a careful revision, not a desk reject. read the letter →

arxiv 2412.03800 v1 pith:E4AEWERV submitted 2024-12-05 cs.LG cs.AI

Hongming Li , Shujian Yu , Bin Liu , Jose C. Principe This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearningintrinsicmotivationmaximumentropyexplorationepisodiclifelongk-nearestneighborsestimationtask-agnosticpre-trainingoffline
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ELEMENT is an intrinsically motivated reinforcement learning framework that explores environments with no extrinsic reward by maximizing state entropy at two time scales at once: the entropy of states visited within a single episode, and the entropy of all states visited across the agent's lifetime. The paper's central claim is that these two objectives are complementary and that combining them outperforms four state-of-the-art intrinsic rewards—RE3, RISE, NGU, and RND—on episodic state entropy, lifelong state coverage, offline reinforcement learning data collection, and unsupervised pre-training for downstream tasks. It introduces two mechanism-level contributions: a non-parametric 'average episodic state entropy' reward that turns the trajectory-level episodic entropy objective into a per-state Markovian signal, and a k-nearest-neighbors graph that makes lifelong entropy estimation practical as the number of visited states grows. If the claims hold, ELEMENT offers a reusable intrinsic reward module for reward-free exploration and for gathering better datasets for offline reinforcement learning.

What carries the argument

The two load-bearing mechanisms are (1) the average episodic state entropy reward, $r_{\mathrm{ep}}(s)=\mathbb{E}_{\tau \ni s}[H_{s\in\tau}(s)]$, a Markovian per-state bonus that approximates the non-Markovian trajectory reward $H_{s\in\tau}(s)$; Proposition 1 identifies it as the optimal solution to an upper bound of a trajectory-reward decomposition loss, and Proposition 2 justifies the choice of estimator by showing that KDE, kNN, and matrix-based Rényi entropy are all proportional to sums of kNN distances. (2) A k-nearest-neighbors graph with fast approximate search and online updates provides the lifelong reward $r_l(s)=\log(\|s-\mathrm{GNNS}(G,s)\|_2+1)$ and reduces the lifelong entropy computation from $O(N^2)$ to roughly $O(Nk)$ with fixed search hyperparameters, making lifelong exploration scalable to millions of states. The reward used in training is the sum $r_i(s)=r_{\mathrm{ep}}(s)+\beta r_l(s)$, with the graph-update hyperparameters $U$ and $T_u$ controlling how often the lifelong memory refreshes.

What would settle it

Take ELEMENT's episodic reward and, on Ant-v4 or Walker2D-v4, compare it against an agent trained with the true trajectory-level episodic entropy delivered only at episode end and redistributed by a learned credit-assignment method; if ELEMENT does not match or beat that oracle, the additive decomposition is the failure point. A simpler diagnostic: record episodes during training and compute the per-trajectory error $|H_{s\in\tau}(s)-\sum_t r_{\mathrm{ep}}(s_t)|$; if the error grows as episodes get longer or terminate stochastically, the proxy reward is not actually maximizing episodic state entropy.

Watch

Extended reading notes

Core claim

The paper claims that maximizing Shannon state entropy at two complementary scales—episodic and lifelong—yields better reward-free exploration than either scale alone or than predictive-uncertainty novelty. The episodic reward is defined as $r_{\mathrm{ep}}(s)=\mathbb{E}_{\tau \ni s}[H_{s\in\tau}(s)]$, the average entropy of the episodes that contain $s$, and Proposition 1 shows this function is the optimal solution to an upper-bounded reward-redistribution loss rather than to the exact decomposition of episodic entropy into per-state rewards. The lifelong reward is $r_l(s)=\log(\|s-\mathrm{GNNS}(G,s)\|_2+1)$, where GNNS returns $k$ nearest neighbors from a fast approximate kNN graph; Proposition 2 shows that kernel density estimation, kNN, and matrix-based Rényi entropy estimators are each proportional to sums of kNN distances, which justifies using the graph to estimate lifelong entropy. In experiments with SAC and A3C backbones, ELEMENT is reported to beat RE3, RISE, NGU, and RND across Hopper, Walker2D, Ant, Humanoid, and Mario in episodic entropy, unique visited states, offline CQL scores, and downstream task returns.

Load-bearing premise

All results depend on the assumption that an episode's entropy score can be approximated by adding up separate per-state rewards, even though the paper's proof establishes optimality only for an upper-bound loss and assumes constant episode length, while MuJoCo episodes can terminate early on health violations.

Editorial extensions

If this is right

  • If ELEMENT is right, an agent needs no task reward to build a good exploration policy: the dual-scale entropy signal alone produces diverse, reusable behaviors in continuous-control and platform environments.
  • The same reward can be used at pre-training time, so downstream tasks start from policies that already cover multiple motion skills, giving faster and sometimes zero-shot transfer.
  • Datasets collected by ELEMENT exploration policies improve offline reinforcement learning: normalized CQL scores in the paper increase on four of five environments compared with baseline exploration data.
  • The kNN graph makes lifelong entropy maximization computationally feasible, so the method can escape the first-in-first-out queues or downsampled buffers used by earlier kNN-based intrinsic rewards.
  • Because Proposition 2 links KDE, kNN, and Rényi estimators, ELEMENT can in principle be instantiated with any of these estimators depending on the environment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: the episodic reward is a credit-assignment scheme for non-Markovian exploration feedback, so the same average-entropy construction could be applied to other trajectory-level objectives such as coverage or skill-diversity scores, not just entropy.
  • A likely bottleneck not tested here is ELEMENT's fixed encoder; if the encoder were updated online, representation drift could inflate novelty and break the entropy estimates, so a natural test is to combine ELEMENT with an adaptively learned representation and compare state-coverage curves.
  • The Proposition 1 upper bound depends on within-episode reward variance, which suggests a testable correction: weighting or clipping $r_{\mathrm{ep}}$ by its per-episode variance should tighten the bound and may improve performance on environments with stochastic termination.
  • The maze reward visualization shows that states near the start keep high episodic reward because they lead to high-entropy episodes, suggesting ELEMENT implicitly discovers bottleneck states; one could test whether the episodic reward correlates with state visitation betweenness in graph-theoretic terms.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ELEMENT, an intrinsic-reward method for reward-free reinforcement learning that combines an episodic reward, defined as the average state entropy of episodes containing a given state (Eq. 11), with a lifelong reward based on kNN distances in a fast approximate kNN graph (Eq. 15). The authors claim that the episodic reward is the optimal solution to an upper bound of a trajectory-reward decomposition loss (Proposition 1) and that common entropy estimators are proportional to kNN-distance sums (Proposition 2). They evaluate ELEMENT against RND, NGU, RE3, and RISE in MuJoCo and Mario environments on episodic state entropy, lifelong state coverage, offline RL data collection, and task-agnostic pre-training.

Significance. If the claims hold, ELEMENT would be a practically useful, multiscale exploration module: the episodic/lifelong combination addresses the known vanishing-reward problem of purely lifelong entropy rewards, and the kNN graph reduces the per-step cost of lifelong entropy estimation from O(N^2) to a much smaller search-based cost. The paper contains a concrete algorithmic pipeline (Algorithms 1-3), qualitative behavioral visualizations, a broad set of environments, and an explicit discussion of limitations (fixed encoder, episode-length trade-off) in Section V. However, the theoretical support is currently not sound: Proposition 1's stated solution does not follow from the minimization problem, Proposition 2 contains sign and approximation errors, and the empirical validation of the key decomposition assumption is performed on task-reward trajectories rather than on ELEMENT's exploration distribution. These issues are load-bearing for the central claim that ELEMENT maximizes episodic and lifelong state entropy, so despite the promising empirical results, the manuscript requires substantial revision.

major comments (4)
  1. [Section III-B, Eqs. (12)-(13) and Appendix A] The stated optimal solution in Proposition 1 is not the minimizer of the stated loss. For a per-state function rep(s), minimizing E_{τ,t}[(H(τ)/Tτ - rep(s_t))^2] pointwise yields rep(s) = E[H(τ)/Tτ | s∈τ] (with the denominator P(s∈τ)), not the unnormalized expression E[H(τ)/Tτ · I(s∈τ)] in Eq. (13). Because the missing normalization depends on s through P(s∈τ), Eq. (13) is not a constant rescale of the true conditional expectation. Eq. (11) drops the 1/Tτ factor, so its relationship to the derived solution is also only up to an unstated scale. Moreover, the proof assumes constant Tτ, but Section IV-A and Section V(b) acknowledge that MuJoCo episodes terminate stochastically; the appendix's variable-length formula Σ Tτ H I / Σ Tτ^2 I is different from Eq. (11) and is not used. These points need to be corrected or the theoretical claim in the abstract, that the episodic reward provides an optimal solution for an upper bound, must be substantially downgraded.
  2. [Section III-C, Proposition 2 and Appendix A.1.b] The proof of Proposition 2 contains a sign error that makes the expression undefined: Eq. (14c) and the derivation below Eq. (23) state H2(s) ∝ log Σ_i [ - Σ_{j∈kNN} κ(s_i,s_j) ], where the argument of the outer logarithm is negative. The derivation also omits the trace normalization in Eq. (2) and incorrectly replaces log Σ_i Σ_j κ(s_i,s_j)^2 with log Σ_i (Σ_j κ(s_i,s_j)). For the kNN estimator, Eq. (14b) equates log Σ_{j∈kNN} ||s_i-s_j||^2 with log ||s_i - s_i^{kNN}||^2, but the Kozachenko-Leonenko estimator in Eq. (4) applies the logarithm to the d-th power of a single distance, not to a sum of squared distances over k neighbors. Consequently, Proposition 2 does not establish that the lifelong reward r_l in Eq. (15) is proportional to any of the stated entropy estimators. The authors should either provide a correct derivation or reframe the kNN graph reward as a heuristic acceleration of kNN-distance-based intrinsic rewards.
  3. [Section IV-B, Fig. 5] The empirical validation of the additive decomposition in Eq. (9) is not performed on the distribution to which the claim applies. The text states that the authors 'train a SAC agent in the Ant environment using default task-driven rewards and record all historical states,' yielding about 2,000 trajectories, and Fig. 5 reports the resulting regression loss. This validates the decomposition for task-reward SAC trajectories, not for ELEMENT's exploration policies, which are trained without extrinsic reward. The central claim that ELEMENT maximizes episodic state entropy depends on the accuracy of the additive proxy under ELEMENT's own exploration distribution. The authors should re-run the decomposition validation on rollout data collected from ELEMENT checkpoints and report the approximation error on that distribution, or justify why the task-reward distribution suffices.
  4. [Abstract and Section IV, Table I and Figs. 6-8] The claim that ELEMENT 'significantly outperforms state-of-the-art intrinsic rewards' is not supported by the reported statistics. In Table I, RISE has a higher mean than ELEMENT on Hopper (0.534 vs. 0.516), and the RE3 entry on Ant carries a very large error bar (±0.347); no significance tests or seed-level tables are provided for any of the comparisons. Figs. 6-8 report means with variance but do not include pairwise statistical comparisons. Since this claim appears in the abstract and is a central selling point, the authors should add appropriate statistical tests, report per-seed results, and either substantiate or temper the 'significantly outperforms' phrasing.
minor comments (4)
  1. [Throughout and Section II-A] There are numerous typos and inconsistencies, including 'dimentionality' in Eq. (4), 'ElEMENT' in Section IV-B, 'unsuperivisedly' in Section VI, 'fatience' in Table II, and 'wihh' in the author affiliation; the paper also alternates between 'Mujoco' and 'MuJoCo'.
  2. [Section IV and Appendix: experimental settings] The hyperparameter tables (Tables III-VI) cover only the MuJoCo environments; the Mario experiments do not report the ELEMENT-specific values (β, entropy estimator, k, graph search steps, graph update interval) that are needed to reproduce the Mario results in Figs. 6-8 and Table I.
  3. [Algorithm 3, line 14] The pseudocode writes 'Estimate rep = Hs∈Dτ(s)', but Eq. (11) requires assigning a per-state value based on the entropy of episodes containing that state; the algorithm should clarify whether the same episodic entropy value is assigned to every state in the trajectory or whether some count-based normalization is applied.
  4. [Section IV-B and Fig. 6] The episodic evaluation metric is the matrix-based entropy functional with α=1.001, while the episodic reward in Ant and Humanoid uses the same family of estimators (matrix-based Rényi with α=3); this objective-metric alignment is not circular, but it limits the independence of the episodic entropy evaluation and should be acknowledged in the experimental section.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ELEMENT's episodic reward is a least-squares reward-redistribution solution, the lifelong reward is the credited kNN entropy reward with an approximate graph, and the evaluation metrics are external or objective-aligned but not fitted.

full rationale

The derivation chain is self-contained. The episodic reward rep(s) = Eτ[H(τ)·I(s∈τ)] is proposed as a non-parametric solution to the trajectory-reward-redistribution loss (Eq. 10), and Proposition 1 solves a Monte-Carlo upper bound of that loss, following prior independent work [31, 32] explicitly acknowledged in the appendix. Eq. 11 differs from the proven optimum only by a constant scale Tτ, which is removed by the min-max normalization in Algorithm 3; this is a mathematical approximation, not a circular reduction. The episodic evaluation in Fig. 6 uses the same episodic state entropy objective, but that is an objective-metric alignment by design, not a fitted input renamed as a prediction: the reward is fixed before training and the metric is computed from held-out rollouts. The lifelong reward rl(s) = log(||s − GNNS(G,s)|| + 1) is the standard kNN entropy reward from APT/RE3/RISE, explicitly credited, with the novel contribution being the approximate kNN graph and online update; this is engineering, not circularity. The only self-citations are to the matrix-based Rényi entropy estimator [18] and book [22] by co-author Príncipe, both externally published and used as computational tools, not as load-bearing justification for ELEMENT's claims. Comparisons against RND, NGU, RE3, and RISE, along with state-coverage histograms, CQL offline scores, and downstream task returns, are external benchmarks outside the paper's fitted values. The additive decomposition assumption in Eq. (9), the constant-length Tτ assumption in Proposition 1, and the validation of Fig. 5 on task-reward trajectories rather than exploration trajectories are correctness and robustness concerns, but they are not circular: the assumption could in principle fail and is empirically tested, and the approximation error is explicitly discussed in Section V.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

The method leans on a stack of unverified assumptions: the validity of non-parametric entropy estimates in the chosen representation spaces, the additivity of episodic entropy as a sum of per-state rewards, the constancy of episode length in the proof, and the accuracy of approximate kNN graph search. None are established for the tested environments, and one proof step is invalid. The number of hand-picked hyperparameters (β, k, estimator, Rényi α, kernel width, graph update schedule) is not large by deep RL standards, but the per-environment estimator selection is a genuine extra degree of freedom.

free parameters (7)
  • beta (β) = 0.5
    Weight for lifelong reward in Eq. (8); set by hand after normalization, with ablation in Fig. 9 but no systematic tuning.
  • k_lifelong = 3
    Number of neighbors k in the lifelong kNN graph reward (Eq. 15, Tables III-VI).
  • entropy estimator per environment = KDE (Hopper), kNN (Walker2D), Matrix-based Rényi α=3 (Ant, Humanoid)
    The paper states it selects the estimator based on experimental performance (Section III-B), which is a post hoc model choice on the evaluation environments.
  • alpha (Rényi) = 3
    Rényi entropy order used for Ant and Humanoid (Tables V, VI).
  • kernel width σ = 1
    Gaussian kernel width for KDE and matrix-based Rényi estimators (Tables III, V, VI).
  • graph update interval U and steps Tu = U=5e5, Tu=5e4
    Control how often the kNN graph refreshes; chosen for all environments (Tables III-VI).
  • graph search steps R1 and restarts R2 = e.g., R1=20, R2=20 (Hopper)
    Approximate nearest neighbor search parameters, set per environment.
assumptions (5)
  • domain assumption Non-parametric entropy estimators (kNN, KDE, matrix-based Rényi) provide valid estimates of the state entropy of the policy's state distribution.
    The entire reward construction depends on these estimates (Eqs. 2-4); consistency conditions are cited from literature but not verified in the tested environments.
  • domain assumption A fixed pre-trained or random encoder provides a representation space in which kNN distances and entropy estimates are meaningful for exploration.
    Section III-A fixes the encoder to avoid representation drift; the paper acknowledges this as a limitation in Section V.
  • ad hoc to paper Episode state entropy can be decomposed as a sum of per-state Markovian rewards: H_{s∈τ}(s) ≈ Σ_t rep(s_t) (Eq. 9).
    This additivity assumption underlies the episodic reward; the paper only validates it with a single fitting experiment (Fig. 5).
  • ad hoc to paper All episodes have the same length Tτ (constant episode length) for Proposition 1.
    The proof of Proposition 1 fixes Tτ as a constant, but MuJoCo episodes terminate stochastically on health violations, so the assumption is violated in the experiments.
  • ad hoc to paper The kNN graph's approximate nearest neighbor search returns sufficiently accurate neighbors for the lifelong entropy reward.
    The graph methods are cited from prior work; the approximation error is not analyzed, and the tolerance argument in Proposition 2 requires a large k-NN distance that may not hold in dense regions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy." pith.science (2026). https://pith.science/paper/E4AEWERV

@misc{pith2026241203800,
  author       = {Pith},
  title        = {Pith review of: ELEMENT: Episodic and Lifelong Exploration via Maximum Entropy},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E4AEWERV}},
  note         = {Machine review of arXiv:2412.03800}
}
abstract

This paper proposes \emph{Episodic and Lifelong Exploration via Maximum ENTropy} (ELEMENT), a novel, multiscale, intrinsically motivated reinforcement learning (RL) framework that is able to explore environments without using any extrinsic reward and transfer effectively the learned skills to downstream tasks. We advance the state of the art in three ways. First, we propose a multiscale entropy optimization to take care of the fact that previous maximum state entropy, for lifelong exploration with millions of state observations, suffers from vanishing rewards and becomes very expensive computationally across iterations. Therefore, we add an episodic maximum entropy over each episode to speedup the search further. Second, we propose a novel intrinsic reward for episodic entropy maximization named \emph{average episodic state entropy} which provides the optimal solution for a theoretical upper bound of the episodic state entropy objective. Third, to speed the lifelong entropy maximization, we propose a $k$ nearest neighbors ($k$NN) graph to organize the estimation of the entropy and updating processes that reduces the computation substantially. Our ELEMENT significantly outperforms state-of-the-art intrinsic rewards in both episodic and lifelong setups. Moreover, it can be exploited in task-agnostic pre-training, collecting data for offline reinforcement learning, etc.

Figures

Figures reproduced from arXiv: 2412.03800 by the authors.

Figure 1
Figure 1. Overview of our approach. The life long intrinsic reward of a state s (the green node) is computed as the distance to its k nearest neighbors, measured in representations space obtained from a fixed neural encoder. The episodic intrinsic reward is calculated as the average state entropy of these episodes including the state. The lifelong reward is then combined with episodic reward. A separate DRL is introduced for … view at source ↗
Figure 2
Figure 2. Episodic entropy maximization encourages movement in a single direction to capture more distinct states per episode. Deviating from this direction will result in smaller distances to the previ￾ously visited states within the same episode. Lifelong motivation discourages revisiting episodes, hindering a continual exploration towards one direction. ELEMENTs shares merits of both, promoting comprehensive exploration wi… view at source ↗
Figure 3
Figure 3. , it uses its neighbors as starting points to search existing nodes for which the new point is now a nearest neighbor, up to a fixed depth. Details can be found in Algorithm 1 and 2. query state 𝐬 starting point new node 𝐬 𝑘NN Graph Search 𝑘NN Graph Update [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Trajectories obtained by executing policies at different checkpoints (depicted by different colors) on Walker2D, Ant and Mario. At the first checkpoint, the agent quickly converges to a policy which produces similar episodes with many different states. When the learnin…
Figure 5
Figure 5. Figure 5: (a) The learning curve given by parametric modeling of the episodic reward rep(s) using a DNN. (b) Predicted episodic state entropy using rep(s) in ELEMENT. 0.0 0.2 0.4 0.6 0.8 1.0 Training Steps 1e6 0 2 4 6 8 Episodic State Entropy Hopper-v4 0.0 0.2 0.4 0.6 0.8 1.0 Tr…
Figure 6
Figure 6. Figure 6: Performance of episodic state entropy maximization. The y-axis represents Shannon state entropy within an episode, which is estimated using matrix-based entropy functional [18] with α = 1.001. experiments, we employ the Conservative Q-Learning (CQL) 0 2 4 6 8 Training …
Figure 9
Figure 9. Figure 9: Trajectories obtained during the training process using different β. Colored by logarithmic probability. of exploration episodes (radius) decreases with the increasing impact of lifelong exploration. This observation validates that lifelong exploration constrains “deep…
Figure 8
Figure 8. Figure 8: Performance of agents in downstream tasks after unsupervised pre-training the agent. F. Ablation Study for β In this section, we extend the experiments presented in [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 10
Figure 10. Figure 10: The left panel displays the rewards assigned to each state during the training process by episodic and lifelong rewards, while the right panel depicts the corresponding trajectories generated based on agents optimized using these rewards. states, there is a gradual in…
Figure 11
Figure 11. Figure 11: Visual interfaces of robotic environments [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 27 canonical work pages

  1. [1]

    Human-level control through deep reinforcement learning,

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Ve- ness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al. , “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015

    work page 2015

  2. [2]

    Mastering the game of go with deep neural networks and tree search,

    D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V . Panneershelvam, M. Lanctot et al. , “Mastering the game of go with deep neural networks and tree search,” nature, vol. 529, no. 7587, pp. 484–489, 2016

    2016

  3. [3]

    A survey of exploration methods in rein- forcement learning,

    S. Amin, M. Gomrokchi, H. Satija, H. van Hoof, and D. Precup, “A survey of exploration methods in rein- forcement learning,” arXiv preprint arXiv:2109.00157 , 2021

    arXiv 2021

  4. [4]

    Behavior from the void: Unsu- pervised active pre-training,

    H. Liu and P. Abbeel, “Behavior from the void: Unsu- pervised active pre-training,” Advances in Neural Infor- mation Processing Systems , vol. 34, pp. 18 459–18 473, 2021

    work page 2021

  5. [5]

    Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate,

    M. Mutti, L. Pratissoli, and M. Restelli, “Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate,” in Proceedings of the AAAI Confer- ence on Artificial Intelligence , vol. 35, no. 10, 2021, pp. 9028–9036

    work page 2021

  6. [6]

    State entropy maximization with random encoders for efficient exploration,

    Y . Seo, L. Chen, J. Shin, H. Lee, P. Abbeel, and K. Lee, “State entropy maximization with random encoders for efficient exploration,” in International Conference on Machine Learning. PMLR, 2021, pp. 9443–9454

    work page 2021

  7. [7]

    Rényi state entropy maximization for exploration acceleration in reinforce- ment learning,

    M. Yuan, M.-O. Pun, and D. Wang, “Rényi state entropy maximization for exploration acceleration in reinforce- ment learning,” IEEE Transactions on Artificial Intelli- gence, 2023

    work page 2023

  8. [8]

    Provably efficient maximum entropy exploration,

    E. Hazan, S. Kakade, K. Singh, and A. Van Soest, “Provably efficient maximum entropy exploration,” in International Conference on Machine Learning. PMLR, 2019, pp. 2681–2691

    2019

Show all 40 references
  1. [9]

    Exploration by maximizing rényi entropy for reward-free rl framework,

    C. Zhang, Y . Cai, L. Huang, and J. Li, “Exploration by maximizing rényi entropy for reward-free rl framework,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, 2021, pp. 10 859–10 867

    2021

  2. [10]

    k-means maximum en- tropy exploration,

    A. Nedergaard and M. Cook, “k-means maximum en- tropy exploration,” arXiv preprint arXiv:2205.15623 , 2022

    2022 arXiv

  3. [11]

    Rein- forcement learning with prototypical representations,

    D. Yarats, R. Fergus, A. Lazaric, and L. Pinto, “Rein- forcement learning with prototypical representations,” in International Conference on Machine Learning. PMLR, 2021, pp. 11 920–11 931

    2021

  4. [12]

    Fast rates for maximum entropy explo- ration,

    D. Tiapkin, D. Belomestny, D. Calandriello, E. Moulines, R. Munos, A. Naumov, P. Perrault, Y . Tang, M. Valko, and P. Menard, “Fast rates for maximum entropy explo- ration,” arXiv preprint arXiv:2303.08059 , 2023

    2023 arXiv

  5. [13]

    Nearest neighbor estimates of entropy,

    H. Singh, N. Misra, V . Hnizdo, A. Fedorowicz, and E. Demchuk, “Nearest neighbor estimates of entropy,” American journal of mathematical and management sci- ences, vol. 23, no. 3-4, pp. 301–321, 2003

    2003

  6. [14]

    Unifying count-based ex- 14 ploration and intrinsic motivation,

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, “Unifying count-based ex- 14 ploration and intrinsic motivation,” Advances in neural information processing systems , vol. 29, 2016

    2016

  7. [15]

    Deep curiosity search: Intra- life exploration can improve performance on challenging deep reinforcement learning problems,

    C. Stanton and J. Clune, “Deep curiosity search: Intra- life exploration can improve performance on challenging deep reinforcement learning problems,” arXiv preprint arXiv:1806.00553, 2018

    2018 arXiv

  8. [16]

    Go-explore: a new approach for hard- exploration problems,

    A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, “Go-explore: a new approach for hard- exploration problems,” arXiv preprint arXiv:1901.10995, 2019

    1901 arXiv

  9. [17]

    Never give up: Learn- ing directed exploration strategies,

    A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky, A. Pritzel, A. Bolt et al. , “Never give up: Learn- ing directed exploration strategies,” arXiv preprint arXiv:2002.06038, 2020

    2002 arXiv

  10. [18]

    Measures of entropy from data using infinitely divisible kernels,

    L. G. S. Giraldo, M. Rao, and J. C. Principe, “Measures of entropy from data using infinitely divisible kernels,” IEEE Transactions on Information Theory, vol. 61, no. 1, pp. 535–548, 2014

    2014

  11. [19]

    Fast approximate nearest-neighbor search with k-nearest neighbor graph,

    K. Hajebi, Y . Abbasi-Yadkori, H. Shahbazi, and H. Zhang, “Fast approximate nearest-neighbor search with k-nearest neighbor graph,” in Twenty-Second Inter- national Joint Conference on Artificial Intelligence, 2011

    2011

  12. [20]

    Fast online k-nn graph building,

    T. Debatty, P. Michiardi, and W. Mees, “Fast online k-nn graph building,” arXiv preprint arXiv:1602.06819, 2016

    2016 arXiv

  13. [21]

    On measures of entropy and information,

    A. Rényi, “On measures of entropy and information,” in Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Con- tributions to the Theory of Statistics , vol. 4. University of California Press, 1961, pp. 547–562

    1961

  14. [22]

    J. C. Principe, Information theoretic learning: Renyi’s entropy and kernel perspectives . Springer Science & Business Media, 2010

    2010

  15. [23]

    Remarks on some nonparametric estimates of a density function,

    R. A. Davis, K.-S. Lii, and D. N. Politis, “Remarks on some nonparametric estimates of a density function,” Selected Works of Murray Rosenblatt, pp. 95–100, 2011

    2011

  16. [24]

    Efficient exploration via state marginal matching,

    L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov, “Efficient exploration via state marginal matching,” arXiv preprint arXiv:1906.05274 , 2019

    1906 arXiv

  17. [25]

    Trust region policy optimization,

    J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in Interna- tional conference on machine learning . PMLR, 2015, pp. 1889–1897

    2015

  18. [26]

    Curiosity-driven exploration by self-supervised predic- tion,

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised predic- tion,” in International conference on machine learning . PMLR, 2017, pp. 2778–2787

    2017

  19. [27]

    Exploration by random network distillation,

    Y . Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” in Inter- national Conference on Learning Representations , 2019

    2019

  20. [28]

    Learning and information theory,

    E. Pfaffelhuber, “Learning and information theory,” In- ternational journal of neuroscience , vol. 3, no. 2, pp. 83–88, 1972

    1972

  21. [29]

    D. B. Lenat, AM: an artificial intelligence approach to discovery in mathematics as heuristic search. Stanford University, 1976

    1976

  22. [30]

    Unsupervised state representation learning in atari,

    A. Anand, E. Racah, S. Ozair, Y . Bengio, M.-A. Côté, and R. D. Hjelm, “Unsupervised state representation learning in atari,” Advances in neural information pro- cessing systems, vol. 32, 2019

    2019

  23. [31]

    Learning guidance rewards with trajectory-space smoothing,

    T. Gangwani, Y . Zhou, and J. Peng, “Learning guidance rewards with trajectory-space smoothing,” Advances in Neural Information Processing Systems, vol. 33, pp. 822– 832, 2020

    2020

  24. [32]

    Learning long- term reward redistribution via randomized return decom- position,

    Z. Ren, R. Guo, Y . Zhou, and J. Peng, “Learning long- term reward redistribution via randomized return decom- position,” arXiv preprint arXiv:2111.13485 , 2021

    2021 arXiv

  25. [33]

    Reinforcement learning with trajectory feedback,

    Y . Efroni, N. Merlis, and S. Mannor, “Reinforcement learning with trajectory feedback,” in Proceedings of the AAAI conference on artificial intelligence , vol. 35, no. 8, 2021, pp. 7288–7295

    2021

  26. [34]

    Soft actor-critic algorithms and applications,

    T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Kumar, H. Zhu, A. Gupta, P. Abbeel et al. , “Soft actor-critic algorithms and applications,” arXiv preprint arXiv:1812.05905, 2018

    2018 arXiv

  27. [35]

    Asynchronous methods for deep reinforcement learning,

    V . Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in Interna- tional conference on machine learning . PMLR, 2016, pp. 1928–1937

    2016

  28. [36]

    Openai gym,

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016

    2016

  29. [37]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

    S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643 , 2020

    2005 arXiv

  30. [38]

    Reward-free exploration for reinforcement learning,

    C. Jin, A. Krishnamurthy, M. Simchowitz, and T. Yu, “Reward-free exploration for reinforcement learning,” in International Conference on Machine Learning. PMLR, 2020, pp. 4870–4879

    2020

  31. [39]

    Con- servative q-learning for offline reinforcement learning,

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Con- servative q-learning for offline reinforcement learning,” Advances in Neural Information Processing Systems , vol. 33, pp. 1179–1191, 2020

    2020

  32. [40]

    Behavioral cloning from observation,

    F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” arXiv preprint arXiv:1805.01954 , 2018

    2018 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.