Pith. sign in

REVIEW 3 major objections 5 minor 65 references

Unsupervised Skill Discovery through Skill Regions Differentiation

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Unsupervised RL skills can be learned by pushing each skill into states that other skills avoid, with a latent-space bonus driving exploration; the paper reports state-of-the-art adaptation performance on the state- and pixel-based URLB…

desk verdict Competent empirical skill-discovery paper with a real proof error in Theorem 2; the mechanism and results are plausible but the count-based exploration claim is unsupported as written. read the letter →

arxiv 2506.14420 v1 pith:AHO6XK3E submitted 2025-06-17 cs.LG

Ting Xiao , Jiakun Zheng , Rushuai Yang , Kang Xu , Qiaosheng Zhang , Peng Liu , Chenjia Bai This is my paper · ORCID
classification cs.LG
keywords unsupervisedreinforcementlearningskilldiscoverystatedensitydeviationconditionalvariationalautoencodersoftmodularizationcount-basedexplorationintrinsicrewardURLBbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes SD3, an unsupervised reinforcement learning method whose central idea is that skills should be learned by maximizing the deviation of each skill's state density from the state densities of all other skills: a skill is rewarded for occupying states that other skills rarely visit. Because entropy-based exploration and mutual-information skill objectives both scale poorly to large and image-based state spaces, SD3 estimates the densities with a conditional variational autoencoder that uses soft modularization to keep estimates stable when skills occupy very different regions, and it derives a second intrinsic reward from the autoencoder's latent space to push each skill to explore its own region. The paper proves that the density-deviation objective $I_{SD3}$ is sandwiched between the mutual information $I(S;Z)$ and $I(S;Z)+\log\lambda$ (Theorem 1), so the classic MI objective is a boundary case, and that the latent-space exploration reward approximates count-based exploration in tabular MDPs (Theorem 2). If the paper is right, the same two-term reward should learn distinguishable, well-exploring skills that adapt to downstream tasks faster than prior MI-based, entropy-based, and coverage-based methods, and it reports top aggregate scores on the state-based and pixel-based URLB benchmarks along with a robustness advantage under observation noise.

What carries the argument

The object that carries the argument is the pair of rewards derived from one conditional variational autoencoder. The autoencoder's ELBO, $L^{\mathrm{elbo}}_z(s) = \mathbb{E}_{Q(h|s,z)}[\log P(s|h,z)] - D_{\mathrm{KL}}[Q(h|s,z)\|P(h|z)]$, is used as the estimate of each skill's log state density $\log d^\pi_z(s)$; soft modularization makes the encoder and decoder a skill-routed weighted combination of shared modules so that density estimates for very different skills do not interfere. The same network yields the intra-skill exploration reward $r^{\mathrm{exp}}_z(s) = D_{\mathrm{KL}}[Q_\phi(h|s,z)\|r(h)]$, which is small for states a skill has visited often and large for novel ones. The two theorems are the theoretical connectors: Theorem 1 proves the density-deviation objective sits strictly above the mutual information $I(S;Z)$ for $\lambda\geq1$ (recovering it exactly at $\lambda=1$), and Theorem 2 proves the exploration reward is an upper bound on a UCB-style information gain that, under a linear decoder and tabular state-skill pairs, becomes $|S|/(2(N(s,z)+\kappa))$ — the count-based exploration bonus.

What would settle it

Run SD3 in a small tabular MDP where true visit counts $N(s,z)$ and true state densities are computable exactly, and check two predictions: the exploration reward should decrease monotonically with $N(s,z)$ and track $|S|/(2(N(s,z)+\kappa))$, and the density-deviation reward should be higher for states that other skills genuinely under-visit. If either correlation fails, the link between the practical network and the theorems is broken. In an image-based environment, the counterpart test is to measure the overlap between skill state distributions with an independently trained, reward-free encoder and ask whether skills that score high on $I_{SD3}$ actually visit disjoint state regions.

Watch

Extended reading notes

Core claim

The paper claims that maximizing the density-deviation objective $I_{SD3} = \mathbb{E}_{z\sim p(z), s\sim d^\pi_z(s)}[\log \frac{\lambda d^\pi_z(s)}{\lambda d^\pi_z(s)p(z) + \sum_{z'\neq z} d^\pi_{z'}(s)p(z')}]$ — with $\lambda \geq 1$ a weight that weakens the gradient pushing other skills' densities down — learns exactly the kind of skills that empowerment-based methods aim for but often miss: skills whose state-occupancy regions are clearly separated from one another while still covering enough of the state space to be useful. Theorem 1 states $I(S;Z) \leq I_{SD3} \leq I(S;Z) + \log\lambda$, with equality to $I(S;Z)$ at $\lambda=1$, so the new objective is an upper bound on the classic mutual-information skill objective rather than a competing heuristic. Theorem 2 states that the exploration reward $r^{\mathrm{exp}}_z(s) = D_{\mathrm{KL}}[Q_\phi(h|s,z) \| r(h)]$ is approximately $|S|/(2(N(s,z)+\kappa))$ in tabular MDPs, identifying the KL-to-prior term with well-studied count-based exploration. The paper's supporting evidence is empirical: SD3 achieves the highest IQM (77.37%), mean (76.19%), and optimality-gap (23.91%) scores among thirteen methods on the state-based URLB, improves on the strongest baselines on pixel-based Walker and Quadruped tasks using a Dreamer backbone, and retains more performance than CIC when observation noise is added during pre-training.

Load-bearing premise

Everything depends on the autoencoder's reconstruction-based density estimate being a faithful stand-in for how often each skill truly visits each state in high-dimensional and image-based environments, and on the KL-to-prior exploration reward still rewarding rare states like visit counts do once the decoder is a deep network rather than the linear model that the proof of Theorem 2 assumes.

Editorial extensions

If this is right

  • Maximizing the SD3 objective with $\lambda \geq 1$ also increases the mutual information $I(S;Z)$, so the new objective strictly generalizes the MI-based skill discovery family that methods like DIAYN and DADS instantiate.
  • The latent-space exploration reward inherits count-based exploration's guarantees in tabular MDPs, giving practitioners a principled intra-skill exploration term that should prevent skill collapse as the diversity reward is maximized.
  • Because density estimation happens in the latent space of the autoencoder rather than over raw observations, the method is claimed to scale to pixel-based environments, and the reported pixel-based URLB results support that claim.
  • The robustness experiments show that the count-like bonus retains 86.33% of clean-task performance under observation noise, versus 71.68% for the entropy-based CIC, indicating that density-based and count-based exploration rewards are more stable pre-training objectives than entropy estimates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The $\log\lambda$ gap in Theorem 1 is not quantified in practice; an extension would be to characterize how $\lambda$ trades off inter-skill separation against per-skill state coverage, so practitioners could set it to a target coverage level rather than tune it per environment.
  • Because the same network supplies both the density estimates and the exploration bonus, the two rewards are coupled: a density error inflates the exploration bonus and vice versa; freezing or periodically refreshing the autoencoder would be a direct test of whether decoupling the two stabilizes skill learning.
  • The paper's own conclusion notes that soft modularization limits SD3 to discrete skill spaces and that Theorem 2's analysis assumes tabular MDPs; extending density deviation to continuous skill spaces would require Monte Carlo estimates of the density sum in Eq. (1), whose variance is not analyzed.
  • The robustness result suggests that swapping other methods' entropy-based exploration terms for this KL-based count-like bonus, for instance in APS or BeCL, might improve their noise tolerance; the paper does not test that transfer directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes SD3, an unsupervised skill-discovery method that rewards each skill for visiting states whose estimated state density deviates from the densities of other skills, and adds a latent-space KL-divergence reward for intra-skill exploration. A skill-conditional CVAE with soft modularization estimates the state densities and provides the latent representation. The paper claims two theoretical results: Theorem 1 bounds the SD3 objective by the mutual information I(S;Z) plus log λ, and Theorem 2 claims the intrinsic reward is approximately equivalent to count-based exploration in tabular MDPs. Experiments cover maze visualization, state-based URLB, pixel-based URLB, a robustness study, and ablations; SD3 achieves the best or near-best aggregate scores on the benchmarks.

Significance. If the claims hold, SD3 would be a competitive and scalable skill-discovery algorithm that unifies inter-skill diversity and exploration. The empirical evaluation is extensive: 12 state-based and 8 pixel-based downstream tasks, multiple seeds, aggregate statistics (IQM/mean/OG), ablations over λ, α, architecture, and skill selection, and a noise-robustness comparison. The paper does not release code or externally verified proofs, and Theorem 2 is not established as stated. With a corrected or appropriately weakened theoretical claim, the empirical contribution is substantial, but the current version overstates the theory and should not be accepted in its present form.

major comments (3)
  1. [Section III-C, Theorem 2 and Appendix C] The proof of Theorem 2 does not establish the stated equivalence. Eq. (18) equates the fixed-input KL reward DKL[Qϕ(h|s,z)||r(h)] to an entropy difference H(Qmargin)-H(Qϕ(h|s,z)); these are different objects, and the equality is not generally valid because Qmargin is an aggregate encoding distribution while r(h) is a fixed prior. Replacing the expectation over the posterior Φ by a point estimate φ does not fix the mismatch, because Eq. (6) uses the fixed learned network. Consequently, Eqs. (19)-(21) bound an information-gain quantity that is not the implemented reward, and Eq. (36) gives only an upper bound on that different quantity. The final approximation '≈' in Eq. (37) and the phrase 'equivalent to count-based exploration' are therefore unsupported; no lower bound or tightness proof is supplied, and the tightness remarks in the appendix are qualitative. This is load-bearing because the abstract, the contributions list, and Section V-D use this theorem to justify the exploration bonus as provably efficient and UCB-like.
  2. [Section III-A, Eq. (3) and Eq. (7)] The entire inter-skill reward rsd3_z(s) in Eq. (7) depends on treating the CVAE ELBO as an estimate of log dπ_z(s). The paper never validates this estimate directly. The soft-modularization ablation in Section V-E.1 shows downstream-task gains but does not measure whether the estimated densities are accurate; the issue is particularly acute for image observations, where no density-estimation diagnostics are provided. Because the density model is trained on the policy's own samples, the 'deviation' signal is partly self-referential, and an inaccurate model could produce the reported behavior through a different mechanism. Please provide a validation of the density estimates (for example, on a tabular or low-dimensional control task where true occupancy can be approximated) or explicitly reframe the reward as 'deviation of the learned density model' rather than 'deviation of true state density.'
  3. [Appendix C, Eqs. (30)-(36)] The proof also suffers from notation and index problems that prevent verification: m is used both as the number of data points and as the index of the last block in the covariance matrix, the block-diagonal form in Eq. (30) assumes an ordering of state-skill pairs that is not stated, and the final c/2 in Eq. (36) is silently identified with |S|/2 even though c was introduced as the output dimension of the linear decoder. These issues compound the gap in the first major comment, and they should be corrected if the theorem is retained in any form.
minor comments (5)
  1. [Section V-C] The text says 'The detailed scores are attached in Table III' for the pixel-based URLB results, but Table III is the robustness experiment; the pixel scores are in Table II. The reference to 'Fig. 6(a)' in the same section also appears to be a typo for Fig. 5.
  2. [Section V-E.2] Figure 8 reports the λ ablation only on Quadruped; please state whether λ-insensitivity was checked on other domains, since the conclusion is phrased generally.
  3. [Section III-A] The statement that the maximum of ISD3 is H(Z) is asserted without the measure-theoretic caveat that densities may vanish or that the maximum is approached only in a limiting sense; a short proof or reference would be helpful.
  4. [Section V-B and V-C] The state-based and pixel-based experiments use different backbones (DDPG vs. Dreamer), and the CVAE architecture and hyperparameters (number of modules, latent dimension, routing network size, reward scaling α per domain) are not fully specified; this is important for reproducibility, especially since no code release is mentioned.
  5. [Section V-D] The robustness claim is phrased as a general advantage of UCB-style bonuses over entropy-based exploration, but the experiment compares only SD3 against CIC; the conclusion should be restricted to this comparison unless additional baselines are added.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 2 reduces the implemented KL exploration reward to an information-gain quantity by the asserted equality in Eq. (18), so the advertised count-based equivalence is built in rather than derived; empirical benchmark results remain independent.

  1. self definitional [Appendix C, Eq. (18); Sec. III-B, Eq. (6); Theorem 2 statement in Sec. III-C]
    "rexp z (s) = EΦDKL[Qϕ(h|s, z)∥ r(h)] = H(Qmargin) − H(Qϕ(h|s, z)), (18) where Qmargin = Q(s, z)|Dm is the margin distribution of the encoding over the posterior of the parameters Φ."

    The implemented reward (Eq. 6) is the per-input KL divergence DKL[Qφ(h|s,z)∥r(h)] evaluated with one learned network. Eq. (18) asserts this equals H(Qmargin)−H(Qφ(h|s,z)), an entropy difference / information gain about the parameter posterior; these are different objects and the equality is not generally true. All subsequent inequalities (Eqs. 19-22 and 29-36) bound that information-gain quantity under a linear-decoder assumption, not the KL reward actually optimized, and the tabular conclusion (Eq. 37) is read off from the one-hot count expression for that quantity. The theorem's advertised equivalence rexp≈|S|/2/(N+κ) is therefore imposed by the redefinition in Eq. (18), not derived from the reward used in the algorithm.

full rationale

SD3's main loop—estimating skill densities with a CVAE trained on the policy's own samples and feeding the estimates back as rewards—is self-referential in the way density-ratio skill-discovery methods generally are, but that is an RL objective, not a fitted quantity being passed off as an independent prediction. Theorem 1's identity ISD3=I(S;Z) at λ=1 is definitional (the ratio in Eq. (1) becomes exactly log(dπ_z(s)/p(s)) when λ=1), but the paper states it transparently as a special case, so it is not a load-bearing circular step by itself. The genuine reduction is in the proof of Theorem 2: the implemented exploration reward in Eq. (6) is a per-input KL divergence to a fixed prior, yet Eq. (18) asserts it equals H(Qmargin)−H(Qφ(h|s,z)), an information gain about the parameter posterior. The inequalities that follow bound that information-gain object under a linear-decoder assumption and then translate it to a count bonus via one-hot features; they never establish the stated bound for the KL reward actually optimized. Thus the advertised 'count-based exploration' property is injected by the redefinition in Eq. (18), not derived from the method's own reward. The benchmark results are external evidence and remain independent, and no load-bearing self-citation was found (refs. [32] and [62] are ordinary technique citations). Hence the circularity is partial, concentrated in the central theoretical claim, yielding a score of 6.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method rests on a variational density estimator whose accuracy is assumed rather than validated, and on an exploration reward whose count-based interpretation is only partially proven. The free parameters λ and α are tuned on benchmarks where the final results are reported, and the discrete skill count is another design choice.

free parameters (3)
  • λ (density deviation weight) = not reported for final runs; ablated over 0.5, 1.0, 1.5, 2.0, 3.0
    Controls the strength of inter-skill separation in the SD3 objective; the authors conclude λ >= 1.5 works but do not state the value used in the main experiments.
  • α (exploration reward coefficient) = 0.04
    Chosen from ablation studies on the maze and Quadruped domains and then applied across all benchmark experiments, without a held-out tuning protocol.
  • n (number of discrete skills) = 10 in maze; not reported for URLB runs
    The discrete skill count is a design choice; the paper notes density estimation requires all skills but does not analyze sensitivity to n in the main benchmarks.
assumptions (5)
  • standard math ELBO is a valid lower bound on log skill-conditional state density
    Equation (3) uses stochastic gradient variational Bayes; the lower bound property is standard.
  • domain assumption The learned CVAE ELBO is an accurate proxy for the true state density dπ_z(s) in high-dimensional and image spaces
    The SD3 reward in Eq. (7) is computed from this estimate, but no validation of density estimation accuracy is provided, especially for pixel observations.
  • domain assumption DKL[Qφ(h|s,z)||r(h)] is a count-based novelty signal in deep RL
    Theorem 2 only proves an upper bound under a linear model; the equivalence is assumed to hold approximately in practice.
  • ad hoc to paper Decoder can be analyzed as a linear function of the encoder feature
    Equations (16)-(17) replace the actual neural decoder with a linear model solely for the proof, without justifying that the real architecture satisfies the same assumptions.
  • domain assumption Uniform skill prior p(z)=1/n and discrete skill space
    Used throughout Eq. (1) and the analysis; the paper states extension to continuous skills is possible but not developed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unsupervised Skill Discovery through Skill Regions Differentiation." pith.science (2026). https://pith.science/paper/AHO6XK3E

@misc{pith2026250614420,
  author       = {Pith},
  title        = {Pith review of: Unsupervised Skill Discovery through Skill Regions Differentiation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AHO6XK3E}},
  note         = {Machine review of arXiv:2506.14420}
}
read the original abstract

Unsupervised Reinforcement Learning (RL) aims to discover diverse behaviors that can accelerate the learning of downstream tasks. Previous methods typically focus on entropy-based exploration or empowerment-driven skill learning. However, entropy-based exploration struggles in large-scale state spaces (e.g., images), and empowerment-based methods with Mutual Information (MI) estimations have limitations in state exploration. To address these challenges, we propose a novel skill discovery objective that maximizes the deviation of the state density of one skill from the explored regions of other skills, encouraging inter-skill state diversity similar to the initial MI objective. For state-density estimation, we construct a novel conditional autoencoder with soft modularization for different skill policies in high-dimensional space. Meanwhile, to incentivize intra-skill exploration, we formulate an intrinsic reward based on the learned autoencoder that resembles count-based exploration in a compact latent space. Through extensive experiments in challenging state and image-based tasks, we find our method learns meaningful skills and achieves superior performance in various downstream tasks.

Figures

Figures reproduced from arXiv: 2506.14420 by the authors.

Figure 1
Figure 1. An overview of the CVAE architecture. (a) The encoder-decoder network with soft modularization. The feature extractor [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. An illustration of skill discovery in SD3. The skills start [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Results for maze experiment. We visually demonstrate the agent’s ability to explore the environment and the diversity [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Results for state-based URLB. The aggregate statistics [ [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Results for Pixel-based URLB. We conduct experi [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Results for robustness experiment. It can be observed [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Ablation on the soft modularization structure. [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Results for the impact of weight parameter in the [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Results for the impact of exploration ratio. (a) We conduct experiments with different [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Skill adaption strategies ablation. We test several [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Additional experiments in the tree-like Maze with different numbers of skills. Under different environmental conditions, [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Skill visualization in DMC. It can be observed that SD3 learns dynamic and valuable skills, which enable the agent [PITH_FULL_IMAGE:figures/full_fig_p012_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 49 canonical work pages

  1. [1]

    A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,

    D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel et al. , “A general reinforcement learning algorithm that masters chess, shogi, and go through self-play,” Science, vol. 362, no. 6419, pp. 1140–1144, 2018

    2018

  2. [2]

    Mastering atari games with limited data,

    W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y . Gao, “Mastering atari games with limited data,” Advances in Neural Information Processing Systems, vol. 34, pp. 25 476–25 488, 2021

    work page 2021

  3. [3]

    Continu- ous improvement of self-driving cars using dynamic confidence-aware reinforcement learning,

    Z. Cao, K. Jiang, W. Zhou, S. Xu, H. Peng, and D. Yang, “Continu- ous improvement of self-driving cars using dynamic confidence-aware reinforcement learning,” Nature Machine Intelligence, vol. 5, no. 2, pp. 145–158, 2023

    work page 2023

  4. [4]

    Uncertainty-aware model-based reinforce- ment learning: Methodology and application in autonomous driving,

    J. Wu, Z. Huang, and C. Lv, “Uncertainty-aware model-based reinforce- ment learning: Methodology and application in autonomous driving,” IEEE Transactions on Intelligent Vehicles , vol. 8, no. 1, pp. 194–203, 2022

    work page 2022

  5. [5]

    Temporal difference learning for model predictive control,

    N. A. Hansen, H. Su, and X. Wang, “Temporal difference learning for model predictive control,” in International Conference on Machine Learning. PMLR, 2022, pp. 8387–8406

    work page 2022

  6. [6]

    Learning robust perceptive locomotion for quadrupedal robots in the wild,

    T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics, vol. 7, no. 62, p. eabk2822, 2022

    work page 2022

  7. [7]

    Relay hindsight experience replay: Self-guided continual reinforcement learning for sequential object manipulation tasks with sparse rewards,

    Y . Luo, Y . Wang, K. Dong, Q. Zhang, E. Cheng, Z. Sun, and B. Song, “Relay hindsight experience replay: Self-guided continual reinforcement learning for sequential object manipulation tasks with sparse rewards,” Neurocomputing, vol. 557, p. 126620, 2023

    2023

  8. [8]

    Reward design with language models,

    M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh, “Reward design with language models,” in The Eleventh International Conference on Learning Representations, 2023

    work page 2023

Show all 65 references
  1. [9]

    Maniskill2: A unified benchmark for generalizable manipulation skills,

    J. Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y . Tang, S. Tao, X. Wei, Y . Yao, X. Yuan, P. Xie, Z. Huang, R. Chen, and H. Su, “Maniskill2: A unified benchmark for generalizable manipulation skills,” in The Eleventh International Conference on Learning Representations , 202...

    2023

  2. [10]

    Pre-trained models: Past, present and future,

    X. Han, Z. Zhang, N. Ding, Y . Gu, X. Liu, Y . Huo, J. Qiu, Y . Yao, A. Zhang, L. Zhang et al., “Pre-trained models: Past, present and future,” AI Open, vol. 2, pp. 225–250, 2021

    2021

  3. [11]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    2023 arXiv

  4. [12]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  5. [13]

    Llama 2: Open foundation and fine-tuned chat models,

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al. , “Llama 2: Open foundation and fine-tuned chat models,” arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  6. [14]

    Masked au- toencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked au- toencoders are scalable vision learners,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 16 000–16 009

    2022

  7. [15]

    V-JEPA: Latent video prediction for visual representation learning,

    A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y . LeCun, M. Assran, and N. Ballas, “V-JEPA: Latent video prediction for visual representation learning,” 2024. [Online]. Available: https: //openreview.net/forum?id=WFYbBOEOtv

    2024

  8. [16]

    Bootstrap your own latent-a new approach to self-supervised learning,

    J.-B. Grill, F. Strub, F. Altch ´e, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar et al. , “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems, vol. 33, pp. ...

    2020

  9. [17]

    Where are we in the search for an artificial visual cortex for embodied intelligence?

    A. Majumdar, K. Yadav, S. Arnaud, J. Ma, C. Chen, S. Silwal, A. Jain, V .-P. Berges, T. Wu, J. Vakil et al. , “Where are we in the search for an artificial visual cortex for embodied intelligence?” in Advances in Neural Information Processing Systems , vol. 36, 2023

    2023

  10. [18]

    R3m: A universal visual representation for robot manipulation,

    S. Nair, A. Rajeswaran, V . Kumar, C. Finn, and A. Gupta, “R3m: A universal visual representation for robot manipulation,” in Conference on Robot Learning . PMLR, 2023, pp. 892–909

    2023

  11. [19]

    URLB: Unsupervised reinforcement learning benchmark,

    M. Laskin, D. Yarats, H. Liu, K. Lee, A. Zhan, K. Lu, C. Cang, L. Pinto, and P. Abbeel, “URLB: Unsupervised reinforcement learning benchmark,” in Neural Information Processing Systems (Datasets and Benchmarks Track), 2021

    2021

  12. [20]

    Variational intrinsic control,

    K. Gregor, D. J. Rezende, and D. Wierstra, “Variational intrinsic control,” arXiv preprint arXiv:1611.07507 , 2016. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 15

    2016 arXiv

  13. [21]

    Behavior from the void: Unsupervised active pre-training,

    H. Liu and P. Abbeel, “Behavior from the void: Unsupervised active pre-training,” in Advances in Neural Information Processing Systems , vol. 34, 2021, pp. 18 459–18 473

    2021

  14. [22]

    Understanding the limitations of variational mutual information estimators,

    J. Song and S. Ermon, “Understanding the limitations of variational mutual information estimators,” in International Conference on Learning Representations, 2020

    2020

  15. [23]

    Diversity is all you need: Learning skills without a reward function,

    B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, “Diversity is all you need: Learning skills without a reward function,” in International Conference on Learning Representations , 2019

    2019

  16. [24]

    Behavior contrastive learning for unsupervised skill discovery,

    R. Yang, C. Bai, H. Guo, S. Li, B. Zhao, Z. Wang, P. Liu, and X. Li, “Behavior contrastive learning for unsupervised skill discovery,” in Proceedings of the 40th International Conference on Machine Learning , 2023, pp. 39 183–39 204

    2023

  17. [25]

    METRA: Scalable unsupervised RL with metric-aware abstraction,

    S. Park, O. Rybkin, and S. Levine, “METRA: Scalable unsupervised RL with metric-aware abstraction,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=c5pwL0Soay

    2024

  18. [26]

    Lipschitz-constrained unsupervised skill discovery,

    S. Park, J. Choi, J. Kim, H. Lee, and G. Kim, “Lipschitz-constrained unsupervised skill discovery,” in International Conference on Learning Representations, 2022

    2022

  19. [27]

    Controllability-aware unsuper- vised skill discovery,

    S. Park, K. Lee, Y . Lee, and P. Abbeel, “Controllability-aware unsuper- vised skill discovery,” inInternational Conference on Machine Learning, vol. 202, 2023, pp. 27 225–27 245

    2023

  20. [28]

    Unsupervised reinforcement learning with contrastive intrinsic control,

    M. Laskin, H. Liu, X. B. Peng, D. Yarats, A. Rajeswaran, and P. Abbeel, “Unsupervised reinforcement learning with contrastive intrinsic control,” in Advances in Neural Information Processing Systems , 2022

    2022

  21. [29]

    Mastering the unsupervised reinforce- ment learning benchmark from pixels,

    S. Rajeswar, P. Mazzaglia, T. Verbelen, A. Pich ´e, B. Dhoedt, A. Courville, and A. Lacoste, “Mastering the unsupervised reinforce- ment learning benchmark from pixels,” in International Conference on Machine Learning. PMLR, 2023, pp. 28 598–28 617

    2023

  22. [30]

    Auto-encoding variational bayes,

    D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv preprint arXiv:1312.6114, 2013

    2013 arXiv

  23. [31]

    An introduction to variational autoencoders,

    D. P. Kingma, M. Welling et al. , “An introduction to variational autoencoders,” Foundations and Trends® in Machine Learning, vol. 12, no. 4, pp. 307–392, 2019

    2019

  24. [32]

    Multi-task reinforcement learning with soft modularization,

    R. Yang, H. Xu, Y . Wu, and X. Wang, “Multi-task reinforcement learning with soft modularization,” Advances in Neural Information Processing Systems, vol. 33, pp. 4767–4777, 2020

    2020

  25. [33]

    Near-bayesian exploration in polynomial time,

    J. Z. Kolter and A. Y . Ng, “Near-bayesian exploration in polynomial time,” in Proceedings of the 26th annual international conference on machine learning, 2009, pp. 513–520

    2009

  26. [34]

    An analysis of model-based interval estimation for markov decision processes,

    A. L. Strehl and M. L. Littman, “An analysis of model-based interval estimation for markov decision processes,” Journal of Computer and System Sciences, vol. 74, no. 8, pp. 1309–1331, 2008

    2008

  27. [35]

    Unifying count-based exploration and intrinsic motivation,

    M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos, “Unifying count-based exploration and intrinsic motivation,” Advances in neural information processing systems , vol. 29, 2016

    2016

  28. [36]

    Count-based exploration with neural density models,

    G. Ostrovski, M. G. Bellemare, A. Oord, and R. Munos, “Count-based exploration with neural density models,” in International conference on machine learning. PMLR, 2017, pp. 2721–2730

    2017

  29. [37]

    Dynamics- aware unsupervised discovery of skills,

    A. Sharma, S. Gu, S. Levine, V . Kumar, and K. Hausman, “Dynamics- aware unsupervised discovery of skills,” in International Conference on Learning Representations, 2020

    2020

  30. [38]

    Explore, discover and learn: Unsupervised discovery of state-covering skills,

    V . Campos, A. Trott, C. Xiong, R. Socher, X. Gir´o-i Nieto, and J. Torres, “Explore, discover and learn: Unsupervised discovery of state-covering skills,” in International Conference on Machine Learning . PMLR, 2020, pp. 1317–1327

    2020

  31. [39]

    Unsupervised skill discovery via recurrent skill training,

    Z. Jiang, J. Gao, and J. Chen, “Unsupervised skill discovery via recurrent skill training,” in Advances in Neural Information Processing Systems , A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, Eds., 2022

    2022

  32. [40]

    Learning to discover skills with guidance,

    H. Kim, B. Lee, S. Park, H. Lee, D. Hwang, K. Min, and J. Choo, “Learning to discover skills with guidance,” in Advances in Neural Information Processing Systems , 2023

    2023

  33. [41]

    Aps: Active pretraining with successor features,

    H. Liu and P. Abbeel, “Aps: Active pretraining with successor features,” in International Conference on Machine Learning . PMLR, 2021, pp. 6736–6747

    2021

  34. [42]

    Efficient exploration via state marginal matching,

    L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov, “Efficient exploration via state marginal matching,”

  35. [43]

    Learning more skills through optimistic exploration,

    D. Strouse, K. Baumli, D. Warde-Farley, V . Mnih, and S. S. Hansen, “Learning more skills through optimistic exploration,” in International Conference on Learning Representations , 2022

    2022

  36. [44]

    Choreographer: Learning and adapting skills in imagination,

    P. Mazzaglia, T. Verbelen, B. Dhoedt, A. Lacoste, and S. Rajeswar, “Choreographer: Learning and adapting skills in imagination,” in International Conference on Learning Representations , 2023. [Online]. Available: https://openreview.net/forum?id=PhkWyijGi5b

    2023

  37. [45]

    Curiosity-driven exploration by self-supervised prediction,

    D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-driven exploration by self-supervised prediction,” in International Conference on Machine Learning . PMLR, 2017, pp. 2778–2787

    2017

  38. [46]

    Self-supervised exploration via disagreement,

    D. Pathak, D. Gandhi, and A. Gupta, “Self-supervised exploration via disagreement,” in International Conference on Machine Learning . PMLR, 2019, pp. 5062–5071

    2019

  39. [47]

    Exploration by random network distillation,

    Y . Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” in International Conference on Learning Representations, 2019

    2019

  40. [48]

    Reinforcement learn- ing with prototypical representations,

    D. Yarats, R. Fergus, A. Lazaric, and L. Pinto, “Reinforcement learn- ing with prototypical representations,” in International Conference on Machine Learning. PMLR, 2021, pp. 11 920–11 931

    2021

  41. [49]

    Unsupervised skill-discovery and skill-learning in minecraft,

    J. J. Nieto, R. Creus, and X. Giro-i Nieto, “Unsupervised skill-discovery and skill-learning in minecraft,” arXiv preprint arXiv:2107.08398, 2021

    2021 arXiv

  42. [50]

    Rethinking mutual information for language conditioned skill discovery on imitation learning,

    Z. Ju, C. Yang, F. Sun, H. Wang, and Y . Qiao, “Rethinking mutual information for language conditioned skill discovery on imitation learning,” in 34th International Conference on Automated Planning and Scheduling, 2024. [Online]. Available: https://openreview.net/forum?id= 8VdptRkRYW

    2024

  43. [51]

    Robust policy learning via offline skill diffusion,

    W. K. Kim, M. Yoo, and H. Woo, “Robust policy learning via offline skill diffusion,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 12, 2024, pp. 13 177–13 184

    2024

  44. [52]

    EUCLID: Towards efficient unsupervised reinforcement learning with multi-choice dynamics model,

    Y . Yuan, J. HAO, F. Ni, Y . Mu, Y . ZHENG, Y . Hu, J. Liu, Y . Chen, and C. Fan, “EUCLID: Towards efficient unsupervised reinforcement learning with multi-choice dynamics model,” in The Eleventh International Conference on Learning Representations , 2023. [Online]. Available:...

    2023

  45. [53]

    Variational empowerment as representation learning for goal-based reinforcement learning,

    J. Choi, A. Sharma, H. Lee, S. Levine, and S. S. Gu, “Variational empowerment as representation learning for goal-based reinforcement learning,” CoRR, vol. abs/2106.01404, 2021. [Online]. Available: https://arxiv.org/abs/2106.01404

    2021 arXiv

  46. [54]

    Deep reinforcement learning at the edge of the statistical precipice,

    R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Belle- mare, “Deep reinforcement learning at the edge of the statistical precipice,” Advances in neural information processing systems , vol. 34, pp. 29 304–29 320, 2021

    2021

  47. [55]

    Deepmind control suite,

    Y . Tassa, Y . Doron, A. Muldal, T. Erez, Y . Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq et al., “Deepmind control suite,” arXiv preprint arXiv:1801.00690 , 2018

    2018 arXiv

  48. [56]

    Continuous control with deep reinforcement learning

    T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y . Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning.” in ICLR (Poster) , 2016. [Online]. Available: http://arxiv.org/abs/1509.02971

    2016 arXiv

  49. [57]

    Mastering visual continuous control: Improved data-augmented reinforcement learning,

    D. Yarats, R. Fergus, A. Lazaric, and L. Pinto, “Mastering visual continuous control: Improved data-augmented reinforcement learning,” in International Conference on Learning Representations , 2022. [Online]. Available: https://openreview.net/forum?id= SJ- yyes8

    2022

  50. [58]

    Mastering atari with discrete world models,

    D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” in ICLR, 2021

    2021

  51. [59]

    Provably efficient rein- forcement learning with linear function approximation,

    C. Jin, Z. Yang, Z. Wang, and M. I. Jordan, “Provably efficient rein- forcement learning with linear function approximation,” Mathematics of Operations Research, vol. 48, no. 3, pp. 1496–1521, 2023

    2023

  52. [60]

    Reward-free model-based reinforcement learning with linear function approximation,

    W. ZHANG, D. Zhou, and Q. Gu, “Reward-free model-based reinforcement learning with linear function approximation,” in Advances in Neural Information Processing Systems , A. Beygelzimer, Y . Dauphin, P. Liang, and J. W. Vaughan, Eds., 2021. [Online]. Available: https://openrevi...

    2021

  53. [61]

    Deep variational information bottleneck,

    A. A. Alemi, I. Fischer, J. V . Dillon, and K. Murphy, “Deep variational information bottleneck,” in International Conference on Learning Rep- resentations, 2017

    2017

  54. [62]

    Dynamic bottleneck for robust self-supervised exploration,

    C. Bai, L. Wang, L. Han, A. Garg, J. Hao, P. Liu, and Z. Wang, “Dynamic bottleneck for robust self-supervised exploration,” Advances in Neural Information Processing Systems , vol. 34, pp. 17 007–17 020, 2021

    2021

  55. [63]

    Provably efficient exploration in policy optimization,

    Q. Cai, Z. Yang, C. Jin, and Z. Wang, “Provably efficient exploration in policy optimization,” in International Conference on Machine Learning. PMLR, 2020, pp. 1283–1294

    2020

  56. [64]

    Logarithmic online regret bounds for undis- counted reinforcement learning,

    P. Auer and R. Ortner, “Logarithmic online regret bounds for undis- counted reinforcement learning,” Advances in neural information pro- cessing systems, vol. 19, 2006

    2006

  57. [2020]

    Available: https://openreview.net/forum?id=Hkla1eHFvS

    [Online]. Available: https://openreview.net/forum?id=Hkla1eHFvS

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.