Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

This paper shows that splitting both the state space and the skill space into per-entity factors, plus a curiosity weight per factor, lets an agent discover a skill for every controllable object — not just the easy ones.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 05:35 UTC pith:66H32YO5

load-bearing objection SUSD is a plausible and well-tested recipe for factorized skill discovery, but the unsupervised framing is undercut by task-informed factorization choices, and the main lemma's proof doesn't hold as written. the 5 major comments →

arxiv 2602.01619 v2 pith:66H32YO5 submitted 2026-02-02 cs.LG cs.AI

SUSD: Structured Unsupervised Skill Discovery through State Factorization

Seyed Mohammad Hadi Hosseini , Mahdieh Soleymani Baghshah This is my paper
classification cs.LG cs.AI
keywords unsupervised skill discoverystate factorizationfactored MDPcuriosity-driven explorationintrinsic motivationhierarchical reinforcement learningdistance-maximizing skillsmulti-object environments
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most unsupervised skill discovery methods treat the whole state as one block, so skills end up covering only the parts of the environment that are easy to control. This paper argues that whenever a state can be decomposed into meaningful factors — separate agents, objects, or body parts — the skill space should be factorized the same way, giving each factor its own skill variable. It then adds a per-factor curiosity mechanism: a density model predicts the next state, and the agent is rewarded more for transitions that are surprising for that specific factor. The result is a skill set that covers all factors, including the hardest ones, and a factored skill representation that makes downstream compositional tasks easier to train with hierarchical RL. In multi-object environments, this method outperforms earlier distance-maximizing and mutual-information skill discovery baselines on downstream tasks, state coverage, and factor-wise reconstruction.

Core claim

The paper's central claim is that whole-state distance-maximizing skill discovery saturates in multi-entity environments because the single embedding cannot do justice to all controllable factors. SUSD's objective replaces the single term with a sum over factors: for each factor i, the inner product between the change in a factor-specific embedding phi_i and a factor-specific skill component z_i, multiplied by sqrt(-log q_theta(s^i_{t+1}|s_t)) — the negative log-likelihood of the factor's next state under a Gaussian density model. That multiplier is the curiosity weight; it is high for transitions the current policy finds hard to produce, so the agent is continually pushed toward the factors

What carries the argument

The load-bearing piece is the factorized DSD objective. It has three components: (1) factor-specific embeddings phi_i, each mapping its own state factor into a latent direction; (2) a factorized skill vector z=(z_1,...,z_N) whose components align with those embeddings via the inner product of latent change; and (3) the curiosity weight, computed by fitting a Gaussian density model q_theta(s'|s) on transitions and extracting the marginal negative log-likelihood for each factor separately. Lemma 4.1 justifies moving the distance function from a Lipschitz constraint into a reward coefficient, and dual gradient descent with a shared Lipschitz bound across the sum of factor embeddings keeps the g

Load-bearing premise

The method assumes a user can supply a state factorization that matches the environment's true controllable entities; with a mismatched, over- or under-factorized partition, the per-factor skills cease to correspond to anything meaningful and the advantage over holistic baselines disappears.

What would settle it

Run SUSD on the Multi-Particle environment with a deliberately wrong factorization — for example, grouping the 20 objects into 10 random pairs instead of the agent-station pairs the paper uses. If the downstream performance falls to the level of METRA/CSD, the result confirms that the factorization, not the learning algorithm, is doing the work. A stronger test is to build a two-factor environment whose reward requires coordinated interaction between the factors; since the objective is a linear combination of per-factor embeddings, SUSD should fail to discover that joint skill while a holistic

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • In any environment whose factor structure is known, the method converts a monolithic skill-learning problem into per-factor subproblems, so the number of controllable entities no longer dilutes coverage of the hardest ones.
  • The factored skill representation means a high-level policy can compose skills at entity level, making hierarchical training on sequential or multi-object tasks substantially more sample-efficient than learning from scratch or using whole-state skills.
  • Because the curiosity weight is per factor, training automatically shifts attention to the worst-performing factor; the paper's worst-agent coverage results suggest the method prevents the 'forgotten entity' failure mode.
  • The method remains competitive on single-agent environments like Ant and HalfCheetah, so the factorization is a safe addition when structure is available and does not hurt when it is not.
  • The zero-mean trick for discrete skill spaces (Appendix F) shows the same framework extends beyond continuous skill latents, widening the class of environments where the factorization applies.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit where the factorization comes from; in practice a user must supply it, so learning the factorization from raw observations is a natural next step.
  • The over/under-factorization experiments suggest a testable diagnostic: in a pixel-based environment, running SUSD with different factor granularities and measuring downstream performance could expose the true object boundaries — a use the paper does not mention.
  • Because the curiosity weight is derived from a learned dynamics model, the method's advantage should degrade in environments with high stochasticity where the density model is inaccurate; this is a direct prediction a reader could test, though the paper does not.
  • If the factorization is aligned with the downstream task (as the authors find in Appendix H), the method effectively performs a form of task-relevant representation learning without ever seeing the task reward, which connects to meta-learning goal-conditioned policies in a way the paper does not develop.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes SUSD, a distance-maximizing (DSD) skill discovery method for factored state spaces. The state is partitioned into factors, each factor gets its own skill sub-vector and embedding function, and a learned Gaussian density model supplies per-factor curiosity weights that scale the factor-wise DSD intrinsic rewards. The resulting factorized skills are then evaluated on downstream tasks via hierarchical RL (a frozen low-level skill policy and a learned high-level skill selector) in Multi-Particle, Kitchen, and 2D-Gunner, plus zero-shot goal reaching in Ant and HalfCheetah. The experiments compare against LSD, CSD, METRA, DIAYN, and DUSDi and report improved downstream returns, state coverage, and factor-decoding accuracy.

Significance. If the method and results hold, SUSD would be a useful contribution: it extends DSD to factored environments, adds an adaptive per-factor weighting mechanism, and provides a broad empirical study with multiple baselines, ablations, and sensitivity analyses. The release of code, the inclusion of over/under-factorization analysis (Appendix M), the increased-skill-dimension control (Appendix I), and the discrete-skill extension (Appendix F) are strengths. However, the theoretical justification of the central objective is currently flawed, and the evaluation does not fully establish the 'unsupervised' claim because the factorization is chosen using downstream-task structure. These issues are load-bearing for the paper's main claims, though they appear addressable with additional analysis and experiments.

major comments (5)
  1. [Lemma 4.1, Appendix A (Eqs. 11–13)] The proof defines \tilde{\phi}(s) := \phi(s)/d(s,s'), but d(s,s') depends on the pair (s,s'), so \tilde{\phi} is not a well-defined function of a single state. Consequently, the constraint ||\tilde{\phi}(s)-\tilde{\phi}(s')||\le 1 does not follow from ||\phi(s)-\phi(s')||\le d(s,s'), and the substitution \phi(s)=\tilde{\phi}(s)d(s,s') in Eq. 12 does not produce Eq. 13. Moreover, Eq. 6 puts the curiosity weight inside the reward while leaving the constraint unweighted, so even a corrected single-pair lemma would not justify the factorized objective. Please either supply a valid derivation (e.g., via the dual norm of the Lipschitz constraint) or present Eq. 6 as a heuristic objective without claiming Lemma 4.1 as its justification.
  2. [Appendix H / §5.2] The claim that SUSD discovers skills 'without supervision' is not yet established because the factorization is chosen using downstream-task structure. Appendix H states that grouping each agent with its corresponding station—the exact agent–station couplings rewarded by the seq and food-poison tasks—improves results, and Appendix M shows that over- or under-factorization degrades performance. Thus the reported advantage in Multi-Particle may depend on a task-aligned factorization rather than on unsupervised discovery. Please report results with a task-agnostic factorization (e.g., automatic grouping based only on state statistics, or a fixed grouping that does not use downstream task labels) and explicitly state whether the method requires such prior knowledge as input.
  3. [§5.2 / Appendix I / Table 3] In Multi-Particle, SUSD's total skill dimension is N·D = 20 (N=10, D=2), while the default baselines use D=2 (Appendix E). The main comparisons in Figure 3, Table 1, and Table 2 may therefore confound method with skill-space capacity. Appendix I's control—increasing METRA/CSD skill dimension to 20—is a useful check and should appear in the main text; it should also be applied to DUSDi, since DUSDi is the main factorized baseline, to ensure a like-for-like comparison.
  4. [§4.2 (Eqs. 5–6)] The statement that \sqrt{-\log q_\theta(s'_i|s_t)} 'can be interpreted as a valid distance metric' is unsupported: a conditional negative log-likelihood is not symmetric and need not satisfy the triangle inequality, so it is not a metric in the sense required by Lemma 4.1. Additionally, Eq. 5 claims proportionality with the Mahalanobis term, but the Gaussian negative log-likelihood contains log-determinant terms that differ per factor and per time step, so the weights are not proportional to -\log q as written. Please define the curiosity weight directly (e.g., as the Mahalanobis term) and either prove the metric property or clearly label the weighting as a heuristic.
  5. [§5.4.2, Table 1] The factor-decoding evaluation partly rewards the factorization used in training. SUSD stores each state factor in a separate embedding component, so a simple decoder can reconstruct each factor from that component with low MSE; baselines must recover each factor from a holistic embedding. This metric is thus partially by construction. It should be supplemented by a metric that does not assume the same factorization, such as zero-shot goal reaching in a factorized environment or downstream task returns (already reported), so that the 'richer latent embedding' claim is independently grounded.
minor comments (5)
  1. [§4.2] Typo: 'Furhermore' should be 'Furthermore'; 'we our final optimization problem' should be 'we formulate our final optimization problem'.
  2. [Eqs. 3, 6, 8] The constraint is written as a sum of squared norms in Eqs. 3 and 6, while Eq. 8 uses `1 - ||...||` (unsquared). Please make the norm convention explicit and consistent across equations.
  3. [Appendix E / Table 3] The text says 'we set N=20 for Multi-partcle (for 10 agents and 10 stations)' and then says 'When grouping agent and station in this environment, we consider N=10 factors.' This is confusing. State clearly that the reported experiments use N=10 grouped factors, and explain how total skill dimension is computed for SUSD versus baselines.
  4. [Appendix B.4] The Kitchen observation space is said to be 142-dimensional and decomposable into seven components, but only six components are listed and their dimensions sum to 128 (33+22+18+19+22+14). Please correct the number of components and dimensions, and include the missing component (apparently the butter).
  5. [§5.1] The sentence 'We further evaluate binning coverage in Appendix K' appears in §5.4.1, not in §5.1; the cross-reference is misplaced.

Circularity Check

4 steps flagged

SUSD's core intrinsic-reward derivation is self-contained, but its supporting factor-wise metrics are aligned with the training objective by construction and the 'unsupervised' claim is weakened by the admission that the factorization was chosen using downstream task structure.

specific steps
  1. self definitional [Section 5.4.1 (State Coverage Across Factors), with objective in Eq. 6]
    "We evaluate SUSD by randomly selecting a skill every 200 steps and collecting 20K rollout steps in the Multi-Particle environment. For each factor (agent), we compute the number of unique states it visits ... As shown in Figure 4, SUSD achieves substantially better coverage than the baselines."

    The SUSD training objective (Eq. 6) is a sum over factors of per-factor embedding displacements weighted by curiosity. The coverage metric counts per-factor unique visited states. Maximizing the training objective directly encourages the quantity being measured, so SUSD's high per-factor coverage is partly a restatement of the objective rather than an independent behavioral prediction. The comparison against unfactorized baselines is still meaningful, but the metric is constructed to match SUSD's factorized design.

  2. self definitional [Section 5.4.2 (Factor Decoding), Table 1]
    "By training a decoder on top of these embeddings to reconstruct the observations, we can evaluate reconstruction quality using factor-wise MSE. A high-quality latent skill embedding will yield low MSE for each factor, outperforming other baselines and indicating that the embedding effectively captures all relevant information necessary to reconstruct the observations."

    Because SUSD's embedding is factorized—each φ_i receives only factor s^i and is trained to track changes in that factor—the factor-wise decoding task is architecturally aligned with SUSD. Holistic baselines such as METRA, CSD, and LSD produce a single global embedding optimized for a different objective, so a per-factor decoder is structurally disadvantaged. Low factor-wise MSE for SUSD is therefore largely a consequence of the factorization itself, not independent evidence of a richer skill representation.

  3. fitted input called prediction [Appendix H (Factorization and Further Experiments)]
    "In the Multi-Particle environment with 20 objects (agents and stations), we group each agent with its corresponding station as a single factor leads to better performance compared to a factorization treating each object (i.e., agent or station) as a separate factor. Figure 9 compares these two factorizations, showing that incorporating prior knowledge (of requiring each agent to interact with its own station) to align factorization with downstream task structure improves results."

    The central framing is 'unsupervised skill discovery' and 'without supervision.' Here the key structural input—the state factorization—is selected using knowledge of the downstream task (each agent must interact with its own station), and the same downstream tasks are then used to demonstrate SUSD's advantage. This is evaluation leakage: the comparison is conditional on a task-aligned inductive bias chosen by the experimenter, so it does not establish that SUSD discovers the relevant structure from reward-free interaction alone. It is not a mathematical equivalence, but it directly weakens the unsupervised claim.

  4. other [Appendix M (Analyzing Overfactorization and Underfactorization Effects)]
    "As shown in Figure 13 performance is highest when the true factorization structure is used, and it degrades as more or less factors are imposed."

    This admission confirms that SUSD's gains depend on knowing the 'true' factorization a priori. Combined with Appendix H, where the Multi-Particle factorization was aligned to downstream task structure, the evidence shows that the method's advantage is contingent on task-informed factorization, not on fully unsupervised discovery. The appendix is honest about the limitation, but the limitation undercuts the generality of the headline claim.

full rationale

Most of SUSD's derivation chain is self-contained: the intrinsic reward (Eq. 10) is built from a factorized DSD objective plus a learned density model, not from downstream rewards, and there are no self-citations or imported uniqueness theorems that carry the argument. The algorithm could in principle be trained and evaluated without circularity. However, three pieces of evidence reduce the weight of the empirical claims. First, the factor-coverage metric (Fig. 4) measures per-factor state displacement that Eq. 6 is directly optimized to maximize, so SUSD's advantage there is partly by construction. Second, the factor-decoding metric (Table 1) is architecturally aligned with SUSD's factorized φ, making it easier for SUSD than for holistic baselines; low factor-wise MSE partly restates the factorization rather than independently demonstrating richer skills. Third, and most important for the 'unsupervised' framing, Appendix H admits that in Multi-Particle the factorization was chosen by aligning agent-station groupings with the downstream task structure, and Appendix M admits performance is highest only when the 'true' factorization is used. The downstream task returns (Fig. 3, Table 2) are not forced by construction—they provide real independent signal—so the core method is not circular. Score 4 reflects partial circularity in the supporting metrics plus task-aligned factorization leakage, while stopping short of 6 because the main downstream results still require learning and are not equivalent to the method's inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 7 axioms · 0 invented entities

The central contribution is an algorithmic structure: factorization choices, a learned curiosity density model, and skill-space dimensions. Most axioms are inherited from the DSD framework; the questionable Lemma 4.1 and the metric assertion are ad hoc assumptions. No invented physical or ontological entities are introduced.

free parameters (5)
  • State factorization N and factor assignment = N=3 (Gunner), N=7 (Kitchen), N=10 or 20 (Multi-Particle); agent+station grouping in MP
    Hand-chosen per environment; Appendix H shows grouping with downstream task knowledge improves results.
  • Skill dimension per factor D = 2
    Hyperparameter chosen for all factors and environments.
  • Density model q_theta parameters = learned online
    Curiosity weights are computed from a learned Gaussian density model fit to the policy's transition data.
  • Initial Lagrange multiplier lambda = 3000 per factor
    Set by hand in Table 3; affects the balance between intrinsic reward and Lipschitz constraint.
  • Slack epsilon = 1e-6
    Hyperparameter preventing the lambda gradient from always being nonnegative.
axioms (7)
  • domain assumption State space factorizes as S = S_1 x ... x S_N with known factors and direct access to the underlying state vector.
    Section 3 states 'we assume direct access to the underlying state vector' and factors correspond to objects/entities.
  • domain assumption The DSD objective (Wasserstein dependency, Eq. 2) is a valid objective for skill discovery.
    Section 2.1.2 inherits this from prior DSD works; it is not re-derived.
  • ad hoc to paper Lemma 4.1: DSD objective with constraint ||phi(s)-phi(s')|| <= d(s',s) can be rewritten with reward scaled by d(s',s) and constraint scaled to 1.
    Appendix A's proof defines phi~(s)=phi(s)/d(s,s') although d is pair-dependent, so the equivalence is not actually proven; the paper relies on it to justify curiosity weighting.
  • ad hoc to paper sqrt(-log q_theta(s'_i|s_t)) is a valid distance metric.
    Section 4.2 asserts this without proof of metric axioms; needed for applying Lemma 4.1.
  • domain assumption Gaussian density model q_theta(s'|s)=N(mu_theta(s), Sigma_theta(s)) adequately captures transition likelihood and factor-wise marginals.
    Section 4.2 uses the Gaussian model to compute factor-wise curiosity weights.
  • domain assumption The chosen factorization is aligned with true controllable factors and downstream task structure.
    Appendix H uses knowledge that each MP agent interacts with its own station; Appendix M shows misaligned factorization degrades performance.
  • standard math Dual gradient descent with Lagrange multiplier lambda and slack epsilon converges to a useful solution.
    Standard optimization practice in DSD methods, used in Section 4.3.

pith-pipeline@v1.3.0-alltime-deepseek · 15973 in / 14042 out tokens · 134677 ms · 2026-08-03T05:35:16.727707+00:00 · methodology

0 comments
read the original abstract

Unsupervised Skill Discovery (USD) aims to autonomously learn a diverse set of skills without relying on extrinsic rewards. One of the most common USD approaches is to maximize the Mutual Information (MI) between skill latent variables and states. However, MI-based methods tend to favor simple, static skills due to their invariance properties, limiting the discovery of dynamic, task-relevant behaviors. Distance-Maximizing Skill Discovery (DSD) promotes more dynamic skills by leveraging state-space distances, yet still fall short in encouraging comprehensive skill sets that engage all controllable factors or entities in the environment. In this work, we introduce SUSD, a novel framework that harnesses the compositional structure of environments by factorizing the state space into independent components (e.g., objects or controllable entities). SUSD allocates distinct skill variables to different factors, enabling more fine-grained control on the skill discovery process. A dynamic model also tracks learning across factors, adaptively steering the agent's focus toward underexplored factors. This structured approach not only promotes the discovery of richer and more diverse skills, but also yields a factorized skill representation that enables fine-grained and disentangled control over individual entities which facilitates efficient training of compositional downstream tasks via Hierarchical Reinforcement Learning (HRL). Our experimental results across three environments, with factors ranging from 1 to 10, demonstrate that our method can discover diverse and complex skills without supervision, significantly outperforming existing unsupervised skill discovery methods in factorized and complex environments. Code is publicly available at: https://github.com/hadi-hosseini/SUSD.

Figures

Figures reproduced from arXiv: 2602.01619 by Mahdieh Soleymani Baghshah, Seyed Mohammad Hadi Hosseini.

Figure 1
Figure 1. Figure 1: Illustration of the SUSD Method. (a) In the skill learning stage, factorized embedding ϕ of the current and next state is passed through the corresponding mapping function ϕ(.) to obtain a skill latent embedding. Additional details about factorized embedding are shown in (b), where factor s i , i ∈ {1, ..., N}, is mapped to its embedding through the function ϕi . These embeddings, together with the skill f… view at source ↗
Figure 2
Figure 2. Figure 2: Left: The state s is passed to the density model, which estimates the mean and variance of q(s ′ |s). These statistics are then partitioned by factors to obtain q(s ′ i |st) 2 i=1. Right: Point x shows high probability in factor 1 but low probability in factor 2—a distinction that cannot be leveraged by the CSD method, which assigns a single weight to the entire state transition rather than to individual s… view at source ↗
Figure 3
Figure 3. Figure 3: Training curves of SUSD and baseline methods on multiple downstream tasks in the Multi [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of state/factor coverage across different factorized environments. [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Benchmark Environments B.5 MULTI-PARTICLE As shown in [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Zero-shot goal reaching performance of policies learned by skill discovery methods across [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of discrete and continuous skill spaces in the 2D-Gunner environment. [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Effect of the curiosity-based weighting module [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Factorization in Multi-Particle: grouping each agent with its station outperforms treating [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Impact of increasing skill dimensionality on baseline USD methods. Even when METRA [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Bin coverage comparison. (a) Average bin coverage across factors in MP. (b) Minimum [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Comparison of SUSD and baseline methods on the PoS and PoT downstream tasks. [PITH_FULL_IMAGE:figures/full_fig_p021_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Effect of over/under-factorization on performance in the 2D-Gunner environment. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Learning Object Manipulation from Scratch via Contrastive Interaction

    cs.RO 2026-06 unverdicted novelty 7.0

    IWR improves CRL sample efficiency and performance in interaction-rich manipulation by interaction-aware resampling that preserves mode boundaries, yielding 19.8% average gains and a real-world air-hockey agent.

Reference graph

Works this paper leans on

24 extracted references · 7 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Variational option discovery algorithms.arXiv preprint arXiv:1807.10299,

    Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms.arXiv preprint arXiv:1807.10299,

    Pith/arXiv arXiv

  2. [3]

    The agent receives a reward of 10 upon reaching the goal

    that is randomly sampled from[−10,10]. The agent receives a reward of 10 upon reaching the goal. C.3 GUNNER Unlimited Ammo (unlim)(Hu et al., 2024): In this downstream task, targets appear at random lo- cations, and the agent must approach and shoot each target to score. Since ammunition is unlimited, the agent does not need to collect any. Limited Ammo (...

    2024

  3. [4]

    Agents must decide whether to interact with their station based on a sequence of binary indicators provided at the beginning of each episode

    Food-poison (fp) (easy, medium, hard, difficult)(Hu et al., 2024): In this downstream task, each station delivers either food or poison to its corresponding agent. Agents must decide whether to interact with their station based on a sequence of binary indicators provided at the beginning of each episode. The easy version uses a sequence of length 2, the m...

    2024

  4. [5]

    Do’s and don’ts: Learning desirable skills with instruction videos.Advances in Neural Information Processing Systems, 37:47741–47766,

    11 Accepted as a conference paper at ICLR 2026 Hyunseung Kim, BYUNG KUN LEE, Hojoon Lee, Dongyoon Hwang, Donghu Kim, and Jaegul Choo. Do’s and don’ts: Learning desirable skills with instruction videos.Advances in Neural Information Processing Systems, 37:47741–47766,

    2026

  5. [6]

    Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,

    Pith/arXiv arXiv

  6. [7]

    12 Accepted as a conference paper at ICLR 2026 Richard S Sutton, Andrew G Barto, et al.Reinforcement learning: An introduction, volume

    2026

  7. [10]

    D COMPAREDMETHODS LSD (Park et al., 2022), CSD (Park et al., 2023), and METRA (Park et al.,

    2022

  8. [11]

    Figure 10: Impact of increasing skill dimensionality on baseline USD methods

    19 Accepted as a conference paper at ICLR 2026 Figure 9: Factorization in Multi-Particle: grouping each agent with its station outperforms treating objects independently. Figure 10: Impact of increasing skill dimensionality on baseline USD methods. Even when METRA and CSD increase their skill dimensionality from2to20, their performance remains below SUSD....

    2026

  9. [12]

    The agent has a 18-dimensional observation space, consisting of 3 state factors: Agent Position, Ammo State, Target State

    environment, the blue star marks the position of the agent, the blue line marks its shooting direction, the red diamond marks ammo location, and the orange cross marks the target position. The agent has a 18-dimensional observation space, consisting of 3 state factors: Agent Position, Ammo State, Target State. The action is 6- dimensional, 2 for agent mov...

    2024

  10. [13]

    The agent operates in a 4-dimensional action space, while the observation space is 142-dimensional

    environment contains a robot arm, a piece of butter, a meatball, a pot, a stove with its switch, and a target location marked in red. The agent operates in a 4-dimensional action space, while the observation space is 142-dimensional. This observation can be decomposed into seven components: 33 dimensions for the arm, 22 for the pot, 18 for the meatball, 1...

    2026

  11. [14]

    as shown in Figure

    Assume that thek-th dimension of factorZ i is1: ri = [ϕ(si t+1)−ϕ(s i t)]k − 1 N−1 X j∈{1,2,...,D}\{k} [ϕ(si t+1)−ϕ(s i t)]j.(14) To evaluate the effectiveness of our discrete approach, we implement the discrete skill space in the 2D-Gunner environment and compare its performance side by side with the continuous skill space, 17 Accepted as a conference pa...

    2026

  12. [18]

    As a result, we were unable to evaluate it on the environments introduced in this work

    also adopts environment factorization, it is constrained to environments with a small state space and a limited number of actions. As a result, we were unable to evaluate it on the environments introduced in this work. E IMPLEMENTATIONDETAILS Dimension of the latent space.For unfactorized environments (i.e., Ant and HalfCheetah), we set the latent skill d...

    2024

  13. [19]

    Although we concatenate this skill vector with the observation to feed into the skill policy, using a one-hot representation for each factor can lead to skill learning collapse

    F EXTENSION TODISCRETESKILLSPACE For discrete skills, we construct the skill space asZ:=Z 1 × · · · × ZN , whereNis the number of factors and eachZ i is aD-dimensional one-hot vector,Z i ∈ {0,1}D. Although we concatenate this skill vector with the observation to feed into the skill policy, using a one-hot representation for each factor can lead to skill l...

    2022

  14. [21]

    G ABLATIONSTUDY In this study, we evaluate the individual contributions of the curiosity-based weighting and factor- ization modules by selectively removing them

    As shown, the two methods achieve very similar performance on downstream tasks, indicating that our discrete skill formulation is competitive with the continuous one. G ABLATIONSTUDY In this study, we evaluate the individual contributions of the curiosity-based weighting and factor- ization modules by selectively removing them. This allows us to quantify ...

    2024

  15. [24]

    20 Accepted as a conference paper at ICLR 2026 Figure 12: Comparison of SUSD and baseline methods on the PoS and PoT downstream tasks. M ANALYZINGOVERFACTORIZATION ANDUNDERFACTORIZATIONEFFECTS In environments containing multiple objects, our approach naturally enables the model to encode object-specific information in separate components. Under-factorizat...

    2026

  16. [2012]

    Gymnasium: A standard interface for reinforcement learning environments.arXiv preprint arXiv:2407.17032,

    Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goul˜ao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. Gymnasium: A standard interface for reinforcement learning environments.arXiv preprint arXiv:2407.17032,

    Pith/arXiv arXiv

  17. [2015]

    Training data is collected from100Kroll- out steps

    and mean squared error (MSE) loss for 100 18 Accepted as a conference paper at ICLR 2026 Figure 8: Effect of the curiosity-based weighting module epochs with learning rate0.0001and batch size1024. Training data is collected from100Kroll- out steps. Every 200 steps, we sample a random skill and store the corresponding(state,skill) pair at each step. We use...

    2026

  18. [2016]

    The observation space consists of a single factor representing the state of the Ant that is 29-dimensional

    environment has an episode length of 200 steps. The observation space consists of a single factor representing the state of the Ant that is 29-dimensional. The action space is continuous, corresponding to the control of the Ant’s joints, and has 8 dimensions. B.2 HALFCHEETAH As shown in Figure 5(b), the HalfCheetah (Todorov et al., 2012; Brockman et al.,

    2012

  19. [2017]

    The action space is 50-dimensional, with 5 dimensions per agent controlling their movements and interactions with the landmarks

    environment has a 70-dimensional observation space, composed of 10 state factors that capture the states of each agent and its corresponding landmark. The action space is 50-dimensional, with 5 dimensions per agent controlling their movements and interactions with the landmarks. C DOWNSTREAMTASKS C.1 ANT Multi-goal Ant(Park et al., 2024): The task require...

    2024

  20. [2018]

    Deep learning using rectified linear units (relu).arXiv preprint arXiv:1803.08375,

    Abien Fred Agarap. Deep learning using rectified linear units (relu).arXiv preprint arXiv:1803.08375,

    Pith/arXiv arXiv

  21. [2019]

    Deep reinforcement learning for robotic manipulation.arXiv preprint arXiv:1610.00633, 1(1),

    Shixiang Gu, Ethan Holly, Timothy P Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation.arXiv preprint arXiv:1610.00633, 1(1),

    Pith/arXiv arXiv

  22. [2020]

    T−1X t=0 (ϕ(st+1)−ϕ(s t))⊤z # s.t.∥ ˜ϕ(s)− ˜ϕ(s′)∥2 ≤1,∀(s, s ′)∈S adj.(12) By replacingϕ(s)with ˜ϕ(s)·d(s, s′)in Eq. 12, we obtain: sup π,ϕ Ep(τ,z)

    13 Accepted as a conference paper at ICLR 2026 A PROOF OFLEMMA4.1 We first start with Eq. 2: sup π,ϕ Ep(τ,z) " T−1X t=0 (ϕ(st+1)−ϕ(s t))⊤z # s.t.∥ϕ(s)−ϕ(s ′)∥2 ≤d(s, s′),∀(s, s′)∈S adj.(11) Let the scaled state function be defined as ˜ϕ(s) := ϕ(s) d(s,s′). Then, we can transform the constraint term in Eq. 11 as follows (sinced(s, s′)≥0): sup π,ϕ Ep(τ,z) "...

    2026

  23. [2023]

    Openai gym.arXiv preprint arXiv:1606.01540,

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym.arXiv preprint arXiv:1606.01540,

    Pith/arXiv arXiv

  24. [2024]

    robosuite: A modular simulation framework and benchmark for robot learning.arXiv preprint arXiv:2009.12293,

    Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Mart´ın-Mart´ın, Abhishek Joshi, Soroush Nasiri- any, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning.arXiv preprint arXiv:2009.12293,

    Pith/arXiv arXiv 2009