Pith. sign in

REVIEW 2 major objections 6 minor 1 cited by

Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning

T0 review · 2 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper establishes that diversity and separability of learned skills determine how well they initialize downstream tasks, and that a Wasserstein-distance-based procedure can discover every optimal initial policy.

desk verdict The LSEPIN theory is a real extension of Eysenbach et al., but the WSEP and PWSEP vertex-discovery guarantees are not proven for the Wasserstein distance the paper actually uses. read the letter →

arxiv 2506.10629 v1 pith:CUE66NQ4 submitted 2025-06-12 cs.LG cs.AIcs.ITmath.IT

Yucheng Yang , Tianyi Zhou , Qiang He , Lei Han , Mykola Pechenizkiy , Meng Fang This is my paper
classification cs.LGcs.AIcs.ITmath.IT MSC 68T0562B1060B10
keywords unsupervisedreinforcementlearningskilldiscoverymutualinformationgeometrydisentanglementWassersteindistancedownstreamtaskadaptationvertex
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that for unsupervised reinforcement learning, how well a pretrained skill set transfers to unseen tasks is governed by the diversity and separability of the skills, and that the standard mutual-information objective $I(S;Z)$ does not guarantee either property. It introduces LSEPIN, the minimum over skills of $I(S;\mathbf{1}_z)$, as a disentanglement metric that captures both informativeness and separability, and it proves that higher LSEPIN can tighten an upper bound on a worst-case adaptation cost. To escape the geometric limitations of the KL divergence, the paper replaces it with the Wasserstein distance. The resulting objective WSEP pushes learned skills to vertices of the feasible state-distribution polytope and can discover more potentially optimal skills than MISL, while the projection-based PWSEP algorithm can discover all of them. A sympathetic reader would care because this supplies reward-free pretraining with objectives and evaluation metrics that are tied, in theory, to downstream adaptation.

What carries the argument

The load-bearing object is the convex polytope $C$ of state-occupancy distributions feasible under the MDP's dynamics. Because expected reward is linear in the occupancy distribution, every downstream task has an optimal initial policy at a vertex of $C$, so the quality of a skill set is how many vertices it covers and how far apart they are. $\mathrm{LSEPIN} = \min_z I(S;\mathbf{1}_z)$ measures the least informative, least separable individual skill, and the proof machinery links it to worst-case adaptation cost through the decomposition of $I(S;\mathbf{1}_z)$ into $p(z)$-weighted KL terms. WSEP replaces those KL distances with pairwise Wasserstein distances, whose symmetry and triangle inequality force optimal skill sets onto vertices. PWSEP($i$) projects a candidate skill onto the convex hull of already-discovered skills, and maximizing that projected Wasserstein distance guarantees a freshly discovered vertex at each iteration.

What would settle it

Run both MISL and PWSEP in a small tabular MDP whose feasible state-distribution polytope and vertices are known, using at least $\lvert V\rvert$ skills, and check whether PWSEP recovers all vertices in $\lvert V\rvert$ iterations; a missed vertex would refute Theorem 3.5. Separately, choose a downstream reward whose optimal distribution is close to the skill average $p(S)$ and test whether a higher-LSEPIN skill set actually has lower worst-case adaptation cost than a lower-LSEPIN set.

Watch

Extended reading notes

Core claim

The central claim is that the diversity and separability of discovered skills are what make them useful as initializations for downstream tasks, and that mutual information skill learning (MISL) does not ensure them. The paper defines a downstream task by a state reward and observes that the task's optimal feasible state distribution is a vertex of the convex polytope $C$ of state-occupancy distributions, so skill learning is vertex discovery. It proposes $\mathrm{LSEPIN} = \min_z I(S;\mathbf{1}_z)$ to score the least separable and informative skill, and shows that among MISL solutions sharing a skill, a higher $I(S;\mathbf{1}_z)$ raises that skill's probability and lowers the upper bound on adaptation cost from the closest other skill (Theorem 3.1), with a direct LSEPIN-to-WAC relation under assumptions of downstream distributions far from the average and a large state space (Theorem 3.2). Replacing KL with Wasserstein distance, the paper shows that maximizing WSEP places all positive-probability skills at vertices (Lemma 3.3), bounds a mean adaptation cost, and—via the iterative projection PWSEP—discovers all $\lvert V\rvert$ vertices of the polytope with $\lvert V\rvert$ policies (Theorem 3.5).

Load-bearing premise

The chain from LSEPIN to lower worst-case adaptation cost assumes that downstream tasks' optimal state distributions lie far from the average skill distribution and that the state space is large; if either assumption fails, the per-skill adaptation costs cannot be treated as equal contributors to the worst-case bound.

Editorial extensions

If this is right

  • Optimizing LSEPIN on top of a mutual-information objective should make every skill individually informative and separable from the others, which under the paper's assumptions lowers the worst-case cost of adapting from the closest learned skill.
  • Maximizing WSEP discovers more vertices of the feasible polytope than MISL, so more unseen reward functions will have one of the learned skills as a near-optimal initialization.
  • PWSEP solves the vertex-discovery problem: with one skill per vertex it can recover every optimal initial policy, which MISL cannot do when vertices are not all on the maximum-KL circle around the average distribution.
  • The theory covers the practically used adaptation procedure in which a downstream policy is initialized from a single learned skill, not from the average state distribution of the skill set.
  • For implementations, Wasserstein-based skill objectives admit a dual/GAN-style estimator, so the geometric guarantees transfer to parametric skill-learning algorithms.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the LSEPIN-to-WAC link holds, LSEPIN could serve as a reward-free model-selection signal: between two pretrained skill sets with equal mutual information, the one with higher LSEPIN should be preferred for downstream fine-tuning, and that preference is testable without any task reward.
  • The vertex-discovery guarantee positions PWSEP as a compact alternative to successor-feature transfer methods that may need over-complete skill sets; the paper notes the contrast, but the practical consequence of building smaller pretraining libraries is left implicit.
  • The hand-chosen transport cost is the next bottleneck: the paper's guarantees live at the level of the state-distribution polytope, and they transfer to realistic continuous environments only once a representation makes distances such as L2 reflect reachability, which is the paper's own stated future direction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper studies unsupervised skill discovery in reward-free MDPs and makes three interconnected claims. It introduces LSEPIN, a per-skill mutual-information metric meant to capture diversity and separability, and relates it to a worst-case adaptation cost WAC under assumptions that are stated in Theorem 3.2. It then replaces KL divergence by Wasserstein distance and defines objectives AWD, WSEP, and PWSEP. The central theoretical assertions are Lemma 3.3, which claims that maximizing WSEP forces all learned skills to vertices of the feasible-state-distribution polytope, and Theorem 3.5, which claims that the iterative PWSEP procedure discovers all such vertices. Proofs are given in Appendices C and G, and Appendices H and I contain practical estimators and empirical illustrations.

Significance. If the main results were correct, the paper would provide a useful theoretical complement to MISL: LSEPIN gives a concrete per-skill separability measure, WAC offers a task-oriented adaptation cost, and the WSEP/PWSEP objectives give a candidate answer to the vertex-discovery question raised by Eysenbach et al. (2022). The paper is also unusually explicit about its limitations, including Remarks 3.2.1 and 3.4.1 and the example in Appendix G.7 showing that WSEP can fail to discover all vertices. However, the central geometric claims for WSEP and PWSEP currently rely on a strict-convexity property of the Wasserstein distance that does not hold for the 1-Wasserstein distance used in the paper's objectives and estimators. As stated, the theorems do not establish the advertised vertex guarantees.

major comments (2)
  1. [3.3.4 / Appendix G.4 / Eq. (74)] Theorem 3.5 inherits the same strict-convexity gap. The proof that each PWSEP iteration discovers a new vertex uses the strict inequality in Eq. (74), which again presupposes strict convexity of W in the projected argument. With W1, a non-vertex can achieve the same projected distance as the best vertex: in the two-point example above, the interior point z has W1(z,q)=0.5, equal to the weighted average of the vertex distances, so Eq. (74) becomes an equality. Consequently the claim that PWSEP discovers all vertices is not established for the Wasserstein distance appearing in Eq. (15). The practical implementation in Appendix H.3 optimizes the same projection and does not restrict to W2^2, so this is not merely a gap in a separate theoretical variant.
  2. [3.2 / Appendix C.3 / Theorem 3.2] Theorem 3.2 is not backed by a complete proof. The theorem's assumptions, namely that the downstream optimal state distribution is 'far from p(S)' and that |S| is large, are never quantified, so the claimed implication cannot be checked. More importantly, the proof in Appendix C.3 uses a degree-of-freedom argument after Eq. (58) that is only sketched: it asserts that the maximizer r* is determined by |S|-2 degrees of freedom and that |Z'|=|S|-2, but no rigorous argument is given for this counting. The statement in the proof that Dm(z) >= Cm 'by definition' is also not justified by Eqs. (45), (46), and (48), since KL divergence is not monotone with respect to averaging in the first argument. Because Theorem 3.2 is the bridge between LSEPIN and the practical WAC claim, this gap weakens the paper's second main contribution as well.
minor comments (6)
  1. [3.3.3 / Remark 3.4.1] Remark 3.4.1 contains an incomplete sentence: 'WSEP as a disentanglement metric promotes the distances between learned skills and These two bounds of MAC...' The passage should be reworded so that the thought about the two bounds is expressed as a complete sentence.
  2. [3.3.1 / Eq. (10)] Equation (10) has a typographical parenthesis mismatch: the WSEP formula reads 'W(p(S|zi), p(S|zj)))' with an extra closing parenthesis.
  3. [Appendix H.2] The sentence 'Then, pertaining skill z by the intrinsic reward ...' should read 'pretraining skill z' or 'training skill z'; 'pertaining' appears to be a typo.
  4. [Appendix H.5] The notation H(S;1z) in Eq. (99) is inconsistent with the conditional entropy notation H(S|1z) used in Eq. (92); if this is intentional, it should be defined, otherwise it should be corrected.
  5. [Table 1 / Appendix H.1] Table 1 marks the 'Practical objective' for I(S;1z) as 'No', but Appendix H.1 explicitly describes an intrinsic reward that increases LSEPIN/I(S;1z). The table should clarify whether 'No' means 'no standalone objective in the paper' rather than 'not implementable'.
  6. [Appendix I.2 / Table 5] The correlation coefficients in Table 5 are computed from only three agents (Random, MISL+nLSEPIN, DADS origin). With three data points, a 'strong correlation' claim is not statistically meaningful; the table should be described as illustrative only.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the main theoretical claims follow from stated geometric assumptions and external prior results, with correctness gaps but no self-referential reduction.

full rationale

The paper's central claims—LSEPIN relates to worst-case adaptation cost, WSEP promotes vertex discovery, and PWSEP can discover all vertices—are derived from the geometry of the feasible state-distribution polytope under explicitly stated assumptions. No parameter is fitted and then relabeled as a prediction: the WAC and MAC bounds are inequalities proved from definitions such as eqs. (4) and (11), not from fitted constants. The MISL vertex facts are imported from Eysenbach et al. (2022), an external source with no author overlap, and the paper attempts its own uniqueness argument in Appendix E; there is no load-bearing self-citation chain. The main weakness is mathematical correctness rather than circularity: Lemma 3.3 and Theorem 3.5 require strict convexity of the Wasserstein distance, while the objectives in eqs. (10) and (15) use W, and Appendix G only establishes strict convexity for W2^2, with Appendix H.2 implementing the 1-Wasserstein dual. This is an unsupported assumption, not an equivalence between input and output. The paper also explicitly acknowledges limitations, including Appendix G.7 ('WSEP can not always discover all vertices') and Appendix I.1's caveat that local optima may limit PWSEP in practice. These weaken the guarantees but do not make the derivation circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The theory relies on the convex polytope geometry of feasible state distributions (from prior work), strict convexity of Wasserstein distance, and the practical but unverified assumption that downstream optima are far from the average skill distribution. No fitted parameters or new physical entities are introduced.

assumptions (4)
  • domain assumption The set C of feasible state distributions of the MDP is a convex polytope.
    Adopted from Eysenbach et al. (2022) Lemma 6.5, used throughout Section 2.2 and all geometric results.
  • standard math For any downstream reward r, the optimal state distribution lies at a vertex of C.
    Linear objective over a convex polytope (Boyd & Vandenberghe, 2004); used to define vertices as optimal policies for potential downstream tasks.
  • standard math The Wasserstein distance (or its square) is strictly convex.
    Used in proof of Lemma 3.3 and G.1; the paper cites Carlen & Gangbo (2003) for W^2 strict convexity, but uses generic W in the main claims.
  • domain assumption Downstream task optimal state distributions are far from the average state distribution p(S) and |S| is large.
    Assumption of Theorem 3.2 to treat every IC_z equally and connect LSEPIN to WAC; justified heuristically in Appendix C.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning." pith.science (2026). https://pith.science/paper/CUE66NQ4

@misc{pith2026250610629,
  author       = {Pith},
  title        = {Pith review of: Task Adaptation from Skills: Information Geometry, Disentanglement, and New Objectives for Unsupervised Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CUE66NQ4}},
  note         = {Machine review of arXiv:2506.10629}
}
read the original abstract

Unsupervised reinforcement learning (URL) aims to learn general skills for unseen downstream tasks. Mutual Information Skill Learning (MISL) addresses URL by maximizing the mutual information between states and skills but lacks sufficient theoretical analysis, e.g., how well its learned skills can initialize a downstream task's policy. Our new theoretical analysis in this paper shows that the diversity and separability of learned skills are fundamentally critical to downstream task adaptation but MISL does not necessarily guarantee these properties. To complement MISL, we propose a novel disentanglement metric LSEPIN. Moreover, we build an information-geometric connection between LSEPIN and downstream task adaptation cost. For better geometric properties, we investigate a new strategy that replaces the KL divergence in information geometry with Wasserstein distance. We extend the geometric analysis to it, which leads to a novel skill-learning objective WSEP. It is theoretically justified to be helpful to downstream task adaptation and it is capable of discovering more initial policies for downstream tasks than MISL. We finally propose another Wasserstein distance-based algorithm PWSEP that can theoretically discover all optimal initial policies.

Figures

Figures reproduced from arXiv: 2506.10629 by the authors.

Figure 1
Figure 1. Visualized examples: (a) C is the feasible state distribution set, and the blue simplex is the probability simplex for state distribution with |S| = 3. (b) MISL discovers 3 skills at the vertices {z1, z2, z3} on the “circle” with maximum “radius” centered in their average state distribution p(S). Prior work Eysenbach et al. (2022) shows that the set C of state distributions feasible under the dy￾namics of the MDP co… view at source ↗
Figure 2
Figure 2. Example of concyclic ver￾tices: z1, z2, z3, z4, z5 can all be vertices optimal for MISL, p(S) can be identified by |S| = 3 of them, so eq. (2) can be solved by any 3 of these vertices. The blue arrow is to adapt from learned skills to target distribution for downstream task (our setting) and the brown ar￾row is to adapt from the average state distribution (setting in Eysen￾bach et al. (2022)) We have answered this q… view at source ↗
Figure 3
Figure 3. Example of two MISL solutions: Z ∗ 1 : {z1, z2, z3} and Z ∗ 2 : {z1, z′ 2 , z′ 3}, Z ∗ 2 has higher I(S; 1z1 ) We provide a theoretical justification for the proposed disentan￾glement metric, showing that it can be a complement of I(S;Z) to evaluate how well the URL agent is prepared for downstream tasks by the following theorems. Definition 3.1 (Worst-case Adaptation Cost). Worst-case Adap￾tation Cost (WAC) is defi… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Example MISL with uniform p(Z) does not promote diversity For a case of with |S| = 3, the feasible polytope C allows a maximum "circle" centered at [0.2, 0.4, 0.4] with a maximum "radius" (I(S, Z)) of 0.253 as this fig in shows. There are 5 ver￾tices v1:5 of C, of whic…
Figure 5
Figure 5. Figure 5: A polytope in a 3-dimensional space, there are 4 vertices. In fig. 5. It shows a polytope with 4 vertices in a 3-dimensional space. Suppose in a situation where two skills are learned, and the learned skills are at vertices a and c. This situation would be optimal for …
Figure 6
Figure 6. Figure 6: Example when WSEP discovers more vertices than MISL [PITH_FULL_IMAGE:figures/full_fig_p029_6.png]
Figure 7
Figure 7. Figure 7: Trajectory examples: Trajectories of different skills are in different colors. (a) is learned [PITH_FULL_IMAGE:figures/full_fig_p033_7.png]
Figure 8
Figure 8. Figure 8: Trajectory examples: Trajectories of different skills are in different colors. (a) is a randomly [PITH_FULL_IMAGE:figures/full_fig_p034_8.png]
Figure 9
Figure 9. Figure 9: Trajectory examples: Trajectories of different skills are in different colors. (a) is learned by [PITH_FULL_IMAGE:figures/full_fig_p035_9.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning

    cs.LG 2025-07 conditional novelty 4.0 of 10

    Contrastive Successor Features recover ground-truth RL states up to a linear map whenever the skill-conditioned transition differences follow a von Mises-Fisher distribution and policies are diverse.

Reference graph

Works this paper leans on

18 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    I(S; 1z) can be used to evaluate every single skill

    work page

  2. [2]

    Besides informativeness, I(S; 1z) explicitly encourages separability, more details about separability are discussed in appendix F. B.4 W HY NOT REPLACE THE WASSERSTEIN DISTANCE IN WSEP OR PWSEP WITH KL DIVERGENCE ? Because KL divergence does not satisfy the triangle inequality, maximizing the replaced WSEP objective could jeopardize the diversity of learn...

    work page

  3. [3]

    largest circle

    All Zi share the same skill z In this proof, we use z to denote the skill shared among the skill sets and z′ ∈ Zi to denote a general skill in set Zi. A MISL solution is a skill distribution p(Z) with N skills having positive probabilities. Different solutions have the same p(S) = Ez′∈Zi p(S|z′). Because for MISL solutions, every learned skill z′ ∈ Zi sho...

    work page 2022

  4. [5]

    The downstream task favored state distribution is far from the average state distribution p(S)

    work page

  5. [6]

    Both could commonly happen in high-dimensional practical environments

    The state space is large. Both could commonly happen in high-dimensional practical environments. When the contribution of IC z to WAC can be treated equally for each skill z, the disentanglement metric I(S; 1z) for each skill z can be considered equally important. Practically, existing unsupervised skill learning algorithms such as He et al. (2022); Durug...

    work page 2022

  6. [7]

    All Zi share the same average state distribution p(S) = Ez′∈Zi [p(S|z′)]

    work page

  7. [8]

    DKL(p(S|z′) ∥ p(S)) is constant for all z′ in all Zi

    work page

  8. [10]

    The sum of KL divergences from learned skills to the optimal target distribution for a downstream task is constant

    work page

Show all 18 references
  1. [11]

    The upper bound of IC z depends this constant and p(z), and the upper bound decreases monotonically with higher p(z)

  2. [12]

    distance

    Increasing I(S; 1z) results in higher p(z) thus lower upper bound of IC z. We can see that for a solution Zi the weighted sum of the KL divergence between the optimal vertex of considered downstream task r ∈ Rz and the MISL learned skills is:X z′∈Zi p(z′)DKL(p(S|z′) ∥ pr) = X ...

    2024

  3. [13]

    All Zi share a common skill z and the same average state distribution p(S) = Ez′∈Zi [p(S|z′)]

  4. [14]

    circle" centered at [0.2, 0.4, 0.4] with a maximum

    DKL(p(S|z′) ∥ p(S)) is constant for all z′ in all Zi. In practice, there could be a gap between optimal maxp(Z) Ep(Z)DKL(p(S|z) ∥ p(S)) and a learned suboptimal dm = maxi Ez∈Zi DKL(p(S|z) ∥ Ez′∈Zi [p(S|z′)]), where Zi are the skill sets that can be learned. This gap could be a...

    2021

  5. [15]

    (2022), MISL objective is equivalent to minp(S) maxz DKL(p(S|z) ∥ p(S))

    By Lemma 6.5 of Eysenbach et al. (2022), MISL objective is equivalent to minp(S) maxz DKL(p(S|z) ∥ p(S))

    2022

  6. [16]

    Because of the strict convexity of KL divergence,maxz DKL(p(S|z) ∥ p(S)) is also strictly convex

  7. [17]

    center", the

    p(S) is constrained in a convex polytope C. Therefore it is a convex optimization problem with a strictly convex objective function. The solution is unique. With unique p(S) to be the "center", the "circle" of maximum "radius" DKL(p(S|z) ∥ p(S)) is also unique. F D IFFERENCE B...

    2021

  8. [508]

    Distance

    Springer, 2003. doi: 10.1007/3-540-45071-8\_50. URL https://doi.org/10.1007/ 3-540-45071-8_50 . Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Cather- ine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: unsupervised reinforcement learning benchmark. In J...

    2022 arXiv

  9. [2014]

    doi: 10.1017/CBO9780511804441

    ISBN 978-0-521-83378-3. doi: 10.1017/CBO9780511804441. URL https://web. stanford.edu/%7Eboyd/cvxbook/. Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR ...

    arXiv

  10. [2019]

    distance

    and Projected Wasserstein distance (PWD) (Rowland et al., 2019) have been demonstrated useful for unsupervised skill discovery. Here we propose an alternative approach to inspire future 31 Published as a conference paper at ICLR 2024 algorithm design. We can parametrize a test...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.