{"id":"7b4289a4-fdc4-4ec8-81b0-31c706236269","arxiv_id":"2507.14748","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Contrastive Successor Features recover ground-truth RL states up to a linear map whenever the skill-conditioned transition differences follow a von Mises-Fisher distribution and policies are diverse.","lead":"This paper proves that a popular mutual information skill learning method, Contrastive Successor Features, can recover the true underlying states of an environment up to a linear transformation. The result explains why diverse skills and inner-product critics make these methods work, and it is the first identifiability guarantee for representation learning in reinforcement learning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proof is conditional on Assumption 1, yet the paper never establishes that CSF induces the assumed vMF/uniform/diverse distributions of ground-truth state differences; its cited evidence is about learned features, so the identifiability guarantee is not shown to apply to CSF.","rationale":"The single most load-bearing condition for the central claim is not the linear algebra inside Thm. 1 (taken from [49]) but whether CSF's training dynamics produce observations satisfying Assumption 1. The reader identified Assumption 1(iv) as weakest; I agree that diversity is unproven, but the more specific and decisive problem is that Assumption 1(ii)-(iii) are stated for ground-truth state differences while the only offered evidence is about learned feature differences from prior work [71]. This is not a matter of disagreeing with the community's consensus on vMF features; it is an internal mismatch: the theorem's DGP is about s'-s, and using phi to validate it is circular unless one already knows phi is linear. Lemma 1 is insufficient for (iv) because excluding the uniform policy leaves many non-diverse policies; the reward can be high for a small set of aligned directions without making skills uniquely decodable from transitions. The proposed test directly measures the quantities Assumption 1 names. If the test fails, the paper's contribution reduces to a conditional theorem plus four empirical R2 curves, and the abstract's 'provably recovers' would overstate the result. If the test passes, the conditional verdict can be upgraded. Since the reader already issued a conditional verdict and flagged closely related issues, my read does not move the verdict.","tokens_in":17332,"tokens_out":7425,"duration_ms":88601,"concrete_test":"Using the authors' Fig. 3 checkpoints and code, fix the learned encoder and the final skill-conditioned policies. For each skill z, sample rollouts and collect ground-truth consecutive states (s,s'); standardize d=s'-s to the unit sphere. Then (1) fit a vMF density to each skill's d and compare held-out log-likelihood with a uniform and a projected-Gaussian baseline; (2) test uniformity of the pooled marginal of d over skills (e.g., Rayleigh/energy-distance test); (3) train a flexible MLP to predict z from (s,s') and compare accuracy with the chance level. If vMF fit, uniformity, or discriminability fail at conventional thresholds, Assumption 1(ii)-(iv) is not met on the very data used to validate the theorem, so the high R2 cannot be taken as confirmation of Thm. 1's applicability. If the checks pass, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Thm. 1 is imported from [49]; the novel step is showing CSF satisfies its premises. Assumption 1(ii)-(iii) are about p(s'-s|z) for ground-truth state differences and their marginal over z. The feasibility paragraph in Sec. 3.1 instead cites [71, Fig. 2] showing that the learned feature differences phi(o')-phi(o) are vMF with uniform marginal. Because phi is learned, feature-level vMF/uniformity does not imply state-level vMF/uniformity; and before Thm. 1, phi is not known to be linear, so using phi to validate assumptions about s is circular. Assumption 1(iv) (diverse skills) is also assumed rather than derived: Lemma 1 only excludes the maximum-entropy policy, while the reward (3) aligning phi-differences with z does not by itself guarantee that an ideal discriminator can uniquely recover z from (s,s'). Thus the paper establishes a conditional statement ('if the data are vMF/uniform/diverse, features are identifiable up to a linear map'), not that CSF's optimization creates such data. The R2 results are consistent with the theorem but do not verify its premises on the quantities the theorem requires. A secondary gap: Prop. 1's proof infers phi(o)=As from difference identifiability, but does not rule out an additive offset, since s is not constrained to the sphere.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes Contrastive Successor Features (CSF), a mutual-information skill learning (MISL) method, through the lens of identifiable representation learning. It adapts an identifiability theorem from Reizinger et al. [49] to the RL setting, stating that under Assumption 1—which includes von Mises-Fisher skill-conditioned state differences, a uniform marginal over differences, and diverse policies—CSF's encoder recovers the ground-truth states up to a linear transformation. The authors also present Lemma 1 against maximum-entropy policies, discuss implications of different MI objectives, and empirically evaluate CSF in MuJoCo and DeepMind Control using R² scores between learned features and ground-truth states, state coverage, and oracle returns.","tokens_in":17570,"tokens_out":7310,"duration_ms":75562,"significance":"The paper is the first to attempt an identifiability guarantee for representation learning in RL, and it draws a useful connection between MISL and nonlinear ICA. The conditional theorem is imported and sound, and the practical insight about entropy regularization is interesting. However, the applicability of the theorem to CSF is not established: the assumptions concern the ground-truth state differences, while the paper's feasibility argument uses learned feature differences, and the diversity assumption is assumed rather than demonstrated. The work is therefore best viewed as a promising theoretical framework with a plausibility study, rather than a proof that CSF in practice identifies the ground truth.","major_comments":[{"comment":"The paper cites [71, Fig. 2] to justify Assumption 1(ii)-(iii), but that figure concerns the learned feature differences φ(o')−φ(o), whereas the assumption is about the ground-truth state differences s'−s. Since φ is unknown and not yet proven linear at the point of justification, the distribution of φ(o')−φ(o) does not provide evidence about the distribution of s'−s. This is circular: using the learned features to verify assumptions about the true states presupposes the linear identifiability the theorem aims to establish. The authors should either derive these properties from the training objective or measure them directly on the ground-truth states.","section":"§3.1, Assumption 1 feasibility items (ii) and (iii)"},{"comment":"Assumption 1(iv) asserts that an ideal discriminator can uniquely map (s'−s) to z, i.e., that the learned policies are diverse. Lemma 1 only rules out the maximum-entropy policy as a failure mode; it does not show that the policies obtained by optimizing the reward in Eq. (3) are diverse in the sense of Definition 1. The paragraph following Lemma 1 in the feasibility section is an intuitive argument, not a proof. The theorem requires this assumption to hold for the actual data distribution, so the paper needs to either prove it under additional conditions or empirically verify it (e.g., by training a discriminator on ground-truth state pairs from the learned policies and measuring its accuracy).","section":"§3.1, Assumption 1(iv) and Lemma 1"},{"comment":"The proof claims that an offset is impossible 'as they lie on S^{d-1}', but Assumption 1 only constrains the differences (s'−s) to the sphere, not the states s themselves. From φ(o')−φ(o)=A(s'−s) one can at most conclude φ(o)=As+c for a constant c. The identifiability statement should be amended to 'up to an affine transformation', or an additional assumption (e.g., that the features and states are centered, or that a reference state is known) must be added. As written, Proposition 1 is not proven.","section":"§3.1, Proposition 1 proof"},{"comment":"The paper claims that 'CSF can provably recover the environment's ground-truth features up to a linear transformation'. What is actually proven is a conditional statement: if Assumption 1 holds, then an encoder and linear classifier that globally minimize the cross-entropy objective are linearly identifiable. The paper does not show that CSF's objective produces data satisfying Assumption 1, and the empirical R² scores are consistent with but do not verify the assumptions. The claim is therefore stronger than the evidence supports; the manuscript should separate the conditional theorem from the claim of CSF's practical identifiability.","section":"Abstract and §3.1"}],"minor_comments":[{"comment":"The text says error bars represent two standard deviations unless otherwise noted, but the caption of Figure 3 says 'Error bars represent one standard deviation.' Please reconcile this discrepancy.","section":"§4 vs Figure 3 caption"},{"comment":"The statement that 'this setup is sufficient, but not necessary: a set of discrete skills also leads to high identifiability scores (Fig. 4)' is misleading without qualification: Fig. 4 shows high R² only for large discrete skill sets (100, 500), while small sets perform poorly. Please rephrase to clarify that diversity in terms of spanning the sphere is what matters.","section":"§3.1, feasibility (i)"},{"comment":"The proof of Lemma 1 is informal, particularly the step 'by reordering the terms, we get' and the final contradiction. It would benefit from a more explicit derivation of the expression for the reward under a skill-independent policy.","section":"§3.2, Lemma 1 proof"},{"comment":"The text states that a linear map A is fit by minimizing ||s − Aφ(o)||²₂. Given the potential offset issue in Proposition 1, the paper should state whether an intercept was included in the regression or justify why it is not needed.","section":"§4, linear fit"}],"recommendation":"major_revision","confidential_remarks":"The paper's novelty is largely derivative: Theorem 1 and Assumption 1 are imported from Reizinger et al. [49], and the empirical justification for the assumptions comes from Zheng et al. [71]; both works share authors with this manuscript. While the application to CSF and the analysis of MI objectives are new, the editor may want to verify that the overlap is properly disclosed and that this submission is sufficiently distinct from prior work by the same group. The self-citation pattern is heavy, and the feasibility argument in §3.1 relies on evidence from [71] rather than on new experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The genuinely new thing is the translation of CSF into the nonlinear ICA framework: modeling skills as auxiliary variables, using feature differences as the contrastive target, and showing why the inner-product critic and the absence of entropy regularizers matter. That translation is useful and mostly correct. The experiments back the conditional claim: when the assumptions hold, features are identifiable up to a linear map, and the R2 numbers in MuJoCo/DMC are consistent with that.\n\nThe soft spots are the assumptions. Theorem 1 is imported verbatim from Reizinger et al. [49], so the contribution lives in the feasibility arguments. Assumptions (ii) and (iii) claim the ground-truth state differences s'−s are vMF with uniform marginal, but the cited evidence is empirical plots of learned feature differences φ(o')−φ(o) from [71]. Since φ is not known to be linear before Theorem 1, feature-level vMF/uniformity does not imply state-level vMF/uniformity. That is a real circularity and it needs to be addressed head-on. Assumption (iv), diverse skills, is also assumed rather than derived; Lemma 1 only rules out the maximum-entropy policy, and the CSF reward (3) does not by itself guarantee that an ideal discriminator can uniquely recover z from (s,s'). Those are load-bearing because without (ii)-(iv) the ICA result doesn't apply.\n\nProposition 1 has a smaller gap: difference identifiability gives φ(o')−φ(o)=A(s'−s), which determines A but not an additive offset. The proof says an offset is impossible because the differences lie on the sphere, but that only constrains differences, not φ(o) itself. If the encoder outputs are on the unit sphere there may be an argument to be made, but it isn't in the proof.\n\nNone of this kills the paper. The conditional statement is correct, and the explanatory value is real: it gives a principled account of design choices that were empirical. But the abstract says CSF 'provably recovers' ground-truth features, and that promise is not met until the assumptions are shown to hold for the actual optimization. The limitations section is honest about some of this, but the framing still oversells.\n\nWho is it for? Researchers working on unsupervised skill discovery or representation learning in RL, and anyone interested in applying ICA-style identifiability guarantees to RL. It deserves peer review — a good referee will push for a revised abstract and for either direct evidence on state differences or a proof that the CSF objective enforces the diversity condition.","headline":"A sound conditional identifiability result for CSF, but the paper overstates its reach: the key assumptions are supported by empirical plots of learned features, not ground-truth state differences, so the guarantee is not actually shown to apply to CSF's optimization.","tokens_in":18163,"tokens_out":2985,"would_cite":true,"duration_ms":36392,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that diverse-skill pretraining in reinforcement learning recovers the ground-truth states of the environment up to a linear transformation, providing the first identifiability guarantee for representation learning in RL.","keywords":["mutual information skill learning","identifiable representation learning","contrastive successor features","nonlinear ICA","reinforcement learning","unsupervised skill discovery","linear identifiability","cross-entropy objective"],"falsifier":"In a new simulated environment with known ground-truth states, record the learned feature differences and test whether they follow the assumed von Mises-Fisher distribution with a uniform marginal on the hypersphere, and whether the skill can be decoded from the state pair. If a case is found where those assumptions clearly fail but the $R^2$ between learned features and ground truth still approaches 1, then the theorem's premises are not necessary for the observed identifiability; if the $R^2$ collapses whenever the assumptions fail, they are the load-bearing ingredient.","tokens_in":1704,"feed_emoji":"🤖","tokens_out":2605,"duration_ms":95937,"temperature":0.7,"pith_summary":"This paper aims to prove why mutual information skill learning (MISL) works in reinforcement learning, using the Contrastive Successor Features (CSF) method as the case study. The central claim is that when an agent learns a diverse set of skills by maximizing the inner product between skill vectors and feature differences, the learned features provably match the environment's ground-truth states up to a linear transformation. This is the first identifiability guarantee for representation learning in RL, and it explains why specific design choices—an inner-product critic, a contrastive (cross-entropy) objective, and diversity rather than entropy—are the ones that succeed. If true, it means self-supervised RL pretraining is not just a heuristic for exploration but a principled way to invert the environment's data-generating process.","feed_headline":"Skill diversity provably recovers RL states up to linear map","feed_subtitle":"First identifiability guarantee for RL: pretrained features equal true states up to a linear map.","key_machinery":"The central object is the Contrastive Successor Features (CSF) objective, in which a critic $q(z_i|\\phi(o),\\phi(o'))$ is trained with a contrastive (cross-entropy) loss to infer the skill $z_i$ from consecutive observations via the inner product $(\\phi(o')-\\phi(o))^\\top z_i$. The load-bearing identity is the identifiability theorem (Theorem 1, adapted from nonlinear ICA): under a von Mises-Fisher conditional distribution for feature differences, a uniform marginal on the hypersphere, and diverse skills that span $\\mathbb{R}^d$, a globally optimal encoder and linear classifier yield $\\phi(o')-\\phi(o) = A(s'-s)$. Proposition 1 extends this to the features themselves, $\\phi(o) = A s$, using the inner product parametrization. This theorem is what converts the heuristic of 'learning diverse skills' into a provable recovery of the ground-truth states.","core_discovery":"Under Assumption 1, the paper proves that the features learned by CSF are identifiable up to a linear map: if a continuous encoder and a linear classifier globally minimize the contrastive cross-entropy objective, then $\\phi(o') - \\phi(o) = A(s' - s)$ for a linear map $A \\in \\mathbb{R}^{d\\times d}$, and because the critic uses an inner product parametrization, the same $A$ relates the features themselves, $\\phi(o) = A s$. The proof adapts a known nonlinear-ICA identifiability result to the RL setting, interpreting skills as auxiliary variables and the required 'diverse policies' as the sufficient variability condition. The paper further shows that maximum-entropy policies cannot be diverse, that the choice of mutual information objective $I(s,s';z)$ versus $I(s;z)$ matters for the geometry of the latent space, and that a finite skill set must span the state space for identifiability. Experiments in MuJoCo and DeepMind Control confirm that the learned features and feature differences achieve high $R^2$ with ground-truth states, both from states and from pixels.","pith_inferences":["A natural extension is to test whether the same linear identifiability holds when the generator $g$ is non-injective or discontinuous; if empirical $R^2$ stays high, the injectivity assumption could be relaxed.","The paper's link between policy diversity and ICA's sufficient variability implies that adding an explicit diversity term to the policy loss, rather than letting it emerge from the reward, could make identifiability hold in environments where exploration is initially poor.","If the linear map $A$ is consistent across seeds and tasks, the learned features could serve as a fixed coordinate system for an agent, letting a downstream policy be trained directly in the ground-truth state space."],"forward_implications":["The learned CSF features can be used to read out the ground-truth states of the MDP with a linear map, which should make downstream task solving and planning easier.","Designing MISL methods away from these ingredients—for example replacing the inner-product critic with a more expressive one or adding a maximum-entropy regularizer—should degrade identifiability and, correspondingly, zero-shot transfer.","A finite set of skills is sufficient for identifiability only if the skill vectors form an affine generator system of the state space; using too few or non-spanning skills breaks the guarantee and reduces exploration.","The distinction between optimizing $I(s,s';z)$ and $I(s;z)$ is not cosmetic: only the transition-based mutual information imposes the locality constraint that keeps consecutive-state embeddings close but distinct.","In pixel-based environments, feature identifiability is still achieved, though the linear relationship for feature differences is weaker than for states."],"supporting_citations":[{"why":"Supplies the cross-entropy identifiability theorem (Theorem 1) that the paper adapts to CSF.","marker":"[49]"},{"why":"Defines the Contrastive Successor Features method whose objective, critic parametrization, and empirical feature distributions are analyzed and reproduced.","marker":"[71]"},{"why":"Provides the nonlinear ICA with auxiliary variables framework that motivates the diversity condition and the inner-product critic.","marker":"[18]"},{"why":"Shows that contrastive learning can invert the data generating process, a key connection between SSL and ICA used in the proof approach.","marker":"[72]"},{"why":"Is the specific self-supervised identifiability setup (DIET) whose assumptions Assumption 1 adapts.","marker":"[21]"}],"fun_headline_variants":["RL features identified up to linear map via skill diversity","Skill diversity guarantees identifiable RL representations","First identifiability proof for RL representation learning","Contrastive skills recover true states up to linear transform","Provable feature recovery in RL via policy diversity"],"cache_read_input_tokens":20224,"weakest_assumption_plain":"The proof assumes that the learned skill-conditioned policies are genuinely diverse, meaning an ideal discriminator can always infer which skill produced a given state transition; if the policies ever fail to be diverse, the identifiability conclusion collapses.","fun_headline_variants_meta":{"raw":{"variants":["RL features identified up to linear map via skill diversity","Skill diversity guarantees identifiable RL representations","First identifiability proof for RL representation learning","Contrastive skills recover true states up to linear transform","Provable feature recovery in RL via policy diversity"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000508,"raw_usage":{"total_tokens":2475,"prompt_tokens":948,"completion_tokens":1527,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":1456}},"tokens_in":564,"tokens_out":1527,"duration_ms":11813,"temperature":1.0,"reasoning_tokens":1456,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:49:48.835736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a new simulated environment with known ground-truth states, record the learned feature differences and test whether they follow the assumed von Mises-Fisher distribution with a uniform marginal on the hypersphere, and whether the skill can be decoded from the state pair. If a case is found where those assumptions clearly fail but the $R^2$ between learned features and ground truth still approaches 1, then the theorem's premises are not necessary for the observed identifiability; if the $R^2$ collapses whenever the assumptions fail, they are the load-bearing ingredient.","supporting_citations":[{"cited_title":"V ogt, Randall Balestriero, Wieland Brendel, and David Klindt","cited_arxiv_id":null,"evidence_quote":"Supplies the cross-entropy identifiability theorem (Theorem 1) that the paper adapts to CSF."}],"review_version":1}