Pith. sign in

REVIEW 4 major objections 5 minor 5 references

The paper claims that a hierarchical successor representation, built from temporally extended options, yields state features that stay stable when rewards change, and that factoring them with non-negative matrix factorisation produces a spa

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:43 UTC pith:LX3BNNUA

load-bearing objection Solid incremental extension of SR to options, but the headline stability claim outruns the proof and experiments; worth refereeing with revisions. the 4 major comments →

arxiv 2602.12753 v2 pith:LX3BNNUA submitted 2026-02-13 cs.LG

Hierarchical Successor Representation for Robust Transfer

Changmin Yu , M\'at\'e Lengyel This is my paper
classification cs.LG MSC 68T05
keywords hierarchical successor representationtemporal abstractionoptions frameworknon-negative matrix factorisationtransfer learningstate representationintrinsic motivationreinforcement learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes replacing the classical successor representation, which tracks expected occupancy under a specific policy and goes stale when that policy changes, with a hierarchical version that predicts occupancy under temporally extended 'options' rather than single actions. Because the high-level strategy of moving through bottlenecks between rooms stays similar across tasks even when low-level actions differ, HSR features change less when the reward changes. Applying non-negative matrix factorisation to the HSR produces a sparse, localised, low-rank basis that aligns with topological structure such as bottlenecks. The authors show this basis transfers to new goal locations with many fewer episodes than SR-based features, and that the same structure powers efficient exploration in large random mazes.

Core claim

The central claim is that incorporating temporal abstraction into the successor representation removes the policy dependence that limits classical SRs: HSR features, built from eigenoptions, are stable under task-induced policy changes, and their piecewise-smooth geometry makes them uniquely amenable to non-negative matrix factorisation. The resulting HSR-NMF basis is sparse, localised, and interpretable, over-represents bottleneck states, and supports sample-efficient transfer across reward configurations in multi-compartment environments. The paper further claims that the temporally extended predictive structure can drive intrinsically motivated exploration that scales to procedurally gene

What carries the argument

The Hierarchical Successor Representation: a matrix whose rows give expected discounted occupancy of each state when following a high-level policy over options, defined by the Bellman recursion M = B + G M, where B averages intra-option successor representations and G is the discounted continuation kernel (F = γ M diag(β)). Eigenoptions are discovered from principal eigenvectors of a random-walk SR via pseudo-rewards. Non-negative matrix factorisation of the HSR then extracts the sparse, localised basis. The paper also defines an online TD update rule for HSR and proves the HSR Bellman operator is a contraction.

Load-bearing premise

The method assumes that a fixed set of eigenoptions, discovered once from a random-walk successor representation, remains the right temporal abstraction for every task, and that the optimal high-level policy over those options changes only mildly across the task distribution.

What would settle it

In the four-room setup, construct two pretraining tasks with opposite high-level routing (e.g., G1 in the left room, G2 requiring the agent to first visit the right room), average the HSRs, and measure the relative matrix change and episodes-to-optimal after transfer. If the transfer advantage persists, the stability does not depend on shared high-level strategy; if it collapses, the claim's boundary is confirmed.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Agents using HSR row features re-adapt to a new goal in a four-room maze in significantly fewer episodes than agents using standard SR features, because the HSR matrix changes less when the policy changes.
  • HSR-NMF basis vectors tile the state space sparsely and concentrate on bottleneck states, avoiding the interference of global spectral bases that makes value updates in one region hurt another.
  • HSR-NMF matches the value-function reconstruction efficiency of SVD-based SR features despite its heavy-tailed spectrum, so the positivity constraint costs little.
  • HSR-based intrinsic rewards cover more of a large random maze than SR-based intrinsic rewards within a fixed interaction budget, with the gap growing with maze size.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the stability argument hinges on the high-level strategy being shared across tasks, a natural stress test is a task family where the optimal first-room choice reverses across pretraining tasks; if HSR transfer collapses there, the boundary of the claim is confirmed.
  • The over-complete NMF direction suggested in the paper could give a principled way to alias repeated substructures in memory-limited agents, since sparse codes can index recurring rooms separately.
  • If HSR-like features appear in biological agents, one would predict neural predictive maps should be more stable to reward revaluation than to changes in transition structure; that is a testable dissociation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes the Hierarchical Successor Representation (HSR), an extension of the classical successor representation that conditions predictive occupancy on a high-level policy over temporally extended options rather than on primitive actions. The authors define an HSR Bellman recursion, prove contraction for a fixed high-level policy, and introduce an 'expected HSR' (eHSR) obtained by averaging task-specific HSRs over pretraining tasks. They then apply non-negative matrix factorization (NMF) to HSR/eHSR to obtain sparse, localised state features, and compare these against SVD and SR-based baselines in four-room transfer experiments, reconstruction analyses, and procedurally generated maze exploration tasks. The central claims are that HSR features are stable under task-induced policy changes and that HSR-NMF provides a policy-agnostic, low-rank, topologically interpretable representation that supports sample-efficient transfer and scalable exploration.

Significance. If the main claims hold, the paper makes a worthwhile contribution to representation learning in RL: it combines temporal abstraction with predictive representations in a way that is analytically tractable, and it provides empirical evidence that the resulting features are more stable than SR features in at least one transfer scenario. The derivation of the HSR Bellman recursion, the contraction proof for fixed µ, the use of 20-seed experiments with reported effect sizes, and the interpretability analyses (bottleneck activation, basis trace plots) are concrete strengths. However, the significance is tempered by the fact that the core 'policy-agnostic' claim rests on a stability property that is neither proven nor tested in regimes where the optimal high-level strategy changes; by the use of a single four-room goal-switch as the main transfer evidence; and by the fact that the main transfer experiments use a full-rank basis (K=N), so the low-rank claim is not actually exercised.

major comments (4)
  1. [§3, Theorem 3.1 and Algorithm S1] The central claim of robustness to task-induced policy changes is not supported by the theoretical result. Theorem 3.1 proves only that the HSR Bellman operator for a fixed high-level policy µ is a contraction. It does not provide any bound on ||M^{µ1} − M^{µ2}|| when a reward change shifts the optimal µ, which is precisely the quantity needed for the 'robust to task-induced policy changes' claim. eHSR (Algorithm S1, lines 15–20) averages HSRs computed from optimal policies over pretraining tasks, but no theorem or analysis shows that this average is stable across unseen tasks, or that unseen-task optimal high-level policies are close to the average. This is a load-bearing gap: without it, the paper's 'policy-agnostic' language overstates what a fixed-option, pretraining-dependent construction can guarantee.
  2. [§4.1, Figure 2] The empirical support for the robustness claim is limited to a single G1→G2 switch in the four-room environment, with goal locations explicitly selected not to coincide with eigenoption terminal states. In this setting the optimal high-level strategy (go through the corridor into the goal room) is essentially unchanged, so the experiment does not test cases where a reward change makes a previously useful option's initiation or termination set useless, or where the optimal high-level policy changes qualitatively. Without such a case, the observed HSR stability may be an artefact of the particular task pair rather than a general property. Additional tasks where the optimal option-usage pattern changes are needed before the 'robust transfer' claim can be accepted.
  3. [Appendix C.1] The main transfer experiments (Figure 3 and the 'low-rank basis' claims in §3 and §4.2) use K=N=104 basis vectors, i.e., a full-rank decomposition. This means the 'low-rank' claim is not actually exercised in the central comparisons: NMF with K=N imposes no compression, so the reported advantage of HSR-NMF over HSR-SVD and SR baselines is not a demonstration of the benefits of a low-rank basis. Figure 4 does vary basis size for reconstruction, but the sample-efficiency transfer results use full rank. The paper should either run transfer experiments with K<<N or explicitly restrict the low-rank claim to the reconstruction analyses.
  4. [§3, Eq. (10) and §4.2] The claim that NMF discovers meaningful 'parts' of the HSR relies on the separability conditions of Donoho & Stodden (2003), but the paper does not verify that these conditions hold for HSR matrices. The empirical sparsity and bottleneck alignment of HSR-NMF bases are suggestive, but the argument that 'the locally dispersed geometry of the HSR enables the NMF to discover the ground-truth underlying generative parts' is asserted rather than demonstrated. At minimum, the paper should justify why HSR matrices satisfy the relevant conditions, or soften the 'ground-truth parts' language.
minor comments (5)
  1. [Introduction] Typo: 'one-step lienar update' should be 'one-step linear update'.
  2. [Figure 1, caption (j)] The caption says 'Same as j, but for the RW-SR matrix' for panel j; this appears to refer to panel i and should be corrected.
  3. [Appendix C.1] Typo: 'SVD/NMD basis' should be 'SVD/NMF basis'; also 'tiemsteps' in the offline transfer paragraph.
  4. [§4.2 / Figure 3] The paper compares HSR against SR and eSR baselines but not against successor features with generalized policy improvement (Barreto et al., 2017; 2018), despite the related-work section arguing that successor features suffer from diffusive features. Adding such a baseline would strengthen the claim that HSR is the method of choice for transfer.
  5. [§3, Expected HSR] The eHSR construction depends on a particular set of pretraining tasks G, but there is no sensitivity analysis with respect to the size or diversity of G. Since the method's robustness is supposed to come from averaging, it would be informative to report how the representation and downstream transfer change as G is varied.

Circularity Check

0 steps flagged

No circularity: HSR is constructed from the MDP/options and eHSR averages pretraining tasks; G2 transfer is a genuine out-of-sample prediction.

full rationale

I walked the derivation chain from Eq. 5 through Algorithm S1 and the transfer experiments. The HSR matrix is defined by the recursive Bellman equation over a fixed option set, and the contraction result (Theorem 3.1) is a standard fixed-point property for a given high-level policy. No parameter is fit to the G2 task: the expected HSR is averaged over pretraining tasks only, and the G2 value reconstruction and transfer results use features computed before seeing the G2 reward, so they are genuine predictions. The only self-citation, Yu et al. (2023), is used as a baseline for exploration in Section 4.3 and is not load-bearing for the HSR construction. The paper's own limitation statement, 'limited to transferring between tasks with shared transition dynamics,' is a scope restriction; the fact that Theorem 3.1 does not itself bound cross-policy changes in the HSR is a gap in the strength of the robustness claim, but it is not a circular step. Thus the central claim does not reduce by construction to its inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 1 invented entities

The central contribution rests on standard MDP/SMDP machinery plus three structurally ad hoc-to-paper assumptions: the fixed eigenoption set, the stability of the high-level policy across tasks, and NMF separability of HSR. The free parameters are standard RL hyperparameters plus the K=N basis choice that weakens the low-rank claim.

free parameters (5)
  • Number of eigenoptions K_options = 8
    Chosen by hand for all transfer experiments; affects the stability and expressiveness of HSR.
  • Number of NMF/SVD basis vectors K = 104 (= state count)
    Used in the main transfer experiments (Appendix C.1); the claimed 'low-rank' advantage is not exercised in the headline transfer results.
  • Discount factor gamma = 0.9 (transfer), 0.99 (exploration)
    Chosen by hand; contraction bound and representation geometry depend on it.
  • Intrinsic reward scaling lambda = 1 (pure exploration), 0.01 (sparse-reward navigation)
    Chosen by hand (Appendix C.2); exploration results depend on the balance between intrinsic and extrinsic reward.
  • Pretraining task set G = unspecified size/goals
    eHSR is an average over HSRs from optimal policies for pretraining tasks; the composition of G is not given, so transfer to 'novel' tasks cannot be fully audited.
axioms (5)
  • standard math Standard MDP, Bellman, and TD assumptions; discounting gamma < 1.
    Used throughout Sections 2–3.
  • domain assumption Eigenoption policies trained on pseudo-rewards provide a useful, task-agnostic option set.
    Section 2, Eq. 4; the stability of HSR rests on the permanence of these options across tasks.
  • ad hoc to paper High-level optimal policies across pretraining tasks share stable structure, so averaging HSRs (eHSR) yields a policy-agnostic basis.
    Section 3, 'Expected HSR'; Discussion admits limitation to shared dynamics.
  • ad hoc to paper HSR satisfies NMF separability conditions (Donoho & Stodden 2003) so that NMF recovers meaningful parts.
    Section 3, Eq. 10 and Fig. 1h; assumed, not proven.
  • standard math The option termination kernel factors as F^a = gamma M^a diag(beta^a).
    Appendix A.1 Eq. S2; correct under standard option termination semantics, but relies on M^a being the pre-termination occupancy.
invented entities (1)
  • Hierarchical Successor Representation (HSR) matrix independent evidence
    purpose: State-feature representation that aggregates predictive occupancy under high-level policies over options rather than primitive actions; used as the basis for transfer and exploration.
    Its behavioral consequences (stability, sparse NMF features, exploration efficiency) are falsifiable through the transfer and maze experiments in Section 4; however it is a mathematical construction, not a novel physical entity.

pith-pipeline@v1.3.0-alltime-deepseek · 14534 in / 16795 out tokens · 148557 ms · 2026-08-02T23:43:23.712250+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Hierarchical Successor Representation for Robust Transfer." pith.science (2026). https://pith.science/paper/LX3BNNUA

@misc{pith2026260212753,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Successor Representation for Robust Transfer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LX3BNNUA}},
  note         = {Machine review of arXiv:2602.12753}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The successor representation (SR) provides a powerful framework for decoupling predictive dynamics from rewards, enabling rapid generalisation across reward configurations. However, the classical SR is limited by its inherent policy dependence: policies change due to ongoing learning, environmental non-stationarities, and changes in task demands, making established predictive representations obsolete. Furthermore, in topologically complex environments, SRs suffer from spectral diffusion, leading to dense and overlapping features that scale poorly. Here we propose the Hierarchical Successor Representation (HSR) for overcoming these limitations. By incorporating temporal abstractions into the construction of predictive representations, HSR learns stable state features which are robust to task-induced policy changes. Applying non-negative matrix factorisation (NMF) to the HSR yields a sparse, low-rank state representation that facilitates highly sample-efficient transfer to novel tasks in multi-compartmental environments. Further analysis reveals that HSR-NMF discovers interpretable topological structures, providing a policy-agnostic hierarchical map that effectively bridges model-free optimality and model-based flexibility. Beyond providing a useful basis for task-transfer, we show that HSR's temporally extended predictive structure can also be leveraged to drive efficient exploration, effectively scaling to large, procedurally generated environments.

Figures

Figures reproduced from arXiv: 2602.12753 by Changmin Yu, M\'at\'e Lengyel.

Figure 1
Figure 1. Figure 1: Temporal abstraction yields hierarchical successor representation. a. Schematic of the computational process underlying the construction of hierarchical successor representations, and corresponding low-dimensional basis through NMF(Φ) and singular value decomposition (SVD; V). b. Exemplar pretraining regimes (G). c. SR matrix corresponding to the random-walk policy (RW-SR). Note that state indices were per… view at source ↗
Figure 2
Figure 2. Figure 2: HSR provides a stable state representation and enables sample-efficient transfer across tasks with shared transition dynamics. a. Exemplar four-room environment, with a fixed start location and two different goal locations. b. Training curves (number of steps to reach the goal location; mean ± s.e.) for Q-learning agents with linear function approximation, given different state representations (left: one-h… view at source ↗
Figure 3
Figure 3. Figure 3: NMF basis of HSR supports sample-efficient transfer. a. Training curves (left) and number of training episodes to reach optimal performance (right) in G1 tasks (Figure 2a) for Q-learning agents with linear function approximation, given different low-dimensional basis as state representations. All agents were assumed to have received necessary pretraining for constructing base matrices (SR/HSR) before corre… view at source ↗
Figure 4
Figure 4. Figure 4: HSR-NMF basis yield a sparse, robust, and interpretable state representation. a. Example trajectory in the four-room environment. b. Activation (normalised) of all basis at each timestep along the example trajectory for eSRSVD, eSRNMF, HSRSVD, HSRNMF (from left to right). Gray numbers below the rightmost panel indicates which room the corresponding trajectory segment is in. c. Reconstruction mean-squared e… view at source ↗
Figure 5
Figure 5. Figure 5: Hierarchical temporal abstraction enables scalable intrinsically motivated exploration. a. Exemplar procedurally generated random maze environment. b. Learning curves (mean ± s.e.) of different agents (see main text) in terms of pure exploration (in the absence of extrinsic reward; left) and goal-directed navigation (with only non-zero reward at randomly selected goal locations; right). c. Asymptotic state… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

5 extracted references · 4 linked inside Pith

  1. [1]

    C., Castro, P

    Agarwal, R., Machado, M. C., Castro, P. S., and Bellemare, M. G. Contrastive behavioral similarity embeddings for generalization in reinforcement learning.arXiv preprint arXiv:2101.05265,

    Pith/arXiv arXiv

  2. [2003]

    C., and Bowling, M

    Farebrother, J., Machado, M. C., and Bowling, M. Gen- eralization and regularization in dqn.arXiv preprint arXiv:1810.00123,

    Pith/arXiv arXiv

  3. [2019]

    M., Schaul, T., Leibo, J

    9 Hierarchical Successor Representation Jaderberg, M., Mnih, V ., Czarnecki, W. M., Schaul, T., Leibo, J. Z., Silver, D., and Kavukcuoglu, K. Reinforce- ment learning with unsupervised auxiliary tasks.arXiv preprint arXiv:1611.05397,

    Pith/arXiv arXiv

  4. [2020]

    ∞X t=0 γt1(st, s′)|s0 =s # = X ¯a∈¯A µ(¯a|s) Eπ¯a

    11 Hierarchical Successor Representation A. Derivations and Proofs A.1. Derivation of HSR. As a reminder, the HSR is defined as the expected discounted future occupancy under some high-level policy, µ:S × ¯A → [0,1], where ¯A=A ∪Ωdenotes the augmented action space. Mµ ss′ =E µ " ∞X t=0 γt1(st, s′)|s0 =s # = X ¯a∈¯A µ(¯a|s) Eπ¯a " τs¯aX t=0 γt1(st, s′)|s0 ...

    2023

  5. [2023]

    Learning invariant representations for rein- forcement learning without reconstruction.arXiv preprint arXiv:2006.10742,

    Zhang, A., McAllister, R., Calandra, R., Gal, Y ., and Levine, S. Learning invariant representations for rein- forcement learning without reconstruction.arXiv preprint arXiv:2006.10742,

    Pith/arXiv arXiv 2006