Pith. sign in

REVIEW 2 major objections 4 minor 4 references

Slow Feature Analysis on Markov Chains from Goal-Directed Behavior

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Goal-directed behavior distorts optimal slow features near rewards, making them worse for value-function approximation than uniform features.

desk verdict Clean Markov-chain SFA derivation and a real scaling bound, but the headline empirical ordering is partly an artifact of evaluating MSE under the behavior-weighted stationary distribution; needs a fixed reference distribution before the practical claims hold. read the letter →

arxiv 2506.01145 v1 pith:NVIDBXTC submitted 2025-06-01 cs.LG

Merlin Schüler , Eddie Seabrook , Laurenz Wiskott This is my paper
classification cs.LG
keywords slowfeatureanalysisMarkovchainsgoal-directedbehaviorvaluefunctionapproximationstationarydistributionspectralembeddingsreinforcementlearningrepresentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that when Slow Feature Analysis (SFA) learns a representation from data collected by goal-directed behavior—exactly the kind of data a reinforcement learning agent produces—the optimal features inherit a scaling that is inversely proportional to the square root of how often each state is visited. This flattens features in the high-occupancy region around the reward and inflates them in rarely visited states. As a result, using these features as a linear basis for approximating the optimal value function performs worse than using features learned from a uniform random walk. Goal-averse behavior, which concentrates occupancy away from the reward, produces features whose scaling aligns with the value function and can outperform uniform features. The paper demonstrates this in one- and two-dimensional grid environments and evaluates three corrections: more even exploration, transition reweighting, and a post-hoc scale correction.

What carries the argument

The central object is the generalized eigenvalue problem $(D - M)Y = D Y \Lambda$, where $M$ is the symmetrized transition probability matrix $M_{uv} = \frac{1}{2}(\mu_u P_{uv} + \mu_v P_{vu})$, $D$ is the diagonal matrix of the stationary distribution $\mu$, and the columns of $Y$ are the optimal slow features ordered by eigenvalue. This is the SFA analogue on a Markov chain and coincides with a directed-graph Laplacian. The $1/\sqrt{\mu_i}$ bound on feature amplitude follows directly from the unit-variance constraint $\sum_i \mu_i y_i^2 = 1$, and it is the mechanism that translates nonuniform state occupancy into feature scaling. The work uses this formulation to show why goal-directed behavior misaligns slow features with value functions and why scale correction reverses the misalignment.

What would settle it

Run a parametric SFA variant (e.g., linear SFA or a neural SFA) on the paper's one-dimensional chain with a $\zeta$-greedy goal-directed policy and compare the learned features and value-approximation errors to the paper's predictions: if the features do not show the $1/\sqrt{\mu_i}$ scaling, or if goal-directed features do not underperform uniform features, then the free-response idealization does not carry over to practice.

Watch

Extended reading notes

Core claim

On an ergodic Markov chain, SFA's optimization problem reduces to a generalized eigenvalue problem whose constraints force each feature component $y_i$ to satisfy $|y_i| \le 1/\sqrt{\mu_i}$, where $\mu_i$ is the stationary occupancy of state $i$. Under goal-directed policies the stationary distribution is peaked at the reward, so optimal slow features are flattened where the agent spends most of its time and amplified where it rarely goes. In the paper's spatial experiments, this scaling makes the slow-feature basis worse at linearly approximating the optimal value function than features from a uniform walk: goal-directed features underperform, while goal-averse features—whose occupancy profile happens to match the value function's shape—outperform. The paper also shows that a post-hoc rescaling by $\sqrt{\mu_i}$, or replacing $\zeta$-greedy exploration with Boltzmann exploration, can largely undo the harm, whereas reweighting transitions (learning rate adaptation) does not.

Load-bearing premise

The analysis assumes the slow features can be any function of the state (unrestricted 'free responses'); if a real parametric SFA implementation cannot realize such features, the predicted scaling and corrections may not apply.

Editorial extensions

If this is right

  • In off-policy reinforcement learning where the exploratory policy is goal-directed, SFA-based representations can be systematically worse for value approximation than uniform-data features, especially when the embedding dimension is low.
  • Goal-averse exploration can turn the same scaling effect into an advantage, because the resulting features' amplitudes align with the value function's peak.
  • Boltzmann exploration with scale correction yields the best overall approximation features across almost all tested settings, while learning-rate-adaptation correction never yields the best performance.
  • The choice of behavior policy used to generate SFA training data is not neutral: it materially changes the geometry of the learned representation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • By extension, the $1/\sqrt{\mu}$ bound is a general property of any representation with unit variance under a nonuniform data distribution, so the same flattening should appear in other spectral embedding methods when trained on goal-directed data, not only SFA.
  • A natural test of the paper's caveat: linear or neural SFA features are constrained to a fixed functional family and may show a weaker or distorted version of the scaling, so the effect's magnitude could be architecture-dependent.
  • The scale correction is applied with the most natural setting (factor $\sqrt{\mu_i}$); over-correcting toward goal-averse scaling is a free hyperparameter that could yield further gains in goal-directed regimes.
  • In continuous state spaces, an analogous occupancy scaling should appear wherever the sampling distribution is nonuniform, so similar misalignment could affect representation learning in continuous control from near-optimal policies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper studies slow feature analysis (SFA) on Markov chains induced by goal-directed, uniform, and goal-averse behavior policies in finite spatial environments. The authors formulate SFA as a generalized eigenproblem on the symmetrized Laplacian of the chain, derive the unit-variance bound |y_i| <= 1/sqrt(mu_i), and argue that optimal slow features are flattened in high-occupancy regions and amplified in low-occupancy regions. They then propose three corrections--behavior modification, learning-rate adaptation, and scale correction--and evaluate linear value-function approximation of V* in 1D and 2D grids. The main empirical claims are that goal-directed behavior degrades and goal-averse behavior improves approximation relative to uniform features, and that Boltzmann behavior with scale correction is broadly beneficial.

Significance. The paper's derivation is transparent and connects SFA on Markov chains to existing spectral embeddings; the bound (24)-(25) is a clean, parameter-free consequence of the constraints. If the empirical ranking survives evaluation under a fixed reference distribution, the paper would make a useful contribution by showing that occupancy-driven scaling can misalign unsupervised representations with downstream value approximation and by identifying simple corrections. The experiments are deterministic and exhaustively sweep the considered parameter grids, and the authors are unusually candid about the non-formal status of the scaling claim and about the free-response idealization.

major comments (2)
  1. [Section 3 and Section 7] The central evaluation uses a behavior-dependent weighting: Section 3 states that the OLS fit 'leverages the stationary distribution of the environment,' and Section 7 reports mean-squared error under that same distribution. Because the goal-directed stationary distribution concentrates mass on the peak of V*, where the optimal SFA features are flattened, while the goal-averse distribution down-weights that peak, the reported ranking goal-averse > uniform > goal-directed can arise from the weighting alone rather than from representation quality. This confound is not listed among the Section 8 caveats. Please re-run the main comparisons under a fixed reference distribution (e.g., uniform over states) and report both behavior-weighted and fixed-reference errors; the Section 8 conclusions should be restated only for the conclusions that survive both metrics.
  2. [Section 4 and Section 8] All experimental results concern optimal free-response features, obtained by dropping 'any restriction (or definition) of the actual functional forms' (Section 4). The Discussion correctly says that real SFA with a fixed architecture 'can generally not realize optimal features' and that transfer is unclear, but the Section 8 bullet conclusions and the abstract do not carry this qualification. Since the proposed correction routes are meant to be used in practical representation learning, either add an experiment with a parametric (e.g., linear or gradient-based) SFA learner, or explicitly restrict every conclusion to optimal free responses.
minor comments (4)
  1. [Section 7.1] In the definition of goal-averse behavior, the condition for i<T should be pi(right|si) < 0.5, not pi(right|si) > 0.5; as written it is identical to the goal-directed condition for that case.
  2. [Section 5, Eqs. (24)-(25)] Write |y_i| <= 1/sqrt(mu_i) (and the corresponding statement in (25)); the notation y_i <= +/-1/sqrt(mu_i) is mathematically ambiguous.
  3. [Section 6] The displayed formula for the corrected diagonal D_vv = 1/2 + N_v mu_v/2 appears to assume sum_{u in N(v)} mu_u = 1; generally D_vv = sum_{u in N(v)} (mu_u + mu_v)/2. Please clarify or correct.
  4. [Section 5 and Section 8] The statement that the scaling effect is 'confirmed by all the experiments conducted' should be presented as an empirical finding for the tested grid environments, since the authors themselves note that the bound alone is not a formal proof.

Circularity Check

0 steps flagged · score 0.0 of 10

The derivation is self-contained: the scaling effect follows from the SFA constraints, and the performance claims are experimental outcomes under a stated evaluation metric, not identities built into the assumptions.

full rationale

The central derivation in Section 4 builds the SFA-on-Markov-chain objective (eqs. 8–16) directly from the slowness objective and the stationary distribution, and solves it as a generalized eigenvalue problem (eq. 21). The occupancy-dependent amplitude bound (eqs. 23–25) is a direct mathematical consequence of the variance constraint; the paper explicitly stops short of claiming a formal proof of the full scaling effect and says it is 'confirmed by all the experiments conducted.' The value-function approximation experiments use ground-truth V*, ordinary least squares, and the behavior policy's stationary distribution as an explicitly stated weighting choice; no parameter is fitted to the approximation target in constructing the optimal features. The self-citations (Wiskott 2003; Franzius et al. 2007) serve as conceptual precedent or as the origin of correction mechanisms, but the main argument does not reduce to them—the Lagrange-multiplier derivation and the numerical evaluations stand on their own. The Discussion's caveat that parametric SFA cannot generally realize optimal features is a genuine external-validity limitation, not a circularity. The behavior-weighted MSE could make the goal-averse versus goal-directed ranking sensitive to the evaluation distribution, but that is a robustness or correctness concern, not a circularity reduction: the paper's equations do not define the reported performance ordering into existence.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The derivation itself introduces no fitted constants or new entities; the only free parameters are experimental settings (gamma and the implicit beta calibration). The main epistemic burden is carried by idealizations: ergodicity, free-response optimality, and infinite-sample linear approximation. All three are explicitly acknowledged in the text, but they limit the strength of the practical conclusions.

free parameters (2)
  • Discount factor gamma = 0.95
    Chosen in Section 3 for all experiments; text states results transfer qualitatively to other discount factors but no supporting experiments are shown.
  • Boltzmann inverse temperature beta = implicitly calibrated, no numeric values given
    Section 7.1 defines beta by matching the probability of optimal actions at goal-neighboring states to a specified zeta-greedy policy; the exact formula or values are not provided, which weakens reproduction.
assumptions (5)
  • domain assumption Every behavior policy that is used induces an ergodic Markov chain with a unique stationary distribution
    Invoked throughout Sections 4 and 7; requires connected graphs with nonzero exploration and fails for deterministic policies or enter-only rooms, as noted in Section 4.
  • domain assumption Optimal SFA features with unrestricted functional form are a faithful proxy for features learned by practical parametric SFA
    Section 4 explicitly drops the restriction on g_i; Section 8 caveat states it is unclear to what extent real SFA exhibits the same effects.
  • domain assumption Infinite-sample linear regression against ground-truth V* is representative of practical value-function approximation
    Stated as idealizations in Section 3; the paper assumes transfer to Q* and to sample-based learning without proof.
  • domain assumption The stationary distribution of the behavior policy is the correct data distribution for evaluating representation quality
    Section 4 replaces time averages with expectations under mu; this is exact only in the infinite-sample ergodic limit.
  • standard math Solutions of the generalized eigenvalue problem (D-M)y = D y Lambda provide the global optima of the SFA objective
    Section 4 and Appendix A use Lagrange multipliers and standard spectral theory; no alternative proof is needed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Slow Feature Analysis on Markov Chains from Goal-Directed Behavior." pith.science (2026). https://pith.science/paper/NVIDBXTC

@misc{pith2026250601145,
  author       = {Pith},
  title        = {Pith review of: Slow Feature Analysis on Markov Chains from Goal-Directed Behavior},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NVIDBXTC}},
  note         = {Machine review of arXiv:2506.01145}
}
read the original abstract

Slow Feature Analysis is a unsupervised representation learning method that extracts slowly varying features from temporal data and can be used as a basis for subsequent reinforcement learning. Often, the behavior that generates the data on which the representation is learned is assumed to be a uniform random walk. Less research has focused on using samples generated by goal-directed behavior, as commonly the case in a reinforcement learning setting, to learn a representation. In a spatial setting, goal-directed behavior typically leads to significant differences in state occupancy between states that are close to a reward location and far from a reward location. Through the perspective of optimal slow features on ergodic Markov chains, this work investigates the effects of these differences on value-function approximation in an idealized setting. Furthermore, three correction routes, which can potentially alleviate detrimental scaling effects, are evaluated and discussed. In addition, the special case of goal-averse behavior is considered.

Figures

Figures reproduced from arXiv: 2506.01145 by the authors.

Figure 1
Figure 1. The schematic of a simplified and finite birth-death-process parameterized by a scalar [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. The optimal embeddings for the birth-death-process with [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Optimal features for the birth-death-process with [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (27 more)
Figure 4
Figure 4. Figure 4: The optimal embeddings for the birth-death-process with [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Value function of optimal policy for different [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Illustration of the effect of ζ-greedy behavior on the stationary distribution and slow features of the birth￾death-process. Top: Stationary distributions. Bottom: Optimal value function for γ = 0.95 and overlay of the first ten slow features of the Markov chain. All f…
Figure 7
Figure 7. Figure 7: Illustration of the possible effects of ζ-greedy behavior on the quality of approximation. environments are potentially misaligned objectives in a reinforcement learning setting. This is aggravated by the fact that the discrepancy is most pronounced when a low embeddin…
Figure 8
Figure 8. Figure 8: The regression performance as log mean-squared-error for different reward positions, dimension of embedding, [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Illustration of the effect of scale correction on features from [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]
Figure 10
Figure 10. Figure 10: The regression performance as log mean squared error for different reward positions, dimension of [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]
Figure 11
Figure 11. Figure 11: The difference in regression performance as symlog mean-squared-error after the scale correction. Plotted [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Example illustration of the effect of LRA correction on features from [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: (a) The regression performance for different reward positions, dimension of embedding, and goal-affinities [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: Best approximation performances for ζ-greedy behavior for different settings and corrections. Best performance per feature dimension indicated in red. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_14.png]
Figure 15
Figure 15. Figure 15: Illustration of the effect of Boltzmann behavior on the stationary distribution and slow features of the [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 16
Figure 16. Figure 16: The approximation quality as log mean-squared-error for different reward positions, dimension of embedding, [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: The difference of quality as symlog mean-squared-error when switching from [PITH_FULL_IMAGE:figures/full_fig_p015_17.png]
Figure 18
Figure 18. Figure 18: Best approximation performances for Boltzmann behavior for different settings and corrections. Best [PITH_FULL_IMAGE:figures/full_fig_p015_18.png]
Figure 19
Figure 19. Figure 19: Best approximation performances for ζ-greedy and Boltzmann behavior for different settings and corrections. Best performance per feature dimension indicated in red. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_19.png]
Figure 20
Figure 20. Figure 20: Example of a lattice with 10 × 10 states with the possible actions ↑, ↓, ←, → leading to transitions into the corresponding states. On the sides, if no target node is available in the chosen direction, a self-transition will occur. Thus, the graph is a generalization …
Figure 21
Figure 21. Figure 21: Stationary distribution and example features of [PITH_FULL_IMAGE:figures/full_fig_p017_21.png]
Figure 22
Figure 22. Figure 22: The regression performance as log mean-squared-error for different reward positions, dimension of [PITH_FULL_IMAGE:figures/full_fig_p018_22.png]
Figure 23
Figure 23. Figure 23: Stationary distribution and example features of [PITH_FULL_IMAGE:figures/full_fig_p018_23.png]
Figure 24
Figure 24. Figure 24: Regression results in the 2D environment for [PITH_FULL_IMAGE:figures/full_fig_p019_24.png]
Figure 25
Figure 25. Figure 25: Stationary distribution and example features of [PITH_FULL_IMAGE:figures/full_fig_p019_25.png]
Figure 26
Figure 26. Figure 26: Regression results in the 2D environment for [PITH_FULL_IMAGE:figures/full_fig_p020_26.png]
Figure 27
Figure 27. Figure 27: Stationary distribution and example features of Boltzmann behavior for different degrees of goal-directedness [PITH_FULL_IMAGE:figures/full_fig_p020_27.png]
Figure 28
Figure 28. Figure 28: Regression results in the 2D environment for Boltzmann behavior (a) compared with [PITH_FULL_IMAGE:figures/full_fig_p021_28.png]
Figure 29
Figure 29. Figure 29: Regression results in the 2D environment for Boltzmann behavior after applying feature scale correction (a) [PITH_FULL_IMAGE:figures/full_fig_p022_29.png]
Figure 30
Figure 30. Figure 30: Best performances in the 2D environment between Boltzmann and [PITH_FULL_IMAGE:figures/full_fig_p023_30.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

4 extracted references · 2 canonical work pages

  1. [1]

    L., Paiement, J.-F., Vincent, P., & Ouimet, M

    Bengio, Y ., Delalleau, O., Roux, N. L., Paiement, J.-F., Vincent, P., & Ouimet, M. (2004). Learning eigenfunctions links spectral embedding and kernel pca. Neural Computation,

    work page 2004

  2. [4]

    We use the convention to write the derivative of a scalary with respect to a matrix X = (Xij)ij is again a matrix ∂y ∂X = ∂y ∂Xij ij (28) of similar dimensions and entries corresponding to partial derivatives of the entries of X. Some useful identities: ∂ tr(AT BA) ∂A = ∂ tr(AT BA) ∂Aij ij (29) and for individual entries ∂ tr(AT BA) ∂Aij = ∂ ∂Aij tr(AT BA...

    work page 2025

  3. [16]

    Bertsekas, D., & Tsitsiklis, J. (2002). Introduction to probability. Athena Scientific. Böhmer, W., Grünewälder, S., Shen, Y ., Musial, M., & Obermayer, K. (2013). Construction of approximation spaces for reinforcement learning. Journal of Machine Learning Research, 14(27), 2067–2118. http://jmlr.org/papers/ v14/boehmer13a.html Chung, F. (2005). Laplacian...

    arXiv 2002

  4. [2005]

    Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. A. (2013). Playing atari with deep reinforcement learning. CoRR, abs/1312.5602. http://arxiv.org/abs/1312.5602 Schüler, M., Hlynsson, H. D., & Wiskott, L. (2019). Gradient-based training of slow feature analysis by differentiable approximate whitening. In W....

    arXiv 2013

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.