Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

The paper claims that shaping learned representations into an isotropic Gaussian makes linear value readouts track non-stationary targets stably, and that a lightweight regularizer enforcing this geometry improves deep RL training across At

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:38 UTC pith:NPUXPVLG

load-bearing objection The empirical results are convincing enough to take seriously, but the 'provably stable' claim doesn't hold up; SIGReg as a cheap regularizer for deep RL is the real contribution. the 4 major comments →

arxiv 2602.19373 v3 pith:NPUXPVLG submitted 2026-02-22 cs.LG cs.AI

Stable Deep Reinforcement Learning via Isotropic Gaussian Representations

Ali Saheb Pasand , Johan Obando-Ceron , Aaron Courville , Pouya Bashivan , Pablo Samuel Castro This is my paper
classification cs.LG cs.AI
keywords deep reinforcement learningnon-stationarityrepresentation geometryisotropic GaussianLyapunov stabilitytracking errorSIGRegneuron dormancy
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Deep RL suffers because learning targets and data distributions shift as the policy improves. The paper argues that this instability is fundamentally a representation-geometry problem: when representations become anisotropic or low-entropy, a linear critic chasing a moving target gets pulled along ill-conditioned directions and its tracking error fails to shrink. The authors prove that if the representation distribution is isotropic Gaussian, the tracking error of a linear readout contracts monotonically toward zero, because isotropy equalizes contraction across all directions and Gaussian tails minimize the variance of the drift that pushes the error around. They then introduce a cheap regularizer that projects embeddings onto random directions and matches each projection to a zero-mean Gaussian, and show it improves stability, sample efficiency, feature rank, and neuron dormancy across PQN and PPO on Atari and on Isaac Gym continuous control. A sympathetic reader would take the paper's core claim to be: non-stationary learning is stabilized by forcing representations toward an isotropic Gaussian, and this is a provable property of the tracking dynamics, not just a useful heuristic.

Core claim

For a linear critic Qθ(s,a)=w⊤φ(s,a) under gradient flow with drifting TD target y_t, define tracking error e(t)=w(t)−Σφ^−1 b_t. The paper proves (Thm 3.1) that when Σφ is constant, d(‖e‖²)/dt = −4eᵀΣφe − 2eᵀΣφ^−1 ḃ_t. The first term is a contraction; the second can be destabilizing. Isotropy (Σφ=σ²I) maximizes the weakest contraction direction and minimizes the condition-number bound on the drift. Gaussianity (Stein's lemma) minimizes the drift variance by removing a residual term r(φ)=φ+Σφ∇log p(φ). The authors conclude zero tracking error is stable and SIGReg enforces this geometry.

What carries the argument

The central object is the Lyapunov function Γ=‖w(t)−w∗t‖², the squared distance between the current last-layer weights and the instantaneous minimizer of the expected TD loss. Its time derivative decomposes into a contraction term governed by the feature covariance Σφ and a drift term governed by the change in the correlation b_t=E[φ y_t]. Isotropy (Σφ=σ²I) does two jobs: it equalizes the contraction across all directions and minimizes the condition number κ(Σφ) that upper-bounds the drift term. Gaussianity is then justified by Stein's lemma: for a Gaussian, E[φ f(φ)]=Σφ E[∇f(φ)], so the residual r(φ) vanishes and the drift term's variance is minimized. SIGReg, the paper's proposed mechanism

Load-bearing premise

The stability proof assumes the embedding covariance is fixed and isotropic during training and treats the TD target as a smooth function of the current embedding, but SIGReg only softly encourages this, so the dropped covariance-drift term in Eq. 26 could in principle make the Lyapunov derivative positive and break the contraction.

What would settle it

During a real PQN or PPO run with SIGReg, record w(t), Σφ(t), b_t and e(t) at each step and compute the full Γ̇ including the representation-drift term 2eᵀΣφ^−1 Σ̇φ Σφ^−1 b_t; if Γ̇ is positive over sustained intervals while tracking error grows, the claimed stable equilibrium is not achieved in practice. A cleaner offline test: fit two linear readouts on a drifting target, one with features forced to an isotropic Gaussian and one with identically scaled but anisotropic features, and measure whether the isotropic readout's tracking error decreases monotonically as Eq. 15 predicts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the tracking-error theorem holds, then representation regularization that enforces isotropy is a direct stability mechanism for any deep RL method with a linear value head, complementing target networks and optimizer tricks.
  • Isotropic Gaussian representations maximize entropy under a fixed variance budget and spread variance evenly across dimensions, which directly counteracts rank collapse and neuron dormancy in long-horizon training.
  • Adding SIGReg to PQN improves area-under-the-curve on 51 of 57 Atari games with a median gain of 138%, and the gains extend to PPO and to Isaac Gym continuous control, suggesting the effect is not algorithm-specific.
  • The ablations show that Gaussian tails matter: heavier-tailed isotropic distributions (Laplacian, Logistic) and pure covariance whitening deliver smaller or less consistent gains, so enforcing tail decay is a necessary part of the recipe.
  • Explicitly enforcing isotropic Gaussian structure with SIGReg narrows the performance gap between first-order RAdam and a Kronecker-factored optimizer, implying that part of second-order optimization's benefit may be the implicit geometry it induces.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's theory assumes the covariance is frozen at an isotropic value, but SIGReg is only a soft regularizer; a natural extension is to test whether stronger or explicit covariance clamping (e.g., periodic whitening) makes the predicted monotone tracking-error decay visible in an actual PQN run.
  • Because the proof is for linear readouts, the mechanism may transfer imperfectly to nonlinear heads, actor-critic couplings, or methods whose value head is not a linear function of the penultimate layer; those settings are where the paper's own limitations point.
  • If the implicit-isotropy hypothesis is right, then any stabilization method that equalizes feature covariance should show the same rank and dormancy improvements; a simple test would be to compare SIGReg against spectral normalization or batch normalization on the same Atari games and measure whether the residual performance gap matches the residual anisotropy.
  • The paper leaves open whether isotropic Gaussian structure is globally optimal or merely a strong default; for tasks requiring highly structured features, the regularizer's strength λ could be annealed over training so the representation first stabilizes and later specializes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper argues that enforcing isotropic Gaussian structure on learned representations stabilizes deep RL training under non-stationary targets. It derives a continuous-time tracking-error identity (Theorem 3.1 / Eq. (15)) for a linear readout, claims this identity shows stable tracking when the embedding covariance is isotropic Gaussian, and introduces Sketched Isotropic Gaussian Regularization (SIGReg) to enforce this geometry. Empirical evaluation spans non-stationary CIFAR-10, the full Atari suite with PQN and PPO, and Isaac Gym continuous control, reporting large AUC improvements and reduced representation collapse / neuron dormancy.

Significance. The central hypothesis — that representation geometry, specifically isotropy and Gaussianity, is a principled stabilizer for non-stationary RL — is attractive and, if established, would be a meaningful step beyond architecture- or optimizer-specific fixes. The paper deserves credit for a broad experimental campaign: full Atari with two algorithms, ablations over target distributions and loss components, continuous control, and an analysis linking Kronecker-factored optimization to implicit isotropy. The empirical correlations between SIGReg loss, feature rank, and dormancy are useful observations, and the code release is a practical plus. However, the theoretical result as stated does not prove the advertised stability claim, and the headline empirical numbers are weakened by per-game hyperparameter selection on the evaluation metric. The contribution is therefore currently a promising empirical study plus a heuristic theoretical framing, rather than the provable-stability result claimed in the abstract and Section 6.

major comments (4)
  1. [Theorem 3.1 / Eq. (1), §3.2, §6] The stability conclusion does not follow from the derived identity. With nonzero b_t-dot, e=0 is not an equilibrium: from Eq. (26), ˙e(0) = -Σ^{-1} ˙b_t ≠ 0. Moreover ˙Γ = -4e^TΣe - 2e^TΣ^{-1}˙b_t can be positive for arbitrarily small e when e is anti-aligned with Σ^{-1}˙b_t, regardless of isotropy. The bound in App. B.1.4 only controls |e^TΣ^{-1}˙b_t| by κ(Σ)||e||; it does not ensure negativity. Thus the claims of a stable zero tracking-error equilibrium and bounded decreasing error (Section 6) are unsupported. The paper should either prove a non-autonomous stability/boundedness statement with explicit conditions on ˙b_t, or substantially weaken these conclusions.
  2. [Assumption in Theorem 3.1 and §B.1.3] The theorem assumes Σφ is constant and drops the representation-drift term 2e^TΣ^{-1} ˙Σ Σ^{-1} b_t from Eq. (26). In the experiments SIGReg is a soft regularizer with a finite strength λ (Table 8), so ˙Σ is not zero and no evidence is provided that this term is negligible. Consequently the theory does not apply to the experimental setting; at minimum, the paper should measure or bound the dropped term during training to justify its removal.
  3. [App. B.1.5] The Gaussianity argument is not a proof. Stein's lemma is applied to y_t(φ) as a smooth function of the current embedding, but the TD target y_t = r + γQ_{θ^-}(s',a') depends on the next state's embedding, not solely on the current φ. Furthermore, Eq. (58) is a Taylor expansion heuristic: it shows that non-Gaussianity introduces higher-order moment terms, but it does not establish that a Gaussian distribution minimizes the variance of the drift term, nor that this variance reduction translates into ˙Γ<0 with high probability. This part should be reframed as intuition, or supported by a rigorous stochastic-dominance argument.
  4. [Table 9 and §4.5 / Table 3] The headline improvements (e.g., 89.5% of games improved, average AUC improvement 889% for PQN) are computed after selecting the SIGReg regularization strength λ per game using the same AUC metric that is then reported (Table 9). This is oracle selection on the evaluation metric and overstates the method's expected performance under fixed hyperparameters. The paper should report results with a fixed λ, or with a proper nested validation procedure, and should include per-method confidence intervals across seeds.
minor comments (4)
  1. [App. C.4.1 / Table 8] The text states that Fig. 13 considers regularization factors 10 and 0.2, while Table 8 and Fig. 13 legends indicate 1.0 and 0.2. This inconsistency should be fixed.
  2. [§4.5] An average AUC improvement of 889% is likely dominated by a few low-scoring games; reporting median, interquartile range, or normalized-score summaries would be more informative.
  3. [Fig. 5] The caption notation 'PQN: [0.4,0.2]→[0.9,0.8] vs. PQN+SIGReg: [0.3,0.2]→[0.1,0.1]' is unclear; please explain what the two numbers represent.
  4. [Table 2] The row 'Covariance Whitening' should state explicitly how whitening is implemented as a regularizer (e.g., added loss or gradient operation) and why it is not equivalent to the Gaussian target.

Circularity Check

0 steps flagged

No significant circularity: the tracking-error identity (Eq. 15) is an algebraic derivation under explicit assumptions, the isotropy/Gaussianity arguments are independent mathematical bounds (B.1.4, B.1.5), SIGReg is adopted from external work (LeJEPA, no author overlap), and empirical gains are measured on external benchmarks. The main flagged gaps concern the validity of the stability conclusion

full rationale

The paper's derivation chain is self-contained rather than circular. Theorem 3.1/Eq. (15) is an algebraic identity obtained from the gradient-flow equation (Eq. 10), the definition of the instantaneous minimizer w*_t = Σφ(t)^{-1}b_t (Eq. 12), and the Lyapunov function Γ = ||e||^2. No fitted quantity is defined in terms of the claimed prediction; the identity holds for any fixed positive-definite Σ, and the 'advantage' of isotropy is argued by a genuine maximin computation over eigenvalues under a fixed trace (App. B.1.4), while Gaussianity is argued via a Stein-lemma residual decomposition (App. B.1.5). Neither step reduces to its conclusion by construction. SIGReg is imported from Balestriero & LeCun (2025), an external paper with no overlapping authors, so no 'ansatz smuggled via self-citation' or 'uniqueness imported from authors' pattern applies. The empirical claims are external-benchmark comparisons (CIFAR-10, Atari, Isaac Gym) with fixed architecture and optimizer; no parameter is fit to a subset and then re-reported as a statistically forced prediction, and the paper is not validated by its own prior results. The self-citations that do appear (Castanyer et al., Obando-Ceron et al., Castro et al.) are contextual or used for baseline settings and do not bear the load of the central theorem. Several concerns are flagged in the manuscript itself and weigh on correctness rather than circularity: the paper concedes that the second term in Eq. (15) has uncontrolled sign (Sec. 3.2) and that a magnitude bound does not imply negativity; the Limitations section concedes only 'approximate stationarity of the representation covariance,' and App. B.1.3 drops the representation-drift term by assuming Σφ ≈ 0 even though SIGReg is a soft regularizer with finite λ; and per-game λ is selected by AUC (Table 9), risking benchmark overfitting. These are validity and experimental-design issues that reduce confidence in the 'provably stable tracking' claim, but they do not make the derivation equivalent to its inputs. Therefore the circularity score is minimal.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 0 invented entities

The central claim rests on the assumption that the embedding covariance is fixed (so the representation-drift term vanishes), on linear-readout dynamics, and on treating TD targets as smooth functions of the current embedding. These are simplifications that are only partially acknowledged, and the per-game λ selection is a data-dependent free parameter affecting the empirical headline.

free parameters (1)
  • SIGReg regularization strength λ = 0.2 or 1.0, chosen per game and algorithm by AUC
    Table 9 reports the best λ selected via AUC for each Atari game; the headline improvements compare this per-game-tuned version against an untuned baseline.
axioms (5)
  • domain assumption Σφ(t) is constant over time in Theorem 3.1
    The theorem drops the representation-drift term 2e^T Σ^{-1}_φ Σdot_φ Σ^{-1}_φ b_t (Eq. 26). SIGReg only softly enforces a fixed covariance, so this term is not zero in experiments.
  • domain assumption Linear readout Q_θ(s,a) = w^T φ(s,a)
    All tracking analysis uses a linear critic; PQN/PPO use nonlinear networks. The paper acknowledges this in Limitations.
  • domain assumption Continuous-time gradient flow models discrete optimization
    The Lyapunov analysis uses ˙w = -∇L_t(w), while experiments use RAdam/Adam with minibatches; no formal link is made between these.
  • domain assumption TD target y_t is a smooth function f(φ) of the current embedding in App. B.1.5
    Stein's lemma requires y_t(φ) smooth; in TD, y_t = r + γQ_{θ^-}(s',a') depends on the next state's embedding, not the current φ. The paper does not flag this.
  • standard math Standard facts: Gaussian maximum entropy; Isserlis theorem; Lyapunov stability conditions
    Used in §3.3 and App. B without proof; they are standard results.

pith-pipeline@v1.3.0-alltime-deepseek · 39970 in / 15577 out tokens · 143133 ms · 2026-08-02T21:38:15.196544+00:00 · methodology

0 comments
read the original abstract

Deep reinforcement learning systems often suffer from unstable training dynamics due to non-stationarity, where learning objectives and data distributions evolve over time. We show that under non-stationary targets, isotropic Gaussian embeddings are provably advantageous. In particular, they induce stable tracking of time-varying targets for linear readouts, achieve maximal entropy under a fixed variance budget, and encourage a balanced use of all representational dimensions--all of which enable agents to be more adaptive and stable. Building on this insight, we propose the use of Sketched Isotropic Gaussian Regularization for shaping representations toward an isotropic Gaussian distribution during training. We demonstrate empirically, over a variety of domains, that this simple and computationally inexpensive method improves performance under non-stationarity while reducing representation collapse, neuron dormancy, and training instability.

Figures

Figures reproduced from arXiv: 2602.19373 by Aaron Courville, Ali Saheb Pasand, Johan Obando-Ceron, Pablo Samuel Castro, Pouya Bashivan.

Figure 1
Figure 1. Figure 1: Illustration of two tracking regimes. Left: the norm of the tracking error exhibits non-monotonic behavior and fails to converge, indicating unstable tracking. Right: the tracking error decreases monotonically and converges to zero, corresponding to stable tracking dynamics. Geometric Implications for Stability. The decomposi￾tion in Eq. 1 highlights competing effects governing stability under non-stationa… view at source ↗
Figure 2
Figure 2. Figure 2: Directly shaping a multivariate distribution. SIGReg first projects the embeddings onto a small set of random directions (pi : sketching), producing multiple uni￾variate distributions. Each projection is then matched to the corresponding univariate target distribution. tors {vk} K k=1, SIGReg operates on the projected variables zk = v ⊤ k ϕ. The regularization loss matches the empiri￾cal distribution of ea… view at source ↗
Figure 3
Figure 3. Figure 3: Non-stationary CIFAR-10. Training under repeated label shuffling. The baseline shows poor recovery after each shift, with SIGReg loss spikes, rank collapse, and increased dormancy. Enforcing isotropic Gaussian representations stabilizes training, accelerates recovery, preserves rank, and reduces dormancy. 0.0 0.2 0.4 0.6 0.8 1.0 1e7 0 1 2 3 1e3 0.0 0.2 0.4 0.6 0.8 1.0 1e7 0.0 0.2 0.4 0.6 0.8 0.0 0.2 0.4 0.… view at source ↗
Figure 4
Figure 4. Figure 4: Two Atari-10 games (PQN). Without isotropy regularization, representations exhibit rank collapse, increased neuron dormancy, and early performance saturation. Encouraging isotropic geometry leads to improved representation quality and higher, more stable performance. sentation quality, reduced neuron dormancy, and improved asymptotic performance. These results indicate that promot￾ing isotropic Gaussian st… view at source ↗
Figure 5
Figure 5. Figure 5: 2D PCA of embedding covariance over train￾ing. Without constraints, representations collapse onto a few dominant principal components. Encouraging isotropic Gaussian structure yields more evenly distributed variance and higher effective dimensionality, reflected in reduced con￾centration on the leading components (PQN: [0.4, 0.2] → [0.9, 0.8] vs. PQN+SIGReg: [0.3, 0.2] → [0.1, 0.1]). gains in learning perf… view at source ↗
Figure 6
Figure 6. Figure 6: Full Atari suite. Effect of isotropic Gaussian regularization. Left: IQM human-normalized learning curves as a function of environment steps for PQN and PPO, with and without isotropic regularization. Right: Per-game improvement in AUC obtained by encouraging isotropic representation geometry. Across both algorithms, isotropic regularization improves final performance and sample efficiency [PITH_FULL_IMAG… view at source ↗
Figure 7
Figure 7. Figure 7: Isaac Gym continuous control. Learning curves on two representative locomotion tasks. Isotropic Gaus￾sian representations improve stability and reduce variance. We report returns over 5 runs for each experiment. See Section D for additional results on four Isaac Gym control tasks. tributions, and high-dimensional continuous action spaces. Across a range of locomotion and manipulation tasks, en￾couraging is… view at source ↗
Figure 8
Figure 8. Figure 8: Effect of non-stationarity and SIGReg loss minimization on representation stability and accelerating recovery in non-stationary CIFAR-10 experiment. Non-stationarity induced by label shuffling causes a sharp drop in accuracy, increase in SIGReg loss, collapse of feature rank, and higher neuron dormancy. It can be observed that Kronecker-factored optimizer implicitly lowers the SIGReg loss compared to first… view at source ↗
Figure 9
Figure 9. Figure 9: Atari-10, PQN, Kron Optimizer. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons. In almost all games, Kron leads to implicit minimization of SIGReg loss, higher rank, and lower dormancy. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Atari-10, PQN, Multi-skip Residual Architecture. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons. In almost all games, using Multi-skip Residual Architecture leads to implicit minimization of SIGReg loss, higher rank, and lower dormancy. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Atari-10, PPO, Kron Optimizer. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons. In almost all games, Kron leads to implicit minimization of SIGReg loss, higher rank, and lower dormancy. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Atari-10, PPO, Multi-skip Residual Architecture. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons. In almost all games, using Multi-skip Residual Architecture leads to implicit minimization of SIGReg loss, higher rank, and lower dormancy. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Atari-10, PQN with Explicit SIGReg Loss Minimization. The change of reward, SIGReg loss, feature rank, and percentage of dormant neurons through time. The orange line is the baseline without SIGReg loss minimization, and the other lines are SIGReg loss minimization with different strengths (Larger λ means stronger regularization). 28 [PITH_FULL_IMAGE:figures/full_fig_p028_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: IQM Human-Normalized Results for PQN (top) and PPO (bottom) Across the Atari-10 Benchmark Games. Explicit SIGReg loss minimization encourages isotropic Gaussian embeddings, reducing the performance gap between first-order optimizers (RAdam) and second-order Kronecker-factored optimization. Adding multi-skip residual connections further improves performance. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: PQN. Reward over time for individual games with and without SIGReg loss minimization. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: PPO. Reward over time for individual games with and without SIGReg loss minimization. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_16.png] view at source ↗
Figure 17
Figure 17. Figure 17 [PITH_FULL_IMAGE:figures/full_fig_p032_17.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. The SIGReg Objective as Variational Free Energy: A Theoretical Active-Inference Account of JEPA World Models

    cs.LG 2026-07 conditional novelty 7.0 partial

    Under constant noise and successful SIGReg enforcement, the JEPA objective is an exact variational free-energy/information bottleneck, while VICReg leaves an irreducible anisotropic gap.

  2. Representation Learning Enables Scalable Multitask Deep Reinforcement Learning

    cs.LG 2026-06 unverdicted novelty 5.0

    MR.Q combines predictive auxiliary tasks with high-capacity value functions in a model-free architecture to achieve strong multitask RL performance without planning.

Reference graph

Works this paper leans on

18 extracted references · 5 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Define the tracking error for the weights of last layer

  2. [2]

    Derive the formula for how this tracking error changes

  3. [3]

    Derive the energy, Lyapunov, function for the norm of tracking error and show that isotropic Gaussian structure makes the zero equilibrium stable, meaning that an increase in the norm of tracking error will be damped and converge to zero over time. We define the tracking error at timetas the difference between the weight matrix and the optimal unknown wei...

    1949

  4. [6]

    Lefschetz, S

    URL https://openreview.net/forum? id=jXLiDKsuDo. Lefschetz, S. and LaSalle, J. P. Stability by liapunov’s direct method: with applications.(No Title), 1961. Liu, J., Ceron, J. S. O., Courville, A., and Pan, L. Neuro- plastic expansion in deep reinforcement learning. InThe Thirteenth International Conference on Learning Rep- resentations, 2025a. URL https:...

    arXiv 1961

  5. [8]

    McLean, R., Chatzaroulas, E., McCutcheon, L., R¨oder, F., Yu, T., He, Z., Zentner, K., Julian, R., Terry, J

    URL https://openreview.net/forum? id=cnqyzuZhSo. McLean, R., Chatzaroulas, E., McCutcheon, L., R¨oder, F., Yu, T., He, Z., Zentner, K., Julian, R., Terry, J. K., Woun- gang, I., Farsad, N., and Castro, P. S. Meta-world+: An improved, standardized, RL benchmark. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benc...

  6. [9]

    Mnih, V ., Kavukcuoglu, K., Silver, D., Rusu, A

    URL https://openreview.net/forum? id=1de3azE606. Mnih, V ., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve- ness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wier- stra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforce...

    Pith/arXiv arXiv 2015

  7. [10]

    Seo, Y ., Sferrazza, C., Geng, H., Nauman, M., Yin, Z.- H., and Abbeel, P

    URL https://proceedings.mlr.press/ v202/schwarzer23a.html. Seo, Y ., Sferrazza, C., Geng, H., Nauman, M., Yin, Z.- H., and Abbeel, P. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control, 2025. URL https://arxiv.org/abs/2505.22642. 11 Stable Deep Reinforcement Learning via Isotropic Gaussian Representations Sokar, G., Agarwal, R....

    Pith/arXiv arXiv 2025

  8. [11]

    Sutton, R

    URL https://openreview.net/forum? id=8oCrlOaYcc. Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction.IEEE Transactions on Neural Networks, 16: 285–286, 1988. Tang, H., Obando-Ceron, J., Castro, P. S., Courville, A., and Berseth, G. Mitigating plasticity loss in contin- ual reinforcement learning by reducing churn. InForty- second Intern...

    1988

  9. [12]

    van Hasselt, H., Doron, Y ., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J

    URL https://openreview.net/forum? id=EkoFXfSauv. van Hasselt, H., Doron, Y ., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. Deep reinforcement learning and the deadly triad.arXiv preprint, arXiv:1812.02648, 2018. Vincent, T., Tripathi, Y ., Faust, T., Oren, Y ., Peters, J., and D’Eramo, C. Bridging the performance gap between target-free and target...

    Pith/arXiv arXiv 2018

  10. [13]

    Echchahed, A

    URL https://www.sciencedirect.com/ science/article/pii/S0304397503006042. Echchahed, A. and Castro, P. S. A survey of state represen- tation learning for deep reinforcement learning.Transac- tions on Machine Learning Research, 2025. ISSN 2835-

    2025

  11. [14]

    Zhang, A., McAllister, R

    URL https://openreview.net/forum? id=sQyRQjun46. Zhang, A., McAllister, R. T., Calandra, R., Gal, Y ., and Levine, S. Learning invariant representations for re- inforcement learning without reconstruction. InIn- ternational Conference on Learning Representations,

  12. [2013]

    Bellemare, M

    ISSN 1076-9757. Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. Autonomous navigation of stratospheric balloons using reinforcement learning.Nature, 588:77 – 82, 2020. Castanyer, R. C., Obando-Ceron, J., Li, L., Bacon, P.-L., Berseth, G., Courville, A., and Castro, P. S. Stable gra- dients for...

    arXiv 2020

  13. [2021]

    URL https://openreview.net/forum? id=-2FCwDKRREu. 12 Stable Deep Reinforcement Learning via Isotropic Gaussian Representations Appendix Contents A Related Work 14 B Formal Analysis 14 B.1 In the presence of non-stationary tasks, Isotropic Gaussian makes zero tracking error a stable equilibrium 14 B.1.1 First Term Analysis . . . . . . . . . . . . . . . . ....

    2023

  14. [2022]

    Lyle, C., Zheng, Z., Nikishin, E., Pires, B

    URL https://openreview.net/forum? id=ZkC8wKoLbQ7. Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural net- works. InInternational Conference on Machine Learning, pp. 23190–23211. PMLR, 2023. Lyle, C., Zheng, Z., Khetarpal, K., Martens, J., van Has- selt, H., Pascanu, R., and Dabney, W. Normalizat...

    2023

  15. [2023]

    Ceron, J

    URL https://openreview.net/forum? id=wPqEvmwFEh. Ceron, J. S. O., Ara´ujo, J. G. M., Courville, A., and Castro, P. S. On the consistency of hyper-parameter selection in value-based deep reinforcement learning. InRein- forcement Learning Conference, 2024a. URL https: //openreview.net/forum?id=szUyvvwoZB. Ceron, J. S. O., Courville, A., and Castro, P. S. In...

    2023

  16. [2024]

    Zang, H., Li, X., Zhang, L., Liu, Y ., Sun, B., Islam, R., des Combes, R

    URL https://openreview.net/forum? id=5FFO6RlOEm. Zang, H., Li, X., Zhang, L., Liu, Y ., Sun, B., Islam, R., des Combes, R. T., and Laroche, R. Understanding and addressing the pitfalls of bisimulation-based representa- tions in offline reinforcement learning. InThirty-seventh Conference on Neural Information Processing Systems,

  17. [2025]

    Gelada, C., Kumar, S., Buckman, J., Nachum, O., and Belle- mare, M

    URL https://openreview.net/forum? id=7IzeL0kflu. Gelada, C., Kumar, S., Buckman, J., Nachum, O., and Belle- mare, M. G. Deepmdp: Learning continuous latent space models for representation learning. InInternational con- ference on machine learning, pp. 2170–2179. PMLR, 2019. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy m...

    Pith/arXiv arXiv 2019

  18. [8856]

    Survey Certification

    URL https://openreview.net/forum? id=gOk34vUHtz. Survey Certification. Farebrother, J., Greaves, J., Agarwal, R., Lan, C. L., Goroshin, R., Castro, P. S., and Bellemare, M. G. Proto- value networks: Scaling representation learning with auxiliary tasks. InThe Eleventh International Confer- ence on Learning Representations, 2023. URL https: //openreview.net...

    Pith/arXiv arXiv 2023