Pith. sign in

REVIEW 3 major objections 5 minor 29 references

Partial, utility-scaled neuron resets keep continual RL agents plastic without triggering policy collapse.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 04:01 UTC pith:C6B5UFAW

load-bearing objection Solid empirical stabilizer for long-horizon continual RL; the zero-collapse result holds, but the causal story is still under-isolated. the 3 major comments →

arxiv 2607.24996 v1 pith:C6B5UFAW submitted 2026-07-27 cs.LG cs.AIcs.RO

Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning

Luc McCutcheon , Evangelos Chatzaroulas , Saber Fallah This is my paper
classification cs.LG cs.AIcs.RO
keywords continual reinforcement learningplasticity lossneuron resetspolicy collapseutility-scaled reinitializationdormant neuronsCalibrated Partial Resets
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Deep networks lose the ability to adapt under long non-stationary training because unused neurons go dormant and full reinitialization of those units can shatter a learned policy. This paper argues that the fix is not to wipe low-utility neurons clean, nor to shrink every weight the same way, but to pull each neuron toward its initialization by an amount scaled to how little it is contributing. The resulting optimizer, Calibrated Partial Resets (CPR), is the only method among those tested that never collapses over 400 million steps of friction-changing locomotion, while still recovering peak return after each task switch and beating prior reset and decay baselines on Continual MetaWorld and Continual MinAtar. A single knob—the maximum pull strength—trades plasticity against peak performance, giving a practical handle on long-horizon stability.

Core claim

Binary full-unit resets restore plasticity but introduce abrupt weight changes that destabilize long-horizon continual RL and can cause policy collapse; CPR replaces that all-or-nothing rule with a continuous, utility-scaled pull toward initialization so that low-utility neurons are refreshed more strongly and high-utility ones only weakly, preserving both trainability and peak performance.

What carries the argument

Calibrated Partial Resets (CPR): every f steps, each neuron’s incoming and outgoing weights are moved toward reinitialization by a fraction r = ρ · ϕ(u), where u is a layer-normalized EMA of gradient-magnitude utility and ϕ is a sharp decreasing map of that utility, so adjustment concentrates on the units that need it most without fully wiping them.

Load-bearing premise

That a sustained drop of at least 8000 return lasting at least 4 million steps is the right definition of policy collapse, and that gradient-magnitude utility is a faithful enough signal that scaling the partial reset by it is what actually prevents collapse.

What would settle it

Run the same 400M-step SlipperyAnt protocol with the same collapse definition: if any binary-reset or decay baseline matches CPR’s zero collapses across 15 seeds while matching its peak and final return, or if swapping CPR’s utility signal for a uniform or random score eliminates the collapse-free advantage, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Long-horizon continual RL can keep recovering after environment shifts without early stopping or aggressive learning-rate annealing.
  • The plasticity–stability tradeoff becomes a single tunable scalar (maximum reset fraction ρ) rather than a brittle threshold.
  • Utility-scaled partial reinitialization is a general design point between uniform decay and binary resets for any non-stationary training regime.
  • Diagnostics such as dormant-neuron ratio, gradient-norm spikes, and parameter-norm growth can be kept flat without sacrificing peak return.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same continuous utility-to-reset map may stabilize other feedback-loop settings (e.g., offline-to-online fine-tuning or multi-agent non-stationarity) where full resets currently cause brittle drops.
  • If gradient-magnitude utility is only a proxy, replacing it with a cheaper activation- or age-based score while keeping the partial rule could cut the small wall-clock overhead further.
  • Collapse-free training over hundreds of millions of steps suggests partial resets could let practitioners drop large rollout buffers that today are used only to mask plasticity loss.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Calibrated Partial Resets (CPR), a continual-learning optimizer that periodically pulls each neuron toward its initialization by a coefficient r = ρ·ϕ(u) scaled by a layer-normalized, EMA-smoothed gradient-magnitude utility (Eqs. 3, 5–7). CPR is positioned between binary reset methods (CBP, ReDo, ReGraMa), which fully reinitialize low-utility units and zero their outgoing weights, and uniform decay methods (Shrink & Perturb). The central empirical claim is that, among compared methods, only CPR avoids policy collapse over 400M PPO steps in SlipperyAnt (zero collapses across 15 seeds, robust to the collapse threshold choice), while maintaining peak performance after friction shifts and improving over baselines on SlipperyHumanoid, Continual MetaWorld, and Continual MinAtar. Supporting evidence includes dormancy/linearization diagnostics, gradient- and parameter-norm trajectories, ablations over ρ, κ, and the utility-to-reset shape ϕ, collapse-threshold sensitivity, and stationary-setting controls (MT1, Permuted MNIST). Code is open-sourced with pinned dependencies.

Significance. If the results hold, this is a useful and well-executed empirical contribution to continual RL. The 400M-step horizon with 20 friction shifts goes well beyond typical reset-method evaluations and exposes a genuine long-horizon failure mode of binary resets; the 15-seed IQM/IQR reporting, the collapse-threshold sensitivity analysis in §4.3, and the wide ρ and κ plateaus (Figures 5, 12) make the headline "zero collapses" result credible rather than anecdotal. The work ships a reproducible, MIT-licensed JAX/Optax codebase with pinned dependencies and per-benchmark hyperparameter tables, and the authors are commendably honest about saturated benchmarks (MT1, Permuted MNIST) and about the limits of their causal story (§5). The selective-and-smooth design point is conceptually clean and cheap (~6% wall-clock overhead), so the method is practically adoptable. The main caveat to significance is that the attribution of collapse avoidance to utility-scaled partial resets specifically — rather than to correlated design differences — is not yet established, which matters for how the community should build on the result.

major comments (3)
  1. [§3, Eq. (7) vs. Eq. (4); Appendix H; §5] The paper's named contribution is that utility-scaled *partial* resets are the stabilizer, but CPR differs from the binary-reset baselines in at least three simultaneous ways, and the experiments do not isolate which one prevents collapse: (a) outgoing-weight handling — baselines zero the outgoing column W^{l+1}_{:,i} on reset (Eq. 4) while CPR scales it by (1−r) (Eq. 7); (b) optimizer-state handling — Appendix H states baselines reset Adam's µ, ν for affected units while CPR leaves optimizer state intact; (c) CPR's continuous (1−r) pull acts as utility-scaled shrinkage that constrains parameter norms (Figure 15), a mechanism Dohare et al. (2024) independently identify as a driver of plasticity loss and one achievable by plain weight decay or regenerative regularization (Kumar et al., 2023, cited but not benchmarked). The κ-ablation (Figure 12) isolates targeted-vs-uniform reset *magnitu
  2. [§4.1, Tables 1–2; Abstract] The abstract claims CPR 'outperforms prior decay and reset-based methods on Continual MetaWorld and Continual MinAtar,' but these tables use only 5 seeds and the IQRs overlap substantially: in Table 1 the Avg column has CPR at 0.231±0.005 against CBP 0.204±0.009 and ReDo 0.199±0.017, and in Table 2 CPR's 75.6±14.3 overlaps ReDo's 68.0±7.6. With n=5, IQM/IQR error bars of this width cannot support an unqualified 'outperforms' claim (the authors themselves cite Agarwal et al. 2021, which recommends stratified bootstrap CIs precisely for such comparisons). Please either increase the seed count, report bootstrap confidence intervals on the IQM differences, or temper the abstract and §4.1 language to 'competitive with / improves final performance' where the intervals overlap.
  3. [§4.3, Figure 1] The collapse-threshold sensitivity analysis is the right response to the obvious concern about the 8,000/4M-step definition, and the 4,000-threshold result is convincing. However, the lenient-threshold paragraph is confusingly argued: 'Under a lenient 10,000 threshold, CPR, ReDo, and ReGraMa each have zero collapses, but this threshold excludes any method whose peak return does not exceed it' — a method whose peak is below 10,000 cannot collapse under this definition *by construction*, so the 10,000 row carries no information about those methods and should not be presented as evidence either way. Please report the per-method collapse counts at each threshold in a small table (currently only Figure 1's bar chart), and clarify that the informative thresholds are those below every method's peak (~6,000–8,000).
minor comments (5)
  1. [Algorithm 1, line 10; §3] After each reset step all utilities are recentered to u=1, discarding the EMA history. With β=0.99 the EMA horizon is ~100 steps against a reset period f=1000, so utilities spend much of each cycle re-equilibrating from a uniform state. A brief discussion (or an ablation without recentering) of why this re-initialization of the selection signal is benign would help, since it partially undercuts the 'smoothed over multiple steps' motivation in §2.2.
  2. [§3, Eq. (5)] The gradient-magnitude utility is justified only by citation to ReGraMa/Liu et al. (2025). A neuron can have persistently large incoming-weight gradients because it is unstable or churning rather than useful; one sentence on why high gradient magnitude is treated as high utility (and how this interacts with PPO's advantage normalization) would close the gap.
  3. [Appendix F, Table 5] The tuning budgets are asymmetric across methods (e.g., CPR's ρ grid has 9 points; ReDo/ReGraMa tune an uncapped 'None' reset fraction). Please state the total number of sweep configurations per method so readers can assess whether CPR's advantage could partly reflect a larger effective search budget, even though separate sweep seeds were used.
  4. [References / Abstract] Typographical: 'V olodymyr Mnih' in the Mnih et al. (2013) entry; missing spaces in the abstract ('binary reset methods,partialresets', 'decay,calibratedutility-scaling'); 'linearised' vs 'linearized' inconsistency between Figure 9 caption and main text; Appendix H 'Reset operator' item 2 says outgoing columns are scaled 'towards zero,' which should be harmonized with the exact (1−r) scaling in Eq. (7).
  5. [§4.2, Figure 3] The caption/text claim 'lower linearization correlates with higher representation capacity' sits awkwardly next to the observation that methods which suppress dormancy have *higher* linearized-unit ratios initially, and that CPR 'maintains steady' ratios rather than lowering them. Please tighten this paragraph so the diagnostic narrative matches what the figure shows.

Circularity Check

0 steps flagged

No significant circularity: CPR is an empirical optimizer design evaluated on external returns and collapse counts, not a derivation that reduces to its inputs.

full rationale

This is a standard continual-RL methods paper. The central objects—utility score (Eq. 5), EMA normalization (Eq. 3), shape map ϕ (Eq. 6), and partial-reset operator (Eq. 7)—are design choices, not quantities derived from the claimed outcome (zero policy collapses / higher IQM return). Those outcomes are measured on held-out training trajectories against external baselines (Adam, CBP, ReDo, ReGraMa, Shrink & Perturb) with multi-seed IQM reporting. Hyperparameters (ρ, κ, β, f) are swept on the same task family, which is ordinary optimizer practice and does not make the collapse/return results true by construction. Citations for gradient-magnitude utility (ReGraMa/Liu et al., CBP/Dohare et al.) are to external authors and supply motivation, not a self-citation uniqueness chain that forces the result. No fitted parameter is relabeled as a prediction; no uniqueness theorem is imported from the present authors; no known empirical law is merely renamed. Causal-attribution confounds (outgoing-weight handling, optimizer-state resets, implicit norm control) are real scientific concerns but are not circularity. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Load-bearing content is almost entirely methodological and empirical. The claim rests on a handful of free hyperparameters (ρ, κ, β, f), the choice of gradient-magnitude utility, the partial-reset algebra, and domain assumptions imported from the plasticity literature (dormant units drive plasticity loss; binary resets cause instability). No new physical entities; CPR itself is an invented operator evaluated experimentally.

free parameters (5)
  • ρ (max per-unit reset fraction) = 0.015 (Ant, MetaWorld, MinAtar); 0.05 (Humanoid)
    Directly controls the plasticity–stability tradeoff; swept and selected per environment on average final return (0.015 SlipperyAnt / 0.05 SlipperyHumanoid; 0.015 defaults elsewhere).
  • κ (sigmoid sharpness) = 16
    Shapes utility→reset map; fixed at 16 after a robustness sweep showing a plateau on [2,20].
  • β (utility EMA decay) = 0.99
    Smoothing of per-neuron utility; chosen from {0.9, 0.99}.
  • f (reset update frequency) = 1000 (main continuous-control); 10^3 on MetaWorld/MinAtar
    How often partial resets fire; swept over {100, 1000, 10000} (and larger on some benchmarks).
  • collapse thresholds (Δreturn, persistence) = 8000 return / 4M steps (primary)
    Operational definition of the headline 'zero collapses' claim; primary pair is 8000 return drop sustained 4M steps, with sensitivity at 4000/10000.
axioms (5)
  • domain assumption Mean incoming-weight gradient magnitude is a valid per-neuron utility for deciding reset strength.
    Adopted from ReGraMa/Liu et al. 2025 and Hernandez-Garcia et al. 2025 in §3; not re-derived here.
  • domain assumption Plasticity loss in deep nets under non-stationarity is substantially driven by dormant/linearized units and rising parameter norms.
    Background from Sokar, Dohare, Lyle lines cited in §1–2; motivates the intervention target.
  • domain assumption Binary full reinitialization plus outgoing-weight zeroing introduces abrupt updates that can destabilize long-horizon RL.
    Stated in §2.3 citing Dohare et al. 2023; underpins why partial resets should help.
  • ad hoc to paper Layer-normalized EMA utility with mean recentered to 1 after each reset step is a stable selection signal across layers and time.
    Design choice in Eq. 3 and Algorithm 1; standard-looking but specific to CPR’s schedule.
  • domain assumption Standard PPO/SAC/Adam/Muon training stacks and Brax/MetaWorld/MinAtar task constructions are faithful continual-RL testbeds.
    Experimental regime §4 and Appendix H; results are conditional on these stacks.
invented entities (2)
  • Calibrated Partial Resets (CPR) operator independent evidence
    purpose: Map per-neuron utility to a continuous pull toward initialization on incoming weights and a matching shrink on outgoing weights.
    Core proposed method (Eq. 6–7, Algorithm 1). Evaluated empirically; not an unobserved physical object.
  • Utility-to-reset shape ϕ (anchored sigmoid) independent evidence
    purpose: Convert layer-normalized utility into r_i ∈ [0, ρ] with mean-anchored maximum.
    Specific functional form introduced in Eq. 6; ablated against exp/softplus/linear in Appendix G.

pith-pipeline@v1.2.0-grok45-kimik3 · 21457 in / 3770 out tokens · 79718 ms · 2026-07-31T04:01:49.817146+00:00 · methodology

0 comments
read the original abstract

Neural networks are hindered by accumulating dormant neurons and loss of expressivity throughout training, particularly in non-stationary data settings, such as continual supervised and reinforcement learning. Recently, neuron resets have been used to maintain gradient flow and restore plasticity. However, full unit reinitialization often sacrifices peak performance and can destabilize training, leading to policy collapse. To preserve plasticity without destabilizing training, we propose Calibrated Partial Resets (CPR), an optimizer that periodically pulls low-utility neurons toward their initialization, with pull strength scaled by each neuron's utility. Unlike binary reset methods, partial resets avoid brittleness; unlike uniform decay, calibrated utility-scaling concentrates adjustment on the units that need it most. Among compared methods, only CPR avoids policy collapse over 400M training steps in SlipperyAnt, and it outperforms prior decay and reset-based methods on Continual MetaWorld and Continual MinAtar benchmarks. Ablations reveal a tunable trade-off between plasticity and peak performance, highlighting utility-scaled reinitialization as a promising direction for continual learning.

Figures

Figures reproduced from arXiv: 2607.24996 by Evangelos Chatzaroulas, Luc McCutcheon, Saber Fallah.

Figure 1
Figure 1. Figure 1: Left: Peak vs. final Interquartile Mean (IQM) episodic return in SlipperyAnt with In￾terquartile Range (IQR) error bars. Right: number of seeds with policy collapses. CPR achieves the highest return and is the only method with zero policy collapses across all 15 seeds over 400M steps while Adam collapses and receives a mean episodic return below zero We argue that this conflates two design choices that sho… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of IQM of episodic returns over 15 seeds. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Left: Dormant-neuron ratio. Right: Linearized-neuron ratio. IQM across 15 seeds (lower is better). Lower dormancy correlates with higher plasticity and lower linearization correlates with higher representation capacity. This shows all methods reduce unit dormancy and linearization, though CBP and CPR appear most effective. Gradient norm [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Actor gradient norm IQM over 15 seeds in SlipperyAnt [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean episodic return over 2M steps for 10 MetaWorld continuous-control tasks (IQM [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Permuted MNIST Continual Supervised Learning. The Interquartile Mean (IQM) of [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: IQM of Adam using larger rollout, with IQR for shaded regions over 15 seeds. This [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Effect of larger rollouts on dormant and linearised neuron accumulation. [PITH_FULL_IMAGE:figures/full_fig_p015_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Utility transformation function comparison [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Results from a SlipperyAnt run showing that CPR pushes neurons away from dormant [PITH_FULL_IMAGE:figures/full_fig_p020_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Ablation of the sharpness parameter (κ) on the SlipperyAnt environment. Performance is highly consistent across a wide range of values (κ), indicating that CPR is robust to hyperparameter tuning. Notably, low sharpness values (κ ≤ 1) lead to significant performance degradation, confirm￾ing that resets must be targeted at low-utility neurons rather than applied indiscriminately. units. We did not find base… view at source ↗
Figure 13
Figure 13. Figure 13: Peak vs. final IQM episodic return in SlipperyHumanoid (higher is better). [PITH_FULL_IMAGE:figures/full_fig_p023_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: The gradient norm for SlipperyHumanoid, averaged over both actor and critic with IQM [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: The parameter norm for Top: SlipperyAnt Bottom: SlipperyHumanoid, averaged over both actor and critic with IQM across 15 seeds. This demonstrates that reset methods can prevent parameter-norm growth by periodically refreshing low-utility units [PITH_FULL_IMAGE:figures/full_fig_p025_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: The value network gradient norm for SlipperyAnt, averaged with IQM across 15 seeds. [PITH_FULL_IMAGE:figures/full_fig_p025_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 15 linked inside Pith

  1. [1]

    or training horizons extend into the regime where plasticity loss compounds into collapse (SlipperyAnt, 400M steps). We include MT1 results not as the principal evidence for CPR, but to verify the method imposes no penalty in stationary settings, a necessary property for a general-purpose continual learning optimizer. Robustness and sample efficiency.On c...

    2023

  2. [3]

    Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood

    URLhttp: //github.com/google-deepmind. Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness.arXiv preprint arXiv:2108.06325,

    Pith/arXiv arXiv

  3. [5]

    Brax–a differentiable physics engine for large scale rigid body simulation.arXiv preprint arXiv:2106.13281,

    C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax–a differentiable physics engine for large scale rigid body simulation.arXiv preprint arXiv:2106.13281,

    Pith/arXiv arXiv

  4. [8]

    Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    Pith/arXiv arXiv

  5. [10]

    Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning

    Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Ling Pan. Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning. arXiv preprint arXiv:2505.24061,

    arXiv

  6. [11]

    Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983,

    Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts.arXiv preprint arXiv:1608.03983,

    Pith/arXiv arXiv

  7. [13]

    Meta-world+: An improved, standardized, rl benchmark.arXiv preprint arXiv:2505.11289,

    Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, KR Zentner, Ryan Julian, JK Terry, Isaac Woungang, et al. Meta-world+: An improved, standardized, rl benchmark.arXiv preprint arXiv:2505.11289,

    arXiv

  8. [14]

    Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,

    V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning.arXiv preprint arXiv:1312.5602,

    Pith/arXiv arXiv

  9. [16]

    Continual learning via learning a continual memory in vision transformer.arXiv preprint arXiv:2303.08250,

    Chinmay Savadikar, Michelle Dai, and Tianfu Wu. Continual learning via learning a continual memory in vision transformer.arXiv preprint arXiv:2303.08250,

    arXiv

  10. [17]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

    Pith/arXiv arXiv

  11. [19]

    Deep reinforcement learning and the deadly triad.arXiv preprint arXiv:1812.02648,

    Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Mo- dayil. Deep reinforcement learning and the deadly triad.arXiv preprint arXiv:1812.02648,

    Pith/arXiv arXiv

  12. [21]

    Neural networks under prolonged non-stationary training forget prior knowledge (French, 1999; Kirkpatrick et al.,

    and trust regions (Schulman et al., 2015). Neural networks under prolonged non-stationary training forget prior knowledge (French, 1999; Kirkpatrick et al.,

    2015

  13. [22]

    Dohare et al

    and lose their ability to adapt (Lyle et al., 2024; Nauman et al., 2024; Dohare et al., 2021), a phenomenon that has drawn increasing attention across class-incremental learning (Savadikar et al., 2023), supervised learning (Ash & Adams, 2020), and RL (Lyle et al., 2022). Dohare et al. (2024) links plasticity loss to rising parameter norms, dormant units,...

    2024

  14. [24]

    We chose SAC over PPO as it is standard for the benchmark and allows us to verify CPR’s performance under off-policy methods

    using Soft Actor Critic (SAC) (Haarnoja et al., 2018). We chose SAC over PPO as it is standard for the benchmark and allows us to verify CPR’s performance under off-policy methods. We compare CPR against Adam and other reset-based baselines across 10 distinct manipulation tasks over 2 million training steps. Reset methods hyperparameters are reported in Table

    2018

  15. [26]

    Transformation Peak Average Final Linear 8223.515 5778.549 5873.256 Exponential 8326.043 5999.8797598.191 Softplus 8145.749 5917.624 6896.087 Sigmoid 8360.509 6066.2867498.798 Table 10: Training performance on SlipperyAnt (IQM across 15 seeds) withρ= 0.015andκ=

    arXiv

  16. [27]

    To ensure a fair comparison, we calibrated each function to share the same threshold value and fixed ρ= 0.015

    As shown in Table 10, the Sigmoid and Exponential transformations yield the highest returns. To ensure a fair comparison, we calibrated each function to share the same threshold value and fixed ρ= 0.015. We observe that the choice of transformation shape has a measurable impact on performance. The Exponential, Sigmoid, and Softplus functions all amplify r...

    2014

  17. [28]

    While ContinualWorld has a focus on catas- trophic forgetting we instead measure the adaptation performance using the latest versions of Meta- World environments

    tasks sequentially. While ContinualWorld has a focus on catas- trophic forgetting we instead measure the adaptation performance using the latest versions of Meta- World environments. We use the average and final performance of plasticity preservation baselines on this benchmark. This benchmark would take 3 days to run using an RTX3090. Continual MinAtar.T...

    2025

  18. [29]

    tx", tx), (

    with Optax (DeepMind et al., 2020). Once attached, the Optax optimizer can be used in the same way as a regular Optax optimizer, only if the reset method requires features (i.e. ReDo or CBP) then these are taken as an input to the optimizer.updatefunction. A simple optimizer setup is shown in Listing 1 Listing 1: Demonstration of simple optimizer using a ...

    2020

  19. [1998]

    Mitigating plasticity loss in continual reinforcement learning by reducing churn.arXiv preprint arXiv:2506.00592,

    Hongyao Tang, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Glen Berseth. Mitigating plasticity loss in continual reinforcement learning by reducing churn.arXiv preprint arXiv:2506.00592,

    Pith/arXiv arXiv

  20. [2013]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning.arXiv preprint arXiv:2403.00514,

    10 Michal Nauman, Michał Bortkiewicz, Piotr Miło ´s, Tomasz Trzci ´nski, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning.arXiv preprint arXiv:2403.00514,

    Pith/arXiv arXiv

  21. [2015]

    Reinitializing weights vs units for maintaining plasticity in neural networks.arXiv preprint arXiv:2508.00212,

    J Fernando Hernandez-Garcia, Shibhansh Dohare, Jun Luo, and Rich S Sutton. Reinitializing weights vs units for maintaining plasticity in neural networks.arXiv preprint arXiv:2508.00212,

    Pith/arXiv arXiv

  22. [2016]

    (2025) suggests reinitialization provides additional benefit

    can mitigate plasticity loss, but prior work Hernandez-Garcia et al. (2025) suggests reinitialization provides additional benefit. CBP (Dohare et al., 2021

    2025

  23. [2017]

    Maintaining plasticity in continual learning via regenerative regularization.arXiv preprint arXiv:2308.11958,

    Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization.arXiv preprint arXiv:2308.11958,

    Pith/arXiv arXiv

  24. [2018]

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba

    URLhttp: //github.com/jax-ml/jax. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, june 2016.arXiv preprint arXiv:1606.01540,

    Pith/arXiv arXiv 2016

  25. [2020]

    Layer normalization.arXiv preprint arXiv:1607.06450,

    Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization.arXiv preprint arXiv:1607.06450,

    Pith/arXiv arXiv

  26. [2021]

    Policy shifts can be amplified through feedback loops, spiraling into catastrophic per- formance collapse (Juliani & Ash, 2024)

    A CPR Algorithm Algorithm 1CPR (utilities every step; resets everyfsteps) Require:Update frequencyf, EMA decayβ, steepnessκ, max resetρ 1:Input:current stept 2:Compute per-neuron utilityS ℓ i (t){layer-normalized} 3:Update EMA:u ℓ i ←β uℓ i + (1−β) ˜Sℓ i 4:iftmodf= 0andt >0then{apply resets everyfsteps} 5:foreach layerℓdo 6:foreach neuronido 7:r ℓ i ←ρ·mi...

    1998

  27. [2023]

    press/v202/lyle23b.html

    URLhttps://proceedings.mlr. press/v202/lyle23b.html. Clare Lyle, Zeyu Zheng, Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks.arXiv preprint arXiv:2402.18762,

    Pith/arXiv arXiv

  28. [2024]

    Self-normalized resets for plasticity in continual learning

    Vivek F Farias and Adam D Jozefiak. Self-normalized resets for plasticity in continual learning. arXiv preprint arXiv:2410.20098,

    arXiv

  29. [2025]

    Transient non-stationarity and generalisation in deep reinforcement learning.arXiv preprint arXiv:2006.05826,

    Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning.arXiv preprint arXiv:2006.05826,

    Pith/arXiv arXiv 2006