Pith. sign in

REVIEW 3 major objections 5 minor 46 references

Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a learnable wavelet activation that splits each channel into a low-frequency base and high-frequency wavelets sustains plasticity in continual learning and outperforms both fixed and fully learnable activations.

desk verdict Promising empirical activation for continual learning, but the theory overreaches and the decoupled-LR confound inflates the headline gains. read the letter →

arxiv 2608.12874 v1 pith:Q7QH5XPK submitted 2026-08-13 cs.LG

Zeyang Zhang , Tieliang Gong , Junyan Lu , Weizhan Zhang This is my paper · ORCID
classification cs.LG
keywords continuallearningplasticitylosslearnableactivationwaveletspectralbiasdecoupledrateclass-incrementalcatastrophicforgetting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the activation function itself is a major, underused lever for preventing plasticity loss in continual learning. It proposes ChannelWavAct, which writes each channel's activation as a fixed low-frequency base (SiLU) plus a sum of learnable high-frequency Mexican Hat wavelets, and adds a loss-stagnation trigger that injects new wavelets when learning plateaus. The paper argues that this hybrid structure counters the spectral bias of fixed activations while regularization on wavelet amplitudes keeps earlier tasks from being overwritten. If correct, it gives a drop-in replacement for ReLU in continual learning models that sustains trainability over hundreds of tasks without replay buffers.

What carries the argument

The central object is ChannelWavAct, a channel-wise activation $\Phi_c(x_c)=w_{low,c}\sigma(x_c)+\sum_{k=1}^{K} w_{c,k}\psi((x_c-\tau_{c,k})/s_k)$, with $\sigma$ the SiLU base for low frequencies and $\psi(u)=(u^2-1)e^{-u^2/2}$ the Mexican Hat mother wavelet for high frequencies. Three mechanisms carry the argument: (1) dynamic wavelet injection, triggered when the training loss fails to drop by a relative margin for $P$ iterations, adds new wavelets with zero-initialized weights and translations so capacity grows without perturbing the learned state; (2) slope-specific regularization penalizes only old amplitude weights $w$, leaving translations and scales free; (3) a decoupled optimizer gives backbone and activation parameters a larger learning rate $\eta_{high}=\lambda_{lr}\eta_{base}$ than biases. The theoretical work uses the local dynamic NTK and a frequency-domain covering argument to show why each of these pieces is needed.

What would settle it

Measure the preconditioned NTK of a continual-learning ResNet using ChannelWavAct after several tasks, project the residual onto high-frequency eigendirections, and check whether $\eta_{high}\lambda_{high}(t)+\eta_{base}\epsilon(t)$ actually controls the decay rate. If residual decay stays slow even with a large $\eta_{high}$, or if the wavelet basis fails to align with the directions the new task needs ($\lambda_{high}$ near zero), Theorem 2's mechanism is not what drives the gains; alternatively, replacing the Mexican Hat wavelets with random high-frequency bumps of the same scale should leave performance nearly unchanged if only the learning rate matters.

Watch

Extended reading notes

Core claim

The central claim is that decomposing the activation into a global low-frequency part and localized high-frequency wavelets is both necessary and sufficient to sustain plasticity: the pure-wavelet part cannot approximate non-zero-mean signals without divergent scales, so the low-frequency base is structurally required, and the wavelet part is what lets high-frequency residuals decay at a useful rate. The paper formalizes this with an NTK analysis: under a uniform learning rate, high-frequency eigenvalues are too small to shrink residuals, but a decoupled learning rate that scales up backbone and activation parameters multiplies the high-frequency eigenvalue and restores convergence. It also proves that loss stagnation coincides with the residual entering the approximate null space of the kernel, which justifies the injection trigger, and gives a spectral-covering formula for how many new wavelets to add. Empirically, replacing ReLU with ChannelWavAct in ResNet backbones improves average and last accuracy on class-incremental benchmarks while keeping forgetting comparable.

Load-bearing premise

The proof that decoupled learning rates restore high-frequency plasticity rests on Assumption 1: high-frequency features are nearly orthogonal to the low-frequency base and the injected wavelets genuinely dominate the high-frequency subspace, so the base contributes only negligible leakage to those directions.

Editorial extensions

If this is right

  • In class-incremental settings, swapping ReLU for ChannelWavAct should raise average accuracy by two to three points on replay-based benchmarks and by larger margins in replay-free settings, according to the reported results.
  • The decoupled learning rate is not a tuning convenience but a necessary condition for high-frequency learning under the paper's analysis; without it the wavelet bases cannot converge fast enough.
  • The loss-stagnation trigger gives a practical, parameter-free way to detect capacity saturation: no separate plasticity metric or gradient statistics are required.
  • A hybrid base-plus-wavelet form is required for $L^2$ approximation of non-zero-mean signals; pure wavelet activations would need divergent scales and become unstable in finite networks.
  • Because post-activation batch normalization is part of the module, injected wavelets can be added without destabilizing feature distributions, which supports long sequences of injections.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is to apply ChannelWavAct to deep reinforcement learning, where plasticity loss and dormant neurons are measured directly; the dormant-ratio results in the paper suggest the same mechanism should keep units alive there.
  • The loss-stagnation trigger could be replaced by a rank-based or NTK-based detector; if the Rayleigh-quotient characterization in Theorem 3 is right, any detector of residual null-space entry should give similar injection timing.
  • The spectral robustness result implies that high-frequency components are structurally anchored in dedicated bases, which may also help in fine-tuning or sparse-replay regimes where parameter drift is common.
  • If the computational overhead the paper acknowledges can be reduced, this activation could become a default choice for long-horizon learning, not just class-incremental benchmarks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes ChannelWavAct, a channel-wise learnable wavelet activation for continual learning. The activation is a sum of a fixed SiLU base for low-frequency content and learnable Mexican-hat wavelets for high-frequency detail, with dynamic wavelet injection triggered by loss stagnation, slope-specific regularization, and a decoupled learning-rate schedule. The paper claims rigorous theoretical guarantees: that the hybrid wavelet architecture is necessary for efficient L2 approximation (Theorem 4), that the decoupled learning-rate mechanism restores plasticity for high-frequency information (Theorem 2), and that the loss-driven trigger is formally justified (Theorem 3). It reports extensive experiments on Permuted MNIST, Random Label MNIST, CIFAR-100, Mini-ImageNet, Tiny-ImageNet, and ImageNet-100 in both replay-based and replay-free continual learning settings, plus spectral analysis experiments.

Significance. If the results hold, the paper offers a useful and concrete activation-level intervention for plasticity loss, with the appealing idea of explicitly separating low-frequency stability from high-frequency adaptability. The empirical scope is broad, including multiple benchmarks, two continual learning frameworks, ablations, parameter sensitivity, and spectral diagnostics. The theoretical sections are standard in flavor but do not currently provide the promised 'rigorous mathematical guarantees': Theorem 2 depends on an assumption that is essentially the property being argued for, and Theorem 3 is not connected to the actual trigger hyperparameters. The main empirical comparison also needs a control for the decoupled optimization protocol before the performance claim can be attributed to the wavelet activation. These are fixable concerns, so the paper warrants a major revision rather than rejection.

major comments (3)
  1. [Section IV, Assumption 1 and Theorem 2 (Eqs. 16, 19, 20)] Theorem 2 assumes that the wavelet kernel already dominates the high-frequency direction, with Θ_high(t)v_high = λ_high(t)v_high and negligible leakage ε(t). The theorem then 'proves' that a decoupled learning rate amplifies the decay of high-frequency residuals. But the assumption is precisely the substantive property that the wavelet activation is claimed to provide, so the result does not establish that ChannelWavAct achieves this alignment or that the decoupled rate is necessary. Relatedly, Theorem 1 invokes the Frequency Principle to assert λ_high ≪ λ_low, which is a background empirical observation, not a theorem. The claims in Section IV that 'the results of Theorem 2 demonstrate that our decoupled learning rate mechanism effectively compensates for the deficiency' should be substantially softened, or the paper should provide empirical evidence (e.g., measuring the projection of task residuals onto the learned wavelet span during training) that the assumed alignment actually occurs.
  2. [Section V, Tables I, II and Table IV] The main cross-method comparisons train ChannelWavAct with the decoupled learning rate (η_high = λ_lr·η_base on backbone weights and activation parameters) and with optimizer rebuilding at injection, but the baseline activations are not documented as receiving the identical optimization protocol. Table IV in Section V-C shows that the decoupled rate alone contributes 4.7% average accuracy and 7.0% forgetting reduction for ChannelWavAct, and the same table suggests ReLU also improves by about 5.5 points when the decoupled protocol is applied. Unless all baselines in Tables I and II are rerun with the same decoupled-learning-rate and optimizer-reset protocol, the statement that 'our approach consistently outperforms existing baseline activation functions' over-attributes the gains to the wavelet activation rather than to a generic plasticity-enhancing optimization intervention. The paper should report baseline results under the identical optimization protocol in the main tables.
  3. [Section IV, Theorem 3 and Section V-C, Fig. 8] Theorem 3 derives a minimum injection size Δ* = ⌈μ(Ω_res)/B_ψ⌉ in Eq. (22) and claims to justify the loss-stagnation trigger. However, the trigger mechanism in Eq. (6) uses a counter with patience P and margin δ, and no formal connection is made between P, δ, and the Rayleigh-quotient threshold ε_tol from Eq. (21). The sensitivity analysis in Fig. 8 finds Δ = 1, P = 50, δ = 0.005 optimal, and the paper does not use Δ* to set any hyperparameter. The claim of providing 'a formal derivation of the loss-driven injection trigger mechanism to precisely guide the injection' is therefore not supported by the presented theory. The authors should either derive a practical calibration of P and δ from the theorem or rephrase the contribution.
minor comments (5)
  1. [Section V-A, paragraph on Random Label MNIST] The citation markers '[cite: 1063, 1064]' appear unresolved in the text and should be replaced with proper references.
  2. [Table II] The asterisk on 'ChannelWavAct*' is not defined in the caption or body text; the distinction between ChannelWavAct* and ChannelWavAct, and the relation to 'w/o L2 Regularization' and '+L2 Regularization', should be clarified.
  3. [Section III-A, Eq. (5), and Section IV, Theorem 4] The notation σ is inconsistent: Eq. (5) defines σ(x) = SiLU(x), while Theorem 4 writes φ(x) = x·σ(x) and calls this the SiLU-based scaling function; the reader is left to infer that σ is the sigmoid in one place and SiLU in another. Please use distinct symbols or state the convention explicitly.
  4. [Algorithm 2] The algorithm resets L*_min and C_bad at the start of each epoch, so the stagnation trigger only monitors within a single epoch and cannot detect cross-epoch plateaus. This is not obviously the same as the loss-stagnation notion analyzed in Theorem 3, and the discrepancy should be discussed.
  5. [Section I-B and V] The paper describes the results as 'state-of-the-art,' but the comparisons are limited to activation-function baselines within fixed frameworks (SSD, EWC, W A). Recent plasticity-loss methods that act through weight re-initialization or regularization (e.g., Shrink and Perturb, regenerative regularization) are cited in the related work but not compared against; the authors should either include such comparisons or qualify the claim.

Circularity Check

2 steps flagged · score 6.0 of 10

The theoretical 'necessity' results are built into Assumption 1 and the trigger definition; the empirical claims are externally benchmarked but the decoupled-LR protocol is not applied to baselines.

  1. self definitional [Section IV, Assumption 1 and Theorem 2 (Equations 16-20)]
    "Assumption 1: For a high-frequency task feature direction vhigh, ... the localized wavelet bases dominate this subspace with an instantaneous eigenvalue λhigh(t). Formally: Θbase(t)vhigh = ϵ(t)vhigh, where ϵ(t)≪1; Θhigh(t)vhigh = λhigh(t)vhigh. ... By explicitly setting ηhigh = λlrηbase with λlr≫1, the inherent high-frequency spectral bias is mathematically compensated, thereby restoring the instantaneous fitting capability of the wavelet bases."

    The theorem's conclusion is the assumption rescaled: Assumption 1 already grants that the wavelet subspace has eigenvalue λhigh(t) on the high-frequency direction vhigh, and Theorem 2 merely multiplies that assumed eigenvalue by ηhigh to obtain the residual decay rate. Thus the 'proof' that the decoupled learning rate restores high-frequency plasticity is contained in the premise that the high-frequency information already lives in the wavelet subspace. It does not independently establish that ChannelWavAct's wavelets capture task-relevant high frequencies, nor that the hybrid wavelet architecture is necessary.

  2. self definitional [Section IV, Theorem 3 and Algorithm 2; Section V-C (Figure 8a)]
    "Theorem 3 provides a theoretical foundation for our loss detection mechanism. Since the numerator of the Rayleigh quotient governs the time derivative of the loss, i.e., d/dt L = −uTKΘKuK, its collapse manifests as a loss stagnation plateau. This justifies our monitoring strategy... the minimum optimal incremental capacity Δ* = ⌈µ(Ωres)/Bψ⌉."

    Algorithm 2 defines the injection trigger by counting iterations where the loss fails to drop (C_bad increments when L_CE ≥ L*_min(1−δ)); the trigger is therefore 'loss stagnation' by definition. Theorem 3 restates this same condition as Rayleigh quotient collapse under gradient flow, so calling it a 'formal derivation' of the trigger is a relabeling rather than an independent determination. Moreover, the derived Δ* is never computed or used: Figure 8a shows Δ=1 is selected by tuning, so the 'capacity expansion' quantity is a fitted hyperparameter, not a predicted value.

full rationale

The empirical comparisons are self-contained against external benchmarks (Tables I-II, ablations), so there is no load-bearing self-citation chain and no pattern of renaming a known empirical result. The Frequency Principle citation and the cited wavelet/frame theorems are external mathematical facts. The main circularity is in the theoretical section: Theorem 2's high-frequency plasticity guarantee is a direct consequence of Assumption 1, which already places the high-frequency eigenspace in the wavelet parameters; and Theorem 3's loss-stagnation characterization is essentially the definition of the Algorithm 2 trigger re-expressed in NTK language, with the derived Δ* replaced by a tuned Δ=1. These are partial circularities that weaken the 'rigorous mathematical guarantees' contribution, while the state-of-the-art empirical claim remains externally testable. Note also that the decoupled-learning-rate protocol is not applied to the baseline activations, and the ablation shows it is worth about 4.7% Average Accuracy; that is a comparison-protocol concern rather than a circularity, so it is not counted in the score.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method introduces several new hyperparameters (patience, margin, regularization weight, injection size, learning-rate multiplier) that are either tuned on the evaluation dataset or not reported. The main extra assumption is Assumption 1, which carries the weight of the theory. No genuinely new entities are postulated.

free parameters (6)
  • patience P = 50
    Tuned on CIFAR-100 sensitivity analysis (Fig. 8b); controls when injection triggers.
  • margin δ = 0.005
    Tuned on CIFAR-100 (Fig. 8d); loss stagnation detector threshold.
  • regularization coefficient λ = 1
    Tuned on CIFAR-100 (Fig. 8c); strength of slope-specific penalty on old wavelet weights.
  • injection number Δ = 1
    Tuned on CIFAR-100 (Fig. 8a); number of wavelets added per expansion.
  • learning rate multiplier λ_lr = not reported
    Algorithm 2 takes λ_lr as input; the value used in experiments is not stated in the paper.
  • regularization coefficients λ_low, λ_wav = not reported
    Eq. (8) defines both, Algorithm 2 uses them, but concrete values are not given.
assumptions (4)
  • domain assumption Dynamic Frequency Principle: high-frequency features have eigenvalues orders of magnitude smaller than low-frequency features in NTK dynamics.
    Invoked in Theorem 1 after Eq. (15) and in Theorem 2 to argue uniform learning rate cannot suppress high-frequency residuals. This is an empirical observation from the literature, not proven here.
  • domain assumption Approximate eigensystem assumption: projection directions v_i satisfy Θ(t)v_i ≈ λ_i(t)v_i at each time t.
    Assumed at the start of Theorem 1; in practice the NTK of finite-width networks is not exactly diagonalized by frequency directions.
  • ad hoc to paper Assumption 1: base kernel leakage onto high-frequency directions is negligible (ε(t)≪1) and the wavelet kernel dominates with eigenvalue λ_high(t).
    This is the key premise of Theorem 2; it already encodes the conclusion that wavelet bases are the right high-frequency components, making the later 'proof' conditional on its own desired property.
  • standard math Standard wavelet and frame theory results: Plancherel theorem, Frame Truncation Theorem, and the zero-mean property of wavelets.
    Used in Theorem 4 to show a pure wavelet system cannot approximate nonzero-mean signals and that finite wavelet frames approximate zero-mean residuals; these are textbook facts.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning." pith.science (2026). https://pith.science/paper/Q7QH5XPK

@misc{pith2026260812874,
  author       = {Pith},
  title        = {Pith review of: Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q7QH5XPK}},
  note         = {Machine review of arXiv:2608.12874}
}
abstract

Plasticity loss has emerged as a critical challenge in continual learning that significantly hinders the acquisition of sequential tasks. While optimizing activation designs offers a potential solution, current fixed-form functions suffer from an inherent spectral bias towards low-frequency variations, whereas learnable variants permit unconstrained updates that induce catastrophic forgetting. To address these limitations, we propose a novel learnable wavelet activation that decomposes the activation function into low-frequency and high-frequency components to explicitly counter spectral bias. Furthermore, we employ dynamic wavelet injection to adaptively enhance plasticity for new tasks, alongside a regularization strategy to ensure the stability of previous learned knowledge. Theoretically, we provide rigorous mathematical guarantees for the proposed framework, proving the structural necessity of the hybrid wavelet architecture for efficient $L^2$ approximation and demonstrating that the decoupled learning rate mechanism successfully restores network plasticity for high-frequency information. Additionally, we provide a formal derivation of the loss-driven injection trigger mechanism to precisely guide the injection. Extensive empirical evaluations demonstrate that our approach maintains superior trainability and generalization throughout the learning process and achieves state-of-the-art performance across diverse continual learning benchmarks.

Figures

Figures reproduced from arXiv: 2608.12874 by the authors.

Figure 1
Figure 1. Schematic illustration of the ChannelWavAct framework. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Evolution of the Dormant Neuron Ratio under Input and Label Trainability settings. The left and right panels display the dormant neuron ratio across sequential tasks for Permuted MNIST and Random Label MNIST. combination of a basis function b(x) = silu(x) = x/(1+e −x ) and a B-spline function spline(x) = P i ciBi(x). The resulting learnable activation function ϕ(x) is defined as: ϕ(x) = wb · b(x) + ws · spline(x) (3… view at source ↗
Figure 3
Figure 3. (a) Accuracy of various activation functions on the Permuted MNIST [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: (a) Evolution of the learned amplitude for frequency components [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 6
Figure 6. Figure 6: Validation loss on the clean target τ0 for ReLU (left) and ChannelWavAct (right), evaluated as a function of (a) noise amplitude β with fixed frequencies k˜ ∈ {0.1, 1.0}, and (b) spatial noise frequency k˜ with fixed amplitudes β ∈ {0.5, 1.0}. dard activations, achievi…
Figure 5
Figure 5. Figure 5: Heatmaps of normalized spectral amplitude across frequencies ( [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 7
Figure 7. Figure 7: Heatmaps of the normalized spectral amplitude across learned [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Impact of key hyperparameters on continual learning performance, [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages

  1. [1]

    icarl: Incremental classifier and representation learning,

    S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,” inProceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, pp. 2001–2010

    2017

  2. [2]

    Rethinking the stability-plasticity trade-off in continual learning from an architectural perspective,

    A. Lu, H. Yuan, T. Feng, and Y . Sun, “Rethinking the stability-plasticity trade-off in continual learning from an architectural perspective,”arXiv preprint arXiv:2506.03951, 2025

    arXiv 2025

  3. [3]

    Coreset selection via reducible loss in continual learning,

    R. Tong, Y . Liu, J. Q. Shi, and D. Gong, “Coreset selection via reducible loss in continual learning,” inThe Thirteenth International Conference on Learning Representations, 2025

    work page 2025

  4. [4]

    Summarizing stream data for memory-constrained online continual learning,

    J. Gu, K. Wang, W. Jiang, and Y . You, “Summarizing stream data for memory-constrained online continual learning,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 11, 2024, pp. 12 217–12 225

    work page 2024

  5. [5]

    A new regularization- based continual learning framework,

    S. Hassan, N. Rasheed, and M. A. Qureshi, “A new regularization- based continual learning framework,” in2024 Horizons of Information Technology and Engineering (HITE). IEEE, 2024, pp. 1–5

    work page 2024

  6. [6]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al., “Overcoming catastrophic forgetting in neural networks,”Pro- ceedings of the national academy of sciences, vol. 114, no. 13, pp. 3521–3526, 2017

    2017

  7. [7]

    A model or 603 exemplars: Towards memory-efficient class-incremental learning,

    D.-W. Zhou, Q.-W. Wang, H.-J. Ye, and D.-C. Zhan, “A model or 603 exemplars: Towards memory-efficient class-incremental learning,”arXiv preprint arXiv:2205.13218, 2022

    arXiv 2022

  8. [8]

    Growing a brain with sparsity-inducing generation for continual learning,

    H. Jin, G.-h. Kim, C. Ahn, and E. Kim, “Growing a brain with sparsity-inducing generation for continual learning,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 18 961–18 970

    work page 2023

Show all 46 references
  1. [9]

    Drm: Mastering visual reinforcement learning through dormant ratio minimization,

    G. Xu, R. Zheng, Y . Liang, X. Wang, Z. Yuan, T. Ji, Y . Luo, X. Liu, J. Yuan, P. Huaet al., “Drm: Mastering visual reinforcement learning through dormant ratio minimization,”arXiv preprint arXiv:2310.19668, 2023

    2023 arXiv

  2. [10]

    Directions of curvature as an explanation for loss of plasticity,

    A. Lewandowski, H. Tanaka, D. Schuurmans, and M. C. Machado, “Directions of curvature as an explanation for loss of plasticity,”arXiv preprint arXiv:2312.00246, 2023

    2023 arXiv

  3. [11]

    Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks,

    H. Lee, H. Cho, H. Kim, D. Kim, D. Min, J. Choo, and C. Lyle, “Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks,”arXiv preprint arXiv:2406.02596, 2024

    2024 arXiv

  4. [12]

    Deep reinforcement learning with plasticity injection,

    E. Nikishin, J. Oh, G. Ostrovski, C. Lyle, R. Pascanu, W. Dabney, and A. Barreto, “Deep reinforcement learning with plasticity injection,”Ad- vances in Neural Information Processing Systems, vol. 36, pp. 37 142– 37 159, 2023

    2023

  5. [13]

    The dormant neuron phenomenon in deep reinforcement learning,

    G. Sokar, R. Agarwal, P. S. Castro, and U. Evci, “The dormant neuron phenomenon in deep reinforcement learning,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 32 145–32 168

    2023

  6. [14]

    Understanding plasticity in neural networks,

    C. Lyle, Z. Zheng, E. Nikishin, B. A. Pires, R. Pascanu, and W. Dabney, “Understanding plasticity in neural networks,” inInternational Confer- ence on Machine Learning. PMLR, 2023, pp. 23 190–23 211

    2023

  7. [15]

    Implicit under- parameterization inhibits data-efficient deep reinforcement learning,

    A. Kumar, R. Agarwal, D. Ghosh, and S. Levine, “Implicit under- parameterization inhibits data-efficient deep reinforcement learning,” arXiv preprint arXiv:2010.14498, 2020

    2010 arXiv

  8. [16]

    Dash: Warm-starting neural net- work training in stationary settings without loss of plasticity,

    B. Shin, J. Oh, H. Cho, and C. Yun, “Dash: Warm-starting neural net- work training in stationary settings without loss of plasticity,”Advances in Neural Information Processing Systems, vol. 37, pp. 43 300–43 340, 2024

    2024

  9. [17]

    Loss of plasticity in deep continual learning,

    S. Dohare, J. F. Hernandez-Garcia, Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton, “Loss of plasticity in deep continual learning,”Nature, vol. 632, no. 8026, pp. 768–774, 2024

    2024

  10. [18]

    Addressing loss of plasticity and catastrophic forgetting in continual learning,

    M. Elsayed and A. R. Mahmood, “Addressing loss of plasticity and catastrophic forgetting in continual learning,”arXiv preprint arXiv:2404.00781, 2024

    2024 arXiv

  11. [19]

    Learning continually by spectral regularization,

    A. Lewandowski, M. Bortkiewicz, S. Kumar, A. Gy ¨orgy, D. Schuur- mans, M. Ostaszewski, and M. C. Machado, “Learning continually by spectral regularization,”arXiv preprint arXiv:2406.06811, 2024

    2024 arXiv

  12. [20]

    Weight clip- ping for deep continual and reinforcement learning,

    M. Elsayed, Q. Lan, C. Lyle, and A. R. Mahmood, “Weight clip- ping for deep continual and reinforcement learning,”arXiv preprint arXiv:2407.01704, 2024

    2024 arXiv

  13. [21]

    Adaptive regularization of representation rank as an implicit constraint of bellman equation,

    Q. He, T. Zhou, M. Fang, and S. Maghsudi, “Adaptive regularization of representation rank as an implicit constraint of bellman equation,”arXiv preprint arXiv:2404.12754, 2024

    2024 arXiv

  14. [22]

    Understanding and pre- venting capacity loss in reinforcement learning,

    C. Lyle, M. Rowland, and W. Dabney, “Understanding and pre- venting capacity loss in reinforcement learning,”arXiv preprint arXiv:2204.09560, 2022

    2022 arXiv

  15. [23]

    Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control,

    M. Nauman, M. Ostaszewski, K. Jankowski, P. Miło ´s, and M. Cygan, “Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control,”Advances in neural information processing systems, vol. 37, pp. 113 038–113 071, 2024

    2024

  16. [24]

    In value-based deep reinforcement learning, a pruned network is a good network,

    J. Obando-Ceron, A. Courville, and P. S. Castro, “In value-based deep reinforcement learning, a pruned network is a good network,”arXiv preprint arXiv:2402.12479, 2024

    2024 arXiv

  17. [25]

    Activation by interval-wise dropout: A simple way to prevent neural networks from plasticity loss,

    S. Park, I. Han, S. Oh, and K.-J. Kim, “Activation by interval-wise dropout: A simple way to prevent neural networks from plasticity loss,” arXiv preprint arXiv:2502.01342, 2025

    2025 arXiv

  18. [26]

    Activation function design sustains plasticity in continual learning,

    L. Lillo and N. Cheney, “Activation function design sustains plasticity in continual learning,”arXiv preprint arXiv:2509.22562, 2025

    2025 arXiv

  19. [27]

    Rectifier nonlinearities improve neural network acoustic models,

    A. L. Maas, A. Y . Hannun, A. Y . Nget al., “Rectifier nonlinearities improve neural network acoustic models,” inProc. icml, vol. 30, no. 1. Atlanta, GA, 2013, p. 3

    2013

  20. [28]

    On the spectral bias of neural networks,

    N. Rahaman, A. Baratin, D. Arpit, F. Draxler, M. Lin, F. Hamprecht, Y . Bengio, and A. Courville, “On the spectral bias of neural networks,” inInternational conference on machine learning. PMLR, 2019, pp. 5301–5310

    2019

  21. [29]

    Staf: Sinusoidal trainable activation functions for implicit neural representation,

    A. Morsali, M. Vaez, M. Soltani, A. Kazerouni, B. Taati, and M. Mohammad-Noori, “Staf: Sinusoidal trainable activation functions for implicit neural representation,”arXiv preprint arXiv:2502.00869, 2025

    2025

  22. [30]

    Adaptive rational activations to boost deep reinforcement learning,

    Q. Delfosse, P. Schramowski, M. Mundt, A. Molina, and K. Kersting, “Adaptive rational activations to boost deep reinforcement learning,” arXiv preprint arXiv:2102.09407, 2021

    2021 arXiv

  23. [31]

    Kan: Kolmogorov-arnold networks,

    Z. Liu, Y . Wang, S. Vaidya, F. Ruehle, J. Halverson, M. Solja ˇci´c, T. Y . Hou, and M. Tegmark, “Kan: Kolmogorov-arnold networks,”arXiv preprint arXiv:2404.19756, 2024

    2024 arXiv

  24. [32]

    Continual backprop: Stochastic gradient descent with persistent randomness,

    S. Dohare, R. S. Sutton, and A. R. Mahmood, “Continual backprop: Stochastic gradient descent with persistent randomness,”arXiv preprint arXiv:2108.06325, 2021

    2021 arXiv

  25. [33]

    On the realization of a kolmogorov network,

    J.-N. Lin and R. Unbehauen, “On the realization of a kolmogorov network,”Neural Computation, vol. 5, no. 1, pp. 18–20, 1993

    1993

  26. [34]

    Wav-kan: Wavelet kolmogorov-arnold networks,

    Z. Bozorgasl and H. Chen, “Wav-kan: Wavelet kolmogorov-arnold networks,”arXiv preprint arXiv:2405.12832, 2024

    2024 arXiv

  27. [35]

    Neural tangent kernel: Con- vergence and generalization in neural networks,

    A. Jacot, F. Gabriel, and C. Hongler, “Neural tangent kernel: Con- vergence and generalization in neural networks,”Advances in neural information processing systems, vol. 31, 2018

    2018

  28. [36]

    Frequency principle: Fourier analysis sheds light on deep neural networks,

    Z.-Q. J. Xu, Y . Zhang, T. Luo, Y . Xiao, and Z. Ma, “Frequency principle: Fourier analysis sheds light on deep neural networks,”arXiv preprint arXiv:1901.06523, 2019

    1901 arXiv

  29. [37]

    Convergence beyond the over- parameterized regime using rayleigh quotients,

    D. A. Robin, K. Scamanet al., “Convergence beyond the over- parameterized regime using rayleigh quotients,”Advances in Neural Information Processing Systems, vol. 35, pp. 10 725–10 736, 2022

    2022

  30. [38]

    Discrete-time signal processing,

    H. Pfister, “Discrete-time signal processing,”Lecture Note, pfister. ee. duke. edu/courses/ece485/dtsp. pdf, 2017

    2017

  31. [39]

    A wavelet tour of signal processing: the sparse way

    I. J. Brown, “A wavelet tour of signal processing: the sparse way.” Investigacion Operacional, vol. 30, no. 1, pp. 85–87, 2009

    2009

  32. [40]

    E. M. Stein and R. Shakarchi,Fourier analysis: an introduction. Princeton University Press, 2011, vol. 1

    2011

  33. [41]

    A wavelet tour of signal processing,

    M. Stephane, “A wavelet tour of signal processing,” 1999

    1999

  34. [42]

    Maintaining plasticity in continual learning via regenerative regularization,

    S. Kumar, H. Marklund, and B. Van Roy, “Maintaining plasticity in continual learning via regenerative regularization,”arXiv preprint arXiv:2308.11958, 2023

    2023 arXiv

  35. [43]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,”Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 2002

    2002

  36. [44]

    Pycil: a python toolbox for class-incremental learning,

    D.-W. Zhou, F.-Y . Wang, H.-J. Ye, and D.-C. Zhan, “Pycil: a python toolbox for class-incremental learning,”SCIENCE CHINA Information Sciences, vol. 66, no. 9, p. 197101, 2023

    2023

  37. [45]

    Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning,

    Z. Mai, R. Li, H. Kim, and S. Sanner, “Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 3589–3599

    2021

  38. [46]

    Maintaining dis- crimination and fairness in class incremental learning,

    B. Zhao, X. Xiao, G. Gan, B. Zhang, and S.-T. Xia, “Maintaining dis- crimination and fairness in class incremental learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 13 208–13 217

    2020

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.