The Importance of Being Lazy: Scaling Limits of Continual Learning

Alessandro Breccia; Giulia Lanzillotta; Jacopo Graldi; Lorenzo Noci; Thomas Hofmann

REVIEW 4 major objections 5 minor 13 references

The Importance of Being Lazy: Scaling Limits of Continual Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that in continual learning, whether wider networks forget less is set by the degree of feature learning in the parameterization, not by width alone.

desk verdict A genuinely useful paper that extends DMFT to continual learning and reconciles contradictory scaling results, but the central causal claim about feature learning driving forgetting is confounded with learning-rate scaling and the paper itself concedes this. read the letter →

arxiv 2506.16884 v2 pith:QQTZPFJX submitted 2025-06-20 cs.LG cs.AIstat.ML

Jacopo Graldi , Alessandro Breccia , Giulia Lanzillotta , Thomas Hofmann , Lorenzo Noci This is my paper · ORCID

classification cs.LGcs.AIstat.ML

keywords catastrophicforgettingcontinuallearningfeaturelazytrainingrichregimewidthscalingdynamicalmeanfieldtheoryneuraltangentkernel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks why neural networks trained on a sequence of tasks forget earlier tasks, and whether scaling the network helps. Its answer is that the benefit of width is not intrinsic: it depends on how much the network's features move during training. The authors interpolate between a lazy regime, where features barely evolve, and a rich feature-learning regime, using a parameter gamma0, and find that catastrophic forgetting stays low until a critical amount of feature learning and then rises sharply. The optimal balance between learning new tasks and retaining old ones sits at a fixed intermediate value gamma*_0 around 0.1 that transfers across widths, and it moves toward richer regimes as tasks become more similar. A theoretical infinite-width description, built with dynamical mean field theory, reproduces the finite-width behavior and grounds the claim that feature learning, not scale, is the controlling factor.

What carries the argument

The load-bearing object is a one-parameter family of network parameterizations: the output scale gamma = gamma0 $\sqrt$(N) and learning rate eta = eta0 $gamma0^{2}$ N make gamma0=0 the lazy neural-tangent limit and gamma0=1 the rich mean-field/maximal-update limit. The authors measure the resulting feature movement with centered kernel alignment (CKA) and forgetting with a new relative metric, the catastrophic forgetting rate (CFr), which computes the average relative drop in per-task accuracy. The theoretical engine is a dynamical mean field theory for infinite width: preactivations h and pregradient variables z become stochastic processes, and the forward and backward kernels Phi and G concentrate at their saddle-point expectations, giving a self-consistent system of equations for the output dynamics under sequential tasks. A perturbation expansion in powers of gamma0 around the lazy limit shows the first-order correction to forgetting vanishes, and the second-order terms, expressed through residuals and the across-task neural tangent kernel, carry the leading feature-learning effect.

What would settle it

Run the same sequential-training sweep over gamma0 while re-tuning the learning rate to its per-gamma0 optimum, for example using the sub-quadratic LR scaling for rich regimes suggested by prior optimization-landscape work; if the sharp low-to-high forgetting transition disappears or moves, the transition is driven by non-optimal learning rates rather than by feature learning itself, whereas if it persists the causal role of feature learning is confirmed.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that the effect of model scale on catastrophic forgetting is modulated by the degree of feature learning, not by width alone. Under neural tangent parameterization, where features are effectively frozen, increasing width monotonically reduces forgetting; under maximal update parameterization, where features learn at any width, width does not reduce forgetting and can even increase it in the rich regime. Interpolating with gamma0 between the two parameterizations reveals a lazy-rich transition: below a data-dependent threshold the features barely move and forgetting is low, above it feature evolution and forgetting both rise sharply. Consequently the plasticity-stability optimum is achieved at a critical, data-dependent gamma*_0 < 1, which transfers across widths (and, with a depth-corrected parameterization, across depths), so larger networks beyond this point waste capacity. Task similarity shifts the entire picture: low similarity moves the optimum toward laziness, while high similarity, where first-task features are reused, can make rich learning beneficial and restore the benefits of width. The paper also extends dynamical mean field theory to sequential tasks, giving closed-form infinite-width equations whose simulations match finite networks.

Load-bearing premise

The sweep over gamma0 is not a pure manipulation of feature learning, because the learning rate is scaled as eta0 $gamma0^{2}$ N, so raising gamma0 also moves the learning rate away from the optimal value; the paper itself notes in Section 5.1 that this may trigger the sharp rise it attributes to feature learning.

Editorial extensions

If this is right

Under the paper's account, width scaling is useful in continual learning only while the network stays effectively lazy; past the lazy-rich transition, additional width does not reduce forgetting and can worsen it.
The optimal feature-learning strength gamma*_0 is around 0.1 and transfers across widths and, under the depth-corrected parameterization, across depths, so practitioners could tune gamma0 on a small model and reuse it at scale.
Because task similarity controls the transition, highly non-stationary streams call for closer-to-lazy training, while similar tasks allow richer learning before forgetting becomes destructive.
The extended dynamical mean field theory reproduces finite-width forgetting at infinite width, indicating these effects persist in the scaling limit.
In high-similarity benchmarks, a pretraining effect can make forgetting non-monotonic in gamma0 and restore the benefit of width, connecting the result to earlier observations that pretrained models scale better.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If feature learning is the mechanism, a natural untested remedy is to bound representation drift during later tasks, for instance by penalizing feature-kernel change, rather than reducing global plasticity, which might preserve learning while suppressing the forgetting driver.
The transfer of gamma* suggests a zero-shot continual-learning scaling recipe: tune gamma0 on a small-width model, then scale width; the paper shows this transfer on the benchmarks it studies, leaving breadth of architectures and datasets as an open extension.
The learning-rate confound in Section 5.1 implies a sharper experiment: separate gamma0's effect on feature learning from its effect on learning rate by an alternative parameterization; the paper leaves this to future work, but the distinction determines whether the transition is about features or optimization.
One deeper implication is that scaling laws for continual learning should include task similarity as a parameter; models tuned to a fixed degree of feature learning may be near-optimal only for the non-stationarity level they were tuned on.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this one. It does something real: it gives the first dynamical mean field theory for sequential multi-task training, extends the lazy/rich gamma0 interpolation to forgetting, and introduces a cleaner relative-drop metric (CFr) that fixes a known artifact in prior CF measures. The infinite-width predictions match finite-width runs in Appendix D, and the small-gamma0 perturbation expansion in Appendix E is a nice bonus. That alone is worth a serious referee's time.

The soft spot is exactly what the stress-test note says: the gamma0 sweep is not a pure feature-learning sweep. Table 1 sets the LR as eta0 * gamma0^2 * N, so varying gamma0 changes feature-learning strength and the LR relative to its per-gamma0 optimum simultaneously. Section 5.1 openly floats this: "our LR shifts from the optimal LR towards a larger-than-optimal LR, potentially triggering the sharp rise we observe." The DMFT simulation inherits the same coupling, so it cannot break the confound either. The strong correlation in Fig. 3d shows gamma0 drives both feature evolution and CFr, but it does not show that feature learning per se causes forgetting. The authors are honest about this, but honesty is not the same as resolving it.

Other concerns are smaller, not fatal. Main plots average five seeds without error bars; the optimal gamma* is read off visually; Prop. 4.1 is stated informally; no code or data are shipped. None of these undermine the theoretical core, but they cap the confidence one can have in the empirical claims.

I disagree with the reader's weakest-assumption framing only in one respect: the paper's central narrative does not collapse. Even if the sharp transition is partly an LR artifact, the width-scaling results in µP versus NTP still hold, the DMFT framework still works, and the transfer of gamma* across widths remains an interesting empirical fact. What is weakened is the causal attribution: "feature learning is ultimately responsible for forgetting" is a hypothesis, not a settled conclusion, until someone reruns the sweep with per-gamma0 LR re-tuning (or shows the LR shift is irrelevant). That experiment is straightforward and the authors should be pushed to do it.

Bottom line: send it to good referees. The theory contribution is solid, the phenomenology is interesting, and the confound is fixable. I would cite it, and I would bring it to the reading group, but I would not yet put the causal claim in a lecture slide without a caveat.

Referee Report

4 major / 5 minor

Summary. The paper studies catastrophic forgetting (CF) in wide neural networks as a function of width scaling and the degree of feature learning. The authors introduce a parameter gamma0 that interpolates between the lazy (NTP) and rich (muP) training regimes, and report that width scaling reduces CF only in the lazy regime, while high feature learning is associated with a sharp low-to-high forgetting transition. They extend dynamical mean field theory (DMFT) to sequential multi-task training for a two-layer MLP, validate the infinite-width predictions against finite-width runs, and use this framework to characterize the dependence of forgetting on task similarity. The central quantitative claim is that the optimal plasticity-stability tradeoff occurs at a data-dependent gamma*_0 < 1 that transfers across widths, and that this optimum is controlled by feature learning.

Significance. If the central claim is correct, the paper would reconcile contradictory empirical results on width scaling in continual learning and provide a practically useful transfer rule: tune gamma0 on small models and reuse it at scale. The paper also makes a substantial theoretical contribution by extending DMFT to non-stationary training, which is new relative to prior lazy-regime analyses. The manuscript ships a machine-checkable DMFT simulation, a detailed finite-width comparison in Appendix D, and an explicit perturbation expansion for small gamma0; these are concrete and reproducible. The novel CFr metric is a reasonable response to known shortcomings of the raw CF metric. However, the main causal conclusion is presently weakened by a learning-rate confound in the gamma0 sweep, which the authors themselves flag in Section 5.1.

major comments (4)

[Section 5.1, Table 1] The gamma0 sweep is not a pure feature-learning sweep. Table 1 sets the learning rate to eta0(t) * gamma0^2 * N, so varying gamma0 simultaneously changes the output scale, the feature-learning strength, and the learning rate relative to its per-gamma0 optimum. The manuscript explicitly hypothesizes in Section 5.1 that the sharp transition 'could be related to our choice of LR scaling' and that the LR shifts from optimal to larger-than-optimal as gamma0 increases. This is load-bearing because the paper's central claim is that feature learning is ultimately responsible for forgetting. The infinite-width DMFT simulation inherits the same coupling, since the DMFT equations are derived under the same eta = O(gamma^2) scaling, so it does not break the confound. To establish the causal claim, the authors should re-tune or rescale the learning rate for each gamma0 (for example, following the optimal LR scaling of Atanasov et al.) and check whether the transition and gamma*_0 persist; alternatively, they should provide a direct manipulation of feature evolution at matched LR.
[Appendix E, Proposition E.1] The perturbation theory is explicitly a small-gamma0 expansion for a linear (identity-activation) two-layer network, and it cannot describe the sharp transition observed at gamma0 approximately 0.1. The paper's evidence for the transition is therefore numerical (finite-width experiments and DMFT simulations), not analytical. This is acceptable as a numerical finding, but the abstract and Section 5 should be more careful: the sharp lazy-rich transition is not derived from the perturbation expansion, and the closed-form results in Appendix E.2 apply only to the infinite-time, linear-network, gamma0 = 0 limit of CF. Please state this scope limitation explicitly where the transition is claimed.
[Figures 1, 3, 4] The main quantitative claims, including gamma*_0 approximately 0.1 and its transfer across widths, are based on averages over five random seeds with no error bars, and gamma*_0 is read visually from the curves. Without error bars or per-seed plots, the claim of exact transfer across widths is not statistically supported. The infinite-width simulation provides independent corroboration, but the finite-width curves are the primary evidence for transfer. Please add error bars, confidence intervals, or seed-level scatter plots, and state the number of seeds used for each figure.
[Section 5.1, Figure 3d] The strong correlation between feature evolution (1 - CKA) and CFr shown in Figure 3d does not establish that feature learning per se causes forgetting. Since gamma0 drives both quantities, the correlation is consistent with gamma0 being a common cause. Combined with the LR confound in the same sweep, this correlation is not sufficient for the causal statement in Section 7 that 'the degree of feature learning is ultimately responsible for forgetting.' A mediation analysis, a matched-LR control, or an intervention that changes feature evolution independently of gamma0 would be needed.

minor comments (5)

[Section 5.2] The main text states gamma*_0 is approximately 0.1 for both Split-CIFAR10 and Permuted-MNIST, but Figure 12 in Appendix B.3 labels the MLP optimum as gamma*_0 = 0.2. Please reconcile these values or explain the discrepancy.
[Section 3.1.1, Definition A.4] The notation max_{t in {i,...,T-1}} in the CFr definition is slightly confusing because t is used both as a time index and as a task index. Consider renaming the maximization variable to avoid ambiguity, e.g., max_{k in {i,...,T-1}}.
[Section 5.1, Figure 3d] The paper reports a p-value < 10^-30 for the correlation in Figure 3d but does not specify the statistical test, the number of points, or whether the points are independent. Please provide this information in the caption or appendix.
[Appendix E.2.1] The modeling setup requires D >= 2P for constructing orthogonal task rotations, but this condition is only mentioned in the appendix. It would be helpful to state it in the main text where task similarity rho is introduced, since it limits the generality of the closed-form results.
[General] The caption of Figure 1 mentions that all figures report averages over 5 random seeds, but several later figures do not repeat this information. Please state the number of seeds and the variance measure (or its absence) consistently across all figures.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: finite-width sweeps, DMFT simulation, and perturbation expansion are independent of the CF behavior they predict; the LR-scaling confound is a causal-inference risk, not a circular reduction.

full rationale

The paper's central empirical claim is a sweep over gamma0 with CFr and feature evolution measured independently via CKA, so the gamma0-CFr curve is not defined into existence. The DMFT infinite-width dynamics (Proposition 4.1, Eq. 8) are derived from the model and gradient flow in Appendix C, simulated from sampled initial Gaussian fields with the LR tuned on non-CL data, and then compared with finite-width runs; the finite-width CFr converging to the infinite-width limit is a genuine out-of-sample check, not a fitted reproduction. The perturbation expansion (Appendix E, Proposition E.1) computes CF^(0) and CF^(2) from stated covariance assumptions (Kx_TiTi = I, Kx_TiTj = rho I) and the model equations; no fitted constant is renamed as a prediction. The only load-bearing weakness is the confound admitted in Section 5.1: 'We hypothesize that this transition could be related to our choice of LR scaling... our LR shifts from the optimal LR towards a larger-than-optimal LR, potentially triggering the sharp rise we observe.' That weakens the causal claim that feature learning per se drives forgetting, but it is a correctness risk, not a circularity: feature learning is not defined in terms of forgetting, and the theoretical simulations are not fitted to the finite-width forgetting transition. Self-citations (Noci et al. 2024; Bordelon et al. 2023) are used as background or as a hypothesis for gamma* transfer, and the transfer claim is supported by the paper's own across-width sweeps; hence they are not load-bearing. No circular step found.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central contribution rests on several modeling choices: muP scaling with gamma0-dependent learning rate, gradient-flow idealization, DMFT kernel concentration, and a synthetic task-similarity covariance model. No new physical entities are introduced. The main empirical optimum gamma*_0 and per-dataset base learning rates are selected from data, but they are not retrofitted to force the DMFT curves to match.

free parameters (2)

gamma0 (feature-learning interpolation strength) = swept from 10^-3 to 1; empirical optimum gamma*_0 ~ 0.1 on Split-CIFAR10 and Permuted-MNIST
Central control variable in the muP parameterization. The claim that an intermediate, data-dependent gamma*_0 is optimal and transfers across widths is read off from these sweeps.
Base learning rate eta0(0) = 30.0 (Split-CIFAR10), 2.0 (Permuted-MNIST), 15.0 (Split-TinyImageNet), 0.25 (MLP)
Tuned on stationary data at base width and then scaled as eta0 * gamma0^2 * N. This scaling confounds the gamma0 sweep because large gamma0 also implies an effectively larger learning rate.

assumptions (4)

domain assumption muP scaling from Table 1, with learning rate eta(t) = eta0 * gamma0^2 * N, yields O(1) NTK evolution and a well-defined infinite-width feature-learning limit.
Invoked throughout Section 4.1 and Proposition 4.1; the learning-rate scaling is also the source of the gamma0/LR confound flagged in Section 5.1.
domain assumption Gradient flow with MSE loss and sequential task exposure, modeled by Heaviside-gated task indicators, captures the relevant continual-learning dynamics.
Equation (3) and Appendix C.1 replace SGD with continuous-time gradient flow; this is standard but not verified for the finite ResNet experiments.
standard math DMFT path-integral saddle-point concentration: as width goes to infinity, feature and gradient kernels concentrate to deterministic expectations.
Appendix C.2 uses the Martin-Siggia-Rose-Janssen-De Dominicis formalism and a saddle-point approximation to derive Proposition 4.1.
ad hoc to paper For the closed-form perturbation result, task similarity is modeled by input covariance rho*I across tasks, requiring D >= 2P.
This synthetic covariance model in Appendix E.2 is introduced to obtain the O(gamma0^2) CF expansion and the known zeroth-order expression CF(0) = y^2/2 * rho^2 * (1-rho)^2.

how reviews work

0 comments

Cite this review

Pith. "Pith review of The Importance of Being Lazy: Scaling Limits of Continual Learning." pith.science (2026). https://pith.science/paper/QQTZPFJX

@misc{pith2026250616884,
  author       = {Pith},
  title        = {Pith review of: The Importance of Being Lazy: Scaling Limits of Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QQTZPFJX}},
  note         = {Machine review of arXiv:2506.16884}
}

read the original abstract

Despite recent efforts, neural networks still struggle to learn in non-stationary environments, and our understanding of catastrophic forgetting (CF) is far from complete. In this work, we perform a systematic study on the impact of model scale and the degree of feature learning in continual learning. We reconcile existing contradictory observations on scale in the literature, by differentiating between lazy and rich training regimes through a variable parameterization of the architecture. We show that increasing model width is only beneficial when it reduces the amount of feature learning, yielding more laziness. Using the framework of dynamical mean field theory, we then study the infinite width dynamics of the model in the feature learning regime and characterize CF, extending prior theoretical results limited to the lazy regime. We study the intricate relationship between feature learning, task non-stationarity, and forgetting, finding that high feature learning is only beneficial with highly similar tasks. We identify a transition modulated by task similarity where the model exits an effectively lazy regime with low forgetting to enter a rich regime with significant forgetting. Finally, our findings reveal that neural networks achieve optimal performance at a critical level of feature learning, which depends on task non-stationarity and transfers across model scales. This work provides a unified perspective on the role of scale and feature learning in continual learning.

Figures

Figures reproduced from arXiv: 2506.16884 by the authors.

**Figure 1.** (a) Test error of stationary training (joint CIFAR10). (b) Final average error and (c) Catastrophic Forgetting rate (CFr) of non-stationary sequential training (Split-CIFAR10). The factor γ0 interpolates between lazy (γ0 → 0) and rich (γ0 = 1) regimes. (a) In stationary training, more feature learning and width scaling are strictly beneficial. (b) In non-stationary training the optimal performance is achieved at a c… view at source ↗

**Figure 2.** The effect of width scaling on forgetting with models parameterized with the NTP and muP (γ0 = 1). (a) Split-CIFAR10 with ResNet model; (b) Permuted-MNIST with MLP and infinitewidth asymptotic behavior. Width Scaling Dilemma Our finding provides a new key to interpret the observations of (Mirzadeh et al., 2022a; 2020; Wenger et al., 2023; Ramasesh et al., 2020) regarding the effect of scale on CF. In particular, we… view at source ↗

**Figure 3.** Entanglement between γ0, feature evolution, and forgetting. (a) Non-linear relationship between γ0 and feature evolution (1−CKA) in ResNet model (N = 4096, Split-CIFAR10); a transition happens at γ0 ≈ γ LRT 0 . (b) γ0 and CFr (ResNet, N = 4096). (c) γ0 and CFr at the infinite-width limit (MLP, permuted-MNIST). (d) The amount of feature evolution correlates with forgetting across various widths and values of γ0 (Re… view at source ↗

Figures from the paper (21 more)

**Figure 4.** Figure 4: (a) Average Loss for varying γ0 at finite and infinite widths (MLP, permuted-MNIST, 30 samples). The optimal γ ⋆ 0 transfers to all finite and infinite widths. (b) The plasticity-stability tradeoff is controlled by the amount of feature learning (ResNet, Split-CIFAR10)…

**Figure 5.** Figure 5: Results on the synthetic dataset Permuted-MNIST with varying levels of task similarity; a ResNet with a width of 4096 is used. (a) The evolution of features is modulated by the task similarity for fixed γ0; γ LRT 0 shifts towards 1 for higher task similarity. (b) Foll…

**Figure 7.** Figure 7: The effect of width and training duration with the NTP on Split-CIFAR10. The longer the training, the higher the feature evolution (a). Consequently, forgetting increases (b). Due to the plasticity-stability tradeoff (c), the Average Error shows a width-dependent optim…

**Figure 8.** Figure 8: Performance of the three parameterizations with varying depth and a fixed width of 64. The µP and NTP are equivalent since the experiment is at base-width and they share the same parameterization w.r.t. the depth; γ0 = 1. (a) CFr; (b) learning error; (c) average error.…

**Figure 9.** Figure 9: Depth scaling of the µP, with a fixed width of 512 and various degrees of feature learning. (a) CFr; (b) learning error; (c) average error. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]

**Figure 10.** Figure 10: Depth scaling of the µP+1/ √ L, with a fixed width of 512 and various degrees of feature learning. (a) CFr; (b) learning error; (c) average error. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

**Figure 11.** Figure 11: (a) Finite and infinite-widths relationship between γ0 and CFr (loss). Zoomed in versions for (b) low γ0 and (c) high γ0. In [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

**Figure 12.** Figure 12: Plasticity (Learning Loss) and stability (CFR (loss)) tradeoff with finite and infinite-width simulation. 10−3 10−2 10−1 100 γ0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Features Evolution (1 − CKA ) 64 128 256 512 1024 2048 4096 Width (a) [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]

**Figure 13.** Figure 13: The lazy-rich transition: above a certain threshold of γ0 the evolution sharply increases consistently for all widths (ResNet model on Split-CIFAR10). 22 [PITH_FULL_IMAGE:figures/full_fig_p022_13.png]

**Figure 14.** Figure 14: Average effective sharpness in the µP for varying γ0 and width. (a) Converged; (b) multi-task. A second insight is gained by analyzing the effective normalized trace of the Hessian, finding that it is modulated by both γ0 and width in a non-trivial fashion ( [PITH_FU…

**Figure 15.** Figure 15: Average effective normalized trace in the µP for varying γ0 and width. (a) Converged; (b) multi-task. Lastly, we want to stress the notable consistency of the dependency between γ0 and the landscape properties across widths, meaning that the landscape properties vary …

**Figure 16.** Figure 16: Multi Task sharpness and its correlation with CFr for µPs. B.6. Width Dependency on Split-TinyImagenet In this section, we report the width-dependency of the Split-TinyImagenet results presented in [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]

**Figure 17.** Figure 17: The role of width and γ0 on Split-TinyImagenet with 5 tasks of 2 classes each. (a) CFr; (b) average error. 10−3 10−2 10−1 100 γ0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 CFr 64 128 256 512 1024 2048 4096 Width (a) 10−3 10−2 10−1 100 γ0 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0…

**Figure 18.** Figure 18: The role of width and γ0 on Split-TinyImagenet with 5 tasks of 40 classes each. (a) CFr; (b) average error. 10−3 10−2 10−1 100 γ0 0.0 0.1 0.2 0.3 0.4 0.5 CFr 64 128 256 512 1024 2048 4096 Width (a) 10−3 10−2 10−1 100 γ0 0.0 0.1 0.2 0.3 0.4 0.5 Average Error 64 128 256…

**Figure 19.** Figure 19: Results on CNN model (i.e. architecture as in App. A.2 but without skip connections): (a) CFr, (b) AE. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_19.png]

**Figure 20.** Figure 20: It is clear in above figure the effect of training on the pre-activations,and consequently gradients, distributions: there is a shift from the initial Gaussian curves towards heavily tailed, non-Gaussian distributions. Having the pre-activations and gradients we can e…

**Figure 21.** Figure 21: All primitive kernels and NTK across tasks T1 and T2. The theoretical one are obtained through simulations of the infinite width model, the experimental ones comes from the trained network aforementioned 32 [PITH_FULL_IMAGE:figures/full_fig_p032_21.png]

**Figure 22.** Figure 22: All primitive kernels and NTK across tasks T3 and T4. The theoretical one are obtained through simulations of the infinite width model, the experimental ones comes from the trained network aforementioned Last, from equation 30, we can compute the loss evolution of all…

**Figure 23.** Figure 23: Tasks’ losses in a four task scenario mentioned above. Vertical dot-dashed lines represent the epochs at which the shift between tasks training happens. In order to gain a deeper insight into the learning dynamics, a useful measure is the alignment of the last-layer f…

**Figure 24.** Figure 24: Theoretical and experimental tasks’ alignment A(Φ, Y Y T ) for all four tasks during all training processes. Vertical dot-dashed lines represent the epochs at which the shift between tasks training happens. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_24.png]

**Figure 25.** Figure 25: Last value of the loss on Task T2 after (long) training on Task T1, on a single hidden layer NN, linear activation, and Gaussian data in order to tune the task similarity and obtain the theoretical setup. Black crosses represent the theoretical values at different lev…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages

[4]

Lin, S., Ju, P., Liang, Y ., and Shroff, N

URL http://www.sciencedirect.com/ science/article/pii/S1566253519307377. Lin, S., Ju, P., Liang, Y ., and Shroff, N. Theory on for- getting and generalization of continual learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learn- ing, volume 202 ...

work page
[7]

Lingle, L

URL https://proceedings.mlr.press/ v202/lin23f.html. Lingle, L. A large-scale exploration of µ-transfer. arXiv preprint arXiv:2404.05728, 2024. Luo, Y ., Yang, Z., Meng, F., Li, Y ., Zhou, J., and Zhang, Y . An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023. Mai, Z., ...

arXiv 2024
[10]

, identifying the respective terms in front of the relative γ0 expansion deriving from the other set of variables composing the self consistent system of equations

(53) The aim is to compute the coefficients CF (0)(t),CF (1),... , identifying the respective terms in front of the relative γ0 expansion deriving from the other set of variables composing the self consistent system of equations. Notice that here superscripts index the coefficients and not powers (e.g. CF (2)(t) is the second order coefficient of the expa...

work page 2022
[11]

Our theory characterizes the effect of increasing the degree of feature learningon CF, starting from the lazy training setting – for which CF is already known

can be ignored. Our theory characterizes the effect of increasing the degree of feature learningon CF, starting from the lazy training setting – for which CF is already known. Perturbative methods have been applied in other contexts of deep learning theory, particularly in computing finite size corrections to the infinite width limit (Roberts et al., 2022...

work page 2022
[12]

(86) (87) The zeroth order term represents the NTK limit term and can be solved quite easily in the non-stationary case, keeping in mind the equations above for the residuals’ evolutions, that follow exponential decays. E.2. CF Perturbation inγ0 with Non-Stationarityρ E.2.1. M ODELING SIMPLE NON-STATIONARITIES Calling the input dimensionD and the number o...

work page
[13]

+e−4t− 1 4e−6t ∆(2) µ2 (t) =ρy3 e−2t(4t− 3 4 ) +e−4t− 1 4e−6t This allows us to inspect the infinite-time limit of task 2 residuals ∆µ2(t) = ∆(0) µ2 (t) +γ2 0∆2) µ2(t) +... after training on task 1, that gives us the following results: lim t→∞ ∆(0) µ2 (t) =y(1−ρ) lim t→∞ ∆(2) µ2 (t) = 0 lim t→∞ ∆µ2(t) =y(1−ρ) so that the final loss value expansion till th...

work page 2024
[2018]

10 The Importance of Being Lazy: Scaling Limits of Continual Learning cc/paper_files/paper/2018/file/ 5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper

URL https://proceedings.neurips. 10 The Importance of Being Lazy: Scaling Limits of Continual Learning cc/paper_files/paper/2018/file/ 5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper. pdf. Ke, Z. and Liu, B. Continual learning of natural lan- guage processing tasks: A survey. arXiv preprint arXiv:2211.12701, 2022. Khetarpal, K., Riemer, M., Rish, I., and Precup, D...

arXiv 2018
[2020]

URL https: //doi.org/10.1016/j.tics.2020.09.004

doi: 10.1016/j.tics.2020.09.004. URL https: //doi.org/10.1016/j.tics.2020.09.004. Hanin, B. Which neural net architectures give rise to ex- ploding and vanishing gradients? Advances in neural information processing systems, 31, 2018. Hanin, B. and Nica, M. Finite depth and width cor- rections to the neural tangent kernel. arXiv preprint arXiv:1909.05989, ...

arXiv 2020

Show all 13 references

[2022]

URL https://openreview.net/forum? id=GhVS8_yPeEa. Ring, M. B. Continual learning in reinforcement environ- ments. PhD thesis, USA, 1994. Roberts, D. A., Yaida, S., and Hanin, B. The principles of deep learning theory, volume 46. Cambridge University Press Cambridge, MA, USA, 2...

2022 arXiv
[2023]

Goldfarb, D., Evron, I., Weinberger, N., Soudry, D., and HAnd, P

URL https://proceedings.mlr.press/ v206/goldfarb23a.html. Goldfarb, D., Evron, I., Weinberger, N., Soudry, D., and HAnd, P. The joint effect of task similarity and over- parameterization on catastrophic forgetting — an an- alytical model. In The Twelfth International Confer- e...

2024 arXiv
[2024]

URL https://openreview.net/forum? id=rgwhJ7INtZ. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Parisi, G. I., Kemker, R., Part, J. L., Kanan, C., and Wermter, S. Continual lifelong learning with neural networks: A review. Neural networks, 113:54–71, 20...

2023 arXiv
[2535]

doi: https://doi.org/10.1016/j.inffus.2019.12

2019 doi
[3529]

Kumar, T., Bordelon, B., Gershman, S

PMLR, 2019. Kumar, T., Bordelon, B., Gershman, S. J., and Pehlevan, C. Grokking as the transition from lazy to rich training dynamics. In The Twelfth International Conference on Learning Representations, 2024. Lee, J., Bahri, Y ., Novak, R., Schoenholz, S. S., Penning- ton, J....

2019 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [4]

Lin, S., Ju, P., Liang, Y ., and Shroff, N

URL http://www.sciencedirect.com/ science/article/pii/S1566253519307377. Lin, S., Ju, P., Liang, Y ., and Shroff, N. Theory on for- getting and generalization of continual learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th International Conference on Machine Learn- ing, volume 202 ...

work page

[2] [7]

Lingle, L

URL https://proceedings.mlr.press/ v202/lin23f.html. Lingle, L. A large-scale exploration of µ-transfer. arXiv preprint arXiv:2404.05728, 2024. Luo, Y ., Yang, Z., Meng, F., Li, Y ., Zhou, J., and Zhang, Y . An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arXiv:2308.08747, 2023. Mai, Z., ...

arXiv 2024

[3] [10]

, identifying the respective terms in front of the relative γ0 expansion deriving from the other set of variables composing the self consistent system of equations

(53) The aim is to compute the coefficients CF (0)(t),CF (1),... , identifying the respective terms in front of the relative γ0 expansion deriving from the other set of variables composing the self consistent system of equations. Notice that here superscripts index the coefficients and not powers (e.g. CF (2)(t) is the second order coefficient of the expa...

work page 2022

[4] [11]

Our theory characterizes the effect of increasing the degree of feature learningon CF, starting from the lazy training setting – for which CF is already known

can be ignored. Our theory characterizes the effect of increasing the degree of feature learningon CF, starting from the lazy training setting – for which CF is already known. Perturbative methods have been applied in other contexts of deep learning theory, particularly in computing finite size corrections to the infinite width limit (Roberts et al., 2022...

work page 2022

[5] [12]

(86) (87) The zeroth order term represents the NTK limit term and can be solved quite easily in the non-stationary case, keeping in mind the equations above for the residuals’ evolutions, that follow exponential decays. E.2. CF Perturbation inγ0 with Non-Stationarityρ E.2.1. M ODELING SIMPLE NON-STATIONARITIES Calling the input dimensionD and the number o...

work page

[6] [13]

+e−4t− 1 4e−6t ∆(2) µ2 (t) =ρy3 e−2t(4t− 3 4 ) +e−4t− 1 4e−6t This allows us to inspect the infinite-time limit of task 2 residuals ∆µ2(t) = ∆(0) µ2 (t) +γ2 0∆2) µ2(t) +... after training on task 1, that gives us the following results: lim t→∞ ∆(0) µ2 (t) =y(1−ρ) lim t→∞ ∆(2) µ2 (t) = 0 lim t→∞ ∆µ2(t) =y(1−ρ) so that the final loss value expansion till th...

work page 2024

[7] [2018]

10 The Importance of Being Lazy: Scaling Limits of Continual Learning cc/paper_files/paper/2018/file/ 5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper

URL https://proceedings.neurips. 10 The Importance of Being Lazy: Scaling Limits of Continual Learning cc/paper_files/paper/2018/file/ 5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper. pdf. Ke, Z. and Liu, B. Continual learning of natural lan- guage processing tasks: A survey. arXiv preprint arXiv:2211.12701, 2022. Khetarpal, K., Riemer, M., Rish, I., and Precup, D...

arXiv 2018

[8] [2020]

URL https: //doi.org/10.1016/j.tics.2020.09.004

doi: 10.1016/j.tics.2020.09.004. URL https: //doi.org/10.1016/j.tics.2020.09.004. Hanin, B. Which neural net architectures give rise to ex- ploding and vanishing gradients? Advances in neural information processing systems, 31, 2018. Hanin, B. and Nica, M. Finite depth and width cor- rections to the neural tangent kernel. arXiv preprint arXiv:1909.05989, ...

arXiv 2020