{"id":"661b7b93-88cc-4f32-a359-b06bb1395cd1","arxiv_id":"2506.16884","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Increasing network width reduces forgetting only in lazy training; the best continual-learning performance occurs at a small critical level of feature learning that transfers across model sizes.","lead":"This paper shows that in continual learning, making neural networks larger only reduces catastrophic forgetting when the network is trained in a lazy regime with little feature learning, and that too much feature learning causes forgetting. It unifies earlier conflicting results by varying one parameter that controls feature learning and by simulating infinite-width networks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gamma0 sweep conflates feature learning with learning-rate scaling; without per-gamma0 LR re-tuning, the causal claim that feature learning drives forgetting is not yet established.","rationale":"I read the paper as making a causal claim: the degree of feature learning, not scale or LR, is ultimately responsible for catastrophic forgetting, and the optimal feature-learning strength gamma* transfers across widths. For that claim to hold, the gamma0 sweep must be a clean manipulation of feature learning. It is not: Table 1 couples gamma0 to the LR (eta0 gamma0^2 N). The authors are honest about this, and the reader's conditional verdict already rests on it. I agree with the reader's weakest assumption. The paper has real strengths: the DMFT extension to sequential tasks is non-trivial, the infinite-width simulation matches finite-width MLP dynamics in Appendix D at the level of preactivation/gradient distributions and losses, and the perturbation theory gives a correct small-gamma0 baseline. None of these, however, tests the causal attribution at the transition, because the theory's gamma0 is the same parameter that controls LR. The correlation in Fig. 3d is downstream of the sweep and cannot distinguish 'feature learning causes forgetting' from 'large gamma0 causes both feature evolution and forgetting through an overly large LR.' A per-gamma0 LR re-tuning experiment is the decisive check. I would keep the reader's conditional verdict: the paper should not be rejected, but the central claim should not be accepted as established until this check is run. No code/data and missing error bars are secondary; the confound is the load-bearing issue.","tokens_in":39444,"tokens_out":5391,"duration_ms":57169,"concrete_test":"Re-run the central gamma0 sweep (ResNet width 4096 on Split-CIFAR10, Fig. 1/3) with the learning rate re-tuned at each gamma0: grid-search eta0 on a validation split of the first task (stationary objective) for every gamma0, or use the sub-quadratic LR scaling that Atanasov et al. (2024) show is optimal in rich regimes. Then measure the same (1-CKA) feature evolution, CFr, average error, and the width-transfer of gamma*. If the sharp transition remains at gamma0 approx 0.1 and gamma* still transfers across widths under per-gamma0-optimal LR, the feature-learning interpretation survives. If the transition shifts or disappears, or gamma* becomes width-dependent, the reported effect is a learning-rate artifact rather than a property of feature learning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is that the main gamma0 sweep is confounded with learning-rate scaling. Table 1 sets the LR schedule to eta0(t) gamma0^2 N for the mean-field/muP models, so varying gamma0 changes the output scale, the feature-learning strength, and the LR relative to its per-gamma0 optimum. Section 5.1 concedes this: 'We hypothesize that this transition could be related to our choice of LR scaling... our LR shifts from the optimal LR towards a larger-than-optimal LR, potentially triggering the sharp rise we observe,' citing Atanasov et al. (2024), where quadratic LR scaling is optimal only in the lazy regime. The infinite-width DMFT simulation inherits the same coupling (gamma0 appears both in Eq. 8 and in the LR scaling), so it does not break the confound, and the perturbation theory in Appendix E is explicitly a small-gamma0 expansion that cannot describe the sharp transition at gamma0 approx 0.1. The strong correlation in Fig. 3d between feature evolution and CFr shows only that gamma0 drives both, not that feature learning per se causes forgetting. Until the LR confound is ruled out, the central causal claim—feature learning is ultimately responsible for the forgetting transition and gamma* transfers because of feature learning—is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies catastrophic forgetting (CF) in wide neural networks as a function of width scaling and the degree of feature learning. The authors introduce a parameter gamma0 that interpolates between the lazy (NTP) and rich (muP) training regimes, and report that width scaling reduces CF only in the lazy regime, while high feature learning is associated with a sharp low-to-high forgetting transition. They extend dynamical mean field theory (DMFT) to sequential multi-task training for a two-layer MLP, validate the infinite-width predictions against finite-width runs, and use this framework to characterize the dependence of forgetting on task similarity. The central quantitative claim is that the optimal plasticity-stability tradeoff occurs at a data-dependent gamma*_0 < 1 that transfers across widths, and that this optimum is controlled by feature learning.","tokens_in":39702,"tokens_out":4389,"duration_ms":46071,"significance":"If the central claim is correct, the paper would reconcile contradictory empirical results on width scaling in continual learning and provide a practically useful transfer rule: tune gamma0 on small models and reuse it at scale. The paper also makes a substantial theoretical contribution by extending DMFT to non-stationary training, which is new relative to prior lazy-regime analyses. The manuscript ships a machine-checkable DMFT simulation, a detailed finite-width comparison in Appendix D, and an explicit perturbation expansion for small gamma0; these are concrete and reproducible. The novel CFr metric is a reasonable response to known shortcomings of the raw CF metric. However, the main causal conclusion is presently weakened by a learning-rate confound in the gamma0 sweep, which the authors themselves flag in Section 5.1.","major_comments":[{"comment":"The gamma0 sweep is not a pure feature-learning sweep. Table 1 sets the learning rate to eta0(t) * gamma0^2 * N, so varying gamma0 simultaneously changes the output scale, the feature-learning strength, and the learning rate relative to its per-gamma0 optimum. The manuscript explicitly hypothesizes in Section 5.1 that the sharp transition 'could be related to our choice of LR scaling' and that the LR shifts from optimal to larger-than-optimal as gamma0 increases. This is load-bearing because the paper's central claim is that feature learning is ultimately responsible for forgetting. The infinite-width DMFT simulation inherits the same coupling, since the DMFT equations are derived under the same eta = O(gamma^2) scaling, so it does not break the confound. To establish the causal claim, the authors should re-tune or rescale the learning rate for each gamma0 (for example, following the optimal LR scaling of Atanasov et al.) and check whether the transition and gamma*_0 persist; alternatively, they should provide a direct manipulation of feature evolution at matched LR.","section":"Section 5.1, Table 1"},{"comment":"The perturbation theory is explicitly a small-gamma0 expansion for a linear (identity-activation) two-layer network, and it cannot describe the sharp transition observed at gamma0 approximately 0.1. The paper's evidence for the transition is therefore numerical (finite-width experiments and DMFT simulations), not analytical. This is acceptable as a numerical finding, but the abstract and Section 5 should be more careful: the sharp lazy-rich transition is not derived from the perturbation expansion, and the closed-form results in Appendix E.2 apply only to the infinite-time, linear-network, gamma0 = 0 limit of CF. Please state this scope limitation explicitly where the transition is claimed.","section":"Appendix E, Proposition E.1"},{"comment":"The main quantitative claims, including gamma*_0 approximately 0.1 and its transfer across widths, are based on averages over five random seeds with no error bars, and gamma*_0 is read visually from the curves. Without error bars or per-seed plots, the claim of exact transfer across widths is not statistically supported. The infinite-width simulation provides independent corroboration, but the finite-width curves are the primary evidence for transfer. Please add error bars, confidence intervals, or seed-level scatter plots, and state the number of seeds used for each figure.","section":"Figures 1, 3, 4"},{"comment":"The strong correlation between feature evolution (1 - CKA) and CFr shown in Figure 3d does not establish that feature learning per se causes forgetting. Since gamma0 drives both quantities, the correlation is consistent with gamma0 being a common cause. Combined with the LR confound in the same sweep, this correlation is not sufficient for the causal statement in Section 7 that 'the degree of feature learning is ultimately responsible for forgetting.' A mediation analysis, a matched-LR control, or an intervention that changes feature evolution independently of gamma0 would be needed.","section":"Section 5.1, Figure 3d"}],"minor_comments":[{"comment":"The main text states gamma*_0 is approximately 0.1 for both Split-CIFAR10 and Permuted-MNIST, but Figure 12 in Appendix B.3 labels the MLP optimum as gamma*_0 = 0.2. Please reconcile these values or explain the discrepancy.","section":"Section 5.2"},{"comment":"The notation max_{t in {i,...,T-1}} in the CFr definition is slightly confusing because t is used both as a time index and as a task index. Consider renaming the maximization variable to avoid ambiguity, e.g., max_{k in {i,...,T-1}}.","section":"Section 3.1.1, Definition A.4"},{"comment":"The paper reports a p-value < 10^-30 for the correlation in Figure 3d but does not specify the statistical test, the number of points, or whether the points are independent. Please provide this information in the caption or appendix.","section":"Section 5.1, Figure 3d"},{"comment":"The modeling setup requires D >= 2P for constructing orthogonal task rotations, but this condition is only mentioned in the appendix. It would be helpful to state it in the main text where task similarity rho is introduced, since it limits the generality of the closed-form results.","section":"Appendix E.2.1"},{"comment":"The caption of Figure 1 mentions that all figures report averages over 5 random seeds, but several later figures do not repeat this information. Please state the number of seeds and the variance measure (or its absence) consistently across all figures.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for ICML and makes a serious contribution to the scaling-limits and continual-learning literature. The main issue is the learning-rate confound in the gamma0 sweep, which the authors themselves acknowledge in Section 5.1. If the authors can add a per-gamma0 LR re-tuning control experiment or otherwise decouple feature learning from LR scaling, the central claim would be substantially strengthened. The missing error bars are a weaker but still relevant concern for the quantitative transfer claim. I recommend major revision rather than rejection because the DMFT framework and the empirical observations are valuable even if the causal interpretation needs refinement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this one. It does something real: it gives the first dynamical mean field theory for sequential multi-task training, extends the lazy/rich gamma0 interpolation to forgetting, and introduces a cleaner relative-drop metric (CFr) that fixes a known artifact in prior CF measures. The infinite-width predictions match finite-width runs in Appendix D, and the small-gamma0 perturbation expansion in Appendix E is a nice bonus. That alone is worth a serious referee's time.\n\nThe soft spot is exactly what the stress-test note says: the gamma0 sweep is not a pure feature-learning sweep. Table 1 sets the LR as eta0 * gamma0^2 * N, so varying gamma0 changes feature-learning strength and the LR relative to its per-gamma0 optimum simultaneously. Section 5.1 openly floats this: \"our LR shifts from the optimal LR towards a larger-than-optimal LR, potentially triggering the sharp rise we observe.\" The DMFT simulation inherits the same coupling, so it cannot break the confound either. The strong correlation in Fig. 3d shows gamma0 drives both feature evolution and CFr, but it does not show that feature learning per se causes forgetting. The authors are honest about this, but honesty is not the same as resolving it.\n\nOther concerns are smaller, not fatal. Main plots average five seeds without error bars; the optimal gamma* is read off visually; Prop. 4.1 is stated informally; no code or data are shipped. None of these undermine the theoretical core, but they cap the confidence one can have in the empirical claims.\n\nI disagree with the reader's weakest-assumption framing only in one respect: the paper's central narrative does not collapse. Even if the sharp transition is partly an LR artifact, the width-scaling results in µP versus NTP still hold, the DMFT framework still works, and the transfer of gamma* across widths remains an interesting empirical fact. What is weakened is the causal attribution: \"feature learning is ultimately responsible for forgetting\" is a hypothesis, not a settled conclusion, until someone reruns the sweep with per-gamma0 LR re-tuning (or shows the LR shift is irrelevant). That experiment is straightforward and the authors should be pushed to do it.\n\nBottom line: send it to good referees. The theory contribution is solid, the phenomenology is interesting, and the confound is fixable. I would cite it, and I would bring it to the reading group, but I would not yet put the causal claim in a lecture slide without a caveat.","headline":"A genuinely useful paper that extends DMFT to continual learning and reconciles contradictory scaling results, but the central causal claim about feature learning driving forgetting is confounded with learning-rate scaling and the paper itself concedes this.","tokens_in":40214,"tokens_out":1446,"would_cite":true,"duration_ms":17236,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that in continual learning, whether wider networks forget less is set by the degree of feature learning in the parameterization, not by width alone.","keywords":["catastrophic forgetting","continual learning","feature learning","lazy training","rich regime","width scaling","dynamical mean field theory","neural tangent kernel"],"falsifier":"Run the same sequential-training sweep over gamma0 while re-tuning the learning rate to its per-gamma0 optimum, for example using the sub-quadratic LR scaling for rich regimes suggested by prior optimization-landscape work; if the sharp low-to-high forgetting transition disappears or moves, the transition is driven by non-optimal learning rates rather than by feature learning itself, whereas if it persists the causal role of feature learning is confirmed.","tokens_in":39244,"feed_emoji":"🔄","tokens_out":8541,"duration_ms":79252,"temperature":0.7,"pith_summary":"This paper asks why neural networks trained on a sequence of tasks forget earlier tasks, and whether scaling the network helps. Its answer is that the benefit of width is not intrinsic: it depends on how much the network's features move during training. The authors interpolate between a lazy regime, where features barely evolve, and a rich feature-learning regime, using a parameter gamma0, and find that catastrophic forgetting stays low until a critical amount of feature learning and then rises sharply. The optimal balance between learning new tasks and retaining old ones sits at a fixed intermediate value gamma*_0 around 0.1 that transfers across widths, and it moves toward richer regimes as tasks become more similar. A theoretical infinite-width description, built with dynamical mean field theory, reproduces the finite-width behavior and grounds the claim that feature learning, not scale, is the controlling factor.","feed_headline":"Feature learning, not scale, drives forgetting in continual learning","feed_subtitle":"Networks forget less when features stay put; the optimal feature-learning strength transfers across widths.","key_machinery":"The load-bearing object is a one-parameter family of network parameterizations: the output scale gamma = gamma0 $\\sqrt$(N) and learning rate eta = eta0 $gamma0^{2}$ N make gamma0=0 the lazy neural-tangent limit and gamma0=1 the rich mean-field/maximal-update limit. The authors measure the resulting feature movement with centered kernel alignment (CKA) and forgetting with a new relative metric, the catastrophic forgetting rate (CFr), which computes the average relative drop in per-task accuracy. The theoretical engine is a dynamical mean field theory for infinite width: preactivations h and pregradient variables z become stochastic processes, and the forward and backward kernels Phi and G concentrate at their saddle-point expectations, giving a self-consistent system of equations for the output dynamics under sequential tasks. A perturbation expansion in powers of gamma0 around the lazy limit shows the first-order correction to forgetting vanishes, and the second-order terms, expressed through residuals and the across-task neural tangent kernel, carry the leading feature-learning effect.","core_discovery":"In the paper's own terms, the discovery is that the effect of model scale on catastrophic forgetting is modulated by the degree of feature learning, not by width alone. Under neural tangent parameterization, where features are effectively frozen, increasing width monotonically reduces forgetting; under maximal update parameterization, where features learn at any width, width does not reduce forgetting and can even increase it in the rich regime. Interpolating with gamma0 between the two parameterizations reveals a lazy-rich transition: below a data-dependent threshold the features barely move and forgetting is low, above it feature evolution and forgetting both rise sharply. Consequently the plasticity-stability optimum is achieved at a critical, data-dependent gamma*_0 < 1, which transfers across widths (and, with a depth-corrected parameterization, across depths), so larger networks beyond this point waste capacity. Task similarity shifts the entire picture: low similarity moves the optimum toward laziness, while high similarity, where first-task features are reused, can make rich learning beneficial and restore the benefits of width. The paper also extends dynamical mean field theory to sequential tasks, giving closed-form infinite-width equations whose simulations match finite networks.","pith_inferences":["If feature learning is the mechanism, a natural untested remedy is to bound representation drift during later tasks, for instance by penalizing feature-kernel change, rather than reducing global plasticity, which might preserve learning while suppressing the forgetting driver.","The transfer of gamma* suggests a zero-shot continual-learning scaling recipe: tune gamma0 on a small-width model, then scale width; the paper shows this transfer on the benchmarks it studies, leaving breadth of architectures and datasets as an open extension.","The learning-rate confound in Section 5.1 implies a sharper experiment: separate gamma0's effect on feature learning from its effect on learning rate by an alternative parameterization; the paper leaves this to future work, but the distinction determines whether the transition is about features or optimization.","One deeper implication is that scaling laws for continual learning should include task similarity as a parameter; models tuned to a fixed degree of feature learning may be near-optimal only for the non-stationarity level they were tuned on."],"forward_implications":["Under the paper's account, width scaling is useful in continual learning only while the network stays effectively lazy; past the lazy-rich transition, additional width does not reduce forgetting and can worsen it.","The optimal feature-learning strength gamma*_0 is around 0.1 and transfers across widths and, under the depth-corrected parameterization, across depths, so practitioners could tune gamma0 on a small model and reuse it at scale.","Because task similarity controls the transition, highly non-stationary streams call for closer-to-lazy training, while similar tasks allow richer learning before forgetting becomes destructive.","The extended dynamical mean field theory reproduces finite-width forgetting at infinite width, indicating these effects persist in the scaling limit.","In high-similarity benchmarks, a pretraining effect can make forgetting non-monotonic in gamma0 and restore the benefit of width, connecting the result to earlier observations that pretrained models scale better."],"supporting_citations":[{"why":"Supplies the self-consistent dynamical mean field theory of kernel evolution in wide networks that this paper extends to sequential tasks.","marker":"Bordelon & Pehlevan (2022)"},{"why":"Gives the NTK-overlap analysis of catastrophic forgetting in the lazy regime that this paper's perturbation expansion starts from and extends to feature learning.","marker":"Doan et al. (2021)"},{"why":"Provides generalization guarantees for continual learning in the lazy regime, one of the prior theoretical results the paper generalizes.","marker":"Bennani et al. (2020)"},{"why":"Empirical baseline claiming wider networks forget less; the paper reconciles this with its parameterization-dependent result.","marker":"Mirzadeh et al. (2022a)"},{"why":"Shows the width benefit depends on training time; the paper attributes that dependence to induced feature learning and revisits the experiment.","marker":"Wenger et al. (2023)"},{"why":"Finds scale benefits continual learning mainly for pretrained models, which the paper links to its pretraining-effect results.","marker":"Ramasesh et al. (2022)"},{"why":"Documents how optimal learning-rate scaling depends on feature-learning strength; the paper cites it to hypothesize why its gamma0 sweep produces a sharp transition.","marker":"Atanasov et al. (2024)"},{"why":"Origin of zero-shot hyperparameter transfer that motivates the transfer of gamma* across widths.","marker":"Yang et al. (2022a)"},{"why":"Provides the depth-corrected maximal update parameterization used to show gamma* also transfers across depths.","marker":"Bordelon et al. (2023)"}],"fun_headline_variants":["Lazy networks forget less: feature learning drives forgetting","Width only reduces forgetting when features stay frozen","Optimal continual learning sits at a critical feature learning","Feature learning, not scale, sets the forgetting threshold","The lazy-rich transition: why bigger nets don't always help"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The sweep over gamma0 is not a pure manipulation of feature learning, because the learning rate is scaled as eta0 $gamma0^{2}$ N, so raising gamma0 also moves the learning rate away from the optimal value; the paper itself notes in Section 5.1 that this may trigger the sharp rise it attributes to feature learning.","fun_headline_variants_meta":{"raw":{"variants":["Lazy networks forget less: feature learning drives forgetting","Width only reduces forgetting when features stay frozen","Optimal continual learning sits at a critical feature learning","Feature learning, not scale, sets the forgetting threshold","The lazy-rich transition: why bigger nets don't always help"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":3005,"prompt_tokens":995,"completion_tokens":2010,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":1934}},"tokens_in":611,"tokens_out":2010,"duration_ms":15202,"temperature":1.0,"reasoning_tokens":1934,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:16:45.331293+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same sequential-training sweep over gamma0 while re-tuning the learning rate to its per-gamma0 optimum, for example using the sub-quadratic LR scaling for rich regimes suggested by prior optimization-landscape work; if the sharp low-to-high forgetting transition disappears or moves, the transition is driven by non-optimal learning rates rather than by feature learning itself, whereas if it persists the causal role of feature learning is confirmed.","supporting_citations":[],"review_version":2}