{"id":"bdd38cde-3cb7-4ef6-b091-6de1bc94d9f1","arxiv_id":"2507.16039","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Task transitions in continual learning cause abrupt, width-persistent changes in the Neural Tangent Kernel of past data, a phenomenon the authors call reactivation, which is modulated by semantic novelty of the new classes.","lead":"This paper measures how the Neural Tangent Kernel (the network's gradient-based feature space) changes when a model is trained on a new task, and finds that task switches cause a sudden, temporary 'reactivation' of feature dynamics even in wide networks. It is a first empirical look at NTK dynamics in continual learning and suggests that theories assuming a fixed kernel may miss important adaptation at task boundaries.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Width-scaling analysis is missing: Fig. 12 shows reactivation amplitude decreasing with width, so the claim that it persists 'even at large scale' is not established without a fitted large-width limit.","rationale":"The paper is a good-faith empirical study with a consistent phenomenon across widths, learning rates, datasets, and task sequences, plus a sample-size ablation in Appendix A.3. Those are real supporting observations. My concern is not about fabrication or about the 32-sample estimator being the primary threat. The reader's under-specification concern (fixed evaluation set, logit definition, seed variance) is valid and should be fixed. But the more load-bearing gap is the width-scaling interpretation: the paper's own Figures 11 and 12 show the reactivation amplitude decreasing as width increases. Static-kernel theory is an asymptotic statement; if the amplitude decays to zero as N grows, the result is a finite-width artifact of the lazy regime, not evidence against static-kernel approximations in the limit. The paper needs a scaling-law fit with an intercept and uncertainty quantification to distinguish 'nonzero in the limit' from 'approaching zero.' Until that is provided, the central theoretical claim is conditional. This does not change the reader's verdict (CONDITIONAL), but it sharpens the condition: the requested fit is the decisive check rather than merely additional documentation.","tokens_in":11952,"tokens_out":4472,"duration_ms":49879,"concrete_test":"For the lazy parametrization in Sec. B.1, rerun the two-task CIFAR-10 experiment at widths 64, 128, 256, 512, 1024, 2048 with ≥10 seeds each, using a fixed 32-sample first-task evaluation set. Measure the task-switch drop in NTK max eigenvalue (or the peak of kernel velocity). Fit Δ(N) = a + b N^{-α} by nonlinear least squares and report the 95% CI for a. Also report the same fit for the ImageNet widths 250/500/1000. If the CI for a includes 0, the persistence claim is unsupported; if it excludes 0, this concern is settled.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Sec. 2.2 and the abstract claim that task switches cause abrupt NTK changes 'even under a lazy learning regime' and 'even at large scale.' The paper's own Figure 12 (and Appendix C.4) shows the magnitude of changes decreasing monotonically as width grows from 64 to 2048, and the ImageNet curves (Fig. 11) show a similar smoothing with width 250/500/1000. No seed variance, no error bars, and no scaling fit are reported, so the qualitative statement 'does not disappear' is an eyeball judgment. The central theoretical stake is whether the large-width limit of the reactivation amplitude is nonzero. In the NTK parametrization, any finite-width deviation must decay at least as 1/N (or faster) to recover the static kernel; if the observed drop/velocity scales to zero, the result is a finite-width correction, not a challenge to static-kernel approximations in the infinite-width limit. The paper needs to fit Δ(N) = a + b N^{-α} (with intercept a) and report the confidence interval on a across seeds. If a is consistent with zero, the headline claim overstates the asymptotic relevance; if a is significantly positive, the objection is resolved. This is the load-bearing gap because the paper's own trend line points in the direction that would falsify its large-scale interpretation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of Neural Tangent Kernel (NTK) dynamics in continual learning. On CIFAR-10 and ImageNet100, a CNN is trained on a sequence of tasks while the paper measures, on the first-task data only, the NTK max eigenvalue, kernel distance 1−CKA, kernel velocity, and kernel alignment. The central observation is that at the moment of a task switch the NTK of the past data changes abruptly — a velocity spike and a drop-recovery ('check-mark') in the max eigenvalue — across widths 64–2048, learning rates 1e-3–1e-5, and in both feature-learning and lazy (learning rate scaled by 1/N) regimes. The paper also shows that introducing new classes drives larger NTK changes than reweighting existing classes, and that the magnitude of the reactivation decays with width. The authors argue these findings challenge static-kernel approximations in continual learning.","tokens_in":12090,"tokens_out":9453,"duration_ms":92386,"significance":"If the central observation holds, the paper identifies a real and underappreciated phenomenon: task boundaries transiently move finite-width networks away from lazy dynamics, so static-kernel theories of continual learning miss a boundary effect. The paper's strengths are its direct, operational definition of reactivation through kernel metrics on fixed first-task data, the consistency of the pattern across many widths, learning rates, and datasets, and the clean comparison between semantic novelty and frequency shift, which yields a well-ordered effect. These make the phenomenon falsifiable. However, the significance of the large-scale claim is currently unquantified: the paper's own scaling figures show a decreasing amplitude, and without a fitted intercept the asymptotic relevance is not established. The underspecified NTK estimator also needs to be pinned down before the quantitative conclusions can be trusted.","major_comments":[{"comment":"The claim that reactivation persists 'even at large scale' is a central conclusion, but the only scaling evidence is a monotonic decrease of the response amplitude with width from 64 to 2048 (Fig. 12; Appendix C.4), with a similar smoothing in the ImageNet curves (Fig. 11). Without a fitted asymptote, the data are equally consistent with a finite-width effect that vanishes in the infinite-width limit. I request a scaling analysis: define the reactivation amplitude (e.g., kernel-velocity peak or max-eigenvalue drop at the switch), average over seeds, and fit Δ(N) = a + b N^{-α} with a confidence interval on the intercept a. If a is consistent with zero, the headline claim should be weakened to a finite-width phenomenon.","section":"Appendix C.4, Fig. 12; Sec. 2.2"},{"comment":"The NTK estimator is underspecified in two ways. First, for a multi-class network the NTK is a tensor indexed by output logits; the paper does not state which scalar output or loss gradient is recorded when a 'single sample' is fed. If the recorded vector is the cross-entropy loss gradient rather than the Jacobian of a logit, the computed matrix is not the NTK. Second, it is not stated whether the same 32 random samples are reused at every time step; if the batch is re-sampled, the velocity S(Θ_t, Θ_{t+dt}) includes sampling noise that could inflate the task-switch spike. Please specify the exact computation and report a fixed-vs-resampled batch comparison.","section":"Appendix A.3"},{"comment":"No seed variance or error bars are reported for any of the central curves, although Sec. 2 says experiments were run 'across multiple seeds.' The qualitative check-mark and velocity spike may be robust in the shown single runs, but the decreasing amplitude with width makes run-to-run variation a first-order concern. Please report mean ± standard deviation (or individual seeds) for at least the reactivation amplitude and for the key curves in a few representative settings.","section":"Sec. 2.2, Figs. 1–4, 7–12"}],"minor_comments":[{"comment":"In Experiment 1, 'varying i ∈ [0, 1]' should be 'i ∈ {0, …, 10}' (or [0, 10]) to match the experiments E0–E10 and the similarity formula.","section":"Sec. 2.3"},{"comment":"The displayed CKA formula is uncentered linear alignment; if centering is intended, the definition should use centered kernel matrices, and the text should say so.","section":"Appendix A.2"},{"comment":"The caption for the second row lists 20, 50, and 100 epochs per task, but panels (d)–(f) are labeled 10, 20, and 100 epochs; please make the numbers consistent.","section":"Appendix C.3, Fig. 11"},{"comment":"Please define 'step' and 'dt' in the main text: one step corresponds to 10 training iterations, and the velocity is computed with dt = 10 steps (100 iterations) according to Fig. 7; stating this only in captions is insufficient for reproducibility.","section":"Sec. 2.1, Fig. 7"},{"comment":"References [8] and [9] appear to be the same paper (Jacot, Gabriel, Hongler, NeurIPS 2018); please consolidate to a single entry.","section":"References"},{"comment":"The description 'Pytorch standard parametrization' and 'very similar to NTK parametrization with only factor-wise difference' is too vague to reproduce; please specify the exact initialization, the learning-rate scaling, and any output scaling for each regime.","section":"Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's headline claim is tied to a large-width limit that the current data do not establish; if the authors can supply a seed-averaged scaling fit with a confidence interval on the intercept, the paper would be much stronger. The NTK computation also needs to be specified precisely, since all quantitative conclusions rest on it. I do not see circularity or novelty concerns beyond the missing quantification."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth reading if you care about NTK approximations in continual learning, but don't buy the 'large scale' claim yet. The core observation is new and plausible: the empirical NTK of first-task data undergoes a transient drop and velocity spike at task boundaries, even in a width-scaled lazy regime, and the effect is larger for semantic novelty than for frequency shifts. That's a genuine empirical contribution, and the controlled task-similarity sweep is nicely done.\n\nWhat's good: the paper measures NTK dynamics directly, uses standard metrics (CKA distance, velocity, alignment), and runs a broad grid—widths 64–2048, learning rates 1e-3 to 1e-5, CIFAR and ImageNet100, multiple task switches. The 'check-mark' pattern is consistent across these configurations, which argues against a narrow artifact. The authors also do a small sample-size ablation for the NTK estimator, which shows the trend is stable across 10/20/100 samples.\n\nWhere it's soft. The NTK estimator itself is under-specified: 32 random samples, no statement whether the same set is reused, no logit selection for multi-class gradients, no error bars. The spike at task boundary could partly be estimator variance if the sample set is re-drawn. More importantly, the abstract's 'even at large scale' is not supported. Figure 12 shows the amplitude of reactivation decreasing monotonically with width from 64 to 2048, and the ImageNet curves smooth out similarly. Without a fitted scaling law of the form Δ(N)=a+bN^{-α} and a confidence interval on the intercept a, you cannot distinguish a finite-width effect from a true asymptotic violation. The paper's own trend line points toward the former. Also, they never specify what a static-kernel baseline would predict; the 'challenge' is more of a visual argument than a falsification test.\n\nYou asked about the stress-test note: I checked Fig 12 and it holds up. The velocity spike drops by roughly an order of magnitude from width 64 to 2048, so the width-scaling gap is load-bearing for the paper's strongest claim.\n\nOverall: the phenomenon is real at practical widths and worth a serious referee. I'd send it to review with major revisions: fix the estimator description, add seeds and error bars, fit the width scaling, and soften the asymptotic phrasing. If the scaling fit shows a zero intercept, the correct conclusion is 'finite-width correction persists at large but finite widths,' which is still interesting but not the same as challenging infinite-width NTK theory.","headline":"A genuinely new empirical observation about NTK dynamics at task boundaries, but the 'large scale' claim needs a scaling fit and the estimator needs seeds and error bars.","tokens_in":12708,"tokens_out":3381,"would_cite":true,"duration_ms":36373,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Task shifts force the neural tangent kernel of past data to move abruptly, even in wide, supposedly lazy networks, with the effect size set by how semantically new the incoming task is.","keywords":["neural tangent kernel","continual learning","lazy regime","feature learning","kernel alignment","task similarity","distribution shift","kernel velocity"],"falsifier":"Recompute the NTK diagnostics with the same fixed 32 images at every checkpoint, with freshly sampled 32-image batches, and with 512 or more images; if the drop-recovery check-mark and velocity spike disappear in the fixed-batch or large-batch runs, the reactivation is an artifact of the measurement window rather than a property of training.","tokens_in":11669,"feed_emoji":"🔄","tokens_out":6948,"duration_ms":69970,"temperature":0.7,"pith_summary":"This paper tries to show that the neural tangent kernel (NTK)—the object that supposedly stays frozen when a network is wide enough to be in the 'lazy' regime—does not stay frozen in continual learning. At the moment a model switches from one task to the next, the kernel of the first task's data undergoes an abrupt, temporary reorganization: a drop-and-recovery 'check-mark' in its top eigenvalue, a spike in kernel velocity, and a shift in kernel alignment. The phenomenon appears across widths from 64 to 2048, across learning rates from 1e-3 to 1e-5, and in both feature-learning and lazy parametrizations. Its size tracks how semantically new the second task is: introducing genuinely new classes causes large kernel changes, while reweighting known classes causes almost none. If this is right, static-kernel approximations in continual learning theory are missing the most important moment of training.","feed_headline":"Task switches reboot feature learning even in wide networks","feed_subtitle":"The old task's kernel drops sharply at each boundary; fixed-kernel theory misses that burst of adaptation.","key_machinery":"The machinery is the empirical NTK of the first-task data, computed from gradient inner products $\\Theta_t(x_i, x_j) = \\nabla_\\theta f_t(x_i)^\\top \\nabla_\\theta f_t(x_j)$ on a 32-sample batch, plus four derived diagnostics: its maximum eigenvalue (spectral norm), kernel distance $S(\\Theta, \\Theta') = 1 - \\mathrm{CKA}(\\Theta, \\Theta')$, kernel velocity $v(t) = S(\\Theta_t, \\Theta_{t+dt})/dt$, and kernel alignment $A(t) = \\mathrm{CKA}(\\Theta_t, yy^\\top)$. The eigenmode recursion $e_{t+1} = (1 - \\eta\\Lambda)e_t$ links the spectral norm to convergence speed. The paper's argument works by watching these diagnostics on old-task data across a task boundary, comparing widths, learning rates, and two kinds of distribution shift.","core_discovery":"The central claim is that task boundaries reactivate feature learning even where single-task theory says it should be dormant. Concretely, with all metrics evaluated on the first task's data, the NTK's maximum eigenvalue drops sharply when the second task begins and then recovers, the kernel distance from initialization jumps, and kernel velocity—defined as $S(\\Theta_t, \\Theta_{t+dt})/dt$ with $S$ a CKA-based distance—spikes and decays back to its pre-switch level within a few epochs. The same pattern holds under NTK parametrization with learning rate scaled as $1/N$, where the kernel is nearly static during the first task itself, so the authors take the switch-triggered motion as evidence of reactivation rather than ordinary training drift. They further claim a monotone, sublinear relation between the number of new classes in the second task and the size of the kernel disruption, and contrast this with frequency-only shifts, where no reactivation occurs.","pith_inferences":["A testable extension the paper leaves implicit: if reactivation is real, the same spike should appear when measuring the NTK on the new task's data from the first iteration of task 2, and its magnitude should predict how much accuracy on task 1 drops afterward, connecting kernel velocity at the boundary to forgetting.","The check-mark shape resembles the transient sharpening seen in loss-landscape studies through the NTK-Hessian link, so connecting reactivation to edge-of-stability dynamics is a natural next step not taken here.","The 32-sample estimator needs a stability check before the phenomenon is treated as quantitative: re-measuring with fixed versus freshly sampled batches, and across output logits, would settle whether the spike is intrinsic or an estimator artifact.","If semantic novelty is the driver, then task-construction choices in continual learning benchmarks, such as how many new classes appear per task, directly control how much feature re-learning occurs, so benchmark design could deliberately tune reactivation."],"forward_implications":["At the widths and learning rates tested (64 to 2048 channels, learning rates $10^{-3}$ to $10^{-5}$), a static NTK is not a valid approximation across a task boundary, so lazy-regime guarantees for continual learning do not transfer directly.","The magnitude of reactivation is predictable from semantic overlap: introducing new classes moves the kernel far more than reweighting known ones, so representation drift at a boundary is partially forecastable.","Widening the network shrinks but does not remove the reactivation; the drop-recovery shape persists at width 2048 and on ImageNet100, so scale alone does not restore kernel stationarity in non-stationary settings.","The transient nature of the spike, with velocity decaying back to baseline within a few epochs, suggests the network passes through a short adaptation window at each boundary, after which dynamics settle again.","Frequency-only distribution shifts produce smooth, minor kernel change, so continual-learning theories must distinguish 'new content' from 'reweighted content' rather than treating all shifts alike."],"supporting_citations":[{"why":"Defines the neural tangent kernel and the static-kernel limit that the paper tests in a non-stationary setting.","marker":"[8]"},{"why":"Prior empirical study of NTK time evolution in single-task networks whose metric choices, including kernel distance and velocity, the paper adopts.","marker":"[6]"},{"why":"Supplies centered kernel alignment, which defines both the kernel distance and the kernel alignment used in every measurement.","marker":"[4]"},{"why":"Provides the parametrization and gradient-computation method used to estimate the empirical NTK.","marker":"[14]"},{"why":"Establishes the lazy-regime scaling, learning rate proportional to $1/N$, that the paper uses to benchmark reactivation against a supposedly static kernel.","marker":"[15]"},{"why":"Contemporary study of NTK learning dynamics motivating the use of kernel velocity during training.","marker":"[16]"},{"why":"Kernel alignment and implicit regularization via feature alignment, the basis for the alignment metric and its interpretation.","marker":"[1]"},{"why":"A static-kernel analysis of continual learning whose generalisation guarantees the reactivation finding directly challenges.","marker":"[2]"},{"why":"A theoretical treatment of catastrophic forgetting through the NTK overlap matrix that assumes a frozen kernel.","marker":"[5]"},{"why":"A static-kernel continual-learning analysis whose assumptions the reactivation finding directly challenges.","marker":"[11]"}],"fun_headline_variants":["Task boundaries trigger kernel revival, contradicting static theory","NTK reactivation at task shifts: static-kernel models miss it","Kernel bursts at task boundaries expose limits of static NTK","Task shifts rekindle feature learning in wide lazy nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole phenomenon is read off an empirical NTK estimated from 32 random images of the first task, and the claim stands on that small-sample kernel being a stable, unbiased stand-in for the true kernel of that task.","fun_headline_variants_meta":{"raw":{"variants":["Task boundaries trigger kernel revival, contradicting static theory","NTK reactivation at task shifts: static-kernel models miss it","Kernel bursts at task boundaries expose limits of static NTK","Task shifts rekindle feature learning in wide lazy nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001106,"raw_usage":{"total_tokens":4590,"prompt_tokens":907,"completion_tokens":3683,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":3613}},"tokens_in":523,"tokens_out":3683,"duration_ms":25951,"temperature":1.0,"reasoning_tokens":3613,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:21:38.483780+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the NTK diagnostics with the same fixed 32 images at every checkpoint, with freshly sampled 32-image batches, and with 512 or more images; if the drop-recovery check-mark and velocity spike disappear in the fixed-batch or large-batch runs, the reactivation is an artifact of the measurement window rather than a property of training.","supporting_citations":[{"cited_title":"Neural tangent kernel: Con- vergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"Defines the neural tangent kernel and the static-kernel limit that the paper tests in a non-stationary setting."},{"cited_title":"Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel","cited_arxiv_id":null,"evidence_quote":"Prior empirical study of NTK time evolution in single-task networks whose metric choices, including kernel distance and velocity, the paper adopts."},{"cited_title":"Algorithms for learning kernels based on centered alignment","cited_arxiv_id":null,"evidence_quote":"Supplies centered kernel alignment, which defines both the kernel distance and the kernel alignment used in every measurement."},{"cited_title":"On the Cone Effect in the Learning Dynamics","cited_arxiv_id":"2503.16316","evidence_quote":"Contemporary study of NTK learning dynamics motivating the use of kernel velocity during training."},{"cited_title":"Implicit regularization via neural feature alignment","cited_arxiv_id":null,"evidence_quote":"Kernel alignment and implicit regularization via feature alignment, the basis for the alignment metric and its interpretation."},{"cited_title":"A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix","cited_arxiv_id":"2010.04003","evidence_quote":"A theoretical treatment of catastrophic forgetting through the NTK overlap matrix that assumes a frozen kernel."},{"cited_title":"Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting, 2022","cited_arxiv_id":null,"evidence_quote":"A static-kernel continual-learning analysis whose assumptions the reactivation finding directly challenges."}],"review_version":1}