REVIEW 3 major objections 6 minor 17 references
Reactivation: Empirical NTK Dynamics Under Task Shifts
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Task shifts force the neural tangent kernel of past data to move abruptly, even in wide, supposedly lazy networks, with the effect size set by how semantically new the incoming task is.
desk verdict A genuinely new empirical observation about NTK dynamics at task boundaries, but the 'large scale' claim needs a scaling fit and the estimator needs seeds and error bars. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the empirical NTK of the first-task data, computed from gradient inner products $\Theta_t(x_i, x_j) = \nabla_\theta f_t(x_i)^\top \nabla_\theta f_t(x_j)$ on a 32-sample batch, plus four derived diagnostics: its maximum eigenvalue (spectral norm), kernel distance $S(\Theta, \Theta') = 1 - \mathrm{CKA}(\Theta, \Theta')$, kernel velocity $v(t) = S(\Theta_t, \Theta_{t+dt})/dt$, and kernel alignment $A(t) = \mathrm{CKA}(\Theta_t, yy^\top)$. The eigenmode recursion $e_{t+1} = (1 - \eta\Lambda)e_t$ links the spectral norm to convergence speed. The paper's argument works by watching these diagnostics on old-task data across a task boundary, comparing widths, learning rates, and two kinds of distribution shift.
What would settle it
Recompute the NTK diagnostics with the same fixed 32 images at every checkpoint, with freshly sampled 32-image batches, and with 512 or more images; if the drop-recovery check-mark and velocity spike disappear in the fixed-batch or large-batch runs, the reactivation is an artifact of the measurement window rather than a property of training.
Extended reading notes
Core claim
The central claim is that task boundaries reactivate feature learning even where single-task theory says it should be dormant. Concretely, with all metrics evaluated on the first task's data, the NTK's maximum eigenvalue drops sharply when the second task begins and then recovers, the kernel distance from initialization jumps, and kernel velocity—defined as $S(\Theta_t, \Theta_{t+dt})/dt$ with $S$ a CKA-based distance—spikes and decays back to its pre-switch level within a few epochs. The same pattern holds under NTK parametrization with learning rate scaled as $1/N$, where the kernel is nearly static during the first task itself, so the authors take the switch-triggered motion as evidence of reactivation rather than ordinary training drift. They further claim a monotone, sublinear relation between the number of new classes in the second task and the size of the kernel disruption, and contrast this with frequency-only shifts, where no reactivation occurs.
Load-bearing premise
The whole phenomenon is read off an empirical NTK estimated from 32 random images of the first task, and the claim stands on that small-sample kernel being a stable, unbiased stand-in for the true kernel of that task.
Editorial extensions
If this is right
- At the widths and learning rates tested (64 to 2048 channels, learning rates $10^{-3}$ to $10^{-5}$), a static NTK is not a valid approximation across a task boundary, so lazy-regime guarantees for continual learning do not transfer directly.
- The magnitude of reactivation is predictable from semantic overlap: introducing new classes moves the kernel far more than reweighting known ones, so representation drift at a boundary is partially forecastable.
- Widening the network shrinks but does not remove the reactivation; the drop-recovery shape persists at width 2048 and on ImageNet100, so scale alone does not restore kernel stationarity in non-stationary settings.
- The transient nature of the spike, with velocity decaying back to baseline within a few epochs, suggests the network passes through a short adaptation window at each boundary, after which dynamics settle again.
- Frequency-only distribution shifts produce smooth, minor kernel change, so continual-learning theories must distinguish 'new content' from 'reweighted content' rather than treating all shifts alike.
Reading between the lines
- A testable extension the paper leaves implicit: if reactivation is real, the same spike should appear when measuring the NTK on the new task's data from the first iteration of task 2, and its magnitude should predict how much accuracy on task 1 drops afterward, connecting kernel velocity at the boundary to forgetting.
- The check-mark shape resembles the transient sharpening seen in loss-landscape studies through the NTK-Hessian link, so connecting reactivation to edge-of-stability dynamics is a natural next step not taken here.
- The 32-sample estimator needs a stability check before the phenomenon is treated as quantitative: re-measuring with fixed versus freshly sampled batches, and across output logits, would settle whether the spike is intrinsic or an estimator artifact.
- If semantic novelty is the driver, then task-construction choices in continual learning benchmarks, such as how many new classes appear per task, directly control how much feature re-learning occurs, so benchmark design could deliberately tune reactivation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an empirical study of Neural Tangent Kernel (NTK) dynamics in continual learning. On CIFAR-10 and ImageNet100, a CNN is trained on a sequence of tasks while the paper measures, on the first-task data only, the NTK max eigenvalue, kernel distance 1−CKA, kernel velocity, and kernel alignment. The central observation is that at the moment of a task switch the NTK of the past data changes abruptly — a velocity spike and a drop-recovery ('check-mark') in the max eigenvalue — across widths 64–2048, learning rates 1e-3–1e-5, and in both feature-learning and lazy (learning rate scaled by 1/N) regimes. The paper also shows that introducing new classes drives larger NTK changes than reweighting existing classes, and that the magnitude of the reactivation decays with width. The authors argue these findings challenge static-kernel approximations in continual learning.
Significance. If the central observation holds, the paper identifies a real and underappreciated phenomenon: task boundaries transiently move finite-width networks away from lazy dynamics, so static-kernel theories of continual learning miss a boundary effect. The paper's strengths are its direct, operational definition of reactivation through kernel metrics on fixed first-task data, the consistency of the pattern across many widths, learning rates, and datasets, and the clean comparison between semantic novelty and frequency shift, which yields a well-ordered effect. These make the phenomenon falsifiable. However, the significance of the large-scale claim is currently unquantified: the paper's own scaling figures show a decreasing amplitude, and without a fitted intercept the asymptotic relevance is not established. The underspecified NTK estimator also needs to be pinned down before the quantitative conclusions can be trusted.
major comments (3)
- [Appendix C.4, Fig. 12; Sec. 2.2] The claim that reactivation persists 'even at large scale' is a central conclusion, but the only scaling evidence is a monotonic decrease of the response amplitude with width from 64 to 2048 (Fig. 12; Appendix C.4), with a similar smoothing in the ImageNet curves (Fig. 11). Without a fitted asymptote, the data are equally consistent with a finite-width effect that vanishes in the infinite-width limit. I request a scaling analysis: define the reactivation amplitude (e.g., kernel-velocity peak or max-eigenvalue drop at the switch), average over seeds, and fit Δ(N) = a + b N^{-α} with a confidence interval on the intercept a. If a is consistent with zero, the headline claim should be weakened to a finite-width phenomenon.
- [Appendix A.3] The NTK estimator is underspecified in two ways. First, for a multi-class network the NTK is a tensor indexed by output logits; the paper does not state which scalar output or loss gradient is recorded when a 'single sample' is fed. If the recorded vector is the cross-entropy loss gradient rather than the Jacobian of a logit, the computed matrix is not the NTK. Second, it is not stated whether the same 32 random samples are reused at every time step; if the batch is re-sampled, the velocity S(Θ_t, Θ_{t+dt}) includes sampling noise that could inflate the task-switch spike. Please specify the exact computation and report a fixed-vs-resampled batch comparison.
- [Sec. 2.2, Figs. 1–4, 7–12] No seed variance or error bars are reported for any of the central curves, although Sec. 2 says experiments were run 'across multiple seeds.' The qualitative check-mark and velocity spike may be robust in the shown single runs, but the decreasing amplitude with width makes run-to-run variation a first-order concern. Please report mean ± standard deviation (or individual seeds) for at least the reactivation amplitude and for the key curves in a few representative settings.
minor comments (6)
- [Sec. 2.3] In Experiment 1, 'varying i ∈ [0, 1]' should be 'i ∈ {0, …, 10}' (or [0, 10]) to match the experiments E0–E10 and the similarity formula.
- [Appendix A.2] The displayed CKA formula is uncentered linear alignment; if centering is intended, the definition should use centered kernel matrices, and the text should say so.
- [Appendix C.3, Fig. 11] The caption for the second row lists 20, 50, and 100 epochs per task, but panels (d)–(f) are labeled 10, 20, and 100 epochs; please make the numbers consistent.
- [Sec. 2.1, Fig. 7] Please define 'step' and 'dt' in the main text: one step corresponds to 10 training iterations, and the velocity is computed with dt = 10 steps (100 iterations) according to Fig. 7; stating this only in captions is insufficient for reproducibility.
- [References] References [8] and [9] appear to be the same paper (Jacot, Gabriel, Hongler, NeurIPS 2018); please consolidate to a single entry.
- [Appendix B.1] The description 'Pytorch standard parametrization' and 'very similar to NTK parametrization with only factor-wise difference' is too vague to reproduce; please specify the exact initialization, the learning-rate scaling, and any output scaling for each regime.
Circularity Check
No significant circularity: the reactivation result is an empirical measurement of NTK metrics, not a consequence fitted from or defined by the conclusion.
full rationale
The paper is purely empirical: every central claim is a direct measurement of NTK-derived quantities on first-task data. Section 2.1 defines kernel spectral norm, kernel distance, kernel velocity, and kernel alignment without reference to the phenomenon being claimed; Section 2.2 reports the observed spike in kernel velocity and the check-mark trajectory in NTK max eigenvalue and explicitly states, "We refer to this phenomenon as the re-activation of feature learning." This is a naming of a measured pattern, not a derivation from the conclusion. The theoretical appendix equations (3)-(4) are standard per-eigenmode error dynamics and are used only to motivate why the NTK spectrum matters; they do not encode the task-switch result. There are no fitted parameters renamed as predictions, no uniqueness theorem imported from the authors' prior work, and no load-bearing self-citation chain. The cited references (Fort et al., Zhou et al., Yang, Yang and Hu) provide external context for NTK evolution and parametrization, but the observed reactivation at task boundaries is not forced by any of those citations. The skeptical concerns—that Fig. 12 and Appendix C.4 show reactivation amplitude decreasing with width, and that Appendix A.3 uses 32 random samples with no stated seed or same-batch protocol—are legitimate evidence-quality and extrapolation concerns, not circularity: they question whether the finite-width observation persists at the large-width limit or whether the measurement is stable, but they do not show that any claim reduces by construction to its inputs. Under the stated criteria, a non-finding with score 0 is appropriate.
Assumptions & free parameters
assumptions (4)
- standard math The standard NTK framework (Jacot et al.) relating gradient descent to kernel evolution holds for the finite-width CNN studied.
- domain assumption Kaiming Normal initialization with learning rate scaled as 1/N approximates the NTK parametrization and induces the lazy regime at the tested widths.
- domain assumption A 32-sample random batch yields a reliable estimate of the NTK trajectory.
- domain assumption The squared-loss eigenmode analysis (Eqs. 3-4) transfers to cross-entropy training, so that NTK spectral norm controls convergence speed in the experiments.
Cite this review
Pith. "Pith review of Reactivation: Empirical NTK Dynamics Under Task Shifts." pith.science (2026). https://pith.science/paper/73LLGYKX
@misc{pith2026250716039,
author = {Pith},
title = {Pith review of: Reactivation: Empirical NTK Dynamics Under Task Shifts},
year = {2026},
howpublished = {\url{https://pith.science/paper/73LLGYKX}},
note = {Machine review of arXiv:2507.16039}
}
read the original abstract
The Neural Tangent Kernel (NTK) offers a powerful tool to study the functional dynamics of neural networks. In the so-called lazy, or kernel regime, the NTK remains static during training and the network function is linear in the static neural tangents feature space. The evolution of the NTK during training is necessary for feature learning, a key driver of deep learning success. The study of the NTK dynamics has led to several critical discoveries in recent years, in generalization and scaling behaviours. However, this body of work has been limited to the single task setting, where the data distribution is assumed constant over time. In this work, we present a comprehensive empirical analysis of NTK dynamics in continual learning, where the data distribution shifts over time. Our findings highlight continual learning as a rich and underutilized testbed for probing the dynamics of neural training. At the same time, they challenge the validity of static-kernel approximations in theoretical treatments of continual learning, even at large scale.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Implicit regularization via neural feature alignment
Aristide Baratin, Thomas George, C ´esar Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In International Conference on Artificial Intelligence and Statistics , pages 2269–2277. PMLR, 2021
work page 2021
-
[2]
Generalisation guarantees for continual learning with orthogonal gradient descent
Mehdi Abbana Bennani, Thang Doan, and Masashi Sugiyama. Generalisation guarantees for continual learning with orthogonal gradient descent. arXiv preprint arXiv:2006.11942, 2020
arXiv 2006
-
[3]
Gradient de- scent on neural networks typically occurs at the edge of stability
Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient de- scent on neural networks typically occurs at the edge of stability. In International Confer- ence on Learning Representations , 2021. URL https://openreview.net/forum? id=jh-rTtvkGeM
work page 2021
-
[4]
Algorithms for learning kernels based on centered alignment
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research , 13(1):795–828, 2012
work page 2012
-
[5]
A Theoretical Analysis of Catastrophic Forgetting through the NTK Overlap Matrix
Thang Doan, Mehdi Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix, 2021. URL https://arxiv.org/abs/2010.04003
work page Pith review arXiv 2021
-
[6]
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33:5850–5861, 2020
work page 2020
-
[7]
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015
2015
-
[8]
Neural tangent kernel: Con- vergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Con- vergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc.,
Show all 17 references
-
[9]
Neural tangent kernel: Convergence and generalization in neural networks
Arthur Jacot, Franck Gabriel, and Cl ´ement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31: 8571–8580, 2018
2018
-
[10]
The break-even point on optimization trajectories of deep neural networks
Stanislaw Jastrzebski, Maciej Szymczak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho*, and Krzysztof Geras*. The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020. URL https://openr...
2020
-
[11]
Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting, 2022
Ryo Karakida and Shotaro Akaho. Learning curves for continual learning in neural networks: Self-knowledge transfer and forgetting, 2022. URL https://arxiv.org/abs/2112. 01653
2022
-
[12]
Similarity of neural network representations revisited
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning , pages 3519–3529. PMLR, 2019
2019
-
[13]
A theory of neural tangent kernel alignment and its influ- ence on training
Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influ- ence on training. arXiv preprint arXiv:2105.14301, 2021
2021 arXiv
-
[14]
Tensor programs ii: Neural tangent kernel for any architecture, 2020
Greg Yang. Tensor programs ii: Neural tangent kernel for any architecture, 2020. URL https://arxiv.org/abs/2006.14548
2020 arXiv
-
[15]
Greg Yang and Edward J. Hu. Feature Learning in Infinite-Width Neural Networks, July 2022. URL http://arxiv.org/abs/2011.14522. arXiv:2011.14522 [cs]
2022 arXiv
-
[16]
On the cone effect in the learning dynamics, 2025
Zhanpeng Zhou, Yongyi Yang, Jie Ren, Mahito Sugiyama, and Junchi Yan. On the cone effect in the learning dynamics, 2025. URL https://arxiv.org/abs/2503.16316. 8 REACTIVATION : E MPIRICAL NTK D YNAMICS UNDER TASK SHIFTS Appendix A. The NTK Framework and NTK Spectrum Consider a ...
2025 arXiv
-
[2018]
URL https://proceedings.neurips.cc/paper_files/paper/2018/ file/5a4be1fa34e62bb8a6ec6b91d2462f5a-Paper.pdf
2018
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.