Pith. sign in

REVIEW 2 major objections 2 minor 33 references

Label and residual projections onto NTK eigenvectors scale with eigenvalues, yielding a convergence bound that uses the full spectrum instead of only its smallest value.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-30 11:37 UTC pith:KZKPL2DM

load-bearing objection The paper introduces Label-NTK and Residual-NTK alignments to replace the min-eigenvalue rate with a spectrum-dependent convergence bound, but the key proportionality claims rest on vaguely stated assumptions. the 2 major comments →

arxiv 2605.25275 v1 pith:KZKPL2DM submitted 2026-05-24 cs.LG

Label-NTK Alignments and A Tighter Convergence Bound in the NTK Regime

Ruchirinkil Marreddy , Chaoyue Liu This is my paper
classification cs.LG
keywords Neural Tangent Kernelconvergence boundslabel alignmentresidual alignmentover-parameterized networkstraining dynamicsgeneralization boundseigen-spectrum
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that in the neural tangent kernel regime, the components of training labels and residuals along each NTK eigenvector grow proportionally to the square root of the corresponding eigenvalue. This alignment property replaces the classical worst-case rate, which is governed by the tiniest eigenvalue, with a tighter rate that sums contributions across the entire spectrum. The resulting bound tracks observed gradient-descent trajectories on MLPs and CNNs far more closely than prior guarantees. The authors supply both empirical measurements of the alignments and a theoretical argument resting on mild assumptions about the data distribution. They further obtain correspondingly improved generalization bounds as a direct corollary.

Core claim

The authors identify Label-NTK alignment and Residual-NTK alignment: the inner product of the label vector (or residual vector) with each NTK eigenvector is proportional to the square root of the associated eigenvalue. Substituting these relations into the linearized NTK dynamics produces a convergence bound whose rate is determined by a weighted sum over all eigenvalues rather than by the minimal eigenvalue alone, and the bound matches measured training loss curves on standard image datasets.

What carries the argument

Label-NTK alignment and Residual-NTK alignment, which encode the scaling of label and residual projections with NTK eigenvalues and thereby allow the dynamics to be expressed in terms of the full eigen-spectrum.

Load-bearing premise

The projections of labels and residuals onto NTK eigenvectors scale with the corresponding eigenvalues under mild data assumptions.

What would settle it

Compute the NTK eigen-decomposition on a fixed dataset and architecture, measure the actual per-eigenvector label projections, and check whether the observed residual decay rate under gradient descent matches the new spectrum-dependent bound; systematic mismatch on multiple runs would falsify the alignment claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Convergence speed is governed by an average over the aligned spectrum rather than the worst single eigenvalue.
  • The same alignment directly tightens generalization bounds that depend on the training trajectory.
  • The analysis applies uniformly to both fully connected and convolutional architectures.
  • The refined bound can be evaluated once the NTK spectrum is known, without requiring the minimal eigenvalue in isolation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the alignment strength varies with network depth or width, it could be used to predict which architectures will train faster on a given data distribution.
  • Spectrum estimation from a small number of samples might become sufficient for practical training-time forecasts.
  • The same scaling relation may persist under mild finite-width corrections, offering a bridge between infinite-width theory and observed behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper claims that projections of labels and residuals onto NTK eigenvectors scale proportionally with the corresponding eigenvalues (Label-NTK and Residual-NTK alignments), justified under mild data assumptions with empirical support. Exploiting these alignments, it derives a refined convergence bound for gradient descent in the NTK regime that depends on the full eigen-spectrum rather than being limited by the minimal eigenvalue, yielding rates that better match observed training dynamics, plus improved generalization bounds. The results are validated theoretically and via experiments on MLPs and CNNs across datasets.

Significance. If the alignments can be established with explicit, non-vacuous error controls that preserve a strict improvement over the classical min-eigenvalue bound, the work would meaningfully tighten NTK convergence analyses and help explain practical fast training. The spectrum-dependent form and cross-architecture experiments are strengths, but the overall significance depends on whether the new bound remains rigorous rather than heuristic.

major comments (2)
  1. [§3.2] §3.2 (Label-NTK alignment): The mild data assumptions are stated too vaguely to verify whether they imply the claimed proportionality (projection coefficient exactly proportional to eigenvalue) without additional factors or constants that could nullify the improvement. A precise statement of the assumptions together with a quantitative deviation bound is required for the refined convergence claim to be load-bearing.
  2. [§4, Theorem 4.1] §4, Theorem 4.1 (refined bound): The derivation replaces the classical λ_min factor with a spectrum-weighted sum that relies on the alignments holding with sufficient accuracy across all relevant eigen-directions. No perturbation or approximation-error term is provided; any systematic deviation for small eigenvalues reverts the guarantee to the pessimistic classical rate, undermining the central claim of a tighter, practical bound.
minor comments (2)
  1. [Figure 2] Figure 2: The projection-vs-eigenvalue scatter plots would benefit from overlaid reference lines (y = c·λ) with reported R² or max-deviation values to quantify alignment quality.
  2. Notation: The constant of proportionality in the alignment definitions is introduced without an explicit symbol; introducing one would clarify the subsequent bound derivations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. The comments highlight important points on the precision of our assumptions and the rigor of the convergence bound. We address each major comment below and will revise the manuscript accordingly to strengthen the presentation.

read point-by-point responses

  1. Referee: [§3.2] §3.2 (Label-NTK alignment): The mild data assumptions are stated too vaguely to verify whether they imply the claimed proportionality (projection coefficient exactly proportional to eigenvalue) without additional factors or constants that could nullify the improvement. A precise statement of the assumptions together with a quantitative deviation bound is required for the refined convergence claim to be load-bearing.

    Authors: We agree that the assumptions in §3.2 are presented at a high level. In the revised manuscript we will replace the current informal description with a precise mathematical statement of the data assumptions (including conditions on label-feature alignment with the NTK eigenspace) and add a new lemma that supplies an explicit quantitative bound on the deviation from exact proportionality, expressed in terms of measurable data properties. This will make the improvement over the classical bound verifiable and load-bearing. revision: yes

  2. Referee: [§4, Theorem 4.1] §4, Theorem 4.1 (refined bound): The derivation replaces the classical λ_min factor with a spectrum-weighted sum that relies on the alignments holding with sufficient accuracy across all relevant eigen-directions. No perturbation or approximation-error term is provided; any systematic deviation for small eigenvalues reverts the guarantee to the pessimistic classical rate, undermining the central claim of a tighter, practical bound.

    Authors: The referee correctly notes the absence of an explicit error term. While §3 establishes the alignments under the stated assumptions (with supporting theory and experiments), we will augment Theorem 4.1 with a perturbation analysis that introduces a controlled approximation-error term. The revised statement will show that the spectrum-weighted bound remains strictly tighter than the λ_min bound whenever the alignment deviation is smaller than a spectrum-dependent threshold, which our empirical results indicate holds in practice. A new remark will also clarify the conditions under which the bound reverts to the classical rate. revision: yes

Circularity Check

0 steps flagged

No circularity; alignment properties derived independently before bound refinement

full rationale

The paper first characterizes Label-NTK and Residual-NTK alignments as phenomena supported by empirical evidence and theoretical justification under stated mild data assumptions, then exploits those properties to obtain a spectrum-dependent convergence bound. No equation or step in the provided abstract reduces the refined bound to a fitted input, self-citation, or definitional equivalence; the alignment scaling is presented as an observed and justified input rather than a constructed output. The derivation therefore remains self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review; the only explicit assumption mentioned is 'mild data assumptions' used to justify the alignment scaling. No free parameters or invented entities are described.

axioms (1)
  • domain assumption mild data assumptions
    Invoked to obtain theoretical justification that label and residual projections scale with NTK eigenvalues.

pith-pipeline@v0.9.1-grok · 5695 in / 1144 out tokens · 30454 ms · 2026-06-30T11:37:12.545800+00:00 · methodology

0 comments
read the original abstract

The Neural Tangent Kernel (NTK) framework explains optimization in over-parameterized neural networks via approximately linearized dynamics, yielding exponential convergence guarantees. However, existing results are often overly pessimistic and do not match the fast training in practice, as they depend on the smallest NTK eigenvalue, which is typically extremely small in practice. In this work, we develop sharper convergence guarantees by characterizing the interaction between data labels and the NTK eigen-spectrum. We identify two key phenomena, Label-NTK alignment and Residual-NTK alignment, showing that projections of labels and residuals onto NTK eigenvectors scale with the corresponding eigenvalues. We provide empirical evidence and theoretical justification under mild data assumptions. Exploiting these alignment properties, we derive a refined convergence bound that depends on the full spectrum and closely matches practical training dynamics, significantly improving over classical worst-case results. We further obtain improved generalization bounds. Experiments on MLPs and CNNs across multiple datasets validate our theory.

Figures

Figures reproduced from arXiv: 2605.25275 by Chaoyue Liu, Ruchirinkil Marreddy.

Figure 1
Figure 1. Figure 1: Comparison of theoretical convergence bounds. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Label-NTK alignment and Residual-NTK alignment, of a five-layer CNN (width: 32, 64, 128, 128, 256) trained on the whole CIFAR-10 dataset at different training time t. Each point is computed as an average over 500 batches, each containing 100 samples. Note: slope α = 2 (left) and α = 1 (right). justification of the Label-NTK alignment under mild assumptions satisfied by real-world datasets. Notably, our the… view at source ↗
Figure 3
Figure 3. Figure 3: Label-NTK alignment and Residual-NTK alignment, of a five-layer MLP (width= 512) trained on the whole CIFAR-10 dataset at different training time t. Each point is computed as an average over 500 batches, each containing 100 samples. Note: slope α = 2 (left) and α = 1 (right). Theoretical justification. Now, we theoretically justify this label-NTK correlation with a focus on small eigenvalues. We make the f… view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of theoretical convergence bounds across architectures and datasets. The MLP [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Label-NTK alignment and Residual-NTK alignment for different architectures and datasets at multiple training time t. Left: Label-NTK alignment. Right: Residual-NTK alignment. First row: CNN on the Tiny-ImageNet10; Row 2-4: MLP on SVHN, Fashion-MNIST, and MNIST respectively. Each point is computed as an average over 500 batches, each containing 100 samples. Note: slope α = 2 (left) and α = 1 (right). 19 [P… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages

  1. [1]

    A convergence theory for deep learning via over- parameterization

    Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. InInternational conference on machine learning, pages 242–252. PMLR, 2019

    work page 2019

  2. [2]

    Du, Wei Hu, Zhiyuan Li, and Ruosong Wang

    Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. InProceedings of the 36th International Conference on Machine Learning, 2019

    work page 2019

  3. [3]

    Neural networks as kernel learners: The silent alignment effect

    Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. InInternational Conference on Learning Representations, 2022

    work page 2022

  4. [4]

    Frequency bias in neural networks for input of non-uniform density

    Ronen Basri, Meirav Galun, Amnon Geifman, David Jacobs, Yoni Kasten, and Shira Kritchman. Frequency bias in neural networks for input of non-uniform density. InInternational conference on machine learning, pages 685–694. PMLR, 2020

    work page 2020

  5. [5]

    The convergence rate of neural networks for learned functions of different frequencies.Advances in Neural Information Processing Systems, 32, 2019

    Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies.Advances in Neural Information Processing Systems, 32, 2019

    work page 2019

  6. [6]

    Spectrum dependent learning curves in kernel regression and wide neural networks

    Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. InProceedings of the 37th International Conference on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pages 1024–1034. PMLR, 2020

    work page 2020

  7. [7]

    Spectral bias and task-model alignment explain generalizationin kernelregression andinfinitelywide neuralnetworks.Nature Communications, 12(1):2914, May 2021

    Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalizationin kernelregression andinfinitelywide neuralnetworks.Nature Communications, 12(1):2914, May 2021. 10

    work page 2021

  8. [8]

    On lazy training in differentiable programming

    Lénaïc Chizat, Édouard Oyallon, and Francis Bach. On lazy training in differentiable programming. InAdvances in Neural Information Processing Systems, volume 32, pages 2937–2947, 2019

    work page 2019

  9. [9]

    Gradient descent on neural networks typically occurs at the edge of stability

    Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. InInternational Conference on Learning Representations, 2021

    work page 2021

  10. [10]

    Gradient descent finds global minima of deep neural networks

    Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. InInternational conference on machine learning, pages 1675–1685. PMLR, 2019

    work page 2019

  11. [11]

    Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh

    Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. InInternational Conference on Learning Representations, 2019

    work page 2019

  12. [12]

    On the similarity between the laplace and neural tangent kernels.Advances in Neural Information Processing Systems, 33:1451–1461, 2020

    Amnon Geifman, Abhay Yadav, Yoni Kasten, Meirav Galun, David Jacobs, and Basri Ronen. On the similarity between the laplace and neural tangent kernels.Advances in Neural Information Processing Systems, 33:1451–1461, 2020

    work page 2020

  13. [13]

    Disentangling feature and lazy training in deep neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2020(11):113301, 2020

    Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2020(11):113301, 2020

    work page 2020

  14. [14]

    Neural tangent kernel: Convergence and generalization in neural networks

    Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. InAdvances in Neural Information Processing Systems (NeurIPS), 2018

    work page 2018

  15. [15]

    Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks

    Ziwei Ji and Matus Telgarsky. Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow relu networks. In8th International Conference on Learning Representations, ICLR 2020, 2020

    work page 2020

  16. [16]

    Understanding the evolution of the neural tangent kernel at the edge of stability

    Kaiqi Jiang, Jeremy Cohen, and Yuanzhi Li. Understanding the evolution of the neural tangent kernel at the edge of stability. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    work page 2025

  17. [17]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    work page 2009

  18. [18]

    Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington

    Jaehoon Lee, Lechao Xiao, Samuel S. Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. InAdvances in Neural Information Processing Systems (NeurIPS), 2019

    work page 2019

  19. [19]

    The large learning rate phase of deep learning: The catapult mechanism

    Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: The catapult mechanism. InAdvances in Neural Information Processing Systems, volume 33, 2020

    work page 2020

  20. [20]

    Better NTK conditioning: A free lunch from (reLU) nonlinear activation in wide neural networks

    Chaoyue Liu, Han Bi, Like Hui, and Xiao Liu. Better NTK conditioning: A free lunch from (reLU) nonlinear activation in wide neural networks. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    work page 2025

  21. [21]

    Aiming towards the minimizers: fast convergence of sgd for overparametrized problems.Advances in neural information processing systems, 36:60748–60767, 2023

    Chaoyue Liu, Dmitriy Drusvyatskiy, Misha Belkin, Damek Davis, and Yian Ma. Aiming towards the minimizers: fast convergence of sgd for overparametrized problems.Advances in neural information processing systems, 36:60748–60767, 2023

    work page 2023

  22. [22]

    Loss landscapes and optimization in over-parameterized non-linear systems and neural networks.Applied and Computational Harmonic Analysis, 59:85–116, 2022

    Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks.Applied and Computational Harmonic Analysis, 59:85–116, 2022. 11

    work page 2022

  23. [23]

    On the linearity of large non-linear models: when and why the tangent kernel is constant.Advances in Neural Information Processing Systems, 33:15954–15964, 2020

    Chaoyue Liu, Libin Zhu, and Misha Belkin. On the linearity of large non-linear models: when and why the tangent kernel is constant.Advances in Neural Information Processing Systems, 33:15954–15964, 2020

    work page 2020

  24. [24]

    Transition to linearity of wide neural networks is an emerg- ing property of assembling weak models

    Chaoyue Liu, Libin Zhu, and Misha Belkin. Transition to linearity of wide neural networks is an emerg- ing property of assembling weak models. InInternational Conference on Learning Representations, 2022

    work page 2022

  25. [25]

    Characterizing the spectrum of the ntk via a power series expansion

    Michael Murray, Hui Jin, Benjamin Bowman, and Guido Montufar. Characterizing the spectrum of the ntk via a power series expansion. InThe Eleventh International Conference on Learning Representations, 2023

    work page 2023

  26. [26]

    Global convergence of deep networks with one wide layer followed by pyramidal topology.Advances in Neural Information Processing Systems, 33:11961–11972, 2020

    Quynh N Nguyen and Marco Mondelli. Global convergence of deep networks with one wide layer followed by pyramidal topology.Advances in Neural Information Processing Systems, 33:11961–11972, 2020

    work page 2020

  27. [27]

    Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks.IEEE Journal on Selected Areas in Information Theory, 1(1):84–105, 2020

    Samet Oymak and Mahdi Soltanolkotabi. Toward moderate overparameterization: Global convergence guarantees for training shallow neural networks.IEEE Journal on Selected Areas in Information Theory, 1(1):84–105, 2020

    work page 2020

  28. [28]

    A theory of neural tangent kernel alignment and its influence on training.arXiv preprint arXiv:2105.14301, 2021

    Haozhe Shan and Blake Bordelon. A theory of neural tangent kernel alignment and its influence on training.arXiv preprint arXiv:2105.14301, 2021

    work page arXiv 2021

  29. [29]

    Explicit loss asymptotics in the gradient descent training of neural networks.Advances in Neural Information Processing Systems, 34:2570–2582, 2021

    Maksim Velikanov and Dmitry Yarotsky. Explicit loss asymptotics in the gradient descent training of neural networks.Advances in Neural Information Processing Systems, 34:2570–2582, 2021

    work page 2021

  30. [30]

    Transition to linearity of general neural networks with directed acyclic graph architecture.Advances in neural information processing systems, 35:5363–5375, 2022

    Libin Zhu, Chaoyue Liu, and Misha Belkin. Transition to linearity of general neural networks with directed acyclic graph architecture.Advances in neural information processing systems, 35:5363–5375, 2022

    work page 2022

  31. [31]

    Gradient descent optimizes over- parameterized deep relu networks.Machine Learning, 109:467–492, 2020

    Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks.Machine Learning, 109:467–492, 2020

    work page 2020

  32. [32]

    An improved analysis of training over-parameterized deep neural networks.Advances in neural information processing systems, 32, 2019

    Difan Zou and Quanquan Gu. An improved analysis of training over-parameterized deep neural networks.Advances in neural information processing systems, 32, 2019. 12 A Proofs A.1 Proof of Theorem 3.4 Without loss of generality, we assume thati = 1and j = 2; namely, the angleθ between the pairx1 and x2 is smallθ≪1. We consider the normalized NTK matrix¯K = 1...

    work page 2019

  33. [33]

    Combining all the above, we finally get the desired result. A.3 Proof of Lemma A.1 Using Taylor expansion with Lagrange remainder term, as well as the GD update rulewt+1 − wt = −ηJ(w t)⊤(f(w t)−y), we have the following: f(w t+1)−f(w t) =−ηK(w t)(f(w t)−y) +ζ t =−ηK(w 0)(f(w t)−y)−ηδK t(f(w t)−y) +ζ t (26) whereδK t =K(w t)−K(w 0)and vectorζ t ∈R n has el...

    work page