Pith. sign in

REVIEW 2 major objections 4 minor 67 references

Correlation flow governs learning at criticality

T0 review · 2 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read At the critical initialisation point, the infinite-depth Neural Tangent Kernel converges to a fixed activation-dependent multiple of the fixed-point correlation matrix, linking learning dynamics to correlation propagation.

desk verdict The NTK–correlation equivalence is genuinely new and the derivation is elegant, but the proof covers n_out = O(L), not the proportional-output regime the abstract advertises. read the letter →

arxiv 2608.08350 v1 pith:FDICGZEG submitted 2026-08-08 cs.LG cond-mat.dis-nn

Andrea Combette , Nelly Pustelnik , Antoine Venaille This is my paper · ORCID
classification cs.LGcond-mat.dis-nn MSC 60B2068T07
keywords neuraltangentkernelmean-fieldtheoryrandommatrixcriticalinitialisationcorrelationpropagationdynamicalisometryorthogonalsequentiallimit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to prove that, in the sequential limit of networks whose width and then depth go to infinity, the Neural Tangent Kernel (the matrix governing gradient descent in the infinite-width regime) is asymptotically proportional to the fixed-point correlation matrix of the pre-activations, provided the network is initialised at the single critical point of the weight-bias variance plane. The proportionality constant is determined only by the activation function, and the result holds at the level of the whole NTK block, meaning the eigenvectors of the kernel align with the correlation eigenvectors. If correct, this unifies three previously separate descriptions of deep networks: correlation propagation, Jacobian spectra, and learning dynamics. The paper also establishes that this critical point is the only place where non-trivial input-dependent correlation survives to infinite depth, that the end-to-end Jacobian nevertheless decays algebraically so dynamical isometry fails, and that orthogonal initialisation suppresses the leading finite-size fluctuations which otherwise accumulate with depth. Finite-width and finite-depth numerical experiments on sine networks support the predicted scalings.

What carries the argument

The load-bearing structures are the mean-field correlation recursion (the deterministic layer-to-layer map for the pre-activation covariance), the critical point at sigma_w = 1/phi_1 and sigma_b = 0, and the end-to-end Jacobian, whose mean first moment is the product $Xi_L^{{alpha beta}}$ = prod_{ell=1}^L chi^perp_ell and decays algebraically as ~c/L at criticality. The proof then works through the rescaled overlap Jacobians, $P^{{alpha beta}}$_{L -> ell} = A L $\sqrt$($K^{{alpha alpha}}$_ell $K^{{beta beta}}$_ell) J^alpha_{L -> ell} $J^{{beta top}}$_{L -> ell}, whose normalised traces tend to 1 uniformly, so by Cesaro's theorem (1/L) sum_{ell=1}^L $P^{{alpha beta}}$_{L -> ell} converges to the identity. Asymptotic freeness of the layer-wise Jacobians supplies the trace and variance rules used to compute the first two spectral moments of the NTK; orthogonality enters by zeroing the weight non-orthogonality term in the finite-size perturbation, which removes the leading O(L/N) correction to the kernel. The result is a three-way identity where correlation flow, gradient transport, and the training kernel are controlled by the same fixed-point correlation.

What would settle it

Train a finite-width sinusoidal network with orthogonal initialisation at criticality, with n_out comparable to N and L large enough to satisfy the paper's scaling condition, and measure the NTK block during training; if the largest eigenvalue of the empirical NTK changes by an order-one factor relative to its initialisation value as training progresses, or if the initial NTK block is not proportional to the empirical correlation matrix with prefactor 1/(A $sigma_w^{2}$) (with A = 1 for sine), the central claim fails. Alternatively, compute the Frobenius norm of (1/L) sum_{ell=1}^L $P^{{alpha beta}}$_{L -> ell} minus the identity for increasing L; it should decay as $L^{{-gamma}}$ for orthogonal weights and should fail to vanish for Gaussian weights under the same scaling.

Watch

Extended reading notes

Core claim

The central claim is Eq. (92): in the sequential limit with orthogonal initialisation and a bounded analytic activation with the second Taylor coefficient vanishing, each NTK block converges to (1/(A $sigma_w^{2}$)) $rho^{{alpha beta}}$_{infty} I_{n_out}, where $rho^{{alpha beta}}$_{infty} is the infinite-depth correlation of the pre-activations for inputs x^$\alpha$ and x^$\beta$ and A is a positive constant built from the activation's Taylor coefficients (A = 1 for sine, A = 2 for tanh). The mechanism is that at criticality the end-to-end Jacobian decays as Xi_L ~ c/L, so the rescaled overlap Jacobians $P^{{alpha beta}}$_{L -> ell} have asymptotic unit normalised trace; their Cesaro average over layers then converges to the identity, and the NTK, expressed as a sum over layers pulled back by these Jacobians, inherits exactly the correlation sequence $rho^{{alpha beta}}$_ell. The convergence is sufficiently strong, in Frobenius norm, when L grows as L ≳ ($n_out^{2}$/N)^{1/(1-gamma)}, while Gaussian initialisation leaves relative fluctuations of order L/N that prevent the identity from holding deep into the network. A necessary companion result is that correlation propagation to infinite depth forces the second Taylor coefficient of the activation to vanish; otherwise the limiting correlation collapses to the input-independent values 1/2 or 1.

Load-bearing premise

The whole spectral story describes learning only if the Neural Tangent Kernel stays frozen during gradient descent; the paper proves freezing only for fixed output dimension and assumes, without proof, that it persists under the depth-width scaling used for the central theorem.

Editorial extensions

If this is right

  • In the frozen-NTK regime, the learning rate of every residual mode is set by the eigenvalue of rho^{alpha beta}_infty, so training speed and spectral bias are determined by the correlation structure of the data rather than by a frequency-specific kernel property.
  • Orthogonal initialisation at criticality makes the NTK block spectrum concentrate as depth grows, whereas Gaussian initialisation produces fluctuations that grow with depth, extending the depth range over which the sequential-limit description is predictive.
  • Because the NTK eigenvectors align with the principal modes of the normalised data correlation matrix, the learned directions in the linearised regime are the dataset's principal components, and input encodings can steer which frequencies or features are learned first.
  • The same machinery predicts that a tanh network at its critical point behaves like a sine network with the constant A adjusted, so the activation function enters the asymptotic learning dynamics only through the prefactor A and the condition that its second Taylor coefficient vanishes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the authors leave implicit is that the NTK-correlation identity should give quantitative decay rates for finite depth: the Frobenius distance between Theta_L^{alpha beta} and (A sigma_w^2)^{-1} rho^{alpha beta}_infty I_{n_out} should itself decay like L^{-gamma}, giving a testable convergence curve that is not reported in the paper.
  • The proof leans on the frozen-NTK hypothesis in the proportional-output regime, which the paper states as a belief rather than a theorem; if the kernel does evolve during training in that regime, Eq. (92) still describes the initial kernel but no longer directly controls the residual dynamics.
  • The finite-size analysis suggests a practical width-depth budget: under orthogonal initialisation, depth can be increased without widening the network and the kernel fluctuations stay of order 1/sqrt(N), whereas Gaussian initialisation forces N to grow with L, so an experiment sweeping (N, L) at fixed n_out would map where the two regimes deviate.
  • The input-norm dependence of rho^{alpha beta}_infty implies that dataset normalisation is not just a practical convenience but a condition for the theorem's predictions to be visible: large or anisotropic input radii can decorrelate even aligned inputs in the mean-field limit, so the NTK-correlation link should be tested on normalised data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. This paper develops a mean-field and random-matrix theory of fully connected networks at critical initialization (sigma_w = 1/phi_1, sigma_b = 0). It derives algebraic decay of the covariance and of the end-to-end Jacobian, compares finite-size fluctuations under Gaussian and orthogonal weight initialization, and proves that, at criticality, the NTK block converges in the infinite-depth limit to (1/(A sigma_w^2)) rho_infty^{alpha beta} I_{nout}. The authors also analyze the global NTK spectrum and its alignment with the input correlation, and support the theory with numerical experiments on sine networks.

Significance. The NTK-correlation proportionality is a novel and potentially unifying statement: it connects information propagation to learning dynamics, with a parameter-free prefactor fixed by the activation Taylor coefficients. The derivations of the algebraic Jacobian decay and of the first-moment convergence are coherent, and the numerical checks in Figs. 7 and 8 are quantitatively supportive. The main caveat is regime: the Frobenius convergence (92) is proven only under a depth-width scaling that rules out proportional outputs in the advertised L/N -> 0 sequential limit, and the frozen-NTK interpretation is not yet justified outside the fixed-output regime. If the scope is narrowed accordingly, the paper remains a substantial contribution.

major comments (2)
  1. [Sec. VI B, Eqs. (90)-(92), and Appendix E3] The proof of (92) is not valid for the proportional-output regime in the L/N -> 0 sequential limit. Substituting nout = cN into (90) gives L >= (c^2 N)^{1/(1-gamma)}, so L/N -> infinity; the Cesaro average in (91) is instead O(nout^2 log L/(N L)) = O(N log L/L), which diverges as L/N -> 0. The bound (E39) omits the ell-dependence of the orthogonal error (the correct term behaves as nout^2/(N ell) rather than nout^2/(N L)); the averaged bound therefore carries a logarithmic factor. Consequently Eq. (92) is proven only for nout = O(1) or nout = O(L) with L = o(N), and the abstract's unqualified 'sequential limit' statement is overbroad.
  2. [Sec. VI A, footnote 3] The learning-dynamics interpretation rests on the frozen-NTK residual dynamics (76), which the paper acknowledges is established only for nout = O(1). For the scalings under which (92) is proved with nout growing, kernel evolution during training is not ruled out; the sentence 'we believe that the NTK remains frozen' is a conjecture, not a theorem. The authors should either prove freezing in the relevant depth-width scaling or explicitly limit the claim that correlation 'governs learning' to the fixed-output regime.
minor comments (4)
  1. [Abstract] The abstract should state the precise regime for Eq. (92): fixed output dimension, or nout = O(L) with L = o(N), rather than the unqualified 'sequential limit of infinitely wide, infinitely deep networks'.
  2. [Sec. VI B, Eq. (89)] Equation (89) is typeset ambiguously: the scalings should be written as sqrt(nout L / N) and sqrt(nout / (N L)) so that the depth exponent is unmistakable.
  3. [Appendix E3] The bound (E39) should be stated with its ell-dependence; as written it suggests a uniform O(nout^2/(N L)) estimate for the orthogonal case, which the derivation does not support, and the Cesaro average carries an additional logarithmic factor.
  4. [Secs. IV C, VI B, and Discussion] The term 'sequential limit' is used both for N -> infinity followed by L -> infinity with L/N -> 0 and for the distinct depth-width scaling in (90); the paper should define the ordering once and use it consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the NTK–correlation proportionality (Eq. 92) is derived from the algebraic-decay estimate and free-probability variance bounds, not from a fit or self-citation.

full rationale

The central claim is the limit Theta_L -> rho/(A sigma_w^2) I (Eq. 92). The derivation is self-contained: Eq. (78) rewrites NTK blocks as covariance times Jacobian overlaps; the rescaling (80) defines P_{L->ell} explicitly; the first moment uses (83)-(85) and Cesaro; the Frobenius bound (91) is proved in Appendix E3 from the free-probability variance formulas (E36) and the algebraic decay of V[D], with the prefactor A fixed by Taylor coefficients in (36). No parameter is fitted to the quantity being predicted, and no uniqueness theorem is imported from the authors' prior work. The only self-citation to [39] supplies the sine kernel recursion and a prior critical-initialisation proposal; it is an illustrative input, not the load-bearing premise for Eq. (92). Footnote 3 and the scaling restriction around (90) are limitations or correctness caveats (frozen NTK is established only for nout=O(1), and condition (90) is incompatible with L/N->0 when nout=O(N)), but they do not make the derivation circular. I therefore find no significant circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the sequential limit, the activation assumptions H1-H4, asymptotic freeness of layer-wise Jacobians, and the unproven extension of NTK freezing to the proportional output regime. No parameters are fitted to data; the critical point is fixed by the activation's linear coefficient, and the prefactor A is computed from Taylor coefficients. No new entities are introduced.

assumptions (5)
  • domain assumption Activation function satisfies H1-H4: bounded, differentiable, Taylor expansion with phi1 > 0, monotonicity of the kernel recursion.
    Used to prove uniqueness and stability of the variance fixed point and convergence of the covariance recursion; excludes ReLU and other unbounded activations.
  • domain assumption Sequential limit: width N goes to infinity before depth L, with L/N to 0.
    Defines the baseline regime; finite-size corrections are treated as perturbations around this limit, and asymptotic freeness applies layer by layer.
  • standard math Asymptotic freeness of layer-wise Jacobians for Gaussian and orthogonal weights.
    Adopted from cited theorems (Yang 2021; Collins and Hayase 2023); underpins Eq. (59) for moments of products of freely independent matrices.
  • domain assumption The NTK remains frozen during training under the depth-width scaling (90).
    The paper acknowledges this is established only for nout=O(1) and states in footnote 3 that it believes the extension holds; this is load-bearing for the training interpretation.
  • domain assumption High-dimensional inputs and uniform hidden width: nin=N, hidden width N, output nout=O(N) or O(1).
    The paper assumes this scaling throughout; low-dimensional raw data is assumed lifted by a random Fourier-feature layer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Correlation flow governs learning at criticality." pith.science (2026). https://pith.science/paper/FDICGZEG

@misc{pith2026260808350,
  author       = {Pith},
  title        = {Pith review of: Correlation flow governs learning at criticality},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FDICGZEG}},
  note         = {Machine review of arXiv:2608.08350}
}
read the original abstract

The initialisation of deep neural networks determines whether information and gradients can propagate across depth, yet a unified theory connecting these properties to learning dynamics remains elusive. Combining mean-field theory and random matrix theory, we establish a direct link between correlation propagation and the Neural Tangent Kernel (NTK) that governs learning in the sequential limit of infinitely wide, infinitely deep networks. Correlation propagation to infinite depth is possible only at a single critical point in the weight-bias variance plane. At this point, we show that the end-to-end Jacobian vanishes algebraically with depth, and use this to prove that the NTK becomes exactly proportional to the output correlation at infinite depth. This equivalence between information propagation and learning dynamics had not yet been noticed. We further show that orthogonal initialisation suppresses the leading finite-size corrections present under Gaussian initialisation, clarifying the respective roles of the two initialisation ensembles in this limit. These theoretical predictions are validated quantitatively on finite-width, finite-depth networks. Together, these results demonstrate that orthogonal initialisation at criticality plays a central role in controlling the asymptotic dynamics of deep learning.

Figures

Figures reproduced from arXiv: 2608.08350 by the authors.

Figure 1
Figure 1. FIG. 1 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. FIG. 2 [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. FIG. 3 [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: FIG. 4 [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: FIG. 5 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: FIG. 6 [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]
Figure 7
Figure 7. Figure 7: FIG. 7 [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: FIG. 8 [PITH_FULL_IMAGE:figures/full_fig_p032_8.png]
Figure 9
Figure 9. Figure 9: FIG. 9. The two sufficient conditions [PITH_FULL_IMAGE:figures/full_fig_p039_9.png]
Figure 10
Figure 10. Figure 10: FIG. 10. Taxonomy of the network’s behaviour at criticality for different input pairs. This figure [PITH_FULL_IMAGE:figures/full_fig_p045_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

67 extracted references · 61 canonical work pages

  1. [1]

    In particular, it shows that the asymptotic NTK is strongly governed by the correlation dynamics

    Indeed: 1 nout ETrPαβ L→ℓ =AL √ Kαα ℓ Kββ ℓ 1 nout TrJα L→ℓ Jβ⊤ L→ℓ (83) ∼ L,ℓ→∞ AL 1 Aℓ Ξαβ L Ξαβ ℓ with Ξαβ L Ξαβ ℓ = L∏ ℓ′=ℓ+1 χ⊥ ℓ′ (84) ∼ L,ℓ→∞ 1.(85) Furthermore, using limℓ→∞ϵαβ ℓ = 0and the fact that, for every ℓ, (ETr[PL→ℓ])L≥ℓ is bounded, Cesàro’s theorem leads to: m(1) Θαβ ∞ = 1 Aσ2w ραβ ∞.(86) This remarkably simple result combines our previou...

    work page 2013

  2. [2]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell,et al., Language models are few-shot learners, Advances in neural information processing systems33, 1877 (2020)

    work page 2020

  3. [3]

    Kaplan, S

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, Scaling laws for neural language models, ArXivabs/2001.08361 (2020)

    arXiv 2020

  4. [4]

    Varoquaux, S

    G. Varoquaux, S. Luccioni, and M. Whittaker, Hype, sustainability, and the price of the bigger- is-better paradigm in ai, inProceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’25 (Association for Computing Machinery, New York, NY, USA,

    work page 2025

  5. [5]

    Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, Efficient backprop, inNeural Networks: Tricks of the Trade: Second Edition, edited by G. Montavon, G. B. Orr, and K.-R. Müller (Springer Berlin Heidelberg, Berlin, Heidelberg, 2012) pp. 9–48

    work page 2012

  6. [6]

    Bengio, P

    Y. Bengio, P. Simard, and P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE transactions on neural networks5, 157 (1994)

    work page 1994

  7. [7]

    Glorot and Y

    X. Glorot and Y. Bengio, Understanding the difficulty of training deep feedforward neural networks, inProceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 9, edited by Y. W. Teh and M. Titterington (PMLR, Chia Laguna Resort, Sardinia, Italy, 2010) pp. 249–256

    work page 2010

  8. [8]

    K. He, X. Zhang, S. Ren, and J. Sun, Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, in2015 IEEE International Conference on Computer Vision (ICCV)(2015) pp. 1026–1034

    work page 2015

Show all 67 references
  1. [9]

    D. A. Roberts, S. Yaida, and B. Hanin,The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks(Cambridge University Press, 2022)

    2022

  2. [10]

    Simon, D

    J. Simon, D. Kunin, A. Atanasov, E. Boix-Adserà, B. Bordelon, J. Cohen, N. Ghosh, F. Guth, A. Jacot, M. Kamb,et al., There will be a scientific theory of deep learning, arXiv preprint arXiv:2604.21691 (2026)

    2026 arXiv

  3. [11]

    Sompolinsky, Statistical mechanics of neural networks, Physics Today41, 70 (1988)

    H. Sompolinsky, Statistical mechanics of neural networks, Physics Today41, 70 (1988)

    1988

  4. [12]

    Carleo, I

    G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborová, Machine learning and the physical sciences, Reviews of Modern Physics91, 045002 (2019)

    2019

  5. [13]

    Bahri, J

    Y. Bahri, J. Kadmon, J. Pennington, S. S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli, Statistical Mechanics of Deep Learning, Annual Review of Condensed Matter Physics11, 501 (2020)

    2020

  6. [14]

    R. M. Neal,Bayesian learning for neural networks, Vol. 118 (Springer Science & Business Media, 1996)

    1996

  7. [15]

    J. Lee, Y. Bahri, R. Novak, S. S. Schoenholz, J. Pennington, and J. Sohl-Dickstein, Deep neural networks as gaussian processes, inInternational Conference on Learning Representations(2018)

    2018

  8. [16]

    S. Mei, A. Montanari, and P.-M. Nguyen, A mean field view of the landscape of two-layer neural networks, Proceedings of the National Academy of Sciences115, E7665 (2018)

    2018

  9. [17]

    Rotskoff and E

    G. Rotskoff and E. Vanden-Eijnden, Trainability and accuracy of artificial neural networks: An interacting particle system approach, Communications on Pure and Applied Mathematics75, 1889 (2022)

    2022

  10. [18]

    Chizat, E

    L. Chizat, E. Oyallon, and F. Bach, On lazy training in differentiable programming, inAdvances in Neural Information Processing Systems, Vol. 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., 2019)

    2019

  11. [19]

    Bahri, B

    Y. Bahri, B. Hanin, A. Brossollet, V. Erba, C. Keup, R. Pacelli, and J. B. Simon, Les houches lectures on deep learning at large and infinite width*, Journal of Statistical Mechanics: Theory and Experiment2024, 104012 (2024)

    2024

  12. [20]

    A. Saxe, J. McClelland, and S. Ganguli, Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, inProceedings of the International Conference on Learning Representations 2014(International Conference on Learning Representations 2014, 2014)

    2014

  13. [21]

    Poole, S

    B. Poole, S. Lahiri, M. Raghu, J. Sohl-Dickstein, and S. Ganguli, Exponential expressivity in deep neural networks through transient chaos, Advances in neural information processing systems29(2016). 35

    2016

  14. [22]

    S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, Deep information propagation, inInternational Conference on Learning Representations(2017)

    2017

  15. [23]

    Doshi, T

    D. Doshi, T. He, and A. Gromov, Critical initialisation of wide and deep neural networks using partial jacobians: General theory and applications, inAdvances in Neural Information Processing Systems, Vol. 36, edited by A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and ...

    2023

  16. [24]

    Pennington, S

    J. Pennington, S. Schoenholz, and S. Ganguli, Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice, inAdvances in Neural Information Processing Systems, Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwa...

    2017

  17. [25]

    Collins and T

    B. Collins and T. Hayase, Asymptotic freeness of layerwise jacobians caused by invariance of multilayer perceptron: The haar orthogonal case, Communications in Mathematical Physics 397, 85 (2023)

    2023

  18. [26]

    Jacot, F

    A. Jacot, F. Gabriel, and C. Hongler, Neural tangent kernel: Convergence and generalization in neural networks, Advances in neural information processing systems31(2018)

    2018

  19. [27]

    J. Lee, L. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington, Wide neural networks of any depth evolve as linear models under gradient descent, inProceedings of the 33rd International Conference on Neural Information Processing Systems(2019) pp. ...

    2019

  20. [28]

    Geiger, S

    M. Geiger, S. Spigler, A. Jacot, and M. Wyart, Disentangling feature and lazy training in deep neural networks, Journal of Statistical Mechanics: Theory and Experiment2020, 113301 (2020)

    2020

  21. [29]

    Graldi, A

    J. Graldi, A. Breccia, G. Lanzillotta, T. Hofmann, and L. Noci, The importance of being lazy: Scaling limits of continual learning, inProceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267 (PMLR, 2025)

    2025

  22. [30]

    Yaida, Non-gaussian processes and neural networks at finite widths, inMathematical and Scientific Machine Learning(PMLR, 2020) pp

    S. Yaida, Non-gaussian processes and neural networks at finite widths, inMathematical and Scientific Machine Learning(PMLR, 2020) pp. 165–192

    2020

  23. [31]

    Hanin, Random fully connected neural networks as perturbatively solvable hierarchies, Journal of Machine Learning Research25, 1 (2024)

    B. Hanin, Random fully connected neural networks as perturbatively solvable hierarchies, Journal of Machine Learning Research25, 1 (2024)

    2024

  24. [32]

    Hanin and M

    B. Hanin and M. Nica, Finite depth and width corrections to the neural tangent kernel, in International Conference on Learning Representations(2020)

    2020

  25. [33]

    H. Day, Y. Kahn, and D. A. Roberts, Feature learning and generalization in deep networks with orthogonal weights, Machine Learning: Science and Technology6, 035027 (2025)

    2025

  26. [34]

    G. Yang, Scaling limits of wide neural networks with weight sharing: Gaussian process behaviour, gradient independence, and neural tangent kernel derivation, arXiv preprint arXiv:1902.04760 (2019)

    2019 arXiv

  27. [35]

    Fan and Z

    Z. Fan and Z. Wang, Spectra of the conjugate kernel and neural tangent kernel for linear-width neural networks, inAdvances in Neural Information Processing Systems, Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., 2020) ...

    2020

  28. [36]

    X. Li, X. Han, and Q. Yang, Eigen analysis of conjugate kernel and neural tangent kernel, inProceedings of the 42nd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 267, edited by A. Singh, M. Fazel, D. Hsu, S. Lacoste-Julien, F. Ber...

    2025

  29. [37]

    Sitzmann, J

    V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, Implicit neural representa- tions with periodic activation functions, inAdvances in Neural Information Processing Systems, Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran ...

    2020

  30. [38]

    Raissi, P

    M. Raissi, P. Perdikaris, and G. E. Karniadakis, Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations, Journal of Computational Physics378, 686 (2019)

    2019

  31. [39]

    Hayou, A

    S. Hayou, A. Doucet, and J. Rousseau, On the impact of the activation function on deep neural networks training, inProceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, edited by K. Chaudhuri and R. Salakhutdin...

    2019

  32. [40]

    Combette, A

    A. Combette, A. Venaille, and N. Pustelnik, A new initialisation to control gradients in sinusoidal neural networks, inThe Fourteenth International Conference on Learning Representations (2026)

    2026

  33. [41]

    Tancik, P

    M. Tancik, P. P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. T. Barron, and R. Ng, Fourier features let networks learn high frequency functions in low dimensional domains, Advances in Neural Information Processing Systems (2020). 36

    2020

  34. [42]

    Speicher, Free probability and random matrices (2014), arXiv:1404.3393 [math.OA]

    R. Speicher, Free probability and random matrices (2014), arXiv:1404.3393 [math.OA]

    2014 arXiv

  35. [43]

    Yang, Tensor programs iii: Neural matrix laws (2021), arXiv:2009.10685 [cs.NE]

    G. Yang, Tensor programs iii: Neural matrix laws (2021), arXiv:2009.10685 [cs.NE]

    2021 arXiv

  36. [44]

    Basri, M

    R. Basri, M. Galun, A. Geifman, D. Jacobs, Y. Kasten, and S. Kritchman, Frequency bias in neural networks for input of non-uniform density, inProceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 119, edited by H. ...

    2020

  37. [45]

    Ronen, D

    B. Ronen, D. Jacobs, Y. Kasten, and S. Kritchman, The convergence rate of neural networks for learned functions of different frequencies, inAdvances in Neural Information Processing Systems, Vol. 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, ...

    2019

  38. [46]

    Rahimi and B

    A. Rahimi and B. Recht, Random features for large-scale kernel machines, inAdvances in Neural Information Processing Systems, Vol. 20, edited by J. Platt, D. Koller, Y. Singer, and S. Roweis (Curran Associates, Inc., 2007)

    2007

  39. [47]

    S. Wang, H. Wang, and P. Perdikaris, On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informed neural networks, Computer Methods in Applied Mechanics and Engineering384, 113938 (2021)

    2021

  40. [48]

    Yang and S

    G. Yang and S. Schoenholz, Mean field residual networks: On the edge of chaos, Advances in neural information processing systems30(2017)

    2017

  41. [49]

    L. Xiao, Y. Bahri, J. Sohl-Dickstein, S. Schoenholz, and J. Pennington, Dynamical isometry and a mean field theory of CNNs: How to train 10,000-layer vanilla convolutional neural networks, in Proceedings of the 35th International Conference on Machine Learning, Proceedings of ...

    2018

  42. [50]

    Price, A useful theorem for nonlinear devices having gaussian inputs, IEEE Transactions on Information Theory 1958-jun vol

    R. Price, A useful theorem for nonlinear devices having gaussian inputs, IEEE Transactions on Information Theory 1958-jun vol. 4 iss. 24, 10.1109/tit.1958.1057444 (1958)

    1958

  43. [51]

    Mingo and R

    J. Mingo and R. Speicher,Free Probability and Random Matrices, Fields Institute Monographs (Springer New York, 2018). Appendix A: Susceptibilities and Price theorem In this section we give the Gaussian expression of the susceptibilities and the elements to derive them. We begi...

    2018

  44. [52]

    We begin by rewriting the fixed point condition: K⋆ =F K(K⋆) :=σ 2 wE [ ϕ(z)2] +σ 2 b withz∼N(0,K ⋆).(B2) We consider two distinct cases depending on the value ofσ2 b

    Existence, uniqueness and stability of the diagonal fixed point The primary focus of this first part is to demonstrate why the diagonal elements of any fixed pointΣαβ ⋆ are equal for nonzero inputsxα,xβ, implying a unique stable fixed point for the variance. We begin by rewrit...

  45. [53]

    Basin of attraction of the diagonal fixed point In this section, we show that for each(σw,σb)and for non-zero inputs, there is exactly one basin of attraction coveringR+∗. More precisely, for non-zero inputs, the recurrence relation (20) converges to the unique stable fixed po...

  46. [54]

    Finite Number of Fixed Points forF Now that we have characterized the diagonal recurrence relation generated byFK, to fully determine the global recurrence, we must also studyFC in (B1), the restriction ofF to the off-diagonal elements. a. Asymptotic AutonomyFirst, note that s...

  47. [55]

    In other words, we want to show that, for everyKαβ 0 in this interval, the sequence generated by the limiting map converges

    Basin of convergence of the off-diagonal recursion Now that we know that there is a finite number of fixed points, let us determine whether their basins of convergence cover the entire admissible covariance interval[−K⋆,K⋆]. In other words, we want to show that, for everyKαβ 0...

  48. [56]

    Discussion of HypothesisH2We verify that hypothesis (H2), equivalently stated in (B7), is reasonable and holds for common activation functions

    Discussion on Hypotheses (H1-H4) a. Discussion of HypothesisH2We verify that hypothesis (H2), equivalently stated in (B7), is reasonable and holds for common activation functions. Common sigmoidal functionsA sufficient condition for (B7) is thatϕsatisfies: ∀x̸= 0 d dx ϕ(x) ∥x∥...

  49. [57]

    Recurrence relation Here we derive the correlation expression and show(38). Let’s delve into the calculations, using the recurrence relation (35): ραβ ℓ+1 = Kαβ ℓ+1√ Kαα ℓ+1Kββ ℓ+1 (C1) = Kαβ ℓ√ Kαα ℓ Kββ ℓ   1 + ϕ2 2 4ϕ2 1 ( Kαα ℓ Kββ ℓ Kαβ ℓ + 2Kαβ ℓ ) + ϕ3 2ϕ1 (Kαα ℓ +K ...

  50. [58]

    Finite size effects a. Induced error termsThe mean-field perturbed recurrence for the empirical kernel ˆKαβ is derived as follows: ˆKαβ ℓ+1 = 1 N zα⊤ ℓ+1zβ ℓ+1 (C4) = 1 Nϕ(zα ℓ )⊤ W⊤ ℓ+1Wℓ+1 N ϕ(zβ ℓ )(C5) =ϕ(z α)⊤ ( W⊤ ℓ+1Wℓ+1 N −σ 2 wI ) ϕ(zβ) +σ2 w N ϕ(zα)⊤ϕ(zβ)(C6) =σ 2 w ...

  51. [59]

    Taxonomy of network effects at criticality In this section, we discuss in detail the flow of correlation at large depth for various input norms and correlations as explained in Sec. IV. To this end, we restrict our study to six well-chosen pairs of inputs on two spheres of rad...

  52. [60]

    Concentration for fixedn out In this section we will prove that in the simplest casenout finite andN→∞ we will have: V[Jα LJβ⊤ L ] = 0(D1) 46 To do so, let us consider a matrix element ( Jα LJβ⊤ L ) ij ( Jα LJβ⊤ L ) ij = 1 nL−1 ∑ k,k′ (WL)ik(W⊤ L)k′jCαβ kk′,(D2) whereCαβ conta...

  53. [61]

    Asymptotic Freeness and its consequences on first and second moments of matrix products This appendix derives equation(59) using free probability. We begin by introducing the necessary definitions offree cumulantsandasymptotic freeness, and establish key identities for mixed m...

  54. [62]

    Gaussian weights moments In this section, we derive the first spectral moment of1 nℓ−1 W⊤ ℓ Wℓ and its relative variance V[ 1 nℓ−1 W⊤ ℓ Wℓ]. From the Marchenko-Pastur law: m(1) 1 nℓ−1 WℓW⊤ ℓ =σ 2 w andV[ 1 nℓ−1 WℓW⊤ ℓ ] = nℓ nℓ−1 .(D36) Following similar reasoning to that in t...

  55. [63]

    This gives the scaling ( m(2) Dα ℓ Dβ ℓ /(m(1) Dα ℓ Dβ ℓ )2−1 ) =O(ℓ−2), giving us the convergence of the studied sum

    Convergence of ∑L ℓ=1 V[Dα ℓ Dβ ℓ ] For this, it is sufficient to recall the definition (16) which yields : L∑ ℓ=1 V[Dα ℓ Dβ ℓ ] = L∑ ℓ=1   m(2) Dα ℓ Dβ ℓ ( m(1) Dα ℓ Dβ ℓ )2−1   Forϕ 2 = 0, by Taylor expanding at criticality we get m(1) Dα ℓ Dβ ℓ =ϕ 2 1 [ 1 +ϕ3 ϕ1 1...

  56. [64]

    Expression of the NTK operator Using the definition of the NTK (14) and the relationzα ℓ = 1√nℓ−1 Wℓϕ ( zα ℓ−1 ) (there is no bias at criticality), we get: ˆΘαβ ℓ = ∂zα ℓ ∂θℓ ( ∂zβ ℓ ∂θℓ )⊤ +Jα ℓˆΘαβ ℓ−1Jβ⊤ ℓ (E1) =ˆΦαβ ℓ + ℓ−1∑ i=1 ˆΦαβ i ℓ−i−1∏ k=0 Jα ℓ−k (ℓ−i−1∏ k=0 Jβ ℓ−k ...

  57. [65]

    Second Moment of the NTK Let’s first start by expanding the second moment as follows: m(2) Θαβ L = 1 nout TrE [ (Θαβ L )2 ] (E6) = 1 (AL)2 1 σ4w L∑ ℓ=0 L∑ ℓ′=0 ραβ ℓ ραβ ℓ′ 1 nout TrE [ Pαβ L→ℓPαβ L→ℓ′ ] .(E7) Then, using the definition ofPαβ L→ℓ: 1 nout TrE [ Pαβ L→ℓPαβ L→ℓ′ ...

  58. [66]

    Bound on ‖‖‖1 L ∑L ℓ=1 Pαβ L→ℓ−I ‖‖‖ 2 F In this section we will prove(91)

    Eigenvectors and learning directions a. Bound on ‖‖‖1 L ∑L ℓ=1 Pαβ L→ℓ−I ‖‖‖ 2 F In this section we will prove(91). First, recall that we have: ‖‖‖‖‖ 1 L L∑ ℓ=1 Pαβ L→ℓ−I ‖‖‖‖‖ 2 F ≤ 1 L L∑ ℓ=1 ‖‖‖Pαβ L→ℓ−I ‖‖‖ 2 F .(E32) Then expanding the summand leads to: ‖‖‖Pαβ L→ℓ−I ‖‖‖ 2...

  59. [67]

    From second moments of block NTKΘαβ L to second moments of the global NTK ΘL The full NTK matrixΘL∈R noutD×noutD second moment satisfies the following expansion : m(2) ΘL = 1 noutD TrE [ (ΘL)2] (E47) = 1 D ∑ α,β∈{1,...,D} 1 nout TrE [ Θαβ L Θαβ⊤ L ] (E48) 56 Using Cauchy-Schwa...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.