Pith. sign in

REVIEW 3 major objections 4 minor 53 references

Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper derives closed dynamical mean field theory equations that exactly characterize the typical training and test loss dynamics of randomly initialized deep linear networks in a proportional large-system limit.

desk verdict A genuinely new DMFT for deep linear networks, with the standard saddle-point caveat and a heuristic scaling-law section; worth refereeing and citing, but 'complete characterization' overstates what is proved. read the letter →

arxiv 2502.02531 v3 pith:SSWST7SU submitted 2025-02-04 cs.LG cond-mat.dis-nnstat.ML

Blake Bordelon , Cengiz Pehlevan This is my paper · ORCID
classification cs.LGcond-mat.dis-nnstat.ML MSC 68T07
keywords deeplinearnetworksdynamicalmeanfieldtheorytrainingdynamicshyperparametertransferresidualpower-lawscalingrandominitializationgradientdescent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that gradient descent training of a deep linear network, started from random weights and trained on random data, is exactly described in the limit where input dimension, width, and dataset size grow together at fixed ratios by a small closed set of equations for correlation and response functions at every hidden layer. These equations predict the full training and test loss trajectories, including how they depend on width, depth, dataset size, learning rate, and parameterization. If correct, this is a minimal model that simultaneously reproduces the 'wider is better' effect in mean-field/maximum-update networks, learning rate transfer across width and depth, the failure of such transfer in neural tangent parameterization, overfitting from repeated data, and accelerated power-law scaling on structured data. A careful reader would care because the theory turns questions about hyperparameter transfer and architecture scaling into concrete, checkable formulas.

What carries the argument

The load-bearing object is the dynamical mean field theory (DMFT) action: a Martin--Siggia--Rose path integral over the disorder, namely the initial weights, the training data, and the SGD samples, whose integrand is dominated as $D\to\infty$ by a single saddle point. The saddle-point equations reduce the high-dimensional dynamics to single-site stochastic processes for $v(t)$, $\Delta(t)$, and each layer's $h^\ell(t)$ and $g^\ell(t)$, driven by colored Gaussian noise and by memory terms through the response functions. The key identity is that correlation functions are averages over the single-site noise, while response functions are derivatives of the fields with respect to that noise, forming a closed set whose time-diagonal entries give the train and test losses.

What would settle it

Numerically integrate the DMFT equations (65)--(68) for a depth-$L$ linear network at several ratios $\nu=N/D$ and $\alpha=P/D$, then simulate full-batch gradient descent at the same sizes with many independent initializations and data draws; if the measured average train and test loss curves and optimal-learning-rate curves do not converge to the DMFT predictions as $D$ grows, for instance if the gap does not shrink like $1/D$ or depends on the chosen saddle-point branch, the single-saddle-point closure fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that in the proportional limit $P/D=\alpha$, $N/D=\nu$ as $D\to\infty$, the typical-case training and test loss trajectories of a deep linear network are completely determined by a set of self-consistent DMFT equations, presented as Eqs. (65)--(68), in which each hidden layer is represented by a single-site Gaussian process. The forward fields $h^\ell(t)$ and backward fields $g^\ell(t)$ obey linear stochastic equations with colored Gaussian noise sources plus memory terms built from correlation functions $C(t,t')$ and response functions $R(t,t')$; the response functions are derivatives of the fields with respect to their noise sources. The train loss is $\hat L(t)=C_\Delta(t,t)$ and the test loss is $L(t)=C_v(t,t)+\sigma^2$, with explicit bias, limited-width ($1/\nu$), and limited-data ($1/\alpha$) contributions. Using this description, the paper shows that in NTK parameterization wider networks train more slowly and optimal learning rates shift with width, while in mean-field/$\mu$P parameterization they do not; that residual networks with branch scale $\beta_0/\sqrt{L}$ admit an infinite-depth limit; and that power-law structured data give test loss scaling $t^{-2\beta/(1+\beta)}$ in the hard-task regime, matching and extending recent one-hidden-layer results.

Load-bearing premise

The derivation assumes that in the $D\to\infty$ limit the disorder-averaged generating function is dominated by a single saddle point of the DMFT action, so that the typical-case dynamics coincide with the saddle-point equations.

Editorial extensions

If this is right

  • Learning-rate transfer becomes a prediction of the theory: in $\mu$P/mean-field parameterization the optimal learning rate is nearly independent of width $\nu$ and depth $L$, while in NTK parameterization it shifts, so transfer curves can be computed before training.
  • The bias--variance decomposition of Eq. (13) lets one attribute generalization gaps to finite width ($1/\nu$), finite data ($1/\alpha$), or bias dynamics, guiding how to scale a model.
  • Repeated-data full-batch gradient descent develops a test-loss overfitting peak through the memory response function $R_\Delta$, whereas one-pass online SGD does not, so the theory predicts when early stopping matters.
  • Residual networks with branch scale $\beta_0/\sqrt{L}$ have a well-defined $L\to\infty$ limit described by coupled stochastic differential equations over layer time, while unscaled branches diverge.
  • On power-law covariates, deep linear networks in the rich regime train with improved exponents, giving $t^{-2\beta/(1+\beta)}$ for hard tasks and $t^{-\beta}$ for easy tasks, extending the shallow-network result to any depth in the gradient-flow limit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same saddle-point structure may extend to nonlinear networks whose preactivations are Gaussian, but linearity is what enables the joint $P,D,N$ proportional limit and the dataset-averaged $h^0$ variable; without linearity the closed dataset-averaged description likely needs new ideas.
  • Editorial inference: the DMFT equations could serve as a cheap surrogate for hyperparameter search, since integrating them at small $\nu$ and $\alpha$ gives a predicted learning-rate plateau that can then be transferred to larger models.
  • Editorial inference: the theory's decomposition suggests a concrete scaling rule for SGD noise, namely that shrinking batch size at fixed $\alpha_B$ should behave like adding $1/\alpha_B$ to the width-limited variance, so smaller batches should use a slightly smaller learning rate to preserve transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper develops a dynamical mean field theory (DMFT) for the training dynamics of deep linear networks in a proportional limit where dataset size P, input dimension D, and hidden widths N grow together with fixed ratios α=P/D and ν=N/D. For full-batch gradient descent and online SGD, the authors derive closed single-site stochastic process equations for the correlation and response functions of the forward and backward fields, from which train and test losses are obtained. The framework is extended to residual networks, where a 1/√L branch scaling is shown to admit an infinite-depth limit described by a stochastic differential equation over a continuous layer index. For power-law structured data, the paper derives approximate dimension-free mean field equations and predicts accelerated power-law loss decays in the rich regime. Extensive finite-size simulations are presented in support of the theory across width, depth, data size, batch size, learning rate, and parameterization.

Significance. If correct, this is a substantial contribution: it provides a unified asymptotic description of deep linear network training that simultaneously captures width and data effects, hyperparameter transfer in µP versus NTK parameterizations, residual network depth limits, and power-law scaling laws. The theory's equations are parameter-free: the DMFT curves shown in Figures 1–7 are not fits but direct consequences of the closed equations, and the paper's numerical comparisons span a wide and convincing range of settings. The residual-network infinite-depth limit is a particularly elegant extension. The main caveats are that the core saddle-point evaluation is not rigorously justified, and the power-law exponent derivation is heuristic. These do not undermine the broad empirical value of the framework, but they do affect the strength of the paper's 'complete characterization' and 'scaling-law' claims.

major comments (3)
  1. [Appendix A.1.2, Eq. (56) and following paragraph] The claim that the disorder-averaged generating function is exactly described by the DMFT equations for the typical case rests on the assertion, after Eq. (56), that 'in this limit, the integral over q is dominated by the saddle point of S.' This is a Laplace/saddle-point statement about a complex, generically non-convex action in O(LT²) order parameters. No uniqueness of the relevant saddle, no Hessian control, and no contour-deformation argument are supplied. Without these, the 'completely characterizes' statement in Section 2 is not established; the excellent finite-size matches support the equations empirically but do not close this mathematical gap. I ask the authors to either add a rigorous concentration argument (e.g., via a dynamical version of the convex Gaussian min-max theorem or a Lipschitz-continuity argument for the observables) or to temper the claim to 'conjectured/empirically validated in the proportional limit'.
  2. [Appendix E.1, Eqs. (112)–(116) and Eq. (21)] The advertised power-law exponent in Equation (21), L(t) ~ t^{-2β/(1+β)} for β<1, is derived from the ansatz K(t) ~ 1 + γ0² t^{χ-1} in Eq. (113) plus a self-consistency argument imported from prior work. The ansatz is not derived from the DMFT equations, and the self-consistent argument has no uniqueness or stability check. Since this accelerated scaling law is one of the paper's headline claims, the heuristic status should be stated explicitly in the main text, and ideally the exponent should be verified by numerically solving the full DMFT equations for a range of γ0 and extracting the effective exponent, rather than relying on the closed-form ansatz.
  3. [Section 4, Appendix E] The structured-data dimension-free theory is labeled 'approximate' and the text notes it becomes accurate only around N,B ~ 16. However, the scaling law (21) and the learning-rate transfer statements in Figures 6–7 are presented as predictions of the theory. The paper should specify precisely which terms are neglected in the approximate mean field treatment (e.g., the replacement of the full response memory by a local noise term) and what limit, if any, justifies those approximations as N,B,D → ∞. Without this, the structured-data results are better described as a conjectured effective theory with numerical support, not a strict asymptotic characterization.
minor comments (4)
  1. [Throughout] There are several typos: 'availabel' in Section 4 (Structured Data), 'signficant' in the Limitations paragraph, 'dynaimcs' in Appendix C, and 'recovers the parameterization' (repeated phrase) in Section 3. These should be corrected.
  2. [Figure 6 caption] The caption writes 't 2/(1+ )' with a missing β in the exponent; please fix to t^{2/(1+β)} or similar.
  3. [Section 4, Eq. (18) and surrounding text] The reuse of α and β for both the proportional limits (α, ν) and the power-law exponents (α, β) is potentially confusing; a footnote is provided, but a different symbol for the power-law exponents would improve readability.
  4. [Appendix H, Table 1] The cost table is useful, but the claim that the non-data-averaged residual DMFT costs L²T³P³ should be explained with a one-sentence derivation or a reference, since this value is not directly evident from the cited work.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central DMFT derivation is self-contained and the predicted loss curves follow from the derived equations without fitting; the unproven saddle-point step and the power-law ansatz are rigor risks, not circular reductions.

full rationale

The paper's central claim is the DMFT characterization in Eqs. (65)-(68). The derivation in Appendix A is a first-principles Martin-Siggia-Rose computation: the disorder average over X, epsilon, and initial weights is performed explicitly (Eqs. 33-47), order parameters are introduced with delta-function constraints, and the D->infinity limit is taken by saddle point. The resulting single-site equations (65), response definitions (66), and correlation definitions (67) form a closed system; test and train losses (68) are read off from the diagonals. No parameter is fitted to the loss curves that are then presented as predictions; the DMFT curves in Figures 1-5 are solutions of these equations and the simulations are independent checks. The main caveat is Appendix A.1.2's assertion that 'the integral over q is dominated by the saddle point of S' after Eq. (56): no Hessian or uniqueness control is provided, so the exactness claim is a mathematical gap. This is a correctness risk, not circularity. The power-law exponent argument in Appendix E.1 (Eqs. 112-116) is also not derived from the closed DMFT equations: it explicitly posits K(t) ~ 1 + gamma0^2 t^{chi-1} and invokes the authors' prior self-consistency scheme in Bordelon et al. (2024b). However, the ansatz is transparently labeled, the resulting exponent is checked against simulations in Figure 10, and this heuristic supports only the scaling-law sub-claim, not the central DMFT characterization. The paper's own Limitations and Future Directions section acknowledges limited function approximation power and computational cost, which are honest limitations and do not indicate circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No parameters are fitted to data or target results in this paper. The inputs γ0 (richness), β0 (branch scale), η (learning rate), α and ν (data and width ratios), σ² (label noise), and the power-law exponents α, β are stated model or data parameters, not fit to match the derived loss curves. The exponent χ=2/(1+β) is solved from a self-consistency condition rather than fitted. The correlation and response functions are bookkeeping order parameters, not new physical entities.

assumptions (4)
  • standard math Martin-Siggia-Rose path integral representation and Gaussian disorder averaging
    Appendix A.1.2 represents the dynamics with delta functions and averages over Gaussian weights and data; this formalism is assumed valid.
  • domain assumption Saddle point dominance of the DMFT action as D→∞
    Appendix A.1.2 states the integral over order parameters is dominated by a single saddle point; this closure is not proven for this dynamics.
  • ad hoc to paper Effective kernel ansatz K(t) ~ 1 + γ0² t^{χ-1} in the small-γ0 limit
    Appendix E.1 uses this ansatz to derive the hard-task exponent in Eq. (21); it is an assumption rather than a consequence of the DMFT equations.
  • domain assumption Depth-µP scaling β=β0/√L and Θ(β0/√L) response functions in the infinite-depth limit
    Section 3 and Appendix C.1 adopt the branch scaling and response scaling (Eqs. 83-84) from prior work to obtain the SDE limit.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer." pith.science (2026). https://pith.science/paper/SSWST7SU

@misc{pith2026250202531,
  author       = {Pith},
  title        = {Pith review of: Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SSWST7SU}},
  note         = {Machine review of arXiv:2502.02531}
}
abstract

We theoretically characterize gradient descent dynamics in deep linear networks trained at large width from random initialization and on large quantities of random data. Our theory captures the ``wider is better" effect of mean-field/maximum-update parameterized networks as well as hyperparameter transfer effects, which can be contrasted with the neural-tangent parameterization where optimal learning rates shift with model width. We provide asymptotic descriptions of both non-residual and residual neural networks, the latter of which enables an infinite depth limit when branches are scaled as $1/\sqrt{\text{depth}}$. We also compare training with one-pass stochastic gradient descent to the dynamics when training data are repeated at each iteration. Lastly, we show that this model recovers the accelerated power law training dynamics for power law structured data in the rich regime observed in recent works.

Figures

Figures reproduced from arXiv: 2502.02531 by the authors.

Figure 1
Figure 1. Our theory captures the role of width, depth and dataset size on the test loss L(t) and train loss Lˆ(t) dynamics. The behavior of different models across widths depends on how γ0 scales with N. (a) Test loss of a L = 4 network in NTK parameterization where γ0 ∼ Θ( √1 ν ), which results in a kernel method (linear model) in the ν → ∞ limit. (b) In MFP/µP, the parameter γ0 = Θ(1) and the convergence rate improves as ν… view at source ↗
Figure 2
Figure 2. The theory can capture the failure/success of learning rate transfer in NTK/µP networks. (a) The optimal learning rate increases with model width ν = N/D in NTK parameterization. This is due to the fact that the effective feature learning rate γ0 is decreasing as 1/ √ ν. (b) The optimal learning rate is approxi￾mately preserved across ν = N/D in µP networks where γ0 is constant, enabling learning rate transfer. Effe… view at source ↗
Figure 3
Figure 3. Train and test losses of a depth L = 4 linear network versus dataset size P = αD compared to online SGD with batch size B = αBD. (a) Networks can fit the training data faster for smaller datasets α. (b) The test error can be non-monotonic as a function of data α if training is performed too long. (c) The dynamics of online SGD with ν = 0.5 across batchsizes B = αBD. Smaller batchsizes slow down training through accu… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The training dynamics of large depth residual networks with random initialization. (a) Increasing γ0 leads to an acceleration of training, but the initial kernel governing the updates are not changed so that d dtL(t)|t=0 is preserved. (b) Increasing β0 alters both the …
Figure 5
Figure 5. Figure 5: Learning rate transfer fails for vanilla ResNets but suc￾ceeds for scaled resnets, consistent with theory (dashed lines). (a) ResNets of varying depth L with constant raw branch scale β = 1. The optimal learning rate changes by several orders of magnitude. (b) ResNets …
Figure 6
Figure 6. Figure 6: A dimension free mean field description of the dynamics captures SGD effects and finite width N effects for power law data covariates. (a) SGD with small batchsize B is approximately captured by the mean field theory for L = 4. (b) Deep linear networks (L = 4) trained …
Figure 7
Figure 7. Figure 7: Learning rate transfer experiments for a L = 3 µP lin￾ear network trained with SGD (B = 128) on powerlaw features (α, β) = (2, 1.75) and γ0 = 0.4. (a) The hyperparameter transfer effect fails for NTK parameterization but (b) succeeds for µP and is accurately captured b…
Figure 8
Figure 8. Figure 8: Maximal stable learning rate as a function of width ν for different parameterizations. For very rich dynamics γ0 ≫ 1, the optimal learning rate approximately scales as ηmax ∼ ( Oν (1) MF (Fixed γ0) Oν  ν 1− 1 L  NTK (Fixed γ0 √ ν), (70) based on a simple argument abo…
Figure 9
Figure 9. Figure 9: Early dynamics of the online SGD can exhibit an initial blowup followed by recovery if γ0 > 0. The initial blowup is approximately L(t) ≈ [PITH_FULL_IMAGE:figures/full_fig_p026_9.png]
Figure 10
Figure 10. Figure 10: Scaling laws for the hard task regime are well predicted by the adjusted exponent. Following the self-consistency argument, we find that the kernel must scale as K(t) ∼ t 1−βχ. Combining this with the assumption implies that χ = 2 1 + β , ∀β < 1. (115) This implies th…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 27 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    and Pennington, J

    Adlam, B. and Pennington, J. The neural tangent kernel in high dimensions: Triple descent and a multi-scale theory of generalization. In International Conference on Machine Learning, pp.\ 74--84. PMLR, 2020 a

    work page 2020

  4. [4]

    and Pennington, J

    Adlam, B. and Pennington, J. Understanding double descent requires a fine-grained bias-variance decomposition. Advances in neural information processing systems, 33: 0 11022--11032, 2020 b

    work page 2020

  5. [5]

    S., Saxe, A

    Advani, M. S., Saxe, A. M., and Sompolinsky, H. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132: 0 428--446, 2020

    2020

  6. [6]

    Out-of-equilibrium dynamical mean-field equations for the perceptron model

    Agoritsas, E., Biroli, G., Urbani, P., and Zamponi, F. Out-of-equilibrium dynamical mean-field equations for the perceptron model. Journal of Physics A: Mathematical and Theoretical, 51 0 (8): 0 085002, 2018

    work page 2018

  7. [7]

    Local kernel renormalization as a mechanism for feature learning in overparametrized convolutional neural networks

    Aiudi, R., Pacelli, R., Baglioni, P., Vezzani, A., Burioni, R., and Rotondo, P. Local kernel renormalization as a mechanism for feature learning in overparametrized convolutional neural networks. Nature Communications, 16 0 (1): 0 568, 2025

    work page 2025

  8. [8]

    Neural networks as kernel learners: The silent alignment effect

    Atanasov, A., Bordelon, B., and Pehlevan, C. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=1NvflqAdoom

    work page 2022

Show all 53 references
  1. [9]

    Predictive power of a bayesian effective action for fully connected one hidden layer neural networks in the proportional limit

    Baglioni, P., Pacelli, R., Aiudi, R., Di Renzo, F., Vezzani, A., Burioni, R., and Rotondo, P. Predictive power of a bayesian effective action for fully connected one hidden layer neural networks in the proportional limit. Physical Review Letters, 133 0 (2): 0 027301, 2024

    2024

  2. [10]

    Feature learning in finite-width bayesian deep linear networks with multiple outputs and convolutional layers

    Bassetti, F., Gherardi, M., Ingrosso, A., Pastore, M., and Rotondo, P. Feature learning in finite-width bayesian deep linear networks with multiple outputs and convolutional layers. arXiv preprint arXiv:2406.03260, 2024

    2024 arXiv

  3. [11]

    and Pehlevan, C

    Bordelon, B. and Pehlevan, C. Self-consistent dynamical field theory of kernel evolution in wide neural networks. arXiv preprint arXiv:2205.09653, 2022

    2022 arXiv

  4. [12]

    and Pehlevan, C

    Bordelon, B. and Pehlevan, C. Dynamics of finite width kernel and prediction fluctuations in mean field neural networks. arXiv preprint arXiv:2304.03408, 2023

    2023 arXiv

  5. [13]

    A dynamical model of neural scaling laws

    Bordelon, B., Atanasov, A., and Pehlevan, C. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092, 2024 a

    2024 arXiv

  6. [14]

    How feature learning can improve neural scaling laws, 2024 b

    Bordelon, B., Atanasov, A., and Pehlevan, C. How feature learning can improve neural scaling laws, 2024 b . URL https://arxiv.org/abs/2409.17858

    2024 arXiv

  7. [15]

    B., Hanin, B., and Pehlevan, C

    Bordelon, B., Noci, L., Li, M. B., Hanin, B., and Pehlevan, C. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit. In The Twelfth International Conference on Learning Representations, 2024 c . URL https://openreview.net/forum?id=KZJehvRKGD

    2024

  8. [16]

    and Montanari, A

    Cheng, C. and Montanari, A. Dimension free ridge regression. arXiv preprint arXiv:2210.08571, 2022

    2022 arXiv

  9. [17]

    M., Kaur, S., Li, Y., Kolter, J

    Cohen, J. M., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. Gradient descent on neural networks typically occurs at the edge of stability. arXiv preprint arXiv:2103.00065, 2021

    2021 arXiv

  10. [18]

    and Sompolinsky, H

    Crisanti, A. and Sompolinsky, H. Path integral approach to random neural networks. Physical Review E, 98 0 (6): 0 062120, 2018

    2018

  11. [19]

    Bayes-optimal learning of deep random networks of extensive-width

    Cui, H., Krzakala, F., and Zdeborov \'a , L. Bayes-optimal learning of deep random networks of extensive-width. In International Conference on Machine Learning, pp.\ 6468--6521. PMLR, 2023 a

    2023

  12. [20]

    Error scaling laws for kernel classification under source and capacity conditions

    Cui, H., Loureiro, B., Krzakala, F., and Zdeborov \'a , L. Error scaling laws for kernel classification under source and capacity conditions. Machine Learning: Science and Technology, 4 0 (3): 0 035033, 2023 b

    2023

  13. [21]

    How two-layer neural networks learn, one (giant) step at a time

    Dandi, Y., Krzakala, F., Loureiro, B., Pesce, L., and Stephan, L. How two-layer neural networks learn, one (giant) step at a time. In NeurIPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023. URL https://openreview.net/forum?id=iBDcaBLhz2

    2023

  14. [22]

    C., Anguita, N., Proca, A

    Domin \'e , C. C., Anguita, N., Proca, A. M., Braun, L., Kunin, D., Mediano, P. A., and Saxe, A. M. From lazy to rich: Exact learning dynamics in deep linear networks. arXiv preprint arXiv:2409.14623, 2024

    2024 arXiv

  15. [23]

    and Gur-Ari, G

    Dyer, E. and Gur-Ari, G. Asymptotics of wide networks from feynman diagrams. In International Conference on Learning Representations, 2020

    2020

  16. [24]

    Double trouble in double descent: Bias and variance (s) in the lazy regime

    d’Ascoli, S., Refinetti, M., Biroli, G., and Krzakala, F. Double trouble in double descent: Bias and variance (s) in the lazy regime. In International Conference on Machine Learning, pp.\ 2280--2290. PMLR, 2020

    2020

  17. [25]

    A., Novak, R., Liu, P

    Everett, K., Xiao, L., Wortsman, M., Alemi, A. A., Novak, R., Liu, P. J., Gur, I., Sohl-Dickstein, J., Kaelbling, L. P., Lee, J., et al. Scaling exponents across parameterizations and optimizers. arXiv preprint arXiv:2407.05872, 2024

    2024 arXiv

  18. [26]

    Rigorous dynamical mean field theory for stochastic gradient descent methods

    Gerbelot, C., Troiani, E., Mignacco, F., Krzakala, F., and Zdeborova, L. Rigorous dynamical mean field theory for stochastic gradient descent methods. arXiv preprint arXiv:2210.06591, 2022

    2022

  19. [27]

    and Nica, M

    Hanin, B. and Nica, M. Finite depth and width corrections to the neural tangent kernel. arXiv preprint arXiv:1909.05989, 2019

    1909 arXiv

  20. [28]

    and Zlokapa, A

    Hanin, B. and Zlokapa, A. Bayesian inference with deep weakly nonlinear networks. arXiv preprint arXiv:2405.16630, 2024

    2024 arXiv

  21. [29]

    Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022

    2022 arXiv

  22. [30]

    and Lu, Y

    Hu, H. and Lu, Y. M. Universality laws for high-dimensional learning with random features. IEEE Transactions on Information Theory, 69 0 (3): 0 1932--1964, 2022

    1932

  23. [31]

    Neural tangent kernel: Convergence and generalization in neural networks

    Jacot, A., Gabriel, F., and Hongler, C. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018

    2018

  24. [32]

    Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning

    Kunin, D., Ravent \'o s, A., Domin \'e , C., Chen, F., Klindt, D., Saxe, A., and Ganguli, S. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. arXiv preprint arXiv:2406.06158, 2024

    2024 arXiv

  25. [33]

    Wide neural networks of any depth evolve as linear models under gradient descent

    Lee, J., Xiao, L., Schoenholz, S., Bahri, Y., Novak, R., Sohl-Dickstein, J., and Pennington, J. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32, 2019

    2019

  26. [34]

    and Sompolinsky, H

    Li, Q. and Sompolinsky, H. Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization. Physical Review X, 11 0 (3): 0 031059, 2021

    2021

  27. [35]

    S., Krzakala, F., Urbani, P., and Zdeborova, L

    Mannelli, S. S., Krzakala, F., Urbani, P., and Zdeborova, L. Passed & spurious: Descent algorithms and local minima in spiked matrix-tensor models. In international conference on machine learning, pp.\ 4333--4342. PMLR, 2019

    2019

  28. [36]

    C., Siggia, E., and Rose, H

    Martin, P. C., Siggia, E., and Rose, H. Statistical dynamics of classical systems. Physical Review A, 8 0 (1): 0 423, 1973

    1973

  29. [37]

    and Montanari, A

    Mei, S. and Montanari, A. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75 0 (4): 0 667--766, 2022

    2022

  30. [38]

    Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit

    Mei, S., Misiakiewicz, T., and Montanari, A. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on Learning Theory, pp.\ 2388--2464. PMLR, 2019

    2019

  31. [39]

    and Urbani, P

    Mignacco, F. and Urbani, P. The effective noise of stochastic gradient descent. Journal of Statistical Mechanics: Theory and Experiment, 2022 0 (8): 0 083405, 2022

    2022

  32. [40]

    Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification

    Mignacco, F., Krzakala, F., Urbani, P., and Zdeborov \'a , L. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems, 33: 0 9540--9550, 2020

    2020

  33. [41]

    Why do learning rates transfer? reconciling optimization and scaling limits for deep learning

    Noci, L., Meterez, A., Hofmann, T., and Orvieto, A. Why do learning rates transfer? reconciling optimization and scaling limits for deep learning. arXiv preprint arXiv:2402.17457, 2024

    2024 arXiv

  34. [42]

    A statistical mechanics framework for bayesian deep neural networks beyond the infinite-width limit

    Pacelli, R., Ariosto, S., Pastore, M., Ginelli, F., Gherardi, M., and Rotondo, P. A statistical mechanics framework for bayesian deep neural networks beyond the infinite-width limit. Nature Machine Intelligence, 5 0 (12): 0 1497--1507, 2023

    2023

  35. [43]

    4+ 3 phases of compute-optimal neural scaling laws

    Paquette, E., Paquette, C., Xiao, L., and Pennington, J. 4+ 3 phases of compute-optimal neural scaling laws. arXiv preprint arXiv:2405.15074, 2024

    2024 arXiv

  36. [44]

    Ren, Y., Nichani, E., Wu, D., and Lee, J. D. Emergence and scaling laws in sgd learning of shallow neural networks. arXiv preprint arXiv:2504.19983, 2025

    2025

  37. [45]

    A., Yaida, S., and Hanin, B

    Roberts, D. A., Yaida, S., and Hanin, B. The principles of deep learning theory. Cambridge University Press Cambridge, MA, USA, 2022

    2022

  38. [46]

    and Vanden-Eijnden, E

    Rotskoff, G. and Vanden-Eijnden, E. Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics, 75 0 (9): 0 1889--1935, 2022

    1935

  39. [47]

    M., McClelland, J

    Saxe, A. M., McClelland, J. L., and Ganguli, S. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013

    2013 arXiv

  40. [48]

    C., Xu, J., Haas, M., and Cevher, V

    Vankadara, L. C., Xu, J., Haas, M., and Cevher, V. On feature learning in structured state space models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  41. [49]

    and Hu, E

    Yang, G. and Hu, E. J. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pp.\ 11727--11737. PMLR, 2021

    2021

  42. [50]

    J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J

    Yang, G., Hu, E. J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., and Gao, J. Tuning large neural networks via zero-shot hyperparameter transfer. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Info...

    2021

  43. [51]

    Tensor programs vi: Feature learning in infinite-depth neural networks

    Yang, G., Yu, D., Zhu, C., and Hayou, S. Tensor programs vi: Feature learning in infinite-depth neural networks. arXiv preprint arXiv:2310.02244, 2023

    2023 arXiv

  44. [52]

    Zavatone-Veth, J. A. and Pehlevan, C. Learning curves for deep structured gaussian feature models, 2023

    2023

  45. [53]

    A., Tong, W

    Zavatone-Veth, J. A., Tong, W. L., and Pehlevan, C. Contrasting random and learned features in deep bayesian linear regression. Physical Review E, 105 0 (6): 0 064118, 2022

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.