Pith. sign in

REVIEW 3 major objections 6 minor 2 cited by

Precise gradient descent training dynamics for finite-width multi-layer neural networks

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper establishes a per-iteration state-evolution law for vanilla gradient descent on finite-width multilayer networks, showing first-layer weights become Gaussian-fluctuating while deeper layers concentrate, and gives a consistent…

desk verdict A serious, honest, complete-looking theory paper, but the central claim rests on unproved GFOM state evolution theorems imported from the same author's unreviewed preprint; send it to referees and make that the first question. read the letter →

arxiv 2505.04898 v1 pith:ECMV5OHN submitted 2025-05-08 cs.LG cs.AImath.OCmath.STstat.MLstat.TH

Qiyang Han , Masaaki Imaizumi This is my paper · ORCID
classification cs.LGcs.AImath.OCmath.STstat.MLstat.TH MSC 60E1560G15
keywords gradientdescentdynamicsmulti-layerneuralnetworksstateevolutionfinite-widthproportionalregimesingle-indexregressiongeneralizationerrorestimationgeneralfirstordermethodsnon-asymptotictheory
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish a precise, per-iteration law for what vanilla gradient descent does to a multilayer neural network when sample size and feature dimension grow together while width and depth stay bounded. If true, it replaces vague narratives about feature learning with a quantitative description: the first-layer weights fluctuate as Gaussians whose covariance is dictated by a state-evolution recursion, the deeper-layer weights concentrate on deterministic matrices, and both training and test error are given by low-dimensional Gaussian integrals. Because the description holds for non-Gaussian features and for individual initializations, it covers behavior that infinite-width theories cannot see, including nontrivial movement away from initialization. The practical payoff is an augmented gradient-descent routine that, at every iteration, outputs a consistent estimate of the model's test error without knowing the signal or link function, which could guide early stopping.

What carries the argument

The engine is an iterative reduction scheme: gradient descent is shown, step by step, to be indistinguishable from a sequence of matrix-variate 'general first order methods' (GFOMs), iterative schemes in which each update is a row-wise nonlinear function of previous iterates plus a matrix multiplication by the data matrix. The GFOM iterates admit an entrywise, non-asymptotic Gaussian state evolution, and the debiasing coefficients connecting the two are matrix-valued Onsager correction matrices that absorb the correlations between successive pre-activations and restore approximate Gaussianity.

What would settle it

Fix a smooth activation such as sigmoid, width q=10, m/n≈1/2, and run vanilla gradient descent with learning rate 2 on single-index data with sub-gaussian features; compare the empirical row distribution of $n^{{1/2}}$$W_1^{{(t)}}$ with the state-evolution Gaussian for t=2 and t=5 across n=600, 2400, and 9600. If the pseudo-Lipschitz discrepancy does not decay to zero at the stated polynomial rate, the central claim is false; separately, for ReLU with the second derivative set to zero, the paper's Figure 4 already shows the test-error estimate drifting from the truth.

Watch

Extended reading notes

Core claim

For every iteration t of vanilla gradient descent under proportional sample size and feature dimension, the row-wise empirical distribution of the scaled first-layer weights $n^{{1/2}}$$W_1^{{(t)}}$ is approximated, in a pseudo-Lipschitz mean with error (KΛκ*)^{c_t} $n^{{-1/c_t}}$, by a Gaussian law produced by a state-evolution recursion; the deeper-layer weights concentrate around deterministic matrices $V_α^{{(t)}}$ with the same order of error. The state evolution combines Onsager correction matrices that debias the first-layer pre-activations, deterministic updates depending smoothly on the initialization, and a Gaussian component encoding the high-dimensional noise. From this, the paper derives that both training error and test error are, up to the same error rate, averages of low-dimensional Gaussian integrals over deterministic residual maps; that these integrals reveal a non-vanishing generalization gap in the proportional regime; and that the trained network remains a single-index function whose effective signal is a linear combination of the true signal and the initialization. The paper further shows that these Gaussian integrals can be computed online by augmenting gradient descent with closed-form correction estimates, yielding a consistent estimator of test error that requires no knowledge of the link function or signal.

Load-bearing premise

The whole argument, including the algorithm, requires every activation to be four times differentiable with bounded derivatives and the link function to be three times differentiable; ordinary ReLU networks are therefore excluded, and the paper's own simulations show the estimator becoming unstable for ReLU.

Editorial extensions

If this is right

  • Training and test error can be evaluated to leading order as means over low-dimensional Gaussian vectors involving deterministic residual maps, so no simulation of the full network is needed to predict either quantity.
  • The generalization gap is generically non-vanishing in the proportional regime and shrinks as the sample-to-dimension ratio becomes large.
  • A consistent estimator of test error can be computed at every gradient-descent iteration without algorithmic convergence and without knowing the link function or the underlying signal, enabling principled early stopping and hyperparameter tuning.
  • The learned model retains the structure of a single-index function, with an effective signal that combines the true signal and the initialization and a Gaussian component that vanishes as the sample size grows relative to dimension.
  • The distributional law applies to non-Gaussian feature designs satisfying sub-gaussian tail conditions, not only to Gaussian data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the machinery, though demonstrated for single-index data and full-batch gradient descent, is structurally modular; the same reduction should extend to multi-index signals and to stochastic gradient descent with proportionally large mini-batches, as the paper notes are omitted formal extensions.
  • Editorial inference: the algorithmic estimator should be checked on activations that are only once or twice differentiable; the paper's simulations with a smoothed ReLU already support the conjecture that Lipschitz first derivatives may suffice, while ReLU with a hard kink requires genuinely new correction terms rather than simply setting the second derivative to zero.
  • Editorial inference: the single-index representation of the learned model gives a concrete falsifiable prediction: after training on single-index data, the network's output should depend on a test input only through its projection onto the span of the true signal and the initialization rows, up to the stated Gaussian error.
  • Editorial inference: the state-evolution error bound worsens with iteration count t, so practical guarantees are best for early stopping; whether the accumulated error can be controlled uniformly over all t is a natural next question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper develops a non-asymptotic state-evolution theory for the iterates of vanilla gradient descent on bounded-width, bounded-depth multilayer networks in the proportional regime m/n ≍ 1, under single-index data with sub-Gaussian features and smooth activations. The main result (Theorem 3.2) states that, in an averaged empirical-row sense, the first-layer weights n^{1/2}W_1^{(t)} behave like Gaussian vectors whose law is defined recursively by a state-evolution procedure, while deeper-layer weights concentrate around deterministic matrices; the error is (KΛκ*)^{c_t} n^{-1/c_t}. The framework yields: characterizations of training and test errors as low-dimensional Gaussian integrals (Theorem 4.2); a link- and signal-agnostic online estimator of the test error (Algorithm 1, proved consistent in Theorem 4.3); and a structural result showing the learned function remains effectively single-index with an effective signal combining the true signal and the initialization (Theorem 5.2). The proof (Sections 7-13) reduces GD to an auxiliary GFOM whose state evolution follows from matrix-variate GFOM theorems imported from the preprint [Han24] and stated in Appendix B. Simulations validate the estimator and, unusually, document its breakdown for wide networks and ReLU activations (Section 6.4).

Significance. If correct, this is a substantial contribution relative to the NTK, mean-field, and tensor-program lines: it provides the first precise distributional characterization of GD in the finite-width proportional regime, allows weight movement away from initialization, covers arbitrary depth at bounded width, and yields a test-error estimator requiring no knowledge of the link function or signal. The manuscript's strengths include complete-looking proofs for the main theorems with explicit non-asymptotic error rates, an estimator whose construction is genuinely agnostic to φ* and μ* (Section 4.3), a structural single-index representation of the learned model (Theorem 5.2), and unusually honest empirical reporting: Section 6.4 documents the instability for ReLU and the deviations for wide networks, and Remarks 4(2) and 6 state the regularity conjectures explicitly.

major comments (3)
  1. [Appendix B (Theorems B.6-B.7); Sections 7.2-7.3; Section 1.5] The central proof chain rests on Theorems B.6 and B.7, the matrix-variate GFOM state-evolution results stated in Appendix B and imported 'in its full strength' from the preprint [Han24] (Section 1.5). Proposition 7.1 is exactly an application of these theorems to the auxiliary GFOM (7.7), and Proposition 7.2 then transfers the resulting characterization to the GD iterates (7.3); Theorem 3.2 and its consequences (Theorems 4.2, 4.3, 5.2) therefore inherit any gap in B.6-B.7. Since B.6-B.7 are stated without proofs and [Han24] is an unreviewed preprint, the referee cannot verify from the manuscript that the hypotheses of these theorems hold for the application at hand. Please either include complete proofs of Theorems B.2, B.3, B.6, B.7 and Propositions B.4, B.8, or state explicitly which peer-reviewed results cover the asymmetric entrywise case needed here and verify their hypotheses line by line.
  2. [Section 9.2, Eq. (9.1), Lemma 9.3] Even granting Theorems B.6-B.7, the manuscript does not verify that the truncated GFOM (9.2) satisfies hypothesis (D*2). The update functions in (9.2) contain ˇG^{(2)}_t ≡ −φ^{-1}η^{(t−1)}_1·χ_{b_t}(S(·,·,·)) with the truncation (9.1). Value truncation of S does not make the derivatives of the composition bounded: by Proposition 8.3-(7), ∂_{u_kℓ}S grows polynomially in ∥u∥ and ∥φ_*(w)∥, so ∂(χ_{b_t}∘S) = χ'_{b_t}(S)·∂S is unbounded as a function on R^{q[0:t]}, and the same holds a fortiori for the higher derivatives. Concretely, for L = 2, q = 1, φ* ≡ 0, ξ = 0, and σ_1(x) = x + 2 sin x (which satisfies (A4) with a sufficiently large Λ), S(u) = σ(u)σ'(u) vanishes at points where σ'(u) = 0 while S'(u) grows linearly, so ∂(χ_b∘S) is unbounded. Hence the global C^3 bound required in (D*2) (and similarly (D*2)') is not established, and Lemma 9.3's appeal to the meta GFOM theory is not justified as written. The gap is repairable, for instance by also truncating the arguments u, w, or by proving a version of the GFOM theorem under polynomial-growth derivative bounds whose log n factors would be absorbed by the (KΛκ* log n)^{ct} pre-factors already present in Lemmas 9.3-9.5, but as it stands the step from 'pseudo-Lipschitz plus delocalization' (Section 7.3.1) to 'application of B.6' is missing. This is the single most load-bearing juncture of the paper and needs to be closed.
  3. [Assumption (A4), Section 3.1; Remarks 4(2) and 6; Section 6.4] Assumption (A4), which requires activations in C^4 with bounded derivatives up to order 4, is load-bearing in three structurally different places: the derivative bounds for the theoretical gradient maps that define the Onsager corrections (Propositions 8.2-8.3), the Lindeberg-principle step replacing test inputs by Gaussians (Lemma 11.1 and Lemma 11.2 in Section 11.1, which need third derivatives of the network map), and Algorithm 1 itself, whose step (3) computes σ''_1. The assumption excludes ReLU; the paper's own simulation (Figure 4, left) shows that setting σ'' ≡ 0 makes the estimator unstable and drifting, and the minimal-regularity conjecture in Remarks 4(2) and 6 (Lipschitz σ') is not proved. The paper is admirably candid about this, but the abstract and introduction advertise 'general multi-layer neural networks' consistent with 'practical architectures' (Section 1.1, Table 1); the reader should be told at the abstract level that the main results require smooth activations and that the algorithmic estimator needs user-supplied second derivatives.
minor comments (6)
  1. [Theorem 3.4; Section 10.3] The statement of Theorem 3.4 lists only (A3)-(A5) and ϕ^{-1} ≤ K, but the proof (Step 5 of Section 10.3) invokes Theorem 3.2, which requires the full Assumption A; please state the full hypotheses and the intended limiting order (first m ∧ n → ∞ at fixed ϕ, then ϕ → ∞).
  2. [Section 4.1, after Theorem 4.2] The assertion that the generalization gap satisfies E(0) Gap^{(t)}(X, Y) = Θ(1) is made without proof; since both terms in the gap are expectations over state-evolution variables that depend on t in a coupled way, either supply a one-line justification or explicitly label the claim as a heuristic.
  3. [Remark 4(1); Section 6] Because the dependence of c_t on t, q, L is not tracked, the rates (KΛκ*)^{ct} n^{-1/ct} are not quantitative at the values of n used in the simulations; the numerical validation in Section 6 is consequently the only quantitative evidence at practical sample sizes, and the paper should say so where the rates are advertised.
  4. [Section 1.2] There are two obvious typos: 'converegence' and 'arises ubiquitously is this regime'.
  5. [Section 6.2, bullet (3)] The explanation 'the apparently smaller update observed when L = 5 is likely due to instability in the fifth-layer updates' is unclear; if the intended meaning is that the fifth-layer relative update is small because ∥W_5^{(0)}∥ is large and the update is noisy, please say so explicitly.
  6. [Algorithm 1, Steps (1)-(3)] The claim that the estimator requires only 'a small number of closed-form computations' (Section 1.3) should be quantified: the forward-derivative tensors ∂_ℓ Ĥ^{(t-1)}_α and ∂^{(b)}_ℓ P̂^{(t-1)}_α add an O(q) factor per layer over backpropagation, which is material for larger widths.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central reduction uses an external general-purpose GFOM theorem, and the estimator consistency is proven rather than assumed; the same-author preprint dependency is a rigor risk, not a circular step.

full rationale

I walked the derivation chain from the reformulation of gradient descent in Section 7 through the auxiliary GFOM iterates in (7.7), the state evolution of Proposition 7.1, the GD-vs-auxiliary error control in Proposition 7.2, and the applications in Theorems 4.2, 4.3, and 5.2. No step reduces by construction to its own input. The state evolution in Definition 3.1 is a recursive GFOM-type construction, and the proof shows that the auxiliary GFOM iterates satisfy it via the matrix-variate GFOM state evolution Theorems B.6-B.7. Those theorems are quoted from the first author's prior preprint [Han24] and are stated in Appendix B without proof, but they are general-purpose results on GFOM iterates with explicit hypotheses (D*1), (D*2) on the update functions; they do not assume the gradient descent conclusion. The novel part of the paper--the iterative reduction scheme, the truncation and smoothing in (9.1), and the error estimates in Lemmas 9.3-9.5 and Proposition 7.2--is genuine mathematical work connecting GD to the GFOM framework. The estimator in Algorithm 1 is not a fitted parameter relabeled as a prediction: it is constructed from data without knowledge of the signal or link function, and its consistency is proven in Proposition 11.4 and Theorem 4.3. The structural statement in Theorem 5.2 is derived from the distributional law of the first-layer weights, not assumed. The main caveat is that the central theorem's proof depends on an unreviewed same-author preprint, and if Theorems B.6-B.7 contain an unstated hypothesis or an error, the chain would fail; this is a correctness and verifiability concern, not a circularity concern under the stated criteria.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

No data-fitted free parameters: the state evolution matrices {τ, ρ, Σ, Ω, δ, D, V} are deterministic functions of the model primitives, computed by Gaussian expectations; the truncation threshold b_t (Eq. 9.1) and the error constants c_t are proof devices, and the paper explicitly leaves c_t untracked (Remark 4.1). The axioms are the modeling frame (single-index data, sub-gaussian design, proportional regime, smooth activations, O(1) rates, data-independent initialization) plus the imported GFOM machinery. No invented entities: the effective signal U_{*,eff}^{(t)} and the Onsager matrices are derived quantities, not postulates.

assumptions (7)
  • standard math Matrix-variate GFOM state evolution theorems (Appendix B), imported from [Han24]
    Used as black boxes in the proofs of Propositions 7.1 and 9.2: Theorems B.6, B.7 give entrywise and averaged state evolution for iterates u(t)=AF_t(v)+G_t(u); Propositions B.4, B.8 give delocalization. Their assumptions (Lipschitz/C^3 row-separate maps, sub-gaussian design) do not include the GD-training result, so this is independent support rather than circular input.
  • domain assumption Assumption (A2): X has independent, mean-zero, unit-variance sub-gaussian entries
    Underlies the GFOM universality and delocalization used throughout; excludes heavy-tailed and dependent features, which the theory does not cover.
  • domain assumption Single-index regression model (1.2): Y_i = φ*(<X_i, μ*>) + ξ_i with φ* ∈ C^3 and ||μ*|| = O(1)
    The state evolution depends on μ* only through its norm (Remark 3.2); the multi-index extension is conjectured, not proven.
  • ad hoc to paper Assumption (A4): activations σ_α ∈ C^4 with bounded derivatives up to order 4
    Load-bearing for the derivative-based Onsager corrections (Propositions 8.2-8.3), for the Lindeberg step in the test-error proof (Lemmas 11.1-11.2), and for Algorithm 1 which uses σ''_1. Excludes ReLU; the authors only conjecture minimal regularity (Lipschitz σ', Remark 6).
  • domain assumption Assumption (A1): m/n ∈ [1/K, K] (proportional regime) with bounded width q and depth L
    Defines the limit regime; results are non-asymptotic in n with constants depending on t, q, L and error n^{-1/c_t}.
  • domain assumption Initialization W(0) independent of X with κ* = 1 + ||n^{1/2}μ*||_∞ + ||ξ||_∞ + ||n^{1/2}W_1(0)||_∞ + max_α ||W_α(0)||_op slowly growing (Eq. 3.1)
    The theorems allow κ* ≲ n^ε; Gaussian initialization satisfies this. Data-dependent initialization structures are not covered.
  • domain assumption Assumption (A3): learning rates η_α^(s) ≤ Λ = O(1)
    O(1) rates keep the non-lazy finite-width evolution; different rate scalings lead to NTK/MF infinite-width limits (Remark 2.3). No convergence of GD is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Precise gradient descent training dynamics for finite-width multi-layer neural networks." pith.science (2026). https://pith.science/paper/ECMV5OHN

@misc{pith2026250504898,
  author       = {Pith},
  title        = {Pith review of: Precise gradient descent training dynamics for finite-width multi-layer neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ECMV5OHN}},
  note         = {Machine review of arXiv:2505.04898}
}
read the original abstract

In this paper, we provide the first precise distributional characterization of gradient descent iterates for general multi-layer neural networks under the canonical single-index regression model, in the `finite-width proportional regime' where the sample size and feature dimension grow proportionally while the network width and depth remain bounded. Our non-asymptotic state evolution theory captures Gaussian fluctuations in first-layer weights and concentration in deeper-layer weights, and remains valid for non-Gaussian features. Our theory differs from existing neural tangent kernel (NTK), mean-field (MF) theories and tensor program (TP) in several key aspects. First, our theory operates in the finite-width regime whereas these existing theories are fundamentally infinite-width. Second, our theory allows weights to evolve from individual initializations beyond the lazy training regime, whereas NTK and MF are either frozen at or only weakly sensitive to initialization, and TP relies on special initialization schemes. Third, our theory characterizes both training and generalization errors for general multi-layer neural networks beyond the uniform convergence regime, whereas existing theories study generalization almost exclusively in two-layer settings. As a statistical application, we show that vanilla gradient descent can be augmented to yield consistent estimates of the generalization error at each iteration, which can be used to guide early stopping and hyperparameter tuning. As a further theoretical implication, we show that despite model misspecification, the model learned by gradient descent retains the structure of a single-index function with an effective signal determined by a linear combination of the true signal and the initialization.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Approximate Message Passing with Random Initialization for Phase Retrieval

    math.ST 2026-08 conditional novelty 7.0 of 10

    Randomly initialized Bayes-optimal AMP provably achieves the weak-recovery threshold δ=1/2 and arbitrarily accurate recovery for δ>1.13 in proportional-regime noiseless phase retrieval.

  2. The Geometry of Statistical Feature Learning in Mean-Field Langevin Dynamics

    math.ST 2026-06 unverdicted novelty 7.0 of 10

    The stationary distribution of entropy-regularized spherical mean-field Langevin dynamics provably concentrates on hidden indices in Gaussian index models, yielding d/N and Md/N estimation rates up to log factors.

Reference graph

Works this paper leans on

21 extracted references · 4 canonical work pages · cited by 2 Pith papers

  1. [2]

    A leave-one-out approach to approxi- mate message passing

    [BHX23] Zhigang Bao, Qiyang Han, and Xiaocong Xu. A leave-one-out approach to approxi- mate message passing. arXiv preprint arXiv:2312.05911,

    arXiv

  2. [3]

    The LASSO risk for Gaussian matrices

    [BM12] Mohsen Bayati and Andrea Montanari. The LASSO risk for Gaussian matrices. IEEE Trans. Inform. Theory, 58(4):1997–2017,

    work page 1997

  3. [9]

    A random matrix theory perspective on the spectrum of learned features and asymp- totic generalization capabilities

    [DPC+24] Yatin Dandi, Luca Pesce, Hugo Cui, Florent Krzakala, Yue M Lu, and Bruno Loureiro. A random matrix theory perspective on the spectrum of learned features and asymp- totic generalization capabilities. arXiv preprint arXiv:2410.18938,

    arXiv

  4. [10]

    Entrywise dynamics and universality of general first order methods

    [Han24] Qiyang Han. Entrywise dynamics and universality of general first order methods. arXiv preprint arXiv:2406.19061,

    arXiv

  5. [11]

    The distribution of ridgeless least squares interpola- tors

    [HX23] Qiyang Han and Xiaocong Xu. The distribution of ridgeless least squares interpola- tors. arXiv preprint arXiv:2307.02044,

    arXiv

  6. [12]

    Gradient descent inference in empirical risk minimiza- tion

    [HX24] Qiyang Han and Xiaocong Xu. Gradient descent inference in empirical risk minimiza- tion. arXiv preprint arXiv:2412.09498,

    arXiv

  7. [13]

    A theory of non-linear feature learning with one gradient step in two-layer neural networks.arXiv preprint arXiv:2310.07891,

    [MLHD23] Behrad Moniri, Donghwan Lee, Hamed Hassani, and Edgar Dobriban. A theory of non-linear feature learning with one gradient step in two-layer neural networks.arXiv preprint arXiv:2310.07891,

    arXiv

  8. [14]

    Dynamical decoupling of generalization and overfitting in large two-layer networks

    [MU25] Andrea Montanari and Pierfrancesco Urbani. Dynamical decoupling of generalization and overfitting in large two-layer networks. arXiv preprint arXiv:2502.21269,

    arXiv

Show all 21 references
  1. [15]

    [RVE22] G. M. Rotsko ff and E. Vanden-Eijnden. Trainability and accuracy of artificial neu- ral networks: an interacting particle system approach. Comm. Pure Appl. Math. , 75(9):1889–1935,

    1935

  2. [18]

    Tensor programs II: Neural tangent kernel for any architecture

    [Yan20a] Greg Yang. Tensor programs II: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14548,

    2006 arXiv

  3. [19]

    Tensor programs III: Neural matrix laws

    [Yan20b] Greg Yang. Tensor programs III: Neural matrix laws. arXiv preprint arXiv:2009.10685,

    2009 arXiv

  4. [20]

    Feature learning in infinite-width neural networks

    [YH20] Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522,

    2011 arXiv

  5. [21]

    Wide residual networks

    [ZK16] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146,

    arXiv

  6. [2012]

    Language models are few-shot learners

    [BMR+20] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901,

    1901

  7. [2013]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

    [SMG13] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120,

    arXiv

  8. [2018]

    The high-dimensional asymptotics of first order methods with random data

    [CCM21] Michael Celentano, Chen Cheng, and Andrea Montanari. The high-dimensional asymptotics of first order methods with random data. arXiv preprint arXiv:2112.07572,

    arXiv

  9. [2019]

    An image is worth 16x16 words: Transformers for image recogni- tion at scale

    [DBK+20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, and Sylvain Gelly. An image is worth 16x16 words: Transformers for image recogni- tion at scale. arXiv prepri...

    2010 arXiv

  10. [2020]

    Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation

    [Yan19] Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760,

    1902 arXiv

  11. [2021]

    HAN AND M

    82 Q. HAN AND M. IMAIZUMI [Cha06] Sourav Chatterjee. A generalization of the Lindeberg principle. Ann. Probab. , 34(6):2061–2076,

  12. [2023]

    Repetita iuvant: Data repetition allows sgd to learn high-dimensional multi-index functions

    [ADK+24] Luca Arnaboldi, Yatin Dandi, Florent Krzakala, Luca Pesce, and Ludovic Stephan. Repetita iuvant: Data repetition allows sgd to learn high-dimensional multi-index functions. arXiv preprint arXiv:2405.15459,

    arXiv

  13. [2024]

    Bellec and Kai Tan

    [BT24] Pierre C. Bellec and Kai Tan. Uncertainty quantification for iterative algorithms in linear models with application to early stopping. arXiv preprint arXiv:2404.17856 ,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.