Pith. sign in

REVIEW 3 major objections 3 minor 46 references

Exact Dynamics of Multi-class Stochastic Gradient Descent

T0 review · 3 major / 3 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read One-pass SGD on multi-class Gaussian mixtures is exactly tracked by an ODE in the high-dimensional limit, and anisotropy triggers a clean-direction alignment phase transition.

desk verdict Solid framework, conditional results: the ODE theorem is a real advance, but the phase-transition claims rest on an unproved uniform bound. read the letter →

arxiv 2510.14074 v2 pith:NPBCEGLP submitted 2025-10-15 stat.ML cs.LGmath.OCmath.PR

Elizabeth Collins-Woodfin , Inbar Seroussi This is my paper · ORCID
classification stat.MLcs.LGmath.OCmath.PR
keywords stochasticgradientdescenthigh-dimensionalasymptoticsGaussianmixtureanisotropiccovariancelearningcurvesphasetransitionlogisticregressionmean-fieldODE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that one-pass stochastic gradient descent on data drawn from a mixture of anisotropic Gaussian classes has a deterministic, high-dimensional limit: the risk, the overlap with class means, and a broad class of statistics converge to curves given by a system of ODEs, with polynomial-in-dimension error that holds with overwhelming probability. The authors extend this exact dynamics to multiple classes, including a number of classes growing logarithmically with dimension, and to covariances that are not positive definite. Applying the ODEs to binary logistic regression, they identify a structural phase transition: when the covariance spectrum is extreme (zero eigenvalues or a steep power law), SGD aligns with low-variance clean directions and the loss decays to zero polynomially; for isotropic or mild power-law data the loss saturates at a positive constant. A separate least-squares analysis yields a learning-rate threshold governing convergence. If the results are right, anisotropic data geometry becomes a quantitative predictor of sample-wise scaling laws.

What carries the argument

The load-bearing object is the product resolvent R_{ℓ*}(z) = ∏_{i=1}^{ℓ*}(K_i − z_i I)^{-1}, constructed from the commuting class covariances, together with the ODEs for the per-eigenspace norm Vρ and mean overlap mρ,j. Because the covariances share eigenvectors, each direction contributes independently, and contour integrals of the resolvent package all polynomial statistics into a finite-dimensional autonomous system. The same resolvent also yields an equivalent homogenized SGD process, which concentrates to the same deterministic dynamics.

What would settle it

Simulate one-pass SGD for the zero-one or power-law binary logistic model with a fixed learning rate and measure a(t) = E[w12]/(E[w12] − E[w12²]) over a long horizon (e.g., t up to 10^4). If a(t) grows without bound, or cannot be uniformly bounded by C_w(γ) near the stated thresholds, Proposition 2 and Proposition 4's polynomial risk-decay claims fail. Alternatively, run the same experiment at two growing dimensions and check whether the ODE prediction error decays as d^{−ε}; failure would indicate the deterministic limit does not hold in the claimed regime.

Watch

Extended reading notes

Core claim

The central claim, Theorem 1, is that under commuting class covariances and mild smoothness of the losses, the stochastic iterate Xk is well approximated by deterministic quantities: per-eigenspace norms Vρ(t) and mean overlaps mρ,j(t) solve a closed system of ODEs, and any statistic φ with a composite polynomial structure satisfies sup_{0≤t≤T} |φ(X_{⌊td⌋}) − ϕ(t)| ≤ C d^{−ε} with overwhelming probability. The deterministic risk is L(t) = Σ_i p_i E[f_i(θ_{t,i})], where θ_{t,i} is a Gaussian variable built from the deterministic overlaps. This makes entire learning curves computable without simulation for anisotropic, multi-class Gaussian mixtures.

Load-bearing premise

The argument's load-bearing premise is Assumption 12, an unproved uniform bound on the ratio a(t) = E[w12]/(E[w12] − E[w12²]); the authors justify it numerically and heuristically, and the logarithmic growth of the zero-variance overlap and the polynomial risk decay collapse if this bound fails.

Editorial extensions

If this is right

  • Learning curves for high-dimensional logistic regression become exactly computable for anisotropic Gaussian mixtures; the paper's ODE predictions match SGD simulations at d=1000.
  • For zero-one and extreme power-law spectra, the risk decays to zero as a polynomial t^{−c}, and the normalized iterate aligns with the zero-variance subspace up to O((log t)^{-1}) relative error.
  • For identity and mild power-law spectra, the risk is bounded below by a positive constant, and SGD does not achieve perfect classification.
  • The structural phase transition is governed by the power-law exponents: when β < 2α and α ≥ β+1, the overlap grows like log t and the loss falls polynomially; otherwise the loss saturates.
  • For multi-class least squares, SGD converges exactly when the learning rate satisfies γ_t < 1 / max_i [ (Tr(K_i)+∥μ_i∥²)/d ].

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if Assumption 12 is eventually proved, the same ODE machinery would yield rigorous learning-rate tuning rules based on the covariance spectrum, including the threshold that separates saturation from clean-direction alignment.
  • Editorial inference: because Gaussian-mixture equivalence is known to hold for broader data distributions, the phase transition predicted for logistic regression plausibly extends to other losses and to neural-network final layers; the paper suspects this, and the ODE structure makes the conjecture directly testable by simulation.
  • Editorial inference: the theorem currently depends heavily on commuting covariances (a common eigenbasis). A natural extension would test whether the same ODE approximation survives for non-commuting anisotropic covariances or for mini-batch SGD; the paper's resolvent proof does not directly apply there.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This paper develops a rigorous high-dimensional limit theory for one-pass SGD on multi-class Gaussian mixture data with possibly anisotropic, commuting class covariances and class means. The main result, Theorem 1, states that under smoothness and growth assumptions, any sufficiently regular statistic of the SGD iterates concentrates, with overwhelming probability, on a deterministic function of a finite-dimensional ODE system; this is also extended to a number of classes growing logarithmically with the dimension. The proof follows the resolvent-based approach of prior work [15], adapted to multiple covariance matrices and nonzero means, and includes martingale bounds, a net argument, a stability analysis, and a non-explosiveness result. The paper then specializes to binary logistic regression under identity, zero-one, and power-law covariance models, claiming a structural phase transition: in the zero-one and extreme power-law regimes the loss decays polynomially to zero and the iterates align with 'clean directions,' whereas in the identity and mild power-law regimes the loss remains bounded away from zero. A separate section analyzes linear least squares with a growing number of classes and gives a learning-rate threshold.

Significance. If the main theorem and the example analyses were fully rigorous, this would be a substantial contribution: it extends the existing exact-dynamics theory for isotropic single-class Gaussian data to multi-class anisotropic mixtures, including a growing number of classes, and it provides concrete, falsifiable asymptotic predictions for logistic regression under anisotropic covariances. The proof of Theorem 1 is detailed and builds carefully on [15], with explicit error bounds and stability statements; the numerical experiments are extensive and support the ODE predictions. However, the headline phase-transition propositions are currently conditional on an unproved technical assumption, and one of them requires an even stronger unsupported condition. These gaps are load-bearing for the paper's advertised asymptotic results, even though the central concentration theorem itself may be sound.

major comments (3)
  1. [§3.1/§3.2; Remark 2; §7.3–§7.5] Propositions 2, 3, and 4 all invoke Assumption 12, which is explicitly not proved: Remark 2 states it is 'justified numerically and heuristically' and that the authors believe it holds generically. The assumption is used in Lemma 20, Eqs. (239)–(241), to turn the integral representation for m_ρ(t) into the bounds (240)–(241), and Lemma 16 uses it to obtain m_00(t)=log t+O(1) and W_1(t)≍t^{-1}. Without a uniform bound on a(t), the claimed log t growth of m_00 and the t^{-c} loss asymptotics are not established. Since these asymptotics are headline contributions, the paper must either prove the bound or explicitly present the propositions as conditional on it.
  2. [§3.2, Proposition 4; §7.4 around Eq. (214)] The extreme-power-law conclusion requires more than Assumption 12 with some finite C_w(γ); Proposition 4(2) additionally requires C_w ≤ 2−ε. This stronger condition is used around Eq. (214) to get B(t) ≤ (2−ε)m(t) + subleading terms, which is exactly what forces B(t)≍m(t) and the t^{-c} risk decay. The paper does not prove this uniform-in-t condition for the claimed parameter range; Figure 7 only shows a(t) below 2 for selected parameter values on a finite simulation horizon. If C_w ≥ 2, the argument that B(t) is dominated by 2m(t) fails and the vanishing-risk conclusion is unsupported.
  3. [§5.3.1, after Eq. (60)] The proof of Proposition 5 ends by saying to set ζ=1−2δ for δ∈(0,1/2). With this choice, the first error term in Eq. (60) is d^{-1/2+ζ}=d^{1/2−2δ}, which is not d^{-δ} (and is larger than d^{-δ} for small δ). The correct choice appears to be ζ=1/2−δ. As written, the final estimate of Proposition 5 does not follow, although this is likely a fixable typo. The issue should be corrected before the proof of Theorem 1 can be considered complete.
minor comments (3)
  1. [§7.4, Assumption 13 and Lemma 19] The text after Assumption 13 says Lemma 19 gives κ2 = 1/α + 2, but Lemma 19 states κ = 1/α + 1 and defines K with λρ rather than λρ^2 as in K2. Please align the notation and exponent so that the kernel used in the proofs is the one stated in the assumption.
  2. [§3.2, Proposition 3(1)] The conclusion m(t)→c μ^T[K]^{-1}μ with c∈[1,C_w] depends on the unspecified constant C_w from Assumption 12. If C_w is not explicitly controlled, the claimed range for the limit is not quantitatively informative; this should be clarified once Assumption 12 is addressed.
  3. [§5.5, Lemmas 9–11] The lemmas are stated under conditions such as ℓ+ ≤ C d^ω with 0≤ω<2/(5+α), etc. It would help readers if the proof of Proposition 5 explicitly tracked which regime of ω is used for the final d^{-δ} rate, since the text only says ℓ⋆=O(log d).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the main ODE theorem is proved from the SGD update via Doob decomposition and martingale bounds; the example asymptotics are conditional on an explicitly unproved technical assumption, which is a rigor gap, not a circular reduction.

full rationale

The paper's central claim does not reduce by construction to its inputs. Theorem 1 asserts that the SGD statistics concentrate on the deterministic ODE system (8), and the proof derives this: it starts from the Taylor/Doob decomposition of the SGD update (Eqs. 13 and 18), introduces resolvent statistics (Section 5.2), proves their equivalence with the spectral ODEs (Lemma 2), and then establishes in Proposition 5 that Z(W_{td}, z) is an approximate solution, with error controlled by the martingale and Hessian-error bounds of Lemmas 9–11. The deterministic ODEs are not fitted to the SGD trajectories; they are obtained from the conditional expectations of the exact updates. The self-citations to [15] supply supporting lemmas and the single-class foundation, but the multi-class, anisotropic, growing-class statement is proved in this paper, so the citations are genuine evidence rather than a circular premise. The example-specific results (Propositions 2–4) are conditional on Assumption 12, which the authors explicitly flag in Remark 2 as 'justified numerically and heuristically' and believed to hold generically; this is an unproved domain/technical hypothesis, not a restatement of the predicted loss or overlap asymptotics. Proposition 4 also explicitly conditions on C_w ≤ 2−ε. If Assumption 12 or the C_w bound fails, the proofs of those propositions do not close, but that is a correctness gap, not circularity. The apparent scaling slip after Eq. (60) (setting ζ = 1−2δ) is likewise a proof error rather than a definitional identity. No fitted parameter is renamed as a prediction, and no uniqueness/ansatz is imported from the authors' prior work to force the conclusion.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The main theorem relies on standard Gaussian-tail and concentration tools plus explicit domain assumptions: commuting covariances and, for growing ℓ*, logarithmic class count with orthogonal means. The illustrative logistic-regression phase transition additionally depends on Assumption 12, which is unproved and only numerically/heuristically supported. No new physical entities are introduced; the GMM-HSGD process is a mathematical construct, not an independent entity with falsifiable predictions.

free parameters (3)
  • learning rate γ = chosen by hand in simulations, e.g., γ=0.9 or 0.001–1 in Figure 1
    The learning rate is an input hyperparameter, not fitted to data. The paper derives thresholds as functions of γ.
  • power-law exponents α, β = e.g., α=1.2, β=0,1.2 in Figures 4–6
    These define the data model in Assumption 10; they are chosen data-generating parameters, not fitted. They determine the mild vs extreme regimes.
  • C_w(γ) in Assumption 12 = unknown; assumed finite or ≤2−ε
    An assumed uniform bound on a(t) that is not proven or estimated; it is a free constant introduced to allow the example proofs to go through.
assumptions (5)
  • domain assumption All class covariance matrices commute: [K_i, K_j]=0 (Assumption 2).
    Enables common eigenbasis and the spectral decomposition into V_ρ and m_{ρ,j}; without it, Eqs. (6)–(8) and the product-resolvent proof do not apply.
  • domain assumption When the class number grows, ℓ* = O(log d), means are orthogonal, and f_i are Lipschitz (Assumption 1B).
    This is required for the growing-class stability and union-bound arguments; it limits the growing-class result to at most logarithmically many classes and orthogonal means.
  • ad hoc to paper Assumption 12: a(t)=E[w12]/(E[w12]−E[w12^2]) < C_w(γ) uniformly in t.
    The paper states 'we justify this assumption numerically and heuristically and we believe that it holds generically ... although we do not prove this analytically' (Remark 2). Propositions 2–4 depend on it for the phase-transition conclusions.
  • standard math Assumption 5 / 11: risk derivatives are (pseudo-)Lipschitz in the Gaussianized arguments.
    Standard regularity needed for unique ODE solutions, stability, and martingale concentration. This is a technical condition, not a physical input.
  • domain assumption Assumption 13: power-law kernels satisfy F_μ(x)≍x^{-κ_μ} and K_2(x)≍x^{-κ_2}.
    Used in the power-law proofs; Lemma 19 shows it follows from Assumption 10, so it is a repackaging of the power-law model rather than a separate physical assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exact Dynamics of Multi-class Stochastic Gradient Descent." pith.science (2026). https://pith.science/paper/NPBCEGLP

@misc{pith2026251014074,
  author       = {Pith},
  title        = {Pith review of: Exact Dynamics of Multi-class Stochastic Gradient Descent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NPBCEGLP}},
  note         = {Machine review of arXiv:2510.14074}
}
read the original abstract

We develop a framework for analyzing the learning dynamics of high-dimensional problems trained using one-pass stochastic gradient descent (SGD) with data from multiple anisotropic classes. Our main theorem provides exact expressions for quantities of interest, including the risk and the overlap with the true signal, in terms of a deterministic system of ODEs, valid in the high-dimensional limit. The theorem holds for a broad class of optimization problems and extends to settings where the number of classes grows with dimension. To illustrate its utility, we investigate in detail the effect of the data's anisotropic structure on the problems of binary logistic regression and least-squares (LS) loss. We study the LS in a linear multiclass setup and derive a learning-rate threshold that depends on the average eigenvalue of the covariance matrices. In the binary logistic regression, we study three cases: isotropic covariances, data covariance matrices with a large fraction of zero eigenvalues (denoted as the zero-one model), and covariance matrices with power-law spectra. We show that a structural phase transition occurs. In particular, for the zero-one model and the power-law model with sufficiently large power, SGD aligns more closely with values of the class mean that are projected onto the ``clean directions'' (i.e., directions of smaller variance). This is supported by analytical studies and numerical simulations, which show the exact asymptotic behavior of the loss in the high-dimensional limit. The effects of data anisotropy that we demonstrate are likely to hold beyond these examples and illustrate one application of the broader theorem that we prove.

Figures

Figures reproduced from arXiv: 2510.14074 by the authors.

Figure 1
Figure 1. Population risk concentration on a binary logistic regression problem for different values of learning rate γ, and for (a) identity covariance with standard normal mean. (b) zero-one model with mean non-zero only in the subspace for which both covariance has zero eigenvalues (c)/(d) power-law diagonal covariance matrices i.e., λ (1) ρ = ( ρ d ) 1.1 and λ (2) ρ = ( ρ d ) 1.5 and mean µρ = √ 1 d ( ρ d ) β for two diff… view at source ↗
Figure 2
Figure 2. Identity and zero-one model with d = 1000 and γ = 0.9. We display L (t), m(t), V (t) and the ratio m(t)/ √ V , i.e. the normalized version of m(t). On each graph, we plot both the SGD simulation and the theoretical prediction (ODE). denote the projection of m(t) into the eigenspace associated with Iij by m(ij)(t) = X ρ∈Iij mρ(t) (11) so we have m(00)(t) + m(01)(t) + m(10)(t) + m(11)(t) = m(t). Likewise, we define µ(… view at source ↗
Figure 3
Figure 3. Zero-one model with d = 1000 and γ = 0.9 showing subspace alignment [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Power-law model with d = 1000, with a1 = a2 = 1.2 and different β 0 25 50 75 100 125 150 175 200 SGD iterations/d 0.0 0.2 0.4 0.6 0.8 m(t)/ V(t) Stream SGD m(t)/ V(t) ODE (a) Alignment, β = 0 0 25 50 75 100 125 150 175 200 SGD iterations/d 0.0 0.1 0.2 0.3 0.4 m(t)/ V(t…
Figure 5
Figure 5. Figure 5: Power-law model with d = 1000, with a1 = a2 = 1.2 and different β 0 25 50 75 100 125 150 175 200 SGD iterations/d 0.0 0.1 0.2 0.3 0.4 0.5 0.6 m_U(t)/ V(t) Stream SGD m_U(t)/ V(t) ODE m_L(t)/ V(t) Stream SGD m_L(t)/ V(t) ODE (a) Subspace alignment, β = 0 0 25 50 75 100 …
Figure 6
Figure 6. Figure 6: Power-law model with d = 1000, with a1 = a2 = 1.2 and different β. Here we project m(t) into subspaces associated with the lower half of the eigenvalues and the upper half of the eigenvalues and we refer to these projections as mL(t) and mU (t) respectively. 12 [PITH_…
Figure 7
Figure 7. Figure 7: Comparison of E[w12]/(E[w12] − E[w 2 12]) between Streaming SGD and ODE for various data models with γ = 1. The pictures for other values of γ exhibit similar long-term behavior, although the rates of convergence are different. Remark 9. We note that a(t) = W1(t) W1(t)…
Figure 8
Figure 8. Figure 8: Comparison of simulation results (loss and [PITH_FULL_IMAGE:figures/full_fig_p048_8.png]
Figure 9
Figure 9. Figure 9: Extreme power-law (α = 1.3) and random means 0 25 50 75 100 125 150 175 200 SGD iterations/d 10 3 10 2 10 1 lo g( (t)) SGD, = 0.2 ODE, = 0.2 (a) ℓ ⋆ = 5 0 25 50 75 100 125 150 175 200 SGD iterations/d 10 3 10 2 10 1 lo g( (t)) SGD, = 0.2 ODE, = 0.2 (b) ℓ ⋆ = 10 0 25 50…
Figure 10
Figure 10. Figure 10: Extreme power-law (α = 1.3) and random means 0 25 50 75 100 125 150 175 200 SGD iterations/d 10 4 6 × 10 5 2 × 10 4 1 ij pim2 ij D SGD 1 ij pim2 ij D ODE 1 d ij pim2 ij D SGD 1 d ij pim2 ij D ODE (a) ℓ ⋆ = 5 0 25 50 75 100 125 150 175 200 SGD iterations/d 10 4 5 × 10 …
Figure 11
Figure 11. Figure 11: Mild power-law (α = 0.5) and random means Acknowledgments and Disclosure of Funding The work of Elizabeth Collins-Woodfin was supported by Fonds de recherche du Québec – Nature et technologies (FRQNT) postdoctoral training scholarship (DOI https://doi.org/10.69777/344…
Figure 12
Figure 12. Figure 12: Mild power-law (α = 0.5) and random means by the Israel Science Foundation grant no. 777/25, the NSF-BSF grant no. 0603624011, and by the Alon fellowship. We thank Courtney Paquette and Elliot Paquette for fruitful discussions. References [1] Luca Arnaboldi, Florent K…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 2 linked inside Pith

  1. [15]

    Hitting the high- dimensional notes: An ode for sgd learning dynamics on glms and multi-index models.Information and Inference: A Journal of the IMA, 13(4):iaae028, 2024

    Elizabeth Collins-Woodfin, Courtney Paquette, Elliot Paquette, and Inbar Seroussi. Hitting the high- dimensional notes: An ode for sgd learning dynamics on glms and multi-index models.Information and Inference: A Journal of the IMA, 13(4):iaae028, 2024

    2024

  2. [1]

    Escaping mediocrity: how two-layer networks learn hard single-index models with SGD.CoRR, 2023

    Luca Arnaboldi, Florent Krzakala, Bruno Loureiro, and Ludovic Stephan. Escaping mediocrity: how two-layer networks learn hard single-index models with SGD.CoRR, 2023

    2023

  3. [2]

    From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks

    Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks. In The Thirty Sixth Annual Conference on Learning Theory, pages 1199–1227. PMLR, 2023

    2023

  4. [3]

    Minimax theory for high-dimensional gaussian mixtures with sparse mean separation.Advances in Neural Information Processing Systems, 26, 2013

    Martin Azizyan, Aarti Singh, and Larry Wasserman. Minimax theory for high-dimensional gaussian mixtures with sparse mean separation.Advances in Neural Information Processing Systems, 26, 2013

    2013

  5. [4]

    High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance.arXiv preprint arXiv:2304.00707, 2023

    Krishnakumar Balasubramanian, Promit Ghosal, and Ye He. High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance.arXiv preprint arXiv:2304.00707, 2023

    arXiv 2023

  6. [5]

    Spectrally-normalized margin bounds for neural networks.Advances in neural information processing systems, 30, 2017

    Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks.Advances in neural information processing systems, 30, 2017

    2017

  7. [6]

    Local geometry of high-dimensional mixture models: Effective spectral theory and dynamical transitions.arXiv preprint arXiv:2502.15655, 2025

    Gerard Ben Arous, Reza Gheissari, Jiaoyang Huang, and Aukosh Jagannath. Local geometry of high-dimensional mixture models: Effective spectral theory and dynamical transitions.arXiv preprint arXiv:2502.15655, 2025

    arXiv 2025

  8. [7]

    Online stochastic gradient descent on non- convex losses from high-dimensional inference.The Journal of Machine Learning Research, 22(1):4788– 4838, 2021

    Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non- convex losses from high-dimensional inference.The Journal of Machine Learning Research, 22(1):4788– 4838, 2021

    2021

Show all 46 references
  1. [8]

    High-dimensional limit theorems for SGD: Effective dynamics and critical scaling

    Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. High-dimensional limit theorems for SGD: Effective dynamics and critical scaling. Advances in Neural Information Processing Systems, 35:25349–25362, 2022

    2022

  2. [9]

    On-line learning with a perceptron.Europhysics Letters, 28(7):525, 1994

    Michael Biehl and Peter Riegler. On-line learning with a perceptron.Europhysics Letters, 28(7):525, 1994

    1994

  3. [10]

    Learning by on-line gradient descent

    Michael Biehl and Holm Schwarze. Learning by on-line gradient descent. Journal of Physics A: Mathematical and general, 28(3):643, 1995

    1995

  4. [11]

    Learning curves for sgd on structured features

    Blake Bordelon and Cengiz Pehlevan. Learning curves for sgd on structured features. InInternational Conference on Learning Representations, 2022

    2022

  5. [12]

    The high-dimensional asymptotics of first order methods with random data.arXiv preprint arXiv:2112.07572, 2021

    Michael Celentano, Chen Cheng, and Andrea Montanari. The high-dimensional asymptotics of first order methods with random data.arXiv preprint arXiv:2112.07572, 2021. 55

    2021 arXiv

  6. [13]

    Sharp global convergence guarantees for iterative nonconvex optimization with random data.Ann

    Kabir Aladin Chandrasekher, Ashwin Pananjady, and Christos Thrampoulidis. Sharp global convergence guarantees for iterative nonconvex optimization with random data.Ann. Statist., 51(1):179–210, 2023

    2023

  7. [14]

    Achieving optimal clustering in gaussian mixture models with anisotropic covariance structures

    Xin Chen and Anderson Ye Zhang. Achieving optimal clustering in gaussian mixture models with anisotropic covariance structures. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  8. [16]

    High-dimensional limit of one-pass SGD on least squares

    Elizabeth Collins-Woodfin and Elliot Paquette. High-dimensional limit of one-pass SGD on least squares. Electronic Communications in Probability, 29:1–15, 2024

    2024

  9. [17]

    Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models.Advances in Neural Information Processing Systems, 36:752–784, 2023

    Alex Damian, Eshaan Nichani, Rong Ge, and Jason D Lee. Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models.Advances in Neural Information Processing Systems, 36:752–784, 2023

    2023

  10. [18]

    Universality laws for gaussian mixtures in generalized linear models.Advances in Neural Information Processing Systems, 36, 2024

    Yatin Dandi, Ludovic Stephan, Florent Krzakala, Bruno Loureiro, and Lenka Zdeborová. Universality laws for gaussian mixtures in generalized linear models.Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [19]

    The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents

    Yatin Dandi, Emanuele Troiani, Luca Arnaboldi, Luca Pesce, Lenka Zdeborova, and Florent Krzakala. The benefits of reusing batches for gradient descent in two-layer networks: Breaking the curse of information and leap exponents. InInternational Conference on Machine Learning, p...

    2024

  12. [20]

    High-dimensional asymptotics of prediction: Ridge regression and classification

    Edgar Dobriban and Stefan Wager. High-dimensional asymptotics of prediction: Ridge regression and classification. The Annals of Statistics, 46(1):247–279, 2018

    2018

  13. [21]

    Rigorous dynamical mean-field theory for stochastic gradient descent methods

    Cedric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zdeborova. Rigorous dynamical mean-field theory for stochastic gradient descent methods. SIAM Journal on Mathematics of Data Science, 6(2):400–427, 2024

    2024

  14. [22]

    Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup.Advances in neural information processing systems, 32, 2019

    Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup.Advances in neural information processing systems, 32, 2019

    2019

  15. [23]

    The gaussian equivalence of generative models for learning with shallow neural networks

    Sebastian Goldt, Bruno Loureiro, Galen Reeves, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. The gaussian equivalence of generative models for learning with shallow neural networks. InMathematical and Scientific Machine Learning, pages 426–471. PMLR, 2022

    2022

  16. [24]

    Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020

    Sebastian Goldt, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020

    2020

  17. [25]

    Minimax-optimal covariance projected spectral clustering for high- dimensional nonspherical mixtures.arXiv preprint arXiv:2502.02580, 2025

    Chengzhu Huang and Yuqi Gu. Minimax-optimal covariance projected spectral clustering for high- dimensional nonspherical mixtures.arXiv preprint arXiv:2502.02580, 2025

    2025

  18. [26]

    Fast margin maximization via dual acceleration

    Ziwei Ji, Nathan Srebro, and Matus Telgarsky. Fast margin maximization via dual acceleration. In International Conference on Machine Learning, pages 4860–4869. PMLR, 2021

    2021

  19. [27]

    Characterizing the implicit bias via a primal-dual analysis

    Ziwei Ji and Matus Telgarsky. Characterizing the implicit bias via a primal-dual analysis. InAlgorithmic Learning Theory, pages 772–804. PMLR, 2021

    2021

  20. [28]

    Trajectory of mini-batch momentum: batch size saturation and convergence in high dimensions.Advances in Neural Information Processing Systems, 35:36944–36957, 2022

    Kiwon Lee, Andrew Cheng, Elliot Paquette, and Courtney Paquette. Trajectory of mini-batch momentum: batch size saturation and convergence in high dimensions.Advances in Neural Information Processing Systems, 35:36944–36957, 2022. 56

    2022

  21. [29]

    Phase transitions and optimal algorithms in high-dimensional gaussian mixture clustering

    Thibault Lesieur, Caterina De Bacco, Jess Banks, Florent Krzakala, Cris Moore, and Lenka Zdeborová. Phase transitions and optimal algorithms in high-dimensional gaussian mixture clustering. In2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allert...

    2016

  22. [30]

    Optimality of spectral clustering in the gaussian mixture model.The Annals of Statistics, 49(5):2506–2530, 2021

    Matthias Löffler, Anderson Y Zhang, and Harrison H Zhou. Optimality of spectral clustering in the gaussian mixture model.The Annals of Statistics, 49(5):2506–2530, 2021

    2021

  23. [31]

    Learning curves of generic features maps for realistic datasets with a teacher-student model

    Bruno Loureiro, Cedric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Marc Mezard, and Lenka Zdeborová. Learning curves of generic features maps for realistic datasets with a teacher-student model. Advances in Neural Information Processing Systems, 34:18137–18151, 2021

    2021

  24. [32]

    Learning gaussian mixtures with generalized linear models: Precise asymptotics in high- dimensions

    Bruno Loureiro, Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning gaussian mixtures with generalized linear models: Precise asymptotics in high- dimensions. Advances in Neural Information Processing Systems, 34:10144–10157, 2021

    2021

  25. [33]

    High dimensional classification via regularized and unregularized empirical risk minimization: Precise error and optimal loss.stat, 1050:25, 2020

    Xiaoyi Mai and Zhenyu Liao. High dimensional classification via regularized and unregularized empirical risk minimization: Precise error and optimal loss.stat, 1050:25, 2020

    2020

  26. [34]

    Dynamical mean- field theory for stochastic gradient descent in gaussian mixture classification

    Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical mean- field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems, 33:9540–9550, 2020

    2020

  27. [35]

    Convergence of gradient descent on separable data

    Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. InThe 22nd International Conference on Artificial Intelligence and Statistics, pages 3420–3428. PMLR, 2019

    2019

  28. [36]

    The full spectrum of deepnet hessians at scale: Dynamics with SGD training and sample size

    Vardan Papyan. The full spectrum of deepnet hessians at scale: Dynamics with SGD training and sample size. arXiv e-prints, pages arXiv–1811, 2018

    2018

  29. [37]

    Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties.Mathematical Programming, pages 1–90, 2024

    Courtney Paquette, Elliot Paquette, Ben Adlam, and Jeffrey Pennington. Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties.Mathematical Programming, pages 1–90, 2024

    2024

  30. [38]

    Classifying high-dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed

    Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdeborová. Classifying high-dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed. InInternational Conference on Machine Learning, pages 8936–8947. PMLR, 2021

    2021

  31. [39]

    Dynamics of on-line gradient descent learning for multilayer neural networks

    David Saad and Sara Solla. Dynamics of on-line gradient descent learning for multilayer neural networks. Advances in neural information processing systems, 8, 1995

    1995

  32. [40]

    Exact solution for on-line learning in multilayer neural networks.Physical Review Letters, 74(21):4337, 1995

    David Saad and Sara A Solla. Exact solution for on-line learning in multilayer neural networks.Physical Review Letters, 74(21):4337, 1995

    1995

  33. [41]

    Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures

    Mohamed El Amine Seddik, Cosme Louart, Mohamed Tamaazousti, and Romain Couillet. Random matrix theory proves that deep learning representations of gan-data behave as gaussian mixtures. In International Conference on Machine Learning, pages 8573–8582. PMLR, 2020

    2020

  34. [42]

    The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19(70):1–57, 2018

    Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19(70):1–57, 2018

    2018

  35. [43]

    Theoretical insights into multiclass classification: A high-dimensional asymptotic view.Advancesin Neural Information Processing Systems, 33:8907–8920, 2020

    Christos Thrampoulidis, Samet Oymak, and Mahdi Soltanolkotabi. Theoretical insights into multiclass classification: A high-dimensional asymptotic view.Advancesin Neural Information Processing Systems, 33:8907–8920, 2020

    2020

  36. [44]

    High-dimensional probability, volume 47 of Cambridge Series in Statistical and Probabilistic Mathematics

    Roman Vershynin. High-dimensional probability, volume 47 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge, 2018. An introduction with applications in data science, With a foreword by Sara van de Geer. 57

    2018

  37. [45]

    A solvable high-dimensional model of GAN.Advances in Neural Information Processing Systems, 32, 2019

    Chuang Wang, Hong Hu, and Yue Lu. A solvable high-dimensional model of GAN.Advances in Neural Information Processing Systems, 32, 2019

    2019

  38. [46]

    Data-dependence of plateau phenomenon in learning with neural network—statistical mechanical analysis.Advancesin Neural Information Processing Systems, 32, 2019

    Yuki Yoshida and Masato Okada. Data-dependence of plateau phenomenon in learning with neural network—statistical mechanical analysis.Advancesin Neural Information Processing Systems, 32, 2019. 58

    2019

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.