Pith. sign in

REVIEW 4 major objections 4 minor 26 references

Random Matrix Theory for Stochastic Gradient Descent

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In the stationary limit, SGD-driven weight eigenvalues obey random matrix theory, with fluctuation width set by learning rate over batch size.

desk verdict Clean RBM evidence and a genuinely new two-species Coulomb gas result, but the general linear scaling rule rests on an unproven noise-isotropy assumption. read the letter →

arxiv 2412.20496 v1 pith:7UF2ZZ4G submitted 2024-12-29 hep-lat cond-mat.dis-nncs.LG

Chanju Park , Matteo Favoni , Biagio Lucini , Gert Aarts This is my paper · ORCID
classification hep-latcond-mat.dis-nncs.LG
keywords stochasticgradientdescentrandommatrixtheoryDysonBrownianmotionCoulombgaslinearscalingruleWignersemicircleRestrictedBoltzmannMachineteacher-studentnetwork
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the eigenvalues of weight matrices trained by stochastic gradient descent behave as a Coulomb gas in the stationary limit, so their statistics follow random matrix theory predictions such as the Wigner semicircle. The central quantitative claim is the variance formula $\sigma_i^2 = \frac{\alpha}{|B|} \frac{V_B[\Delta p]_{ii}}{2\Omega_i}$, which separates the optimizer-controlled factor $\alpha/|B|$ from model-dependent gradient fluctuations. This derives the linear scaling rule—keeping the ratio of learning rate to batch size fixed keeps the fluctuation level of learning unchanged—from first-principles matrix dynamics. The claim is tested in a Gaussian Restricted Boltzmann Machine, where the fit to the Wigner semicircle and the Binder cumulant agree, and in a linear one-hidden-layer teacher-student network, where the spectral density becomes a generalized Wigner semicircle. If correct, the paper provides a physics origin for a widely used empirical training rule and a way to separate universal optimizer effects from architecture-specific ones.

What carries the argument

The central object is the eigenvalue Langevin equation obtained from Dyson Brownian motion for SGD, Eq. (5): $$x_i' = x_i + \$\alpha$ \tilde K_i + \frac{\$alpha^{2}$}{|B|} \sum_{j\neq i} \frac{\tilde $g_i^{2}$}{x_i-x_j} + \frac{\$\alpha$}{\sqrt{|B|}} \sqrt{2\tilde $g_i^{2}$}\,\eta_i.$$ The Vandermonde determinant from the change of variables to eigenvalues produces the pairwise $1/(x_i-x_j)$ repulsion, which is what converts independent noise into Wigner statistics. The associated Fokker-Planck equation has a stationary Coulomb gas solution whose potential, expanded quadratically around each target eigenvalue, yields the variance formula Eq. (9).

What would settle it

Train a Gaussian RBM on a known target spectrum twice with the same $\alpha/|B|$ but very different absolute values, for instance $\alpha=0.01$, $|B|=10$ and $\alpha=0.001$, $|B|=1$, and measure the width of an eigenvalue peak after convergence. If the two widths disagree beyond statistical error, or if the Binder cumulant of a doubly degenerate peak departs from $-4/27$, the universal scaling and Wigner spectral shape claimed by the paper are ruled out.

Watch

Extended reading notes

Core claim

The paper establishes that in the stationary limit the eigenvalue distribution of weight matrices under SGD is governed by Dyson Brownian motion: each eigenvalue $x_i$ performs a drift-plus-noise motion with an additional Coulomb repulsion from every other eigenvalue, $1/(x_i-x_j)$. Solving the stationary Fokker-Planck equation gives a Coulomb gas distribution in which each eigenvalue fluctuates around its target value with a width $\sigma_i^2 = \frac{\alpha}{|B|} \frac{V_B[\Delta p]_{ii}}{2\Omega_i}$. The ratio $\alpha/|B|$ enters solely through the stochasticity of the optimizer, while all model dependence sits in the ratio of gradient variance to potential curvature. The resulting spectral density around a learned eigenvalue is a Wigner semicircle, and nearest-neighbour spacings follow the Wigner surmise, as verified in the Gaussian RBM; with an extra linear layer the Coulomb gas acquires two species with different variances and the spectral density becomes a generalized Wigner semicircle while the level-spacing law survives.

Load-bearing premise

The whole derivation hinges on the assumption that discrete SGD can be represented by a continuous-time Langevin and Fokker-Planck process whose stationary solution is the Coulomb gas; if finite-learning-rate corrections change that stationary distribution, the predicted Wigner semicircle and $\sqrt{\alpha/|B|}$ width scaling would not hold exactly for actual discrete updates.

Editorial extensions

If this is right

  • Scaling the learning rate and batch size together by the same factor leaves the stationary eigenvalue fluctuation width unchanged, giving a first-principles derivation of the practical linear scaling rule.
  • The spectral density around each learned eigenvalue is a Wigner semicircle, not a Gaussian, so the Binder cumulant $-4/27$ can be used as a model-independent signature of the Coulomb gas regime.
  • Adding hidden layers changes the effective potential curvature per eigenvalue, turning the Coulomb gas into a multi-species system; the level-spacing Wigner surmise survives, but the spectral density generalizes and develops wider tails.
  • Fluctuation control separates cleanly into hyperparameters ($\alpha/|B|$) and model properties ($V_B[\Delta p]_{ii}/(2\Omega_i)$), so each can be tuned or measured independently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the stationary Coulomb gas description extends to finite learning rates in deeper networks, then the same ratio $\alpha/|B|$ should govern the fluctuation level of layer-wise singular vectors and gradient statistics, not just eigenvalues.
  • A testable extension beyond the paper is to measure the eigenvalue distribution of a nonlinear teacher-student network with known target spectrum; the deviation of the fitted width from Eq. (9) would quantify where the continuous-time Langevin approximation breaks down.
  • The two-component Coulomb gas result suggests that in heterogeneous architectures, eigenvalue fluctuations should be characterized by a variance profile rather than a single scale, which could be used to detect layer-dependent training instabilities before they appear in the loss.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. Park et al. present a lattice-field-theory proceedings contribution that applies Dyson Brownian motion and Coulomb-gas random matrix theory to the eigenvalue dynamics of weight matrices trained by stochastic gradient descent. From a central-limit representation of minibatch gradients (Eq. 4), they quote an eigenvalue Langevin equation (Eq. 5) with a Coulomb repulsion term, solve the associated Fokker-Planck equation for a stationary Coulomb gas (Eqs. 6-7), and obtain the variance formula sigma_i^2 = (alpha/|B|) V_B[Delta p]_{ii}/(2 Omega_i) (Eq. 9), which expresses the linear scaling rule that alpha/|B| fixes the fluctuation level. They test the resulting spectral density and Binder cumulant in a Gaussian RBM and introduce a linear one-hidden-layer teacher-student network in which an added noise term yields a two-species Coulomb-gas spectral density (Eq. 32).

Significance. If Eq. 9 is valid, the paper offers a physically clear separation of an optimiser-universal factor alpha/|B| from a model-dependent factor V_B/Omega, and it would provide a first-principles route to a widely used empirical scaling rule. The Gaussian-RBM evidence is the strongest part: the fitted spectral shape, the Binder cumulant near -0.147, and the independent variation of alpha and |B| in Fig. 3 support the predicted scaling form, and the data/code release in Ref. [26] makes the check reproducible. The value of the manuscript as submitted is nevertheless limited because the key reduction from SGD noise to Dyson Brownian motion is assumed rather than demonstrated, and the new linear-network test does not validate that reduction because it injects isotropic noise by hand (Eq. 28).

major comments (4)
  1. [§3, Eq. (5)] The derivation of the eigenvalue process is the load-bearing step, but Eq. (5) is quoted from Ref. [2] without proving that the DBM structure applies to SGD noise. For X = W^T W, the covariance of the noise in the eigenbasis is the image of the gradient-noise covariance under the singular-vector frame, and it is not isotropic for a generic loss. The DBM stationary measure (7) requires the special relation 2 tilde g_i^2 = V_B[Delta p]_{ii} = 2 V_B[Delta p]_{i neq j}: the noise variance must satisfy a precise relation between diagonal and off-diagonal entries in every instantaneous eigenbasis. This condition is not derived, and no simulation reported here measures it directly. Without it, additional noise-induced drift appears and Eq. (9) is not the stationary width. Please either supply the argument that SGD gradient covariance automatically satisfies this isotropy (which may be true for the Gaussian RBM class) or state explicitly that the derivation applies only to that restricted class.
  2. [§4.2, Eqs. (16), (28)] The linear teacher-student test is not evidence for the SGD-to-DBM reduction. Eq. (16) replaces the batch correlation by delta_{ij}, explicitly discarding the mini-batch stochasticity that is the source of SGD noise, and Eq. (28) then inserts an independent Gaussian eta with variance 0.01 because, in the authors' words, 'the algorithm by itself is not noisy enough.' Under this artificial noise prescription, the two-species spectral density (32) is expected by construction; the agreement in Fig. 5 therefore verifies the calculation of a two-species DBM fit, not the claim that SGD on a linear network produces that distribution. The abstract's claim that the paper derives the linear scaling rule for SGD remains unsupported by this section.
  3. [§2, transition to continuous time] The stationary calculation in Eqs. (6)-(7) is performed with a continuous-time Fokker-Planck equation, but the algorithm is the discrete map (1). The paper itself cites Refs. [17-19] for the failure of the naive alpha -> 0 limit to give a correct Ito stochastic differential equation. Since Eq. (5) is a discrete update, one needs a controlled stochastic-modified-equation limit to justify that the continuous-time drift and diffusion are those used in Eq. (6). Unless finite-learning-rate corrections are shown to be irrelevant for the stationary measure, the predicted Wigner semicircle and the alpha/|B| scaling are not guaranteed for discrete SGD. Please address this point explicitly.
  4. [§3, Eqs. (8) and (9)] There is an inconsistency in the potential expansion. Eq. (8) defines tilde V_i = tilde V_i(x_s) + (1/(2 Omega_i))(x_i - x_s)^2 and calls Omega_i the curvature; with that convention, substituting Eq. (8) into the exponent of Eq. (7) gives sigma_i^2 = (alpha/|B|) Omega_i tilde g_i^2, not the tilde g_i^2/Omega_i form written in Eq. (9). If Eq. (9) is the intended result, Eq. (8) should read (Omega_i/2)(x_i - x_s)^2. Please correct the factor and define Omega_i unambiguously in terms of V''.
minor comments (4)
  1. [Fig. 3 (right)] The axis labels of the right panel are garbled in the arXiv text ('sqrt(alpha) |B| kappa^2 Omega'); please redraw the panel with clearly separated symbolic axis labels.
  2. [§4.1] Please state explicitly how many independent RBM runs and how many samples per run are used for the histograms and for the Binder cumulant, and report the measured U4 value with a statistical uncertainty rather than only showing the value in the figure.
  3. [Introduction and Ref. [2]] Because the paper summarizes Ref. [2] and adds a new section, it would be helpful to state explicitly which results are new to this contribution and which are taken from Ref. [2].
  4. [§4.2] The linear-network simulations are restricted to 2 x 2 matrices; while acceptable as a proof of concept, the claim of a 'generalised Wigner semi-circle' would be considerably strengthened by showing at least one larger-N example.

Circularity Check

2 steps flagged · score 5.0 of 10

The alpha/|B| scaling is inherited from the assumed CLT noise amplitude, and the SGD-to-DBM mapping (isotropic eigenbasis noise) is imported from the authors' own prior work rather than derived or tested; the RBM check also fits sigma as a parameter.

  1. ansatz smuggled in via citation [Section 3, Eq. (5)]
    "A key result from Dyson Brownian motion [8, 9] is that the equation satisfied by the eigenvalues of X can be written down directly in terms of the drift and fluctuations of X, as well as a Coulomb term, namely [2] ... where ˜K_i = −E_B[Δp]_{ii} and 2 ˜g^2_i = V_B[Δp]_{ii} = 2V_B[Δp]_{i≠j}."

    Equation (5) is the load-bearing step that turns SGD into Dyson Brownian motion, and it is not derived in this paper: it is cited to the authors' own prior work [2]. The specific condition V_B[Δp]_{ii} = 2 V_B[Δp]_{i≠j} is the isotropy of the gradient noise in the instantaneous eigenbasis; without it the stationary measure is not the Coulomb gas (7) and the variance formula (9) does not follow. Since this condition is asserted rather than demonstrated for generic SGD noise on W, the central prediction inherits an ansatz through a same-author citation rather than from an independent derivation.

  2. fitted input called prediction [Section 4.1, Eq. (9) and Fig. 3]
    "... and σ ∼ sqrt(α/|B|) is given by Eq. (9) and used as a fit parameter. ... (Right) The width of eigenvalue distributions scales with the predicted universal scaling factor sqrt(α/|B|)."

    The model-dependent coefficient V_B[Δp]_{ii}/(2Ω_i) in the predicted width is not evaluated from the model; σ is a free parameter in each Wigner-semicircle fit. The observed proportionality of the fitted σ to sqrt(α/|B|) therefore tests only the functional form that is already contained in the assumed Langevin noise amplitude α/sqrt(|B|) of Eq. (4). The 'universal' part of the prediction is thus the CLT input, while the model-dependent part is fitted, so the verification is weaker than a parameter-free prediction.

full rationale

The paper does real RMT work: it maps the eigenvalue update to a Coulomb gas, produces the Wigner-semicircle density and Wigner surmise for degenerate modes, and tests the alpha/|B| scaling by varying alpha and |B| independently. However, the central scaling rule is not fully independent of the input assumptions. The alpha/|B| factor in Eq. (9) is the square of the CLT noise amplitude already present in Eq. (4), so the linear-scaling prediction is inherited by construction from the central-limit approximation rather than emerging as a new universal law of SGD. More importantly, the step that makes the Coulomb-gas stationary distribution (7) valid - the relation V_B[Δp]_{ii} = 2 V_B[Δp]_{i≠j}, i.e. isotropic noise in the eigenbasis - is imported from the authors' own Ref. [2] and is not derived here or verified for generic weight matrices. The linear-network test in Sec. 4.2 does not close this gap: the authors inject artificial isotropic noise (Eq. (28)) because 'the algorithm by itself is not noisy enough,' so that experiment verifies the Coulomb gas for the injected noise, not the reduction of SGD noise to DBM. These issues make the central claim partially circular and partially assumption-dependent, but the empirical scaling tests and the exact Coulomb-gas calculations give the paper independent content; hence a moderate score of 5 rather than a higher one.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central derivation rests on a small set of modeling assumptions: CLT Gaussianity of batch gradients, existence of a continuous-time Langevin/DBM limit (flagged by the authors as nontrivial), the DBM eigenvalue evolution quoted from the authors' own Ref. [2], a delta-function approximation for input correlations, and a quadratic expansion around a unique minimum. The paper's contribution is concentrated in the scaling prediction; the coefficient of that scaling is model-dependent and fitted.

free parameters (4)
  • sigma (Wigner semicircle width) = about 0.05 for the peak shown in Fig. 2; varies with alpha/|B|
    In Eq. (11) the spectral density is fitted to the histogram with sigma as a free parameter; the scaling of sigma with sqrt(alpha/|B|) is then tested.
  • sigma_1, sigma_2 (two-species variances) = not quoted
    In Eq. (32) the generalized Wigner semicircle is fitted to the linear-network eigenvalue histograms with two independent variances; these are not predicted from the SGD noise statistics.
  • injected noise variance in Eq. (28) = 0.01
    An artificial Gaussian noise term eta ~ N(0, 0.01) is added to the linear-network update because the algorithm by itself is not noisy enough; this hand-chosen variance controls the spectral widths and is not derived from SGD.
  • a_i (mode-dependent learning-rate parameter) = not quoted
    The fits to Eq. (22) in Fig. 4 treat a_i as an adjustable rate; in principle a_i is alpha times the diagonal of U^T Z^T Z U, but the paper does not verify the fitted a_i against this prediction.
assumptions (5)
  • domain assumption Mini-batch gradients are i.i.d. and the CLT applies, so the batch gradient is mean plus Gaussian noise of variance V_B[Delta p] (Eq. 3).
    Fundamental to the Langevin description; breaks for small batch sizes or non-Gaussian gradient distributions, and is not justified for all loss landscapes.
  • domain assumption A continuous-time Langevin and Fokker-Planck limit exists for the discrete SGD update, with diffusion proportional to alpha^2/|B| (Eqs. 4-6).
    The paper itself notes in Sec. 2 that the naive alpha to 0 limit does not give a correct SDE (Refs. [17-19]); the stationary distribution is nonetheless derived in continuous time.
  • domain assumption The eigenvalue dynamics of X = W^T W follow Dyson Brownian motion with the Coulomb term as in Eq. (5), quoted from Ref. [2].
    This is the key RMT input; the derivation is not shown in this paper and is taken from the authors' earlier work.
  • domain assumption In the linear network, the batch-averaged input correlation is delta_ij (Eq. 16), ignoring mini-batch sampling fluctuations.
    The paper says this can be ignored for sufficiently large batches, but then injects artificial noise to compensate, suggesting the assumption is not accurate in the tested regime.
  • standard math The effective potential for each eigenvalue has a unique minimum, allowing a quadratic expansion (Eq. 8) and Gaussian variance (Eq. 9).
    Standard expansion around a minimum; valid near convergence but not during the full transient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Random Matrix Theory for Stochastic Gradient Descent." pith.science (2026). https://pith.science/paper/7UF2ZZ4G

@misc{pith2026241220496,
  author       = {Pith},
  title        = {Pith review of: Random Matrix Theory for Stochastic Gradient Descent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7UF2ZZ4G}},
  note         = {Machine review of arXiv:2412.20496}
}
read the original abstract

Investigating the dynamics of learning in machine learning algorithms is of paramount importance for understanding how and why an approach may be successful. The tools of physics and statistics provide a robust setting for such investigations. Here we apply concepts from random matrix theory to describe stochastic weight matrix dynamics, using the framework of Dyson Brownian motion. We derive the linear scaling rule between the learning rate (step size) and the batch size, and identify universal and non-universal aspects of weight matrix dynamics. We test our findings in the (near-)solvable case of the Gaussian Restricted Boltzmann Machine and in a linear one-hidden-layer neural network.

Figures

Figures reproduced from arXiv: 2412.20496 by the authors.

Figure 1
Figure 1. General structure of a Restricted Boltzmann Machine, with 𝑁𝑣 (𝑁ℎ) visible (hidden) nodes. freedom are Gaussian fields, interacting via a bilinear coupling, 𝜙𝑖𝑊𝑖𝑎ℎ𝑎, one obtains a Gaussian RBM, with a probability distribution 𝑝(𝜙, ℎ) ∼ exp[−𝑆(𝜙, ℎ)] and the “action”, 𝑆(𝜙, ℎ) = 1 2 𝜇 2 𝜙 𝑇 𝜙 + 1 2𝜎 2 ℎ (ℎ − 𝜂) 𝑇 (ℎ − 𝜂) − 𝜙 𝑇𝑊 ℎ. (10) Here 𝜇 2 and 𝜎 2 ℎ are hyperparameters and we put the bias 𝜂 = 0. A full analysis of… view at source ↗
Figure 2
Figure 2. (Left) Target eigenvalues (dashed lines) and model eigenvalues (histograms) after training. The middle 8 target eigenvalues are doubly degenerate due to periodic boundary conditions. (Right) Close-up of one of the peaks: the learnt eigenvalue distribution of the RBM follows the Wigner semi-circle (solid line). 0.000 0.002 0.004 0.006 0.008 α/|B| 0.94 0.96 0.98 1.00 1.02 1.04 λfit/κ |B| α 0.00 0.05 0.10 0.15 0.20 r α… view at source ↗
Figure 3
Figure 3. (Left) Deviation of the fitted centres of the model eigenvalue distributions from the target ones due to the inter-mode Coulomb interaction. The offset decreases as stochasticity in the model decreases. (Right) The width of eigenvalue distributions scales with the predicted universal scaling factor √︁ 𝛼/|B|. In both cases 𝛼 and |B| are varied independently. and find excellent agreement. One may note in [PITH_FULL_I… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training dynamics of the square of the singular values of a 2 × 2 student matrix, given a teacher matrix with doubly degenerate eigenvalues, using 𝑍 as in Eq. (26) (left) and Eq. (27) (right). The presence of 𝑍 affects the rate of convergence. Shown are an ensemble of …
Figure 5
Figure 5. Figure 5: Histogram of the spectral density 𝜌(𝑥) in the presence of a hidden layer, with 𝑍 as in Eq. (26) (left) and Eq. (27) (right). Also shown are fits to the standard Wigner semi-circle (11) (dashed line) and the generalised Wigner semi-circle (32) for two species (solid lin…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 15 canonical work pages

  1. [2]

    Aarts, B

    G. Aarts, B. Lucini and C. Park,Stochastic weight matrix dynamics during learning and Dyson Brownian motion,Phys. Rev. E(to appear) (2024) [2407.16427]

    arXiv 2024

  2. [26]

    C. Park, G. Aarts and B. Lucini,chanjure/Stochastic_weight_matrix_dynamics_during_ learning_and_Dyson_Brownian_motion-data_release: v1.0.1, Aug., 2024. 10.5281/zenodo.13310439. 13

    work page doi:10.5281/zenodo.13310439 2024

  3. [1]

    Carleo, I

    G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby et al.,Machine learning and the physical sciences, Reviews of Modern Physics91(2019) 045002 [1903.10563]

    arXiv 2019

  4. [3]

    Wigner,Characteristic vectors of bordered matrices with infinite dimensions, Annals of Mathematics 62(1955) 548

    E.P. Wigner,Characteristic vectors of bordered matrices with infinite dimensions, Annals of Mathematics 62(1955) 548

    work page 1955

  5. [4]

    Wigner,Conference on Neutron Physics by Time-of-Flight, p

    E.P. Wigner,Conference on Neutron Physics by Time-of-Flight, p. 67, 1956

    work page 1956

  6. [5]

    Dyson,Statistical theory of the energy levels of complex systems

    F.J. Dyson,Statistical theory of the energy levels of complex systems. I, J. Math. Phys.3 (1962) 140

    work page 1962

  7. [6]

    Dyson,Statistical theory of the energy levels of complex systems

    F.J. Dyson,Statistical theory of the energy levels of complex systems. II,J. of Math. Phys.3 (1962) 157. 11 Random Matrix Theory for Stochastic Gradient Descent Chanju Park and Matteo Favoni

    work page 1962

  8. [7]

    Dyson,Statistical Theory of the Energy Levels of Complex Systems

    F.J. Dyson,Statistical Theory of the Energy Levels of Complex Systems. III,J. Math. Phys.3 (1962) 166

    1962

Show all 26 references
  1. [8]

    Dyson,A Brownian-Motion Model for the Eigenvalues of a Random Matrix, J

    F.J. Dyson,A Brownian-Motion Model for the Eigenvalues of a Random Matrix, J. Math. Phys. 3 (1962) 1191

    1962

  2. [9]

    Mehta,Random Matrices, Academic Press, New York, 3rd ed

    M.L. Mehta,Random Matrices, Academic Press, New York, 3rd ed. (2004)

    2004

  3. [10]

    Martin and M.W

    C.H. Martin and M.W. Mahoney,Traditional and Heavy-Tailed Self Regularization in Neural Network Models, 1901.08276

    1901 arXiv

  4. [11]

    Baskerville, D

    N.P. Baskerville, D. Granziol and J.P. Keating,Applicability of Random Matrix Theory in Deep Learning, 2102.06740

    arXiv

  5. [12]

    Goyal, P

    P. Goyal, P. Dollár, R.B. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola et al.,Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, 1706.02677

    arXiv

  6. [13]

    Smith and Q.V

    S.L. Smith and Q.V. Le,A Bayesian Perspective on Generalization and Stochastic Gradient Descent, 1710.06451

    arXiv

  7. [14]

    Smith, P

    S.L. Smith, P. Kindermans and Q.V. Le,Don’t Decay the Learning Rate, Increase the Batch Size, 1711.00489

    arXiv

  8. [15]

    Smith, D

    S.L. Smith, D. Duckworth, Q.V. Le and J. Sohl-Dickstein,Stochastic natural gradient descent draws posterior samples in function space, 1806.09597

    arXiv

  9. [16]

    Aarts, O

    G. Aarts, O. Hajizadeh, B. Lucini and C. Park,Dyson Brownian motion and random matrix dynamics of weight matrices during learning, in38th conference on Neural Information Processing Systems, 2024 [2411.13512]

    2024 arXiv

  10. [17]

    Mandt, M.D

    S. Mandt, M.D. Hoffman and D.M. Blei,Continuous-time limit of stochastic gradient descent revisited, in8th NIPS Workshop on Optimization for Machine Learning, 2015

    2015

  11. [18]

    Q. Li, C. Tai and W. E,Stochastic modified equations and adaptive stochastic gradient algorithms, inProceedings of the 34th International Conference on Machine Learning, vol. 70, pp. 2101–2110, 2017 [1511.06251]

    2017 arXiv

  12. [19]

    Yaida,Fluctuation-dissipation relations for stochastic gradient descent, inInternational Conference on Learning Representations, 2019 [1810.00004]

    S. Yaida,Fluctuation-dissipation relations for stochastic gradient descent, inInternational Conference on Learning Representations, 2019 [1810.00004]

    2019 arXiv

  13. [20]

    Smolensky,Chapter 6: Information processing in dynamical systems: Foundations of harmony theory, inParallel Distributed Processing: Volume 1, D

    P. Smolensky,Chapter 6: Information processing in dynamical systems: Foundations of harmony theory, inParallel Distributed Processing: Volume 1, D. Rumelhart and J. McLelland, eds., pp. 194–281, MIT Press, 1986

    1986

  14. [21]

    Hinton,Training Products of Experts by Minimizing Contrastive Divergence,Neural Computation 14(2002) 1771

    G.E. Hinton,Training Products of Experts by Minimizing Contrastive Divergence,Neural Computation 14(2002) 1771

    2002

  15. [22]

    Decelle and C

    A. Decelle and C. Furtlehner,Restricted Boltzmann machine: Recent advances and mean-field theory,Chinese Physics B30 (2021) 040202 [2011.11307]. 12 Random Matrix Theory for Stochastic Gradient Descent Chanju Park and Matteo Favoni

    2021 arXiv

  16. [23]

    Aarts, B

    G. Aarts, B. Lucini and C. Park,Scalar field restricted Boltzmann machine as an ultraviolet regulator, Phys. Rev. D109 (2024) 034521 [2309.15002]

    2024 arXiv

  17. [24]

    Bahri, J

    Y. Bahri, J. Kadmon, J. Pennington, S.S. Schoenholz, J. Sohl-Dickstein and S. Ganguli, Statistical Mechanics of Deep Learning, Annual Review of Condensed Matter Physics11 (2020) 501

    2020

  18. [25]

    Goldt, M.S

    S. Goldt, M.S. Advani, A.M. Saxe, F. Krzakala and L. Zdeborová,Dynamics of stochastic gradient descent for two-layer neural networks in the teacher–student setup,Journal of Statistical Mechanics: Theory and Experiment2020(2020) 124010

    2020

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.