Pith. sign in

REVIEW 2 major objections 6 minor 34 references

Analytic theory of dropout regularization

T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper derives exact ODEs for dropout-trained two-layer networks and shows the optimal dropout rate grows with label noise.

desk verdict A serious extension of the Goldt et al. ODE framework to dropout, with real analytic results; the headline noise-dependence claim in the later phases needs the hidden Mathematica coefficients made visible before I'd fully trust it. read the letter →

arxiv 2505.07792 v2 pith:NUQMBH4D submitted 2025-05-12 stat.ML cond-mat.dis-nncond-mat.stat-mechcs.LG

Francesco Mori , Francesca Mignacco This is my paper · ORCID
classification stat.MLcond-mat.dis-nncond-mat.stat-mechcs.LG MSC 68T0782C32
keywords dropoutregularizationteacher-studentframeworkonlinestochasticgradientdescentgeneralizationerrororderparameterssoftcommitteemachinelabelnoisehigh-dimensionallimit
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dropout regularization is standard practice, but its effect on learning is usually understood only heuristically. This paper tries to change that for a canonical setting: a two-layer student network trained by online stochastic gradient descent on Gaussian inputs, with labels produced by a fixed teacher network plus Gaussian noise. In the high-dimensional limit the paper derives a closed system of ordinary differential equations for the training dynamics, and from those equations it obtains explicit formulas showing that dropout suppresses the harmful correlations between hidden units and that the optimal activation probability $r^*$ decreases as the label-noise variance $\sigma^2$ grows. If these results are right, dropout stops being a free hyperparameter: the best dropout rate is a calculable function of the noise level, the network width, and the stage of training.

What carries the argument

The central object is the set of order parameters $Q_{ik}$, $R_{in}$, $T_{nm}$ and the Gaussian integrals $I_2,I_3,I_4$ that express the averaged SGD updates in terms of pre-activation correlations. Dropout is encoded in the factors $N[r,\{i,j,k,l\}] = r^{\#\text{distinct student indices}}$, so that self-overlaps carry an extra factor of $r$ relative to cross-overlaps; this is the mechanism that decorrelates hidden nodes. The analytic results are obtained by substituting symmetric or specialized ansätze for the order parameters and expanding either in small training time $\alpha$ or small learning rate $\eta$, with the unspecialized phase fixed point given by the transcendental equation for $R_0$ in Eq. (29).

What would settle it

Run the same soft committee ($K=2$, $M=1$, with the same error-function activation used in the paper, $\eta=1$, second layer fixed) with $N=10^5$ Gaussian inputs and labels $\phi^*(\mathbf{x})+\sigma\zeta$. Measure the generalization error at $\alpha=0.1$ for activation rates $r\in\{0.7,0.8,0.9,1\}$ at noise levels $\sigma\in\{0,0.3,1\}$: if the best $r$ does not decrease with $\sigma$, or if the ODE trajectories deviate from the simulations beyond finite-size fluctuations, the central claim is wrong.

Watch

Extended reading notes

Core claim

In the limit where the input dimension $N$ and the number of training steps $\mu$ go to infinity at fixed ratio $\alpha=\mu/N$, the paper claims that the stochastic dynamics of a two-layer soft committee machine under Bernoulli dropout and online SGD are exactly captured by the ODEs for the order parameters $Q_{ik}=\mathbf{w}_i\cdot\mathbf{w}_k/N$, $R_{in}=\mathbf{w}_i\cdot\mathbf{w}_n^*/N$, and readout weights $v_i$, given in Eq. (10). Dropout enters through a simple counting rule: each averaged update carries a factor $r$ for every distinct student index in the overlap, so correlations between different hidden nodes are suppressed by higher powers of the dropout probability. Solving these equations at early times yields the closed-form optimal activation rate in Eq. (24), at the unspecialized plateau it yields the generalization error in Eq. (30) and a linear-in-$\eta$ correction showing $r^*$ decreases with $\sigma^2$, and a stability analysis shows that below a critical activation rate the network never specializes. The consistent message is that dropout protects generalization by making the noisy parts of the hidden-unit weights more independent, at a small cost in signal alignment.

Load-bearing premise

The whole theory assumes that with infinitely many input dimensions the stochastic training run behaves like its averaged limit—the overlaps between student and teacher weights become deterministic and the hidden-unit outputs become Gaussian; the authors adopt this closure from earlier work without testing it here, so all dropout formulas depend on it.

Editorial extensions

If this is right

  • The optimal activation probability $r^*$ falls as label noise $\sigma^2$ grows, in the early-time regime, in the unspecialized plateau, and near full specialization.
  • Dropout improves generalization mainly by reducing the correlation $\tilde{\Delta}$ between the noise components of distinct hidden nodes, even though it slightly reduces each node's overlap with the teacher.
  • For $K=M$ and small learning rate there is a critical activation rate $r_c$; below it the unspecialized plateau is stable and specialization never occurs.
  • At intermediate noise levels the best constant dropout rate depends on training time: strong dropout wins early, weaker dropout wins later once specialization becomes favorable.
  • At short times dropout is unnecessary if the learning rate is tuned optimally; the theory gives the joint optimum at $r^*=1$ and $\eta=M/(2K(\sigma^2+1/3))$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A schedule that starts with strong dropout and gradually raises $r$ should outperform any constant rate, because the early-time, plateau, and long-time optima are ordered differently; the paper hints at this via optimal control but leaves it untested.
  • Applying the same order-parameter counting to DropConnect, where the Bernoulli variables act on individual weights, would likely show a similar noise-dependent optimum with decorrelation acting at a finer scale.
  • The plateau result that $r_0^*\to 11/12$ for wide students suggests a noise-independent architectural benefit of dropout; a testable prediction is that finite-width networks trained with $r\approx 11/12$ in the symmetric phase will beat both $r=1$ and much smaller $r$ even with no label noise.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper analyzes online stochastic gradient descent training of a two-layer student network with dropout in the teacher-student framework. In the high-dimensional limit N -> infinity, the authors derive a closed system of ODEs for the order parameters Q, R, v, extending the no-dropout framework of Ref. [19]. They validate these ODEs against finite-size simulations (Fig. 2) and then use them to study three training regimes. At early times, they obtain an explicit closed-form optimal activation probability r* as a function of model size and label noise (Eq. 24). At intermediate times, in the small-learning-rate limit, they derive the plateau generalization error (Eqs. 13-14) and a first-order correction in eta showing r* ~ r0* + eta(a+b sigma^2). At long times, for K=M, they analyze the specialization transition and obtain a similar perturbative result (Eq. 16). The main qualitative conclusions are that dropout decorrelates hidden nodes and that the optimal activation probability decreases with label noise.

Significance. Assuming the results hold, this is a valuable analytic contribution to the theory of dropout. The early-time formula is explicit, derived from the model rather than fitted, and directly checked against full-ODE optimization (Fig. 3). The paper also validates the ODE description against finite-size simulations (Fig. 2) and the small-eta plateau prediction against numerical integration of the ODEs (Fig. 5), and it provides Mathematica notebooks that reproduce the heavy algebra. The prediction that dropout mitigates label noise and that the optimal dropout probability increases with noise is falsifiable and practically relevant. The main weakness is that the noise-dependence results in the plateau and specialized phases depend on coefficients that are not displayed in the paper and whose signs are only illustrated for a finite set of parameters.

major comments (2)
  1. [Sections 3.3.2 and 3.5.1, Eqs. (15)-(16), (38), (44)] The central claim that the optimal activation probability decreases with label noise in the unspecialized and specialized phases rests on the signs of the coefficients b(K,M) and b(M), since r* is expressed as r0* + eta(a+b sigma^2). The text reports these coefficients only through Figs. 7 and 9 for a finite set of (M,K) values; the explicit expressions are not given in the manuscript and are relegated to the external Mathematica notebook [23]. Moreover, unlike the early-time formula (24), which is validated against full-ODE optimization in Fig. 3, the perturbative predictions (15)-(16) are not compared with numerical minimization of the full ODEs (46)-(48) in the corresponding regimes; Fig. 10 varies r at fixed sigma and Fig. 8 addresses stability rather than optimal r. Please either display the coefficients (e.g., in an appendix) with a proof or systematic numerical evidence for their signs, or add a direct comparison of the perturbative r*(sigma) with the full-ODE optimum for representative (M,K) in both regimes. Without this, the headline result for these phases is not verifiable from the manuscript.
  2. [Appendix A and Section 2.1] The ODEs (46)-(48) are introduced as following Ref. [19], but the derivation is not sketched. In particular, the paper does not state explicitly how the factors N[r,{...}] arise from averaging the dropout masks, nor does it spell out the Gaussian-closure and self-averaging assumptions that justify the deterministic ODE description in the high-dimensional limit. The numerical validation in Fig. 2 supports the ODEs for the tested setting, but for a paper whose results all derive from these equations, the derivation should be made self-contained, at least to the level of a sketch of the mask averaging and a statement of the closure assumptions. This is important for readers to assess the regime of validity beyond the cases simulated.
minor comments (6)
  1. [Section 3.2, Eqs. (21)-(23)] The phrase 'to leading order' after Eq. (21) should specify that the expansion is in the small initial norms or in small training time alpha, so that the order of the approximation is unambiguous.
  2. [Fig. 2 caption] The observable 'R11/√Q11T11' should be written as R_11 / sqrt(Q_11 T_11) to avoid ambiguity in the square-root scope.
  3. [Eq. (45)] The notation N[r,{i,j,k,...,l}] would benefit from a brief note stating that it is the probability that all nodes in the set are active, i.e., r^|S|.
  4. [Section 3.4, Eq. (42)] The matrix A(M,r) in Eq. (42) is not displayed; please include its explicit form in the appendix so that the linear stability analysis is reproducible without the external notebook.
  5. [Abstract and Section 2.2] The abstract says the optimal dropout probability increases with noise while the summary says the optimal activation probability decreases; adding a parenthetical that dropout probability equals 1 - r would prevent confusion.
  6. [Eq. (54)] The asterisk in (1 + c11) * (1 + c33) appears to be a typographical artifact; it should be ordinary multiplication, and the notation should be harmonized with the rest of the appendix.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the optimal-dropout predictions are derived from the model ODEs and checked against simulations, not fitted or imported as self-citations.

full rationale

The paper's central results are derived, not fitted. The ODEs in Eq. (10), written explicitly in Appendix A, extend the independent teacher-student framework of Ref. [19] by adding Bernoulli dropout variables; the derivation is not outsourced to a same-author citation. The early-time optimal rate Eq. (24) is obtained by expanding the generalization error Eq. (11) under the stated symmetric ansatz Eq. (20) and minimizing the resulting small-α expression, and it is then compared with numerical optimization of the full ODEs in Fig. 3. The plateau results Eqs. (13)-(14) and (29)-(30) follow from solving the fixed-point equations and minimizing the closed-form plateau error; the specialization results follow from an explicit linear-stability analysis around that fixed point. The perturbative results Eqs. (15), (38) and (44) are computed by inserting the stated ansatz into the ODEs and expanding in η, with the coefficients a and b supplied in the companion notebook and plotted; the sign of b is an output of those computations, not an input. The only self-citations (Refs. [33,34]) appear in the conclusions as future directions on adaptive dropout schedules and are not load-bearing for any derivation. The main text's omission of the lengthy algebraic forms of a(K,M) and b(K,M) is a transparency/verification gap, but it does not make the derivation circular.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no fitted free parameters: optimal dropout rates are derived from the model. The main axioms are the high-dimensional self-averaging and Gaussian closure from the framework of Ref. [19], the restriction to erf activations, orthogonal teachers, and fixed second-layer weights, and symmetry ansatze for the unspecialized phase. No invented entities are posited.

assumptions (5)
  • domain assumption The high-dimensional limit N, mu -> infinity with alpha = mu/N fixed yields closed ODEs for order parameters via a Gaussian and self-averaging assumption on pre-activations (framework of Ref. [19]).
    Invoked in Appendix A when stating Eqs. (46)-(48); not re-derived in the paper.
  • domain assumption Teacher and student use the same erf activation function g*(x) = g(x) = erf(x/sqrt(2)), and teacher vectors are orthogonal T_nm = delta_nm.
    Stated in Sec. 2.1; the closed-form generalization error Eq. (11) depends on erf. Results do not generalize to other activations without re-derivation.
  • domain assumption The student second-layer weights are fixed to one (eta_v = 0), so the analysis is restricted to soft committee machines.
    Stated in Sec. 2.1: 'we do not train the second layer, by fixing v_i = v*_n = 1 and eta_v = 0.'
  • domain assumption Symmetric and unspecialized ansatz Q_ik = Q delta_ik + C(1 - delta_ik), R_in = R is assumed for the plateau fixed point (Eq. (20) and Eq. (27)).
    Used in Secs. 3.2 and 3.3; verified a posteriori by numerical integration of the ODEs.
  • domain assumption Small-eta expansion is valid, and the leading-order eta correction captures the sigma-squared dependence.
    Used in Secs. 3.3.2 and 3.5.1; the paper notes that at leading order in eta the noise sigma-squared drops out, so the first-order correction is needed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analytic theory of dropout regularization." pith.science (2026). https://pith.science/paper/NUQMBH4D

@misc{pith2026250507792,
  author       = {Pith},
  title        = {Pith review of: Analytic theory of dropout regularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NUQMBH4D}},
  note         = {Machine review of arXiv:2505.07792}
}
read the original abstract

Dropout is a regularization technique widely used in training artificial neural networks to mitigate overfitting. It consists of dynamically deactivating subsets of the network during training to promote more robust representations. Despite its widespread adoption, dropout probabilities are often selected heuristically, and theoretical explanations of its success remain sparse. Here, we analytically study dropout in two-layer neural networks trained with online stochastic gradient descent. In the high-dimensional limit, we derive a set of ordinary differential equations that fully characterize the evolution of the network during training and capture the effects of dropout. We obtain a number of exact results describing the generalization error and the optimal dropout probability at short, intermediate, and long training times. Our analysis shows that dropout reduces detrimental correlations between hidden nodes, mitigates the impact of label noise, and that the optimal dropout probability increases with the level of noise in the data. Our results are validated by extensive numerical simulations.

Figures

Figures reproduced from arXiv: 2505.07792 by the authors.

Figure 1
Figure 1. a) Illustration of the model presented in Section 2.1. b) Sketch of the generalization error dynamics highlighting the three phases of learning. At early times, the weights remain close to their small-norm initialization. During the plateau (unspecialized phase), the hidden units are symmetric, being equally aligned with the teacher units. After specialization, this symmetry is broken. the mechanisms driving the per… view at source ↗
Figure 2
Figure 2. Comparison of learning dynamics with and without dropout. Left: Generalization error vs. training time α = µ/N with (r = 0.68) and without (r = 1) dropout. Center: The observable ∆ = ( ˜ Q12−R11R21)/ √ Q11Q22, which quantifies detrimental correlations between the student’s hidden nodes (see main text for details). Right: Cosine similarity between the student’s first hidden node weight vector w1 and the teacher weigh… view at source ↗
Figure 3
Figure 3. Optimal activation rate r ∗ as a function of the variance σ 2 of the label noise. Parameters: K = 2, M = 1, η = 1, ηv = 0, α = 0.1. The teacher weight vector w ∗ is taken to be of unit norm, the student weights w are initialized to zero. The second layer weights are fixed to one. The continuous line represents the optimal rate obtained by numerical optimization by using the full equations of motion. The crosses corr… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Early-times phase diagram. Above the lines, obtained for different values of the number K of hidden nodes in the student, it is beneficial to use dropout, with the optimal activation probability given in Eq. (24). Below the line, it is optimal to avoid dropout, corresp…
Figure 5
Figure 5. Figure 5: Left. Generalization error at the plateau vs. the student’s width K at fixed activation rate r = 0.8. Right. Generalization error at the plateau vs. the activation rate r at fixed student’s width K = 3. In both panels, the symbols are obtained by numerical integration …
Figure 6
Figure 6. Figure 6: Left: Optimal activation probability r ∗ 0 in the plateau phase as a function of K − M for different values of M. The curves are obtained via numerical minimization of the generalization error in Eq. (30), derived in the small-η limit. For large K, all curves converge …
Figure 7
Figure 7. Figure 7: Coefficients a(K, M) (left) and b(K, M) (right) for the plateau phase as a function of K −M for different values of M. To leading order in η, the generalization error takes the form ϵg ≈ ϵ 0 g (r) + η ϵ1 g (r, σ), (36) where ϵ 0 g (r) is given in Eq. (30), and an expli…
Figure 8
Figure 8. Figure 8: Specialization phase diagram in the (M, r) plane. In the non-specialized phase, the plateau fixed point is linearly stable. As r increases, the fixed point becomes unstable, signaling a transition to the specialized phase where student units align with individual teach…
Figure 9
Figure 9. Figure 9: Coefficients a(M) (left) and b(M) (right) as a function of M for the specialized phase with K = M. 10 2 10 1 10 0 10 1 10 2 10 3 10 4 10 5 10 3 10 2 10 1 Error r=0.7 r=0.8 r=0.9 r=0.96 r=1.0 10 2 10 1 10 0 10 1 10 2 10 3 10 4 10 5 0.014 0.027 0.053 0.103 0.200 Error r=…
Figure 10
Figure 10. Figure 10: Generalization error as a function of time [PITH_FULL_IMAGE:figures/full_fig_p014_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

34 extracted references · 24 canonical work pages

  1. [19]

    Dy- namics of stochastic gradient descent for two-layer neural networks in the teacher-student setup

    Sebastian Goldt, Madhu Advani, Andrew M Saxe, Florent Krzakala, and Lenka Zdeborov´ a. Dy- namics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. Advances in neural information processing systems, 32, 2019

    2019

  2. [23]

    For full details on the calculations see the Mathematica notebooks athttps://github.com/ francescomori/analytic_dropout.git

    work page

  3. [1]

    Improving neural networks by preventing co-adaptation of feature detectors.arXiv preprint arXiv:1207.0580, 2012

    Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors.arXiv preprint arXiv:1207.0580, 2012

    arXiv 2012

  4. [2]

    Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958, 2014

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting.The journal of machine learning research, 15(1):1929–1958, 2014

    1929

  5. [3]

    A review on dropout regularization approaches for deep neural networks within the scholarly domain.Electronics, 12(14):3106, 2023

    Imrus Salehin and Dae-Ki Kang. A review on dropout regularization approaches for deep neural networks within the scholarly domain.Electronics, 12(14):3106, 2023

    work page 2023

  6. [4]

    Regularization of neural networks using dropconnect

    Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. InInternational conference on machine learning, pages 1058–1066. PMLR, 2013

    2013

  7. [5]

    Curriculum dropout

    Pietro Morerio, Jacopo Cavazza, Riccardo Volpi, Ren´ e Vidal, and Vittorio Murino. Curriculum dropout. InProceedings of the IEEE International Conference on Computer Vision, pages 3544– 3552, 2017. 15

    work page 2017

  8. [6]

    Dropout reduces under- fitting

    Zhuang Liu, Zhiqiu Xu, Joseph Jin, Zhiqiang Shen, and Trevor Darrell. Dropout reduces under- fitting. InInternational Conference on Machine Learning, pages 22233–22248. PMLR, 2023

    2023

Show all 34 references
  1. [7]

    Dropout training, data-dependent reg- ularization, and generalization bounds

    Wenlong Mou, Yuchen Zhou, Jun Gao, and Liwei Wang. Dropout training, data-dependent reg- ularization, and generalization bounds. In Jennifer Dy and Andreas Krause, editors,Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine ...

    2018

  2. [8]

    Dropout: Explicit forms and capacity control

    Raman Arora, Peter Bartlett, Poorya Mianjy, and Nathan Srebro. Dropout: Explicit forms and capacity control. InInternational Conference on Machine Learning, pages 351–361. PMLR, 2021

    2021

  3. [9]

    Regularization of neural networks using dropconnect

    Li Wan, Matthew Zeiler, Sixin Zhang, Yann Le Cun, and Rob Fergus. Regularization of neural networks using dropconnect. In Sanjoy Dasgupta and David McAllester, editors,Proceedings of the 30th International Conference on Machine Learning, volume 28 ofProceedings of Machine Lear...

    2013

  4. [10]

    Adaptive dropout with rademacher complexity regularization

    Ke Zhai and Huan Wang. Adaptive dropout with rademacher complexity regularization. In International Conference on Learning Representations, 2018

    2018

  5. [11]

    Understanding dropout

    Pierre Baldi and Peter J Sadowski. Understanding dropout. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors,Advances in Neural Information Processing Sys- tems, volume 26. Curran Associates, Inc., 2013

    2013

  6. [12]

    Dropout training as adaptive regularization

    Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. Advances in neural information processing systems, 26, 2013

    2013

  7. [13]

    On the implicit bias of dropout

    Poorya Mianjy, Raman Arora, and Rene Vidal. On the implicit bias of dropout. InInternational conference on machine learning, pages 3540–3548. PMLR, 2018

    2018

  8. [14]

    Implicit regularization of dropout.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(6):4206–4217, 2024

    Zhongwang Zhang and Zhi-Qin John Xu. Implicit regularization of dropout.IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(6):4206–4217, 2024

    2024

  9. [15]

    Deep information propagation

    Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. InInternational Conference on Learning Representations, 2017

    2017

  10. [16]

    Learning by on-line gradient descent.Journal of Physics A: Mathematical and general, 28(3):643, 1995

    Michael Biehl and Holm Schwarze. Learning by on-line gradient descent.Journal of Physics A: Mathematical and general, 28(3):643, 1995

    1995

  11. [17]

    Exact solution for on-line learning in multilayer neural networks

    David Saad and Sara A Solla. Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21):4337, 1995

    1995

  12. [18]

    On-line learning in soft committee machines.Physical Review E, 52(4):4225, 1995

    David Saad and Sara A Solla. On-line learning in soft committee machines.Physical Review E, 52(4):4225, 1995

    1995

  13. [20]

    Three unfinished works on the optimal storage capacity of networks.Journal of Physics A: Mathematical and General, 22(12):1983, 1989

    Elizabeth Gardner and Bernard Derrida. Three unfinished works on the optimal storage capacity of networks.Journal of Physics A: Mathematical and General, 22(12):1983, 1989

    1983

  14. [21]

    H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples. Phys. Rev. A, 45:6056–6091, Apr 1992

    1992

  15. [22]

    Cambridge University Press, 2001

    Andreas Engel.Statistical mechanics of learning. Cambridge University Press, 2001

    2001

  16. [24]

    Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020

    Sebastian Goldt, Marc M´ ezard, Florent Krzakala, and Lenka Zdeborov´ a. Modeling the influence of data structure on learning in neural networks: The hidden manifold model.Physical Review X, 10(4):041044, 2020. 16

    2020

  17. [25]

    Transient dynamics of on-line learning in two-layered neural networks.Journal of Physics A: Mathematical and General, 29(16):4769, 1996

    Michael Biehl, Peter Riegler, and Christian W¨ ohler. Transient dynamics of on-line learning in two-layered neural networks.Journal of Physics A: Mathematical and General, 29(16):4769, 1996

    1996

  18. [26]

    On-line learning dynamics of relu neural networks using statis- tical physics techniques.arXiv preprint arXiv:1903.07378, 2019

    Michiel Straat and Michael Biehl. On-line learning dynamics of relu neural networks using statis- tical physics techniques.arXiv preprint arXiv:1903.07378, 2019

    1903 arXiv

  19. [27]

    A theory of initialisation’s impact on specialisation

    Devon Jarvis, Sebastian Lee, Cl´ ementine Carla Juliette Domin´ e, Andrew M Saxe, and Ste- fano Sarao Mannelli. A theory of initialisation’s impact on specialisation. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  20. [28]

    Soft mode in the dynamics of over- realizable online learning for soft committee machines.Phys

    Frederieke Richert, Roman Worschech, and Bernd Rosenow. Soft mode in the dynamics of over- realizable online learning for soft committee machines.Phys. Rev. E, 105:L052302, May 2022

    2022

  21. [29]

    Hidden unit specialization in layered neural networks: Relu vs

    Elisa Oostwal, Michiel Straat, and Michael Biehl. Hidden unit specialization in layered neural networks: Relu vs. sigmoidal activation.Physica A: Statistical Mechanics and its Applications, 564:125517, 2021

    2021

  22. [30]

    The committee machine: Computational to statistical gaps in learning a two-layers neural network

    Benjamin Aubin, Antoine Maillard, Florent Krzakala, Nicolas Macris, Lenka Zdeborov´ a, et al. The committee machine: Computational to statistical gaps in learning a two-layers neural network. Advances in Neural Information Processing Systems, 31, 2018

    2018

  23. [31]

    Out-of- equilibrium dynamical mean-field equations for the perceptron model.Journal of Physics A: Mathematical and Theoretical, 51(8):085002, 2018

    Elisabeth Agoritsas, Giulio Biroli, Pierfrancesco Urbani, and Francesco Zamponi. Out-of- equilibrium dynamical mean-field equations for the perceptron model.Journal of Physics A: Mathematical and Theoretical, 51(8):085002, 2018

    2018

  24. [32]

    Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification.Advances in Neural Information Processing Systems, 33:9540–9550, 2020

    Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborov´ a. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification.Advances in Neural Information Processing Systems, 33:9540–9550, 2020

    2020

  25. [33]

    Optimal protocols for continual learning via statistical physics and control theory.arXiv preprint arXiv:2409.18061, 2024

    Francesco Mori, Stefano Sarao Mannelli, and Francesca Mignacco. Optimal protocols for continual learning via statistical physics and control theory.arXiv preprint arXiv:2409.18061, 2024

    2024 arXiv

  26. [34]

    in preparation

    Francesca Mignacco and Francesco Mori. in preparation. A Equations for the training dynamics In this appendix, we present the dynamical equations discussed in the main text. The derivation of these equations follows closely that of Ref. [19], with the difference that an additi...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.