Statistical Physics of Deep Neural Networks: Generalization Capability, Beyond the Infinite Width, and Feature Learning

Sebastiano Ariosto

REVIEW 3 major objections 4 minor 12 references

Statistical Physics of Deep Neural Networks: Generalization Capability, Beyond the Infinite Width, and Feature Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A statistical-mechanics bound says deep networks generalize by last-layer width alone.

desk verdict Real result for random-feature readouts, dressed up as a general deep-learning bound; the scope gap is the decisive weakness. read the letter →

arxiv 2501.19281 v1 pith:L6SVVYMG submitted 2025-01-31 cond-mat.dis-nn cs.LG

Sebastiano Ariosto This is my paper · ORCID

classification cond-mat.dis-nncs.LG MSC 82C3268T07

keywords deepneuralnetworksgeneralizationgapstatisticalmechanicsoflearningreplicamethodquenchedaveragefeatureinfinite-widthlimitStudent'st-process

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This thesis argues that physics-based, data-averaged methods can say concrete things about deep neural networks that worst-case learning theory cannot. Its central result is an asymptotic bound on the generalization gap—the gap between test and training error—that depends only on the width of the final layer, improving classical statistical-learning bounds by orders of magnitude. The same formalism is extended beyond the infinite-width limit to give a closed-form generalization error for finite-width one-hidden-layer networks, an approximate partition function for deep networks, and a connection between such networks and Student's t-processes. A third line of work asks when networks genuinely internalize dataset structure rather than memorize it. If these claims hold, the effective complexity of a deep network is its readout capacity, not its parameter count.

What carries the argument

The central machinery is the quenched average of the partition function for a network split into a fixed random feature map and a trained last layer, evaluated with the replica trick under a replica-symmetric ansatz. A Gaussian approximation of the residuals between predictions and labels turns the disorder average into a tractable integral whose saddle-point equations yield order parameters from which both generalization and training errors are read off.

What would settle it

One could settle the central claim by training a family of deep networks with identical last-layer width and widely varying hidden-layer widths on the same regression task: if the generalization gap grows with hidden-layer width in the large-sample limit, the last-layer-only bound is wrong. A complementary check is to measure the non-Gaussianity of the training residuals, since the derivation's Gaussian step predicts those higher cumulants should vanish.

Watch

Extended reading notes

Core claim

The paper's central claim is that when the hidden layers of a deep network are treated as fixed random features and only the last layer is optimized, the average generalization gap is set by the number of output-layer neurons—not by the total number of trainable weights. Using the replica method, the thesis derives closed expressions for generalization and training errors, and shows that in the large-data, large-width limit the bound improves on worst-case capacity bounds by orders of magnitude. The same data-dependent framework yields a finite-width thermodynamic limit with a closed-form generalization error for a one-hidden-layer network, and identifies deep networks in this regime with Student's t-processes. The final part examines a controlled teacher-student task and finds conditions under which networks collapse to the teacher—learning structure—rather than memorizing the dataset.

Load-bearing premise

The bound assumes that hidden layers act as fixed random feature generators and that the deviations between predictions and labels can be treated as Gaussian, an approximation the paper itself calls 'somewhat uncontrolled'.

Editorial extensions

If this is right

In the asymptotic regime, the generalization gap of a deep network with an optimized readout is controlled by the last layer width and the number of samples, not the full parameter count.
The bound improves on worst-case capacity bounds by several orders of magnitude for typical large networks.
A closed-form generalization error exists for finite-width one-hidden-layer networks in the proportional regime where sample count scales with width.
Deep networks in this thermodynamic limit are statistically equivalent to Student's t-processes, giving a tractable non-Gaussian prior beyond the infinite-width Gaussian process.
Under a controlled teacher-student task, the volume of solutions and Gaussianity breaking determine whether a network learns the teacher's structure or merely memorizes training labels.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One consequence not spelled out in the thesis is a 'readout bottleneck' principle: if the last-layer-only bound survives fully trained hidden layers, then deep representations store information that a small readout can exploit, so generalization capacity is set by the readout rather than by the whole network.
The link to Student's t-processes suggests a testable prediction for Bayesian deep learning: finite-width posteriors should show heavier tails than their infinite-width Gaussian-process limits, with the tails controlled by network width and depth.
The Gaussian approximation flagged as uncontrolled in the thesis could be tested directly by computing the fourth cumulant of residual distributions in numerical experiments; a large deviation from zero in the proportional regime would signal where the bound should break down.
The memorize-vs-learn criterion from the final chapter could be turned into a practical probe: measure solution-space volume or Gaussianity breaking on real datasets to detect when a model is forced to internalize data structure.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear X,

The reader's conditional verdict is about right, and the stress-test note lands. The main result—generalization gap asymptotically controlled by last-layer width—is a real result for a random-feature model with readout-only training. The derivation is careful, the saddle-point equations are spelled out, and the numerical checks in 2.3 give the formulas a run for their money. The finite-width closed-form generalization error and the Student's t-process connection in Chapter 3 are genuinely new relative to the GLK20/CBP21 line. Credit where due: no fitted constants enter, the replica calculation is internally consistent, and the text flags the Gaussian approximation as uncontrolled instead of hiding it.

The soft spot is scope, not arithmetic. The bound is derived in a setting where the first L−1 layers are frozen random features and only the last-layer weights v are optimized (Eqs. 2.18, 2.22). The abstract and introduction state the result for deep networks in general: "only depends on the last layer size rather than on the total number of network parameters." No argument—no coupling bound, no monotonicity, no comparison theorem—transfers the frozen-feature bound to end-to-end trained networks. The generalization gap is not monotone under enlarging the hypothesis class, and Chapter 4 itself argues feature learning changes representations. So this is a scope failure, not just a missing epsilon of rigor. A rigorous Gaussian approximation theorem would not repair it; you would still need a separate argument that trained features behave no worse than frozen random ones.

The Gaussian approximation is the second soft spot, minor by comparison but real: the text calls it "somewhat uncontrolled," and it rests on weak correlations of feature maps, which is plausible but unproven. For the last-layer-only claim as advertised, that is a secondary issue.

Who gets value: people working on random features, kernel regression, and statistical physics of learning will find useful formulas and a solid pedagogical exposition of the replica method. The thesis is not a careful general theory of deep-learning generalization. I would not cite the advertised claim as a statement about fully trained neural networks.

Send it to a serious referee, but the referee should require the authors to either prove a transfer argument or restate the main claim as a property of readout-only training on frozen random features. That restatement alone would make the paper honest and still valuable.

Referee Report

3 major / 4 minor

Summary. The thesis applies equilibrium statistical-mechanics methods to deep neural networks. Chapter 2 develops a teacher-student regression model in which the first L-1 layers are treated as a frozen random feature map and only the last-layer weights are optimized; a replica calculation yields average training and generalization errors (Eqs. 2.52, 2.54) and an asymptotic bound that the abstract advertises as depending only on the last-layer width. Chapter 3 studies a finite-width, proportional thermodynamic limit beyond the infinite-width regime and reports a closed-form generalization error for one-hidden-layer networks, an approximate partition function for deeper networks, and a connection to Student-t processes. Chapter 4 presents a preliminary task-explicit analysis of when a network learns an internal representation of the teacher rather than memorizing the training set.

Significance. Chapter 2's saddle-point formulas are genuinely parameter-free: no constants are fitted, and the numerical tests in Section 2.3.1 provide evidence that the bound tracks simulated generalization gaps. Chapter 3's closed-form finite-width generalization error is a constructive step beyond the infinite-width limit. If the advertised last-layer-only bound applied to fully trained deep networks, it would be a major improvement over VC/Rademacher bounds. However, the derivation in Chapter 2 is for a random-feature model with readout-only training, and the extension to fully trained networks is asserted rather than proved; this scope gap is the main weakness. The thesis is therefore best read as a technically rich contribution to random-feature and finite-width statistical mechanics, with the fully trained DNN claim currently an overreach.

major comments (3)

[§2.2.1, Eq. (2.18); Abstract and §0.2] The central derivation starts from f_DNN(x)=v·φ(x) with only v optimized and φ a fixed (random) feature map, and the text explicitly says 'we assume fixed (and random) features and only optimize the last layer weights.' The abstract and introduction, however, state the result for deep networks in general: the generalization bound 'only depends on the last layer size rather than on the total number of network parameters.' No theorem, coupling bound, or monotonicity argument is provided showing that end-to-end training of hidden layers preserves or improves the frozen-feature bound. The order parameters J and Φ in Eq. (2.26) are functionals of the feature map; training the hidden layers changes them, and the generalization gap is not monotone under enlarging the hypothesis class. The statement in §2.1.4 that the formulas apply to a subspace 'which, we will argue, contains trained networks' is not backed by an argument in the visible text. As written, the result is a statement about random-feature/readout-only models; either add a quantitative transfer argument (e.g., a lazy-training/NTK regime) or explicitly restrict the claim and adjust the abstract accordingly.
[§2.2.2, Eqs. (2.24)–(2.27), (2.40)] The Gaussian approximation for the joint distribution of residuals q_a is load-bearing: it is used to evaluate the dataset integral in Eq. (2.27) and enters the action in Eq. (2.40), from which Eqs. (2.52) and (2.54) follow. The text itself calls this approximation 'somewhat uncontrolled' and justifies it only by analogy with the Gaussian equivalence principle and weak mutual correlations of the feature map. The manuscript should state the precise scaling/normalization conditions under which the approximation is controlled, or quantify its failure through finite-N corrections or additional numerical checks. Without this, the asymptotic bound is a plausible mean-field result rather than a proved statement.
[§2.3.1] The numerical tests appear to include state-of-the-art architectures trained end-to-end on real datasets, while the theoretical formulas were derived for readout-only training on frozen features. Empirical agreement for fully trained networks is encouraging but does not substitute for a derivation; the manuscript should separate 'derived for frozen features' from 'observed for fully trained networks,' and should discuss why the bound may survive feature learning (e.g., because deep features approach a random-feature or lazy-training regime in the tested settings).

minor comments (4)

[§2.1.3, Theorem 2.1.2] The Rademacher bound contains a minus sign before the square-root term; a negative term would make the upper bound potentially negative, so the sign should be checked (the standard form has a positive confidence term).
[§2.2.2, gray-box details] The notation in the Gaussian and Hubbard-Stratonovich integrals is inconsistent, with missing normalization factors in some of the z-integrals; please clean up this derivation for readability.
[§2.4 heading] The section title 'Discussion and prespective' contains a typo; it should read 'Discussion and perspective.'
[Abstract and Eq. (2.52)] The phrase 'only depends on the last layer size' should be reconciled with Eq. (2.52), where J and Φ depend on the feature map; please specify precisely whether the dependence on hidden-layer widths and weight distributions is absent asymptotically or merely not tracked.

Circularity Check

0 steps flagged · score 2.0 of 10

No direct circular reduction; the last-layer bound is a genuine consequence of the explicit readout-only model, but the paper's headline claim about fully trained DNNs is a scope extrapolation, and the one self-citation is not load-bearing.

full rationale

The central derivation in Chapter 2 is self-contained for the model as defined: Eqs. (2.18), (2.22), (2.49), (2.52), and (2.54) follow from a quenched replica saddle-point calculation for a teacher-student problem with fixed random features and only the last-layer weights optimized. No fitted constants are introduced, and the generalization and training errors are not defined to equal the claimed bound. The dependence on the last-layer size N is a consequence of the model, since only the readout v of size N is trainable, so the statement that the bound depends on last-layer size rather than total parameter count is an artifact of that readout-only assumption rather than an independent discovery about fully trained DNNs. This is a scope limitation, not a circularity: the random-feature result is honestly derived for the stated model, while the transfer to fully trained networks is asserted without a theorem. The Gaussian approximation is explicitly flagged as 'somewhat uncontrolled due to the absence of a formal result confirming its validity' and is justified by external prior work (e.g., the Gaussian equivalence principle), not by a self-citation chain. The only identifiable self-citation, '[APG+22]' in the Chapter 2 introduction ('as further discussed in [APG+22]'), is a pointer to related discussion and is not load-bearing for any equation. No self-definitional equality, fitted parameter renamed as prediction, or imported uniqueness theorem was found.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central derivation rests on modeling choices that are stated but not proven: teacher-student data, Gaussian equivalence for random features, replica symmetry, and last-layer-only training. No free parameters are fitted. No new physical entities are introduced; the Student's t-process is a mathematical equivalence, not a new entity.

assumptions (4)

domain assumption Inputs are iid draws from rho(x) and labels come from a teacher f_T(x).
Used to define the joint input-output distribution and perform the quenched average over datasets in section 2.2.1.
domain assumption The joint distribution of q_a = f_T - v dot phi can be approximated as multivariate Gaussian, the Gaussian equivalence principle.
Central to Eq. 2.27; the text explicitly states this is "somewhat uncontrolled" in section 2.2.2.
domain assumption The replica trick with a replica symmetric ansatz is valid, with convexity of the last-layer problem making the ansatz exact.
Used to evaluate the quenched free energy in the partition function detail of section 2.2.2.
ad hoc to paper Only the last layer is trained; hidden features are fixed random maps, and trained DNNs are argued to lie in this subspace.
This modeling choice is what makes the bound depend on last-layer size; it is not derived from full training dynamics in section 2.2.1.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Statistical Physics of Deep Neural Networks: Generalization Capability, Beyond the Infinite Width, and Feature Learning." pith.science (2026). https://pith.science/paper/L6SVVYMG

@misc{pith2026250119281,
  author       = {Pith},
  title        = {Pith review of: Statistical Physics of Deep Neural Networks: Generalization Capability, Beyond the Infinite Width, and Feature Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L6SVVYMG}},
  note         = {Machine review of arXiv:2501.19281}
}

read the original abstract

Deep Neural Networks (DNNs) excel at many tasks, often rivaling or surpassing human performance. Yet their internal processes remain elusive, frequently described as "black boxes." While performance can be refined experimentally, achieving a fundamental grasp of their inner workings is still a challenge. Statistical Mechanics has long tackled computational problems, and this thesis applies physics-based insights to understand DNNs via three complementary approaches. First, by averaging over data, we derive an asymptotic bound on generalization that depends solely on the size of the last layer, rather than on the total number of parameters -- revealing how deep architectures process information differently across layers. Second, adopting a data-dependent viewpoint, we explore a finite-width thermodynamic limit beyond the infinite-width regime. This leads to: (i) a closed-form expression for the generalization error in a finite-width one-hidden-layer network (regression task); (ii) an approximate partition function for deeper architectures; and (iii) a link between deep networks in this thermodynamic limit and Student's t-processes. Finally, from a task-explicit perspective, we present a preliminary analysis of how DNNs interact with a controlled dataset, investigating whether they truly internalize its structure -- collapsing to the teacher -- or merely memorize it. By understanding when a network must learn data structure rather than just memorize, it sheds light on fostering meaningful internal representations. In essence, this thesis leverages the synergy between Statistical Physics and Machine Learning to illuminate the inner behavior of DNNs.

Figures

Figures reproduced from arXiv: 2501.19281 by the authors.

**Figure 1.1.** The MNIST dataset. This dataset contains 60, 000 images, each depicting a single handwritten digit ranging from 0 to 9. Every image is formatted as a 28ˆ28 matrix, where each matrix element corresponds to a grayscale value in the image. The values of each pixel vary from 0, denoting the absence of ink, to 255, indicating the highest intensity of ink. Image taken from https://en.wikipedia.org/wiki/MNIST_database [PI… view at source ↗

**Figure 2.** [PITH_FULL_IMAGE:figures/full_fig_p059_2.png] view at source ↗

**Figure 2.2.** Generalisation gap in the lazy-training regime (1). The behavior of the normalized (rescaled by Nout) generalization gap as a function of the training dataset P, for a one-hidden layer student architecture for the three different classes of teacher outlined in the main text: linear (blue symbols), quadratic (green) and one-hidden layer (red). The solid black line marks our mean field upper bound. Data points are the… view at source ↗

Figures from the paper (29 more)

**Figure 2.3.** Figure 2.3: Generalisation gap in the lazy-training regime (2). The behavior of the normalized (multiplied by P) generalization gap as a function of the last layer size Nout, for a one-hidden layer student architecture for the three different classes of teacher outlined in the m…

**Figure 2.4.** Figure 2.4: Generalization Gap in the Lazy-Training Regime (3). Panels A, B, C) illustrate the generalization gap of a lazy-training architecture, plotted against the number of training examples for varying input dimensions D. These are shown for three teacher models: linear, qu…

**Figure 2.5.** Figure 2.5: Generalization Gap in the Lazy-Training Regime (4). This figure illustrates the behavior of the normalized generalization gap as a function of the hidden layer size Nout, observed in two-hidden layer student architectures for a different number of neurons in the last…

**Figure 2.6.** Figure 2.6: Generalization Gap in the Fully-Trained Regime (1). The normalized generalization gap (rescaled by Nout) for neural networks with one hidden layer (1HL NN, shown on the left-hand side in panels A, C, and E) and two hidden layers (2HL NN, on the right-hand side in pa…

**Figure 2.7.** Figure 2.7: Generalization Gap in the Fully-Trained Regime (2). The figure displays the normalized generalization gap (rescaled by Nout) for neural networks with one hidden layer (1HL NN, depicted on the left-hand side in panels A, C, and E) and two hidden layers (2HL NN, shown …

**Figure 2.8.** Figure 2.8: Generalization Gap of State-of-the-Art CNNs. This figure illustrates the behavior of the normalized generalization gap (rescaled by Nout) as a function of the training dataset size P, for three different state-of-the-art convolutional neural networks (CNNs): ResNet1…

**Figure 2.9.** Figure 2.9: Train (blue) and test (magenta) loss of different teacher/student tasks as a function [PITH_FULL_IMAGE:figures/full_fig_p089_2_9.png]

**Figure 2.10.** Figure 2.10: Numerical estimate of the trivial predictor [PITH_FULL_IMAGE:figures/full_fig_p090_2_10.png]

**Figure 2.11.** Figure 2.11: The behavior of the generalization gap in two-hidden-layer deep neural networks [PITH_FULL_IMAGE:figures/full_fig_p092_2_11.png]

**Figure 3.1.** Figure 3.1: Infinity Width vs Proportional regime (1). Learning curves of 1HL architectures with Erf activation (trained with a discretised Langevin dynamics) as a function of the hidden layer size N1 for two regression tasks on the CIFAR10 (above) and MNIST (below) datasets. Z…

**Figure 3.2.** Figure 3.2: Infinity Width vs Proportional Regime (2). Experimental learning curves are presented as a function of N1 for increasing values of the Gaussian prior of the last layer λ1. Error bars are included within the points. Dashed lines connecting the points are provided to a…

**Figure 3.3.** Figure 3.3: Infinity Width vs Proportional Regime (3). Training loss of various one-hidden layer architectures trained on a completely random task (where the inputs x P R N0 with N0 “ 50 and the scalar outputs y are i.i.d. random variables drawn from a normal distribution with z…

**Figure 3.4.** Figure 3.4: Multilayer Neural Network in the Proportional Regime (1). (a) Test loss of an L-HL neural network with ReLU activation trained on the CIFAR10 dataset, as a function of depth L, for P “ 100. The network is trained on a regression task in the small α regime (α “ 0.1), …

**Figure 3.5.** Figure 3.5: Multilayer Neural Network in Proportional Regime (2). (a) Test loss of an LHL neural network with ReLU activation trained on the MNIST dataset, as a function of depth L, for P “ 100. The network is trained on a regression task in the small α regime (α “ 0.1), close …

**Figure 3.6.** Figure 3.6: Multilayer Neural Network in Proportional Regime (3). Numerical evaluation of the solution Q¯˚ is depicted for a ReLU activation function and isotropic network αℓ “ α for all ℓ, across different depths L. As L increases (around L „ 30), the parameter Q¯˚ rapidly con…

**Figure 4.1.** Figure 4.1: Learning Curves as a function of the Dataset Size P. Generalization (circle) and training (diamond) curves as a function of the dataset size P, for different choices of the number of neurons in the hidden layer and input dimensions. The single performance point is ob…

**Figure 4.2.** Figure 4.2: Learning Curves as a function of the Dataset Size P. Generalization (circle) and training (diamond) curves as a function of the rescaled dataset size P{Ntot, for different choices of the number of neurons in the hidden layer and input dimensions. The single performa…

**Figure 4.3.** Figure 4.3: Final Weight Distribution Across Different Dataset Sizes P. This graph illustrates the weight distributions for three different dataset sizes P. On the x-axis, we have ArccospWi ¨ Tq, representing the angle between the weights and the teacher, while the y-axis indica…

**Figure 4.4.** Figure 4.4: Alignment between the Weights and the Teacher in Three Different Scenarios. This graph depicts the alignment between the weights and the teacher in three distinct scenarios. The first row corresponds to a dataset size of P “ 80, the second row to P “ 60, 000, and th…

**Figure 4.5.** Figure 4.5: Distance between the Weights and the Teacher in Three Different Scenarios. This graph depicts the Euclidean distance between the weights v ¨ W and the rescaled teacher T˜ in three distinct scenarios: P “ 80 in green, P “ 60, 000 in blue,and P “ 230, 000 in yellow. W…

**Figure 4.6.** Figure 4.6: Rescaled Singular Values of Matrix J for Different Dataset Sizes P. In the first row, we plot the rescaled singular values of matrix J for different dataset sizes P. The second row features plots of the weight distributions, as a reminder, in the same way that we hav…

**Figure 4.7.** Figure 4.7: Entropy of Singular Values of Matrix J as a Function of Dataset Size P. This graph illustrates the entropy of the singular values of matrix J as a function of the dataset size P. The entropy value obtained at large dataset sizes is consistent with a good alignment be…

**Figure 4.8.** Figure 4.8: Entropy of Singular Values of Matrix J as a Function of Alignment Degree. Starting from a teacher vector, we create a hypothetical matrix J, taking 600 copies of the vector T to which we add Gaussian noise with varying strengths. This graph plots the entropy of the s…

**Figure 4.9.** Figure 4.9: Behavior of Γ as a function of the dataset size and the input size. This graph illustrates the behavior of Γ, rescaled by D, as a function of the ratio between the input size D and the dataset size P. We can see how, for small value of D{P the behaviour of Γ is non l…

**Figure 4.10.** Figure 4.10: The subleading behaviour of Γ in P guarantee that Γ ą e ´1 , at least for large value of P. This graph illustrates the behavior of Γ, as a function of the dataset size P, for fixed value of D{P. We can see how, for large value of P, Γ is always greater than e ´1 […

**Figure 4.11.** Figure 4.11: Kurtosis of the Output Error as a Function of the Generalization Error, for different values of P. This graph illustrates the behavior of the kurtosis of the error as a function of the generalization error during the trining, for dataset size P. In particular, we fo…

**Figure 4.12.** Figure 4.12: Kurtosis of the Output Error as a Function of the Rescaled Dataset Size P{Ntot. This graph illustrates the behavior of the kurtosis of the output as a function of the rescaled dataset size P{Ntot, for fixed values of D “ 100 and Nout “ 600. For greater clarity, the …

**Figure 4.13.** Figure 4.13: Kurtosis of the Output Error as a Function of the Entropy of the Singular Values of Matrix J. This graph illustrates the behavior of the kurtosis of the output as a function of the entropy of the singular values of matrix J, for different choice of P, and fixed valu…

**Figure 4.14.** Figure 4.14: Concavity Changes in Generalization Errors Corresponding to Peaks and Valleys in Kurtosis. The generalization error (top) and Kurtosis (down) as a function of γ “ P{Ntot. The various colored regions highlight changes in the concavity of the generalization error. In-…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 2 canonical work pages

[6]

1088 / 1751 - 8121 / aba028

doi: 10 . 1088 / 1751 - 8121 / aba028. url: https : / / doi . org / 10 . 1088/1751-8121/aba028. [CUH16] Djork-Arne Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus).arXiv preprint arXiv:1511.07289, 2016. [CV95] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Le...

arXiv 2016
[296]

[YSL+13] Dong Yu, Michael L Seltzer, Jinyu Li, Jui-Ting Huang, and Frank Seide

IEEE, 2015. [YSL+13] Dong Yu, Michael L Seltzer, Jinyu Li, Jui-Ting Huang, and Frank Seide. Feature learning in deep neural networks-studies on speech recognition tasks. arXiv preprint arXiv:1301.3605, 2013. [YYY+20] Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Re- thinking bias-variance trade-off for generalization of neural networks....

arXiv 2015
[362]

[RHW86b] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams

MIT Press, 1986. [RHW86b] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learn- ing representations by back-propagating errors.nature, 323(6088):533–536, 1986. [RIG23] Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural net- works trained with SGD learn distributions of increasing complexity. In Andreas Krause, Emma Brunskill, ...

arXiv 1986
[1530]

[AGS87] Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky

url: https://link.aps.org/doi/10.1103/PhysRevLett.55.1530. [AGS87] Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky. Statistical me- chanics of neural networks near saturation.Annals of physics, 173(1):30– 67, 1987. [Ait20] Laurence Aitchison. Why bigger is not always better: on finite and infinite neural networks. In Hal Daumé III and Aarti Singh, e...

doi:10.1103/physrevlett.55.1530 1987
[1983]

url: https ://www.sciencedirect.com/science/article/pii/0047259X83900192

issn: 0047-259X.doi: 10.1016/0047-259X(83)90019-2. url: https ://www.sciencedirect.com/science/article/pii/0047259X83900192. [BMZ19] Carlo Baldassi, Enrico M. Malatesta, and Riccardo Zecchina. Properties of the geometry of solutions and capacity of multilayer neural networks with rectified linear unit activations.Phys. Rev. Lett., 123:170602, 17, October

arXiv
[1999]

187 [GSJ+20] Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart

doi: 10.1162/089976699300016629. 187 [GSJ+20] Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Dis- entangling feature and lazy training in deep neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2020(11):113301, 2020. [GSS14] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar...

arXiv
[2014]

[SZ14] Karen Simonyan and Andrew Zisserman

url: https://proceedings.mlr.press/v33/shah14.html. [SZ14] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556, 2014. [TDC86] Gerard Toulouse, Stanislas Dehaene, and Jean-Pierre Changeux. Spin glass model of learning by selection. Proceedings of the National Academy of Scie...

arXiv 2014
[2017]

[LBB+98] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner

doi: 10.1038/s41467-017-01596-3. [LBB+98] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998. doi: 10.1109/5.726791. [LBN+17a] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep n...

arXiv 1998

Show all 12 references

[2018]

url: https://link.aps.org/ doi/10.1103/PhysRevX.8.031003

doi: 10.1103/PhysRevX.8.031003. url: https://link.aps.org/ doi/10.1103/PhysRevX.8.031003. [COB19a] Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alche-Buc, E. Fox, and R. Gar...

doi
[2019]

url: https://link.aps

doi: 10.1103/PhysRevLett.123.170602. url: https://link.aps. org/doi/10.1103/PhysRevLett.123.170602. [Bot10] Léon Bottou. Large-scale machine learning with stochastic gradient de- scent. InProceedings of COMPSTAT’2010, pages 177–186. Springer, 2010. 182 CHAPTER 6. BIBLIOGRAPHY ...

2010 arXiv
[2020]

url: https://proceedings.mlr.press/v119/aitchison20a.html. [AJB+17] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Em- manuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in d...

2017 arXiv
[2021]

url: https://link.aps

doi: 10.1103/PhysRevE.103.042142 . url: https://link.aps. org/doi/10.1103/PhysRevE.103.042142. 192 CHAPTER 6. BIBLIOGRAPHY [Mur12] Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012. [MV15] Aravindh Mahendran and Andrea Vedaldi. Understanding deep ...

2012

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.

[1] [6]

1088 / 1751 - 8121 / aba028

doi: 10 . 1088 / 1751 - 8121 / aba028. url: https : / / doi . org / 10 . 1088/1751-8121/aba028. [CUH16] Djork-Arne Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus).arXiv preprint arXiv:1511.07289, 2016. [CV95] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Le...

arXiv 2016

[2] [296]

[YSL+13] Dong Yu, Michael L Seltzer, Jinyu Li, Jui-Ting Huang, and Frank Seide

IEEE, 2015. [YSL+13] Dong Yu, Michael L Seltzer, Jinyu Li, Jui-Ting Huang, and Frank Seide. Feature learning in deep neural networks-studies on speech recognition tasks. arXiv preprint arXiv:1301.3605, 2013. [YYY+20] Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Re- thinking bias-variance trade-off for generalization of neural networks....

arXiv 2015

[3] [362]

[RHW86b] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams

MIT Press, 1986. [RHW86b] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learn- ing representations by back-propagating errors.nature, 323(6088):533–536, 1986. [RIG23] Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural net- works trained with SGD learn distributions of increasing complexity. In Andreas Krause, Emma Brunskill, ...

arXiv 1986

[4] [1530]

[AGS87] Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky

url: https://link.aps.org/doi/10.1103/PhysRevLett.55.1530. [AGS87] Daniel J Amit, Hanoch Gutfreund, and Haim Sompolinsky. Statistical me- chanics of neural networks near saturation.Annals of physics, 173(1):30– 67, 1987. [Ait20] Laurence Aitchison. Why bigger is not always better: on finite and infinite neural networks. In Hal Daumé III and Aarti Singh, e...

doi:10.1103/physrevlett.55.1530 1987

[5] [1983]

url: https ://www.sciencedirect.com/science/article/pii/0047259X83900192

issn: 0047-259X.doi: 10.1016/0047-259X(83)90019-2. url: https ://www.sciencedirect.com/science/article/pii/0047259X83900192. [BMZ19] Carlo Baldassi, Enrico M. Malatesta, and Riccardo Zecchina. Properties of the geometry of solutions and capacity of multilayer neural networks with rectified linear unit activations.Phys. Rev. Lett., 123:170602, 17, October

arXiv

[6] [1999]

187 [GSJ+20] Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart

doi: 10.1162/089976699300016629. 187 [GSJ+20] Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Dis- entangling feature and lazy training in deep neural networks.Journal of Statistical Mechanics: Theory and Experiment, 2020(11):113301, 2020. [GSS14] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar...

arXiv

[7] [2014]

[SZ14] Karen Simonyan and Andrew Zisserman

url: https://proceedings.mlr.press/v33/shah14.html. [SZ14] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition.arXiv preprint arXiv:1409.1556, 2014. [TDC86] Gerard Toulouse, Stanislas Dehaene, and Jean-Pierre Changeux. Spin glass model of learning by selection. Proceedings of the National Academy of Scie...

arXiv 2014

[8] [2017]

[LBB+98] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner

doi: 10.1038/s41467-017-01596-3. [LBB+98] Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition.Proceedings of the IEEE, 86(11):2278–2324, 1998. doi: 10.1109/5.726791. [LBN+17a] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep n...

arXiv 1998