Microscopic and collective signatures of feature learning in neural networks

Andrea Corti; Marco Gherardi; Pietro Rotondo; Rosalba Pacelli

REVIEW 3 major objections 4 minor 59 references

Microscopic and collective signatures of feature learning in neural networks

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In standard-scaled Bayesian one-hidden-layer networks, the posterior distance between class manifolds is nonmonotonic in temperature, and hidden weights develop data-dependent correlations—signatures of feature learning even where predictio

desk verdict Genuinely new analytic signatures of feature learning in Bayesian nets, but the headline nonmonotonicity rests on a scaling assumption the SM shows fails for the very datasets used; the dropped correction should be kept or bounded before the curves are trusted. read the letter →

arxiv 2508.20989 v1 pith:5Z7BUNCA submitted 2025-08-28 cond-mat.dis-nn cond-mat.stat-mech

Andrea Corti , Rosalba Pacelli , Pietro Rotondo , Marco Gherardi This is my paper

classification cond-mat.dis-nncond-mat.stat-mech

keywords featurelearningBayesianneuralnetworksproportionallimitclassmanifoldsGaussianprocessequivalenceweightcorrelationsnonmonotonicityintemperaturestatisticalmechanicsof

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether standard-scaled Bayesian one-hidden-layer networks can learn features at all in the proportional regime, where width and training set size grow together—and answers yes, with two quantitative signatures. First, the average squared distance between the two class manifolds in feature space is a nonmonotonic function of the Gibbs temperature: it rises to a peak and then falls, and the peak disappears when labels are randomized or when the network is infinitely wide. Second, the hidden-layer weights acquire finite, data-dependent correlations and displacements relative to their infinite-width prior, even though the network's predictive distribution remains that of Gaussian process regression with a trivially rescaled kernel. The authors interpret the temperature peak as the equilibrium counterpart of the nonmonotonic inversion dynamics of class manifolds seen under gradient descent, and read both effects as fingerprints of feature learning in a regime previously thought to be essentially kernel-like. A careful reader would care because the result gives concrete, computable observables—class-manifold separation and weight correlations—for detecting feature learning where output statistics alone cannot.

What carries the argument

The load-bearing objects are the posterior-averaged squared distance between class manifolds, built from the posterior feature similarity matrix ⟨σ(h_i^μ)σ(h_i^ν)⟩, and the two-point function of first-layer weights, ⟨w_{1h}w_{1k}⟩ = δ_hk/λ0 + (Λ1+Λ2)/λ0. The computation runs through the saddle-point effective action of the proportional limit, whose order parameter Q̄ renormalizes the infinite-width Gaussian process kernel K into K(R)=Q̄K/λ1 and supplies posterior mean outputs f̄±. The O(1/N1) corrections to the feature similarity matrix become finite when weighted by the distance observable, and their competition—a temperature-independent prior term y^T K y/P and a posterior term proportiona

What would settle it

Evaluate the full posterior-averaged distance in Eq. (3), keeping all three terms, for a dataset where the assumption fails—the supplement names MNIST restricted to classes 0 and 1—or for Gaussian data at finite α, and check whether ⟨D²⟩ remains nonmonotonic in T. If the peak disappears or becomes monotonic once the dropped correction is included, the reported signature is an artifact of the approximation; if it survives, the claim is confirmed.

Watch

Extended reading notes

Core claim

The central claim is that nontrivial feature learning occurs in standard-scaled Bayesian one-hidden-layer networks in the proportional limit, and it leaves two calculable traces. Averaging over the posterior, the squared distance between the two class manifolds—the mean post-activations of samples with labels +1 and −1—is not a constant but a nonmonotonic function of temperature, as given by Eq. (4). The feature-dependent term λ1/Q̄ (f̄+−f̄−)², which is absent in the infinite-width limit, grows as the posterior starts to fit the labels and then saturates, producing a peak whose height increases with the load α=P/N1; randomizing labels removes the peak. At the microscopic level, the two-point

Load-bearing premise

The central nonmonotonicity depends on the assumption that the prior contribution to class separation, y^T K y / P, is finite (or at least comparable to the posterior term) as P grows, so that the one correction term dropped in Eq. (4) is negligible; the supplement shows this is only approximately true for CIFAR10 and cannot be pursued for MNIST 0/1.

Editorial extensions

If this is right

The gradient-descent inversion of class manifolds has a thermodynamic counterpart: temperature acts like an early-stopping regularizer, and the easy-to-hard transition in training is an equilibrium phase boundary at the peak temperature.
Class-manifold geometry is a more sensitive probe of feature learning than the predictive distribution, which stays Gaussian-process-like throughout the proportional regime.
The height and location of the peak are controlled by α, so the signature is tunable and can be used to quantify how far a finite-width network is from the lazy infinite-width regime.
Microscopic weight correlations provide a second, independent fingerprint, testable directly from hidden-layer weights without reference to outputs.
Random-label control isolates genuine label-driven feature learning: the peak vanishes when labels carry no class structure.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the nonmonotonicity survives exact evaluation of Eq. (3) (including the dropped second correction) in datasets where the assumption fails, the peak would be a robust feature; if not, it would be an artifact of the approximation on real images. This distinction is testable with the supplement's own numerical scaling plots.
The weight-correlation result suggests the standard 'lazy vs rich' dichotomy may be too coarse: a network can have Gaussian-process-equivalent predictions while its internal representation is actively learning, so output-level comparisons miss representation-level feature learning.
The same saddle-point machinery should yield a similar temperature peak for convolutional or deep linear networks via local kernel renormalization, making D² nonmonotonicity a candidate universal diagnostic of feature learning across architectures.
The correspondence with gradient-descent inversion dynamics suggests an experimental protocol: measure D² during training while injecting label noise or varying effective temperature, and compare the peak temperature with the Bayesian prediction.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should read this if you care about whether feature learning can be detected in the proportional-limit Bayesian setting. The genuinely new claim is that the posterior-averaged squared distance between class manifolds is nonmonotonic in temperature, and that this is the equilibrium analogue of the GD inversion dynamics from [31]. That is a nice, testable idea, and the randomized-label and alpha-to-zero controls make the picture plausible.

What the paper does well: the effective-action framework is used consistently, the derivation through Eq. (3) is clear, and the weight-correlation computation in the SM is real work—the consistency check with linear activations is a good idea. The interpretation that features can be learned even when the predictive distribution is a GP with trivially rescaled prior is worth taking seriously.

The soft spot is the step from Eq. (3) to Eq. (4). The second term is dropped on the assumption that y^T K y / P = O(1). The SM shows this is not strictly true for CIFAR10 (it grows linearly, albeit slowly) and completely fails for MNIST 0/1, where the text says the approach cannot be pursued. That means the headline peak in Fig. 1 is computed under an uncontrolled approximation in exactly the finite-P regime where the theory is applied. The SM plots do show the dropped term is numerically small for the P values used, so the specific curves might survive, but the asymptotic statement is not controlled. This is addressable: redo the calculation keeping the Δ1 term, or provide an explicit bound showing it is subdominant over the P range of interest. Also, the paper gives no code or data for the figures, so the curves are not independently checkable. The weight-correlation part partly overlaps with [25], though the emphasis on data-dependent correlations is more explicit here.

One more thing: the SM text and the Figure 3 caption contradict each other about whether CIFAR10 satisfies the scaling. That needs fixing.

Who is this for? People working on Bayesian networks and feature learning. It is a solid contribution to a specific literature, not a broad new method. It deserves a serious referee—the core derivation is plausible and the controls are good, but the central approximation needs a clear fix or a much more careful finite-P justification. I would send it to review with the request that the authors either include the Δ1 term in the calculation or provide a rigorous bound on it. As it stands, the nonmonotonicity is an interesting observation, not a proven theorem.

Referee Report

3 major / 4 minor

Summary. The paper studies Bayesian one-hidden-layer fully connected networks in the proportional limit (width and training-set size both large), using an effective-action/saddle-point formalism. Its two main claims are: (i) the posterior-averaged squared distance between class manifolds in feature space is a nonmonotonic function of the Gibbs temperature, with a peak that disappears for random labels and in the infinite-width limit; and (ii) the hidden-layer weights acquire finite, data-dependent correlations in the proportional regime. The interpretation advanced is that these are equilibrium signatures of feature learning, related to inversion dynamics observed under gradient descent. The central quantitative object is Eq. (4), obtained from the exact posterior average Eq. (3) by dropping the correction term proportional to y^T Δ1 y on the basis of a scaling assumption on y^T K y / P.

Significance. If the result holds, the paper offers a concrete, analytically tractable equilibrium observable—class-manifold distance—that changes nonmonotonically with temperature and distinguishes the proportional regime from the infinite-width GP regime. The microscopic weight-correlation result is also a useful step beyond the usual statement that the predictive distribution is GP-like. The randomized-label control, the exact linear-activation consistency check in the SM, and the explicit scaling checks in SM Fig. 3 are commendable. However, the nonmonotonicity claim rests on a scaling assumption that the authors themselves show is not satisfied for the datasets used to display the effect (CIFAR10) and is invalid for MNIST 0/1. This makes the headline signature currently uncontrolled at the level of the central derivation, and the quantitative theory needs revision before the claim can be accepted.

major comments (3)

[Eq. (3)–(4), SM 'Derivation of the average squared distance', SM Fig. 3] The passage from Eq. (3) to Eq. (4) drops the term -(4/αP) y^T Δ1 y. The justification is the zero-temperature scaling y^T K y / P = O(1), under which the dropped term is O(1/N1). But the SM (Fig. 3, top-left and middle-left) explicitly shows that for CIFAR10 the quantity (1/P) y^T K y grows linearly with P, and for MNIST 0/1 the same quantity grows to ~800, with the text stating that the scaling approach 'cannot be pursued.' Since Eq. (4) is used to generate the headline nonmonotonic curves in Fig. 1(b,c), the predicted peak is computed under an approximation that fails for the very datasets shown. If y^T K y / P grows with P, the dropped term is not subleading; it can contribute a temperature-dependent correction comparable to the retained terms, potentially shifting or destroying the peak. The authors should recompute the curves from the full Eq. (3) (including Δ1) or provide a rigoro
[Eq. (3)–(4), finite-T discussion] Even if the zero-temperature scaling y^T K y / P were O(1), the finite-temperature behavior of the dropped term is not addressed. The text argues that at β→∞ the term is O(1/N1) and at β→0 it vanishes, and that the system 'continuously interpolates' between these regimes. But Eq. (4) is evaluated at intermediate temperatures where Fig. 1 shows the peak. The interpolation statement does not establish that the magnitude of Δ1 remains small compared to the prior term at those temperatures. At finite β, K̃^{-1}(R) is not proportional to K^{-1}; the spectral structure of K can enhance the bilinear form. A direct numerical evaluation of the dropped term as a function of T for the plotted cases (or an analytical estimate) is required to support Eq. (4).
[Discussion and conclusions, Fig. 1] The claim that the nonmonotonic peak is a 'signature of feature learning' is inferred from the presence of the posterior term (λ1/Q̄)(f̄_+ - f̄_-)^2 in Eq. (4) and from the fact that random labels remove the peak. This inference is reasonable only if Eq. (4) is a controlled approximation. If the dropped Δ1 term is actually comparable to the retained terms, the temperature dependence of ⟨D²⟩ could be dominated by that term, which is a posterior correction but not the feature-separation term emphasized by the authors. The randomized-label control alone does not rescue the interpretation, because randomizing labels also changes the size and temperature dependence of Δ1. The authors should either recompute including Δ1 and show that the peak persists and arises from the Δ2/feature term, or weaken the interpretation accordingly.

minor comments (4)

[SM, 'Derivation of the average squared distance'] Typo: the text says 'the divergence in the infinite-width limit α → ∞' but the infinite-width limit in the main text is α → 0 (P fixed, N1 → ∞). The same paragraph's formula uses α in a way consistent with α → 0. Please correct.
[Main text, p. 3] The phrase 'This numerical observation is an exact statement for iid random data' is imprecise. The preceding argument applies to iid Gaussian (or sufficiently concentrated) data via the central limit theorem, but not to arbitrary iid data with heavy tails or correlations. The SM's own distinction between Gaussian, CIFAR10, and MNIST is helpful and should be reflected here.
[Fig. 1 caption] The caption states 'All curves are computed using the first 1000 CIFAR10 samples.' Given the scaling concern, it would be useful to state P and α explicitly in the caption, and to show error bars or variability over subsamples, since the theory is applied at finite P.
[Eqs. (7)–(8)] The definitions of Λ1 and Λ2 mix the matrix Δ^{μν}_{hk} with the earlier scalars Δ1 and Δ2. The notation is not confusing in context, but a brief pointer to the precise definition of Δ^{μν}_{hk} in the SM would improve readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central nonmonotonicity is a new application of a published effective-action formula, and the main caveat is an acknowledged scaling approximation, not a circular reduction.

full rationale

Walking the derivation chain: the posterior feature similarity Eq. (1) is imported from [24]; substituting it into the definition of D^2 gives the exact decomposition Eq. (3). The passage from Eq. (3) to Eq. (4) drops the Delta_1 term under the stated assumption y^T K y/P = O(1). The claimed nonmonotonicity then follows from the temperature dependence of the GP predictive outputs f_+/- and of the saddle-point order parameter Qbar, not from any parameter fitted to the target observable and later renamed as a prediction. The effective-action framework and Eq. (1) come from prior works with overlapping authorship ([20,24,25]), but those are published, independently tested frameworks, and the present paper's new claims (the nonmonotonic peak and the weight correlations) are not already contained in the cited inputs. The randomized-label and alpha-to-zero controls provide independent checks that the peak is tied to label information. The paper itself flags the main weakness in the SM: "Strictly speaking, the scaling is not satisfied for CIFAR10 dataset, where 1/P z^T K z grows linearly" and "This approach cannot be pursued in the case of MNIST dataset restricted to 0 and 1 classes." This is a genuine approximation risk for the quantitative curves, but it is an acknowledged uncontrolled approximation, not a circularity: Eq. (4) is not equivalent to its inputs by construction, and the failure of the scaling does not make the derivation circular. Therefore no circular step meets the evidentiary bar; score 1.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central computation inherits the effective-action saddle-point machinery and the feature similarity formula from prior work [20,24] with overlapping authorship (Pacelli, Rotondo, Gherardi). The main fitted object is the data-dependent order parameter Q-bar. The scaling assumption that y^T K y / P is effectively O(1) for the datasets shown is explicitly not exact for CIFAR10 and fails for MNIST. No new particles, forces, or entities are introduced; the only new objects are observables such as D2 and weight correlations.

free parameters (3)

Q-bar (saddle-point order parameter) = Data-dependent, defined by dS/dQ = 0, no closed form given
Determines the renormalized kernel and all posterior corrections in Eqs. (1), (4), (6)-(8). It is computed by optimizing an effective action on the training data and labels, so it functions as a fitted quantity in the derivation.
Distance normalization exponents eta, delta = Main text uses eta=0, delta=1; SM alternative uses eta=1, delta=1/2
These exponents are chosen so that D2 has a finite or comparable scaling in the proportional limit. Different choices change the observable and its divergence, so the nonmonotonicity depends on a hand-selected normalization.
Prior inverse variances lambda0, lambda1 = lambda0=1 in Fig. 2; lambda1 not specified for most plots
Scale the priors on readout and internal weights and set the magnitude of posterior corrections. They are model inputs rather than fitted to the feature observables, but at least one is set arbitrarily in the numerical examples.

assumptions (5)

domain assumption The saddle-point evaluation of the partition function is valid in the proportional limit.
Invoked in 'Setting of the learning problem' through references [18,19,25] and used for all posterior averages; it is a non-rigorous asymptotic approximation not proved in this paper.
domain assumption The averaged feature similarity matrix formula, Eq. (1) from [24], is correct and is the starting point for D2.
Taken from prior work by the same group [24]; if this formula is wrong, the collective distance formula and the nonmonotonicity claim fail.
domain assumption Balanced binary labels with y=+/-1 and zero-mean activations are sufficient for the main claims.
Stated in the problem setting; the unbalanced generalization is relegated to the SM and not used in the main plots.
ad hoc to paper The scaling assumption (1/P) y^T K y = O(1), or an 'effectively same order' variant, holds for the datasets used in Fig. 1.
This is the load-bearing approximation behind Eq. (4). The SM admits strict scaling fails for CIFAR10 and cannot be pursued for MNIST 0/1. Without it, the prior term can dominate and erase the temperature peak.
standard math Standard Gaussian integration identities, including Sherman-Morrison, are used in the weight-statistics derivation.
Used in the SM around Eqs. (30)-(43) to express Delta^{mu nu}_{hk} and remove the dependence on C^{-1}.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Microscopic and collective signatures of feature learning in neural networks." pith.science (2026). https://pith.science/paper/5Z7BUNCA

@misc{pith2026250820989,
  author       = {Pith},
  title        = {Pith review of: Microscopic and collective signatures of feature learning in neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5Z7BUNCA}},
  note         = {Machine review of arXiv:2508.20989}
}

read the original abstract

Feature extraction - the ability to identify relevant properties of data - is a key factor underlying the success of deep learning. Yet, it has proved difficult to elucidate its nature within existing predictive theories, to the extent that there is no consensus on the very definition of feature learning. A promising hint in this direction comes from previous phenomenological observations of quasi-universal aspects in the training dynamics of neural networks, displayed by simple properties of feature geometry. We address this problem within a statistical-mechanics framework for Bayesian learning in one hidden layer neural networks with standard parameterization. Analytical computations in the proportional limit (when both the network width and the size of the training set are large) can quantify fingerprints of feature learning, both collective ones (related to manifold geometry) and microscopic ones (related to the weights). In particular, (i) the distance between different class manifolds in feature space is a nonmonotonic function of the temperature, which we interpret as the equilibrium counterpart of a phenomenon observed under gradient descent (GD) dynamics, and (ii) the microscopic learnable parameters in the network undergo a finite data-dependent displacement with respect to the infinite-width limit, and develop correlations. These results indicate that nontrivial feature learning is at play in a regime where the posterior predictive distribution is that of Gaussian process regression with a trivially rescaled prior.

Figures

Figures reproduced from arXiv: 2508.20989 by the authors.

**Figure 1.** (a) Separation of class manifolds in the lazy training infinite-width vs proportional regime. In the infinite-width limit, the separation between class manifolds D 2 is unaffected by training. The weight displacement that occurs in this setting is not enough to produce changes in the collective observable D 2 . In the proportional regime, D 2 undergoes a finite shift, due to the microscopic weight displacements. (b,… view at source ↗

**Figure 2.** Finite corrections to ⟨ww⟩ in the proportional regime. (a) Values of 25 randomly selected components of the proportional correction Λ2, Eq. (8), as a function of α. As α increases, transitioning the system to the proportional regime, each component exhibits a linear trend, consistent with the scaling predicted by GD dynamics. (b) The heatmap shows a submatrix of Λ2 at α = 2. The magnitude of the matrix elements … view at source ↗

**Figure 3.** Gaussian data are associated with the theoretical scaling provided in the text. Strictly speaking, the scaling is not [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: * (a) CIFAR10, even vs. odd 0 100 200 300 400 0 100 200 300 400 Kernel, MNIST 0.0 0.1 0.2 0.3 0.4 0.5 0.6 Kµν [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

59 extracted references · 52 canonical work pages

[31]

J. Lee, S. Schoenholz, J. Pennington, B. Adlam, L. Xiao, R. Novak, and J. Sohl-Dickstein, Finite versus infinite neural networks: an empirical study, in Advances in Neu- ral Information Processing Systems , Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., 2020) pp. 15156–15172

work page 2020
[25]

A Multiscale integrated ap- proach to the study of the nervous system in health and disease

for finite-width deep linear networks. In the infinite-width limit, the terms Λ1 and Λ2 vanish, the components of w become uncorrelated, and averages over the prior and over the posterior coincide. In the pro- portional case, since the kernel ∆ µν hk cannot be expressed in simple terms by using Kµν, the zero-temperature limit is not useful to investigate ...

work page 2022
[1]

D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning representations by back-propagating errors, Na- ture 323, 533 (1986)

work page 1986
[2]

Krizhevsky, I

A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Commun. ACM 60, 84–90 (2017)

work page 2017
[3]

Krizhevsky and G

A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images , Tech. Rep. 0 (University of Toronto, Toronto, Ontario, 2009)

work page 2009
[4]

Yang and E

G. Yang and E. J. Hu, Tensor programs iv: Feature learning in infinite-width neural networks, in Proceedings of the 38th International Conference on Machine Learn- ing, Proceedings of Machine Learning Research, Vol. 139, edited by M. Meila and T. Zhang (PMLR, 2021) pp. 11727–11737

work page 2021
[5]

Seroussi, G

I. Seroussi, G. Naveh, and Z. Ringel, Separation of scales and a thermodynamic description of feature learning in some cnns, Nature Communications 14, 908 (2023)

work page 2023
[6]

A. X. Yang, M. Robeyns, E. Milsom, N. Schoots, and L. Aitchison, A theory of representation learning in deep neural networks gives a deep generalisation of kernel methods (2023), arXiv:2108.13097 [stat.ML]

work page Pith review arXiv 2023

Show all 59 references

[7]

C. Shi, L. Pan, and I. Dokmani´ c, Spring-block theory of feature learning in deep neural networks, Phys. Rev. Lett. 134, 257301 (2025)

2025
[8]

van Meegen and H

A. van Meegen and H. Sompolinsky, Coding schemes in neural networks learning classification tasks, Nature Communications 16, 3354 (2025)

2025
[9]

R. M. Neal, Priors for infinite networks, in Bayesian Learning for Neural Networks (Springer New York, New York, NY, 1996) pp. 29–53

1996
[10]

D. J. MacKay, Introduction to gaussian processes, NATO ASI Series F Computer and Systems Sciences 168, 133 (1998)

1998
[11]

C. E. Rasmussen and C. K. I. Williams, Gaussian pro- cesses for machine learning. , Adaptive computation and machine learning (MIT Press, 2006) pp. I–XVIII, 1–248

2006
[12]

J. Lee, J. Sohl-dickstein, J. Pennington, R. Novak, S. Schoenholz, and Y. Bahri, Deep neural networks as gaussian processes, in International Conference on Learning Representations (2018)

2018
[13]

Novak, L

R. Novak, L. Xiao, Y. Bahri, J. Lee, G. Yang, D. A. Abo- lafia, J. Pennington, and J. Sohl-dickstein, Bayesian deep convolutional networks with many channels are gaussian processes, in International Conference on Learning Rep- resentations (2019)

2019
[14]

Yang, Tensor programs i: Wide feedforward or recur- rent neural networks of any architecture are gaussian pro- cesses, in Neural Information Processing Systems (2019)

G. Yang, Tensor programs i: Wide feedforward or recur- rent neural networks of any architecture are gaussian pro- cesses, in Neural Information Processing Systems (2019)

2019
[15]

Canatar, B

A. Canatar, B. Bordelon, and C. Pehlevan, Spectral bias and task-model alignment explain generalization in ker- nel regression and infinitely wide neural networks, Nature communications 12, 1 (2021)

2021
[16]

Arora, S

S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdi- nov, and R. Wang, On exact computation with an in- finitely wide neural net, in Advances in Neural Informa- tion Processing Systems , Vol. 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d 'Alch´ e-Buc, E. Fox, an...

2019
[17]

Favaro, B

S. Favaro, B. Hanin, D. Marinucci, I. Nourdin, and G. Peccati, Quantitative clts in deep neural networks, Probability Theory and Related Fields 191, 933 (2025)

2025
[18]

Li and H

Q. Li and H. Sompolinsky, Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization, Phys. Rev. X 11, 031059 (2021)

2021
[19]

Hanin and A

B. Hanin and A. Zlokapa, Bayesian interpolation with deep linear networks, Proceedings of the Na- tional Academy of Sciences 120, e2301345120 (2023), https://www.pnas.org/doi/pdf/10.1073/pnas.2301345120

2023 doi
[20]

Pacelli, S

R. Pacelli, S. Ariosto, M. Pastore, F. Ginelli, M. Gher- ardi, and P. Rotondo, A statistical mechanics framework for bayesian deep neural networks beyond the infinite- 6 width limit, Nature Machine Intelligence 5, 1497 (2023)

2023
[21]

Pacelli, L

R. Pacelli, L. Giambagli, and P. Baglioni, Kernel shape renormalization in bayesian shallow networks: a gaussian process perspective, in 2024 IEEE Workshop on Com- plexity in Engineering (COMPENG) (2024) pp. 1–6

2024
[22]

H. Cui, F. Krzakala, and L. Zdeborov´ a, Bayes-optimal learning of deep random networks of extensive-width, Proceedings of the 40th International Conference on Ma- chine Learning (2023)

2023
[23]

Camilli, D

F. Camilli, D. Tieplova, E. Bergamin, and J. Barbier, Information-theoretic reduction of deep neural networks to linear models in the overparametrized proportional regime, arXiv preprint arXiv:2505.03577 (2025)

2025 arXiv
[24]

Aiudi, R

R. Aiudi, R. Pacelli, P. Baglioni, A. Vezzani, R. Buri- oni, and P. Rotondo, Local kernel renormalization as a mechanism for feature learning in overparametrized con- volutional neural networks, Nature Communications 16, 10.1038/s41467-024-55229-3 (2025)

2025 doi
[26]

Bassetti, M

F. Bassetti, M. Gherardi, A. Ingrosso, M. Pastore, and P. Rotondo, Feature learning in finite-width bayesian deep linear networks with multiple outputs and convolu- tional layers, Journal of Machine Learning Research 26, 1 (2025)

2025
[27]

Baglioni, R

P. Baglioni, R. Pacelli, R. Aiudi, F. Di Renzo, A. Vez- zani, R. Burioni, and P. Rotondo, Predictive power of a bayesian effective action for fully connected one hid- den layer neural networks in the proportional limit, Phys. Rev. Lett. 133, 027301 (2024)

2024
[28]

Baglioni, L

P. Baglioni, L. Giambagli, A. Vezzani, R. Burioni, P. Ro- tondo, and R. Pacelli, Kernel shape renormalization ex- plains output-output correlations in finite bayesian one- hidden-layer networks, Phys. Rev. E 111, 065312 (2025)

2025
[29]

H. Shan, Q. Li, and H. Sompolinsky, Order parameters and phase transitions of continual learning in deep neural networks, arXiv preprint arXiv:2407.10315 (2024)

2024 arXiv
[30]

Ingrosso, R

A. Ingrosso, R. Pacelli, P. Rotondo, and F. Gerace, Sta- tistical mechanics of transfer learning in fully connected networks in the proportional limit, Phys. Rev. Lett. 134, 177301 (2025)

2025
[32]

Ciceri, L

S. Ciceri, L. Cassani, M. Osella, P. Rotondo, F. Valle, and M. Gherardi, Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalization, Nature Machine Intelligence 6, 40 (2024)

2024
[33]

Pukowski and H

P. Pukowski and H. Lu, Investigating the impact of hard and easy samples on generalization reveals in-class data imbalance, in AutoML 2024 Methods Track (2024)

2024
[34]

Chung, D

S. Chung, D. D. Lee, and H. Sompolinsky, Classification and geometry of general perceptual manifolds, Phys. Rev. X 8, 031003 (2018)

2018
[35]

Rotondo, M

P. Rotondo, M. Pastore, and M. Gherardi, Beyond the storage capacity: Data-driven satisfiability transition, Phys. Rev. Lett. 125, 120601 (2020)

2020
[36]

Pastore, P

M. Pastore, P. Rotondo, V. Erba, and M. Gherardi, Sta- tistical learning theory of structured data, Phys. Rev. E 102, 032119 (2020)

2020
[37]

Cohen, S

U. Cohen, S. Chung, D. D. Lee, and H. Sompolinsky, Separability and geometry of object manifolds in deep neural networks, Nature Communications11, 746 (2020)

2020
[38]

M. Pastore, Critical properties of the SAT/UNSAT tran- sitions in the classification problem of structured data, Journal of Statistical Mechanics: Theory and Experi- ment 2021, 113301 (2021)

2021
[39]

Farrell, S

M. Farrell, S. Recanatesi, T. Moore, G. Lajoie, and E. Shea-Brown, Gradient-based learning drives robust representations in recurrent neural networks by balancing compression and expansion, Nature Machine Intelligence 4, 564 (2022)

2022
[40]

Kamnitsas, D

K. Kamnitsas, D. Castro, L. L. Folgoc, I. Walker, R. Tanno, D. Rueckert, B. Glocker, A. Criminisi, and A. Nori, Semi-supervised learning via compact latent space clustering, in Proceedings of the 35th International Conference on Machine Learning , Proceedings of Ma- chine Lear...

2018
[41]

Jacot, F

A. Jacot, F. Gabriel, and C. Hongler, Neural tangent ker- nel: Convergence and generalization in neural networks, in Advances in Neural Information Processing Systems , Vol. 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Curran...

2018
[42]

Francazi, M

E. Francazi, M. Baity-Jesi, and A. Lucchi, A theoreti- cal analysis of the learning dynamics under class imbal- ance, in Proceedings of the 40th International Conference on Machine Learning , Proceedings of Machine Learn- ing Research, Vol. 202, edited by A. Krause, E. Brun- s...

2023
[43]

F. S. Pezzicoli, V. Ros, F. P. Landes, and M. Baity-Jesi, Anomaly detection with class imbalance: learning from exactly solvable models, in The 28th International Con- ference on Artificial Intelligence and Statistics (2025)

2025
[44]

Frosst, N

N. Frosst, N. Papernot, and G. Hinton, Analyzing and improving representations with the soft nearest neigh- bor loss, in Proceedings of the 36th International Con- ference on Machine Learning , Proceedings of Machine Learning Research, Vol. 97, edited by K. Chaudhuri and R. Sa...

2019
[45]

He and W

H. He and W. J. Su, A law of data separation in deep learning, Proceedings of the National Academy of Sciences 120, e2221704120 (2023), https://www.pnas.org/doi/pdf/10.1073/pnas.2221704120

2023 doi
[46]

A. Ali, J. Z. Kolter, and R. J. Tibshirani, A continuous- time view of early stopping for least squares regression, in Proceedings of the Twenty-Second International Con- ference on Artificial Intelligence and Statistics , Proceed- ings of Machine Learning Research, Vol. 89, e...

2019
[47]

Fischer, J

K. Fischer, J. Lindner, D. Dahmen, Z. Ringel, M. Kr¨ amer, and M. Helias, Critical feature learning in deep neural networks, in Proceedings of the 41st In- ternational Conference on Machine Learning , Proceed- ings of Machine Learning Research, Vol. 235, edited by R. Salakhutd...

2024
[48]

Rubin, K

N. Rubin, K. Fischer, J. Lindner, I. Seroussi, Z. Ringel, M. Kr¨ amer, and M. Helias, From kernels to features: A multi-scale adaptive theory of feature learning, in Forty- second International Conference on Machine Learning (2025)

2025
[49]

Bordelon and C

B. Bordelon and C. Pehlevan, Self-consistent dynamical 7 field theory of kernel evolution in wide neural networks, in Advances in Neural Information Processing Systems , Vol. 35, edited by S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Curran Associates, Inc.,
[50]

Lauditi, B

C. Lauditi, B. Bordelon, and C. Pehlevan, Adaptive ker- nel predictors from feature-learning infinite limits of neu- ral networks, in Forty-second International Conference on Machine Learning (2025)

2025
[51]

Aguirre-L´ opez, S

F. Aguirre-L´ opez, S. Franz, and M. Pastore, Random fea- tures and polynomial rules, SciPost Phys. 18, 039 (2025)

2025
[52]

Barbier, F

J. Barbier, F. Camilli, M.-T. Nguyen, M. Pastore, and R. Skerk, Optimal generalisation and learning transition in extensive-width shallow neural networks near interpo- lation (2025), arXiv:2501.18530 [stat.ML]

2025 arXiv
[53]

Barbier, F

J. Barbier, F. Camilli, M.-T. Nguyen, M. Pastore, and R. Skerk, Statistical mechanics of extensive-width bayesian neural networks near interpolation (2025), arXiv:2505.24849 [stat.ML]

2025 arXiv
[54]

V. Erba, E. Troiani, L. Zdeborov´ a, and F. Krzakala, The nuclear route: Sharp asymptotics of erm in overpa- rameterized quadratic networks (2025), arXiv:2505.17958 [stat.ML]

2025
[55]

Ingrosso and S

A. Ingrosso and S. Goldt, Data-driven emer- gence of convolutional structure in neu- ral networks, Proceedings of the National Academy of Sciences 119, e2201854119 (2022), https://www.pnas.org/doi/pdf/10.1073/pnas.2201854119

2022 doi
[56]

Q. Li, B. Sorscher, and H. Sompolinsky, Rep- resentations and generalization in artificial and brain neural networks, Proceedings of the National Academy of Sciences 121, e2311805121 (2024), https://www.pnas.org/doi/pdf/10.1073/pnas.2311805121

2024 doi
[57]

Ringel, N

Z. Ringel, N. Rubin, E. Mor, M. Helias, and I. Seroussi, Applications of statistical field theory in deep learning, arXiv preprint arXiv:2502.18553 (2025). 8 SUPPLEMENT AR Y MA TERIAL CONTENTS References 5 Supplementary Material 8 Derivation of the average squared distance 8 D...

2025 arXiv
[58]

(15) As mentioned in the main text, the terms that come from the posterior distribution (which involve y) are clearly distinguished from the ones which come from the relative minus sign of the norm (represented by z). We defined ˜D2 1 = 1 N η−1 1 N 2δ + PX µν zµKµνzν, (16) ˜D2...
[59]

( ˜K −1 (R))µν∆µν hk − X λσ ( ˜K −1 (R))µλyλ(∆µν hk)( ˜K −1 (R))νσ yσ # = = Z Q>0 dQe− N1 2 S(Q) Q λ1N1 X µν

In this case the first term grows slowly, allowing us to consider the two contributions effectively of the same order. This approach cannot be pursued in the case of MNIST dataset restricted to 0 and 1 classes. In this case, the kernel strongly correlates with the labels highl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [31]

J. Lee, S. Schoenholz, J. Pennington, B. Adlam, L. Xiao, R. Novak, and J. Sohl-Dickstein, Finite versus infinite neural networks: an empirical study, in Advances in Neu- ral Information Processing Systems , Vol. 33, edited by H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin (Curran Associates, Inc., 2020) pp. 15156–15172

work page 2020

[2] [25]

A Multiscale integrated ap- proach to the study of the nervous system in health and disease

for finite-width deep linear networks. In the infinite-width limit, the terms Λ1 and Λ2 vanish, the components of w become uncorrelated, and averages over the prior and over the posterior coincide. In the pro- portional case, since the kernel ∆ µν hk cannot be expressed in simple terms by using Kµν, the zero-temperature limit is not useful to investigate ...

work page 2022

[3] [1]

D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning representations by back-propagating errors, Na- ture 323, 533 (1986)

work page 1986

[4] [2]

Krizhevsky, I

A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Commun. ACM 60, 84–90 (2017)

work page 2017

[5] [3]

Krizhevsky and G

A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images , Tech. Rep. 0 (University of Toronto, Toronto, Ontario, 2009)

work page 2009

[6] [4]

Yang and E

G. Yang and E. J. Hu, Tensor programs iv: Feature learning in infinite-width neural networks, in Proceedings of the 38th International Conference on Machine Learn- ing, Proceedings of Machine Learning Research, Vol. 139, edited by M. Meila and T. Zhang (PMLR, 2021) pp. 11727–11737

work page 2021

[7] [5]

Seroussi, G

I. Seroussi, G. Naveh, and Z. Ringel, Separation of scales and a thermodynamic description of feature learning in some cnns, Nature Communications 14, 908 (2023)

work page 2023

[8] [6]

A. X. Yang, M. Robeyns, E. Milsom, N. Schoots, and L. Aitchison, A theory of representation learning in deep neural networks gives a deep generalisation of kernel methods (2023), arXiv:2108.13097 [stat.ML]

work page Pith review arXiv 2023