Pith. sign in

REVIEW 3 major objections 3 minor 58 references

Kernel shape renormalization explains output-output correlations in finite Bayesian one-hidden-layer networks

T0 review · 3 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single matrix Q* predicts how outputs correlate in finite Bayesian networks.

desk verdict A well-tested application of an existing proportional-limit effective action; the headline correlation formula was already in the authors' own conference paper, but the validation and generalization-loss analysis make it a useful reference. read the letter →

arxiv 2412.15911 v1 pith:JGHAAEGD submitted 2024-12-20 cond-mat.dis-nn

P. Baglioni , L. Giambagli , A. Vezzani , R. Burioni , P. Rotondo , R. Pacelli This is my paper · ORCID
classification cond-mat.dis-nn
keywords kernelshaperenormalizationoutput-outputcorrelationsproportionallimitBayesianneuralnetworksone-hidden-layerNNGPreadoutweightoverlapsgeneralizationloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Finite-width Bayesian networks with several outputs show correlations between outputs that the infinite-width description cannot produce; this paper claims those correlations are a genuine finite-load effect, controlled by the same object that renormalizes the kernel: a $D \times D$ matrix $Q^*$ that minimizes a free-energy action in the proportional limit. The central formula is $\langle v_a \cdot v_b\rangle/N_1 = Q^*_{ab}/\lambda_1$, which says the expected overlap of readout weight vectors for outputs $a$ and $b$ is exactly the corresponding entry of the saddle-point matrix divided by the readout prior scale. If the claim is right, a practitioner can compute from the data and the architecture how strongly the outputs of a finite one-hidden-layer Bayesian network are coupled and can watch those correlations vanish as the width grows, since $Q^* \to 1$ in the infinite-width limit. The paper also shows that the same effective theory predicts the generalization loss and that Bayesian training matches gradient-descent training in generalization on standard image datasets, so the correlations are not a sampling artifact.

What carries the argument

The central object is the $D \times D$ order-parameter matrix $Q$ in the effective action $S_{\mathrm{FC}}(Q)$, together with the renormalized kernel $K^{(\mathrm{R})}_Q = Q \otimes K_{\mathrm{NNGP}}$ formed by its tensor product with the infinite-width NNGP kernel. $Q$ is integrated out by a saddle point, so its minimizing value $Q^*$ enters every prediction; it renormalizes the shape of the kernel by mixing the $D$ outputs, something a scalar renormalization cannot do. The load-bearing identity is $\langle v v^T\rangle/N_1 = Q^*/\lambda_1$, obtained by differentiating the partition function with respect to a matrix-valued readout prior; it turns an abstract saddle-point matrix into a direct observable, the overlap of readout weight vectors, and provides the mechanism by which shared hidden features become output-output correlations.

What would settle it

Train a fully thermalized one-hidden-layer Bayesian network with $D=3$ outputs on synthetic Gaussian inputs where two classes are built to share a hidden feature and the third is independent, with $P=N_1=1000$, and measure $\langle v_a \cdot v_b\rangle/N_1$ from Langevin samples; Eq. (4) predicts this equals $Q^*_{ab}/\lambda_1$ with $Q^*$ from minimizing Eq. (1), up to thermodynamic-limit corrections. If the off-diagonal overlap stays near zero, or has the wrong sign, while $\alpha$ is held fixed and the sampler has converged, the Gaussian equivalence behind $Q^*$ fails; alternatively, checking that the discrepancy shrinks as $N_1$ grows at fixed $\alpha$ tests the predicted scaling.

Watch

Extended reading notes

Core claim

The paper establishes, in the proportional thermodynamic limit where $P$ and $N_1$ grow together at fixed $\alpha = P/N_1$, that the posterior statistics of a fully connected one-hidden-layer Bayesian network with $D$ outputs are governed by a renormalized kernel $K^{(\mathrm{R})}_Q(X,X) = Q \otimes K_{\mathrm{NNGP}}(X,X)$, with $Q$ a $D \times D$ matrix order parameter and $K_{\mathrm{NNGP}}$ the infinite-width Gaussian process kernel. The minimizing value $Q^*$ of the effective action in Eq. (1) simultaneously fixes the generalization loss and, through Eq. (4), the output-output correlations: after training, the normalized overlap of the readout weight vectors $v_a$ and $v_b$ equals $Q^*_{ab}/\lambda_1$. Off-diagonal entries of $Q^*$ are therefore not fitting noise but a data-dependent prediction: at finite $\alpha$, outputs whose classes share hidden features acquire systematically negative overlaps (for example, vehicle classes in CIFAR10), and in the infinite-width limit $\alpha \to 0$, $Q^* \to 1$ and the correlations vanish, reproducing the classic result that $D$ outputs decouple into $D$ independent networks. The paper validates Eq. (4) by Langevin sampling on synthetic, MNIST, and CIFAR10 data at $\alpha = 0.1$ and $\alpha = 1$, finding quantitative agreement.

Load-bearing premise

The whole calculation assumes the hidden-layer sums that couple the outputs are jointly Gaussian once $P$ and $N_1$ are large in fixed proportion, and the paper does not bound how far finite networks can stray from that approximation.

Editorial extensions

If this is right

  • At any fixed load $\alpha = P/N_1$, output-output correlations are a thermodynamic prediction of the model, not a finite-size accident; they survive in the proportional limit and vanish only as $Q^*$ approaches the identity.
  • The readout-overlap matrix $\langle v v^T\rangle/N_1$ is a direct experimental estimator of $Q^*/\lambda_1$, so measuring trained readout weights is enough to read off the kernel renormalization.
  • Predictions for new data must use the renormalized kernel $K^{(\mathrm{R})}_{Q^*} = Q^* \otimes K_{\mathrm{NNGP}}$; using the bare infinite-width kernel at finite $\alpha$ misestimates both the generalization loss and the covariance of predictions.
  • Classes that share visual or semantic features are predicted to show systematically negative off-diagonal overlaps, giving a concrete signature of feature sharing that can be checked class by class.
  • Single-output networks cannot exhibit kernel shape renormalization, since with $D=1$ the matrix $Q$ is a scalar and cannot change the kernel's shape; the effect is intrinsically multi-output.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If Eq. (4) also holds for networks trained by gradient descent rather than Langevin sampling, the off-diagonal matrix $Q^*/\lambda_1$ would provide a direct, weight-based diagnostic of feature sharing in practical finite networks; the paper does not test readout correlations under gradient-descent training, only aggregate generalization.
  • The same Gaussian-equivalence route should predict output-output correlations in other finite architectures with collective hidden variables, such as convolutional networks; measuring readout-overlap matrices in finite convolutional networks with shared filters would be a direct test.
  • The paper's observation that theory-data discrepancies shrink as the input dimension $N_0$ grows suggests finite-input corrections beyond $\alpha$; a refined effective theory including $1/N_0$ corrections could make Eq. (4) quantitative at smaller $N_0$.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper studies finite-width Bayesian one-hidden-layer networks with multiple outputs in the proportional limit P,N1→∞ at fixed α=P/N1. Starting from an effective action with a D×D order parameter Q (Eq. (1)), the authors obtain a renormalized NNGP kernel K_Q = Q ⊗ K_NNGP and derive closed-form expressions for the posterior predictive mean and covariance (SM Eqs. (17)–(18)). The central result is Eq. (4), which identifies the normalized readout-overlap matrix ⟨vv^T⟩/N1 with Q*/λ1, where Q* minimizes the effective action. The paper validates this prediction against Langevin-dynamics simulations on synthetic data, MNIST, and CIFAR10 for several values of α and D, and additionally compares infinite-width Bayesian predictions with Adam-trained networks. The numerical agreement reported in Figs. 1, 2, and SM Fig. 5 is the main empirical support for the claim that kernel shape renormalization quantitatively explains output-output correlations.

Significance. If the central derivation is sound, Eq. (4) is a valuable quantitative bridge between statistical-mechanics order parameters and observable weight correlations in finite Bayesian networks. The paper provides a simple physical interpretation of negative off-diagonal overlaps (e.g., confusable CIFAR10 classes) and extends the recent proportional-limit program for Bayesian deep learning to multi-output architectures. The Monte Carlo validation is carried out carefully, with blocking error estimates, thermalization checks, and multiple datasets, and the comparison with Adam is a useful practical benchmark. However, the theoretical result rests on a Gaussian equivalence whose asymptotic regime is not fully specified; in particular, the role of the input dimension N0 is not included in the stated proportional limit, and the manuscript itself provides evidence (SM Fig. 4(c)) that N0 is a controlling parameter. Until this gap is closed or explicitly acknowledged as a controlled approximation, the quantitative agreement in Figs. 2 and 5 is evidence for a useful heuristic but not for the theorem stated in the text.

major comments (3)
  1. [SM Eqs. (10)-(12) and main-text proportional limit] The Gaussian equivalence asserted in SM Eq. (11) is not justified in the stated limit. The variables h_μ have covariance C, and for fixed input dimension N0 this covariance has rank at most N0. When N0 < P+1, which is the case in the experiments (P=1000 with N0=784, 144, or 122), det C and C^{-1} in SM Eq. (12) are undefined, and the density entering the Gaussian expectation is not well-defined. The cited Breuer-Major type theorems do not apply to sums over a fixed-rank Gaussian vector. The main text defines the proportional limit as P,N1→∞ at fixed α, without any growth condition on N0; consequently the effective action (1) and the key result (4) are not derived in the stated limit. Please specify a joint limit N0,P,N1→∞ with a rate condition and provide error bounds, or explicitly reformulate the results as a finite-N0 heuristic approximation whose accuracy must be tested separately.
  2. [SM Fig. 4(c) and Fig. 1(b)] SM Fig. 4(c) reports that the relative discrepancy Δ between theory and simulation decreases when N0 increases at fixed P and N1. This is direct evidence that N0 is an essential control parameter of the Gaussian equivalence, and it corroborates the concern raised above. The main text currently presents the agreement in Figs. 1 and 2 as supporting the stated proportional-limit theory, but all experiments use a single fixed N0 per dataset. To make the claim quantitative, the manuscript should include a systematic study varying N0 (e.g., through random projections of MNIST at several N0 values) showing convergence toward the predictions as N0 grows, or explicitly limit the claims to a heuristic regime. As written, the experimental section does not test the limit that the theory invokes.
  3. [SM Sec. II, derivation of Eq. (29)] The overlap formula ⟨va·vb⟩/N1 = Q*_ab/λ1 is derived from the modified partition function in SM Eq. (22), whose saddle-point reduction again relies on the same Gaussian equivalence used for Eq. (1). Thus the central prediction (main-text Eq. (4)) inherits all of the limitations of SM Eq. (11). The paper should state this dependency explicitly; it is not merely a numerical nuisance but a logical link in the derivation.
minor comments (3)
  1. [Fig. 1 caption] The caption says 'N0 = 122' for the random Gaussian data, but SM Sec. III A specifies N0 = 12^2 = 144 for the synthetic dataset. Please correct the typo and make the notation consistent throughout.
  2. [References and text] The dataset name 'CIF AR10' and 'Imagenette' appear with inconsistent spacing; also, Ref. [43] ('A smaller subset of 10 easily classified classes from imagenet') lacks author and year information. These should be cleaned up for a final version.
  3. [SM Sec. IV, Eq. (41)] The perturbative expansion Q = Q(0) + α1 Q(1) + ... is used to produce the dashed 'heuristic proportional th.' curves in Fig. 1, but the expansion parameter α1 is not small in the presented experiments (α up to 1 for N1=1000). The text should note that this is a formal expansion and that its range of validity is not assessed.

Circularity Check

1 steps flagged · score 2.0 of 10

No prediction reduces to a fit: Q* is obtained by minimizing the effective action, not by matching correlations, and Eq. (4) is validated against exact Langevin sampling of the Gibbs posterior. The only caveat is a low-severity cluster of self-citations (Refs. [26, 40, 53]) supplying the theoretical framework, which is mitigated by independent numerical tests.

  1. self citation load bearing [Main text, 'Effective action for shallow FC networks' (Eq. (1) and Refs. [26, 37-42]); SM Sec. II around Eq. (29) and Ref. [53].]
    "Denoting y the vectorized matrix Y , and restricting to odd activation functions, the action is given by [26]: ... For the non-linear case, the derivation leverages a Gaussian equivalence [37–40] informally justified via a class of generalized central limit theorems [41, 42]. ... This result has a natural interpretation in terms of Gaussian Processes, as was shown in [53]."

    The paper's entire theoretical scaffold is imported from prior work with overlapping authorship: the effective action (Eq. (1)) from Ref. [26] (co-authored by Pacelli and Rotondo), the Gaussian-equivalence justification from Refs. [37-40] (including the authors' own Ref. [40]), and the Gaussian-process reading of the headline overlap identity from Ref. [53] (authored by Pacelli, Giambagli and Baglioni, the first three authors of this paper). The central 'prediction' Eq. (4) is thus announced as previously shown in the authors' own Ref. [53], and all generalization-loss predictions flow from the self-cited action. This is the nearest thing to a circular step in the paper, but it is not vicious: the SM re-derives the overlap identity from the modified partition function (Eqs.

full rationale

The claimed derivation chain is Eq. (1) (effective action) -> saddle-point matrix Q* -> renormalized kernel (Eq. (2)) -> predictions for generalization loss (Eq. (3)) and weight-overlap correlations (Eq. (4)). There is no reduction-by-construction: Q* is defined as the minimizer of the action S_FC(Q), which depends on the data, the NNGP kernel, and hyperparameters (lambda0 = lambda1 = 1, T = 0.01), and no free parameter is fitted to the correlation observable. The overlap prediction <v_a . v_b>/N1 = Q*_ab/lambda1 is derived in the SM via the source-field identity (Eq. (21)-(29)) and checked against independent Langevin MC that samples the exact Gibbs posterior; the agreement shown in Fig. 2 and SM Fig. 5 (including the strongly negative vehicle-class overlaps) is therefore a genuine, externally falsifiable test. The enumerated patterns are otherwise absent: no uniqueness theorem is imported, no known result is merely renamed (the output-output correlation phenomenon is explicitly attributed to Neal and MacKay), and the Gaussian equivalence (SM Eq. (11)) is openly declared heuristic and anchored in external Breuer-Major literature [41, 50, 51], making the fixed-N0 concern (SM Fig. 4(c) shows discrepancies decreasing with N0) a correctness risk rather than a circularity. The paper also explicitly flags its own limitations: uncontrolled finite-width effects at small N1 (Discussion), finite-input-dimension deviations (SM Fig. 4(c)), and inapplicability for N0 << P (SM Sec. V.B). The remaining issue is the density of self-citations ([26], [29], [40], [53]) that supply the theoretical framework; but because the central numbers are validated by parameter-free Gibbs sampling, the self-citation is not load-bearing in the epistemic sense. Overall: no prediction is equivalent to its inputs by construction; score 2 reflects the self-citation cluster rather than any fitted-input circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central predictions rest on the previously derived effective action (Ref. [26]) and on a Gaussian equivalence assumption. The only hand-set constants are the prior variances λ0, λ1 and temperature T, all fixed in the experiments. No new physical entities are introduced; the order parameter Q* is a mathematical quantity from the cited framework and is not a new particle, force, or ledger entry.

free parameters (3)
  • λ0 (input-layer prior variance / L2 strength) = 1.0
    Set by hand in all Bayesian experiments; the predictions of Q* and Eq. (4) depend on this scale together with λ1 and T.
  • λ1 (readout prior variance / L2 strength) = 1.0
    Set by hand; appears explicitly in the overlap formula ⟨v_a·v_b⟩/N1 = Q*_ab/λ1.
  • T (temperature) = 0.01
    Set by hand; inverse temperature enters the effective action and the equilibrium posterior, and small T approximates the noiseless regression limit.
assumptions (4)
  • domain assumption Gaussian equivalence for the hidden-layer collective variables: P({q}) converges to a Gaussian in the proportional limit (SM Eq. (11)).
    Invoked to compute the posterior statistics; justified heuristically by Breuer-Major CLTs, not proved for finite α and N0.
  • domain assumption The saddle-point approximation for the partition function integral over Q is exact in the N1 to infinity proportional limit (main Eq. (1) and SM).
    Standard in this statistical mechanics framework; finite-N1 corrections are not controlled, as acknowledged in the Discussion.
  • domain assumption Activation functions are odd (main text, before Eq. (1)); Erf is used in Bayesian experiments.
    The effective action derivation restricts to odd activations; the Adam experiments with ReLU fall outside the Bayesian theory and are compared only to infinite-width predictions.
  • standard math Ingham's formula for the Fourier transform of a Wishart determinant (SM Eq. (14)).
    Used to introduce the matrix order parameter Q; a classical mathematical identity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Kernel shape renormalization explains output-output correlations in finite Bayesian one-hidden-layer networks." pith.science (2026). https://pith.science/paper/JGHAAEGD

@misc{pith2026241215911,
  author       = {Pith},
  title        = {Pith review of: Kernel shape renormalization explains output-output correlations in finite Bayesian one-hidden-layer networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JGHAAEGD}},
  note         = {Machine review of arXiv:2412.15911}
}
abstract

Finite-width one hidden layer networks with multiple neurons in the readout layer display non-trivial output-output correlations that vanish in the lazy-training infinite-width limit. In this manuscript we leverage recent progress in the proportional limit of Bayesian deep learning (that is the limit where the size of the training set $P$ and the width of the hidden layers $N$ are taken to infinity keeping their ratio $\alpha = P/N$ finite) to rationalize this empirical evidence. In particular, we show that output-output correlations in finite fully-connected networks are taken into account by a kernel shape renormalization of the infinite-width NNGP kernel, which naturally arises in the proportional limit. We perform accurate numerical experiments both to assess the predictive power of the Bayesian framework in terms of generalization, and to quantify output-output correlations in finite-width networks. By quantitatively matching our predictions with the observed correlations, we provide additional evidence that kernel shape renormalization is instrumental to explain the phenomenology observed in finite Bayesian one hidden layer networks.

Figures

Figures reproduced from arXiv: 2412.15911 by the authors.

Figure 1
Figure 1. FIG. 1 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. FIG. 2 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. FIG. 3 [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: FIG. 4 [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: FIG. 5 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: FIG. 6 [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: FIG. 7 [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 47 canonical work pages

  1. [53]

    Kernel shape renormalization in bayesian shallow networks: a gaussian process perspective,

    Rosalba Pacelli, Lorenzo Giambagli, and Paolo Baglioni, “Kernel shape renormalization in bayesian shallow networks: a gaussian process perspective,” in 2024 IEEE Workshop on Complexity in Engineering (COMPENG) (2024) pp. 1–6

    work page 2024

  2. [26]

    A statistical mechanics framework for bayesian deep neural networks beyond the infinite-width limit,

    R. Pacelli, S. Ariosto, M. Pastore, F. Ginelli, M. Gherardi, and P. Rotondo, “A statistical mechanics framework for bayesian deep neural networks beyond the infinite-width limit,” Nature Machine Intelligence 5, 1497–1507 (2023)

    work page 2023

  3. [1]

    Priors for infinite networks,

    Radford M. Neal, “Priors for infinite networks,” in Bayesian Learning for Neural Networks (Springer New York, New York, NY, 1996) pp. 29–53

    work page 1996

  4. [2]

    Introduction to gaussian processes,

    David JC MacKay, “Introduction to gaussian processes,” NATO ASI Series F Computer and Systems Sciences 168, 133–166 (1998)

    work page 1998

  5. [3]

    Computing with infinite networks,

    Christopher Williams, “Computing with infinite networks,” in Advances in Neural Information Processing Systems, Vol. 9, edited by M.C. Mozer, M. Jordan, and T. Petsche (MIT Press, 1996)

    work page 1996

  6. [4]

    Gaussian process behaviour in wide deep neural networks,

    Alexander G. de G. Matthews, Jiri Hron, Mark Rowland, Richard E. Turner, and Zoubin Ghahramani, “Gaussian process behaviour in wide deep neural networks,” in International Conference on Learning Representations (2018)

    work page 2018

  7. [5]

    Deep neural networks as gaussian processes,

    Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri, “Deep neural networks as gaussian processes,” in International Conference on Learning Representations (2018)

    work page 2018

  8. [6]

    On exact computation with an infinitely wide neural net,

    Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang, “On exact computation with an infinitely wide neural net,” in Advances in Neural Information Processing Systems, Vol. 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´ e-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., 2019)

    work page 2019

Show all 58 references
  1. [7]

    On lazy training in differentiable programming,

    L´ ena ¨ ıc Chizat, Edouard Oyallon, and Francis Bach, “On lazy training in differentiable programming,” inAdvances in Neural Information Processing Systems , Vol. 32, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d 'Alch´ e-Buc, E. Fox, and R. Garnett (Curran Associ...

    2019

  2. [8]

    Why bigger is not always better: on finite and infinite neural networks,

    Laurence Aitchison, “Why bigger is not always better: on finite and infinite neural networks,” in Proceedings of the 37th International Conference on Machine Learning , Proceedings of Machine Learning Research, Vol. 119, edited by Hal Daum´ e III and Aarti Singh (PMLR, 2020) p...

    2020

  3. [9]

    Quantitative clts in deep neural networks,

    Stefano Favaro, Boris Hanin, Domenico Marinucci, Ivan Nourdin, and Giovanni Peccati, “Quantitative clts in deep neural networks,” arXiv preprint arXiv:2307.06092 (2023)

    2023 arXiv

  4. [10]

    Deep convolutional networks as shallow gaussian processes,

    Adri` a Garriga-Alonso, Carl Edward Rasmussen, and Laurence Aitchison, “Deep convolutional networks as shallow gaussian processes,” in International Conference on Learning Representations (2019)

    2019

  5. [11]

    Bayesian deep convolutional networks with many channels are gaussian processes,

    Roman Novak, Lechao Xiao, Yasaman Bahri, Jaehoon Lee, Greg Yang, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein, “Bayesian deep convolutional networks with many channels are gaussian processes,” in International Conference on Learning Representations (2019)

    2019

  6. [12]

    Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes,

    Greg Yang, “Tensor programs i: Wide feedforward or recurrent neural networks of any architecture are gaussian processes,” in Neural Information Processing Systems (2019)

    2019

  7. [13]

    Feature learning in deep neural networks-studies on speech recognition tasks,

    Dong Yu, Michael L Seltzer, Jinyu Li, Jui-Ting Huang, and Frank Seide, “Feature learning in deep neural networks-studies on speech recognition tasks,” arXiv preprint arXiv:1301.3605 (2013)

    2013 arXiv

  8. [14]

    Learning sparse features can lead to overfitting in neural networks,

    Leonardo Petrini, Francesco Cagnetta, Eric Vanden- Eijnden, and Matthieu Wyart, “Learning sparse features can lead to overfitting in neural networks,” in Advances in Neural Information Processing Systems , edited by Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun ...

    2022

  9. [15]

    Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalization,

    Simone Ciceri, Lorenzo Cassani, Matteo Osella, Pietro Rotondo, Filippo Valle, and Marco Gherardi, “Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalization,” Nature Machine Intelligence 6, 40–47 (2024)

    2024

  10. [16]

    A spring- block theory of feature learning in deep neural networks,

    Cheng Shi, Liming Pan, and Ivan Dokmani´ c, “A spring- block theory of feature learning in deep neural networks,” arXiv preprint arXiv:2407.19353 (2024)

    2024 arXiv

  11. [17]

    A mean field view of the landscape of two- layer neural networks,

    Song Mei, Andrea Montanari, and Phan-Minh Nguyen, “A mean field view of the landscape of two- layer neural networks,” Proceedings of the National Academy of Sciences 115, E7665–E7671 (2018), https://www.pnas.org/doi/pdf/10.1073/pnas.1806579115

    2018 doi

  12. [18]

    On the global convergence of gradient descent for over-parameterized models using optimal transport,

    L´ ena ¨ ıc Chizat and Francis Bach, “On the global convergence of gradient descent for over-parameterized models using optimal transport,” in Advances in Neural Information Processing Systems , Vol. 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianc...

    2018

  13. [19]

    Mean field analysis of neural networks: A law of large numbers,

    Justin Sirignano and Konstantinos Spiliopoulos, “Mean field analysis of neural networks: A law of large numbers,” SIAM Journal on Applied Mathematics 80, 6 725–752 (2020), https://doi.org/10.1137/18M1192184

    2020 doi

  14. [20]

    Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks,

    Grant Rotskoff and Eric Vanden-Eijnden, “Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks,” inAdvances in Neural Information Processing Systems , Vol. 31, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. C...

    2018

  15. [21]

    Tensor programs iv: Feature learning in infinite-width neural networks,

    Greg Yang and Edward J. Hu, “Tensor programs iv: Feature learning in infinite-width neural networks,” in Proceedings of the 38th International Conference on Machine Learning , Proceedings of Machine Learning Research, Vol. 139, edited by Marina Meila and Tong Zhang (PMLR, 2021...

    2021

  16. [22]

    What can be learnt with wide convolutional neural networks?

    Francesco Cagnetta, Alessandro Favero, and Matthieu Wyart, “What can be learnt with wide convolutional neural networks?” (2023)

    2023

  17. [23]

    Locality defeats the curse of dimensionality in convolutional teacher-student scenarios,

    Alessandro Favero, Francesco Cagnetta, and Matthieu Wyart, “Locality defeats the curse of dimensionality in convolutional teacher-student scenarios,” in Advances in Neural Information Processing Systems , edited by A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (2021)

    2021

  18. [24]

    A theory of representation learning in deep neural networks gives a deep generalisation of kernel methods,

    Adam X. Yang, Maxime Robeyns, Edward Milsom, Nandi Schoots, and Laurence Aitchison, “A theory of representation learning in deep neural networks gives a deep generalisation of kernel methods,” (2023), arXiv:2108.13097 [stat.ML]

    2023 arXiv

  19. [25]

    Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization,

    Qianyi Li and Haim Sompolinsky, “Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization,” Phys. Rev. X11, 031059 (2021)

    2021

  20. [27]

    Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks,

    R. Aiudi, R. Pacelli, A. Vezzani, R. Burioni, and P. Rotondo, “Local Kernel Renormalization as a mechanism for feature learning in overparametrized Convolutional Neural Networks,” arXiv e-prints , arXiv:2307.11807 (2023), arXiv:2307.11807 [cs.LG]

    2023 arXiv

  21. [28]

    Statistical mechanics of transfer learning in fully-connected networks in the proportional limit,

    Alessandro Ingrosso, Rosalba Pacelli, Pietro Rotondo, and Federica Gerace, “Statistical mechanics of transfer learning in fully-connected networks in the proportional limit,” (2024), arXiv:2407.07168 [cond-mat.dis-nn]

    2024 arXiv

  22. [29]

    Predictive power of a bayesian effective action for fully connected one hidden layer neural networks in the proportional limit,

    P. Baglioni, R. Pacelli, R. Aiudi, F. Di Renzo, A. Vezzani, R. Burioni, and P. Rotondo, “Predictive power of a bayesian effective action for fully connected one hidden layer neural networks in the proportional limit,” Phys. Rev. Lett. 133, 027301 (2024)

    2024

  23. [30]

    Critical feature learning in deep neural networks,

    Kirsten Fischer, Javed Lindner, David Dahmen, Zohar Ringel, Michael Kr¨ amer, and Moritz Helias, “Critical feature learning in deep neural networks,” in Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, ed...

    2024

  24. [31]

    High-dimensional learning of narrow neural networks,

    Hugo Cui, “High-dimensional learning of narrow neural networks,” (2024), arXiv:2409.13904 [stat.ML]

    2024 arXiv

  25. [32]

    Fundamental limits of overparametrized shallow neural networks for supervised learning,

    Francesco Camilli, Daria Tieplova, and Jean Barbier, “Fundamental limits of overparametrized shallow neural networks for supervised learning,” arXiv preprint arXiv:2307.05635 (2023)

    2023 arXiv

  26. [33]

    Dissecting the interplay of attention paths in a statistical mechanics theory of transformers,

    Lorenzo Tiberi, Francesca Mignacco, Kazuki Irie, and Haim Sompolinsky, “Dissecting the interplay of attention paths in a statistical mechanics theory of transformers,” arXiv preprint arXiv:2405.15926 (2024)

    2024 arXiv

  27. [34]

    Adam: A method for stochastic optimization,

    Diederik P Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 (2014)

    2014 arXiv

  28. [35]

    Tune: A research platform for distributed model selection and training,

    Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica, “Tune: A research platform for distributed model selection and training,” arXiv preprint arXiv:1807.05118 (2018)

    2018 arXiv

  29. [36]

    Feature learning in finite-width bayesian deep linear networks with multiple outputs and convolutional layers,

    Federico Bassetti, Marco Gherardi, Alessandro Ingrosso, Mauro Pastore, and Pietro Rotondo, “Feature learning in finite-width bayesian deep linear networks with multiple outputs and convolutional layers,” (2024), arXiv e-prints:2406.03260 [stat.ML]

    2024 arXiv

  30. [37]

    The generalization error of random features regression: Precise asymptotics and the double descent curve,

    Song Mei and Andrea Montanari, “The generalization error of random features regression: Precise asymptotics and the double descent curve,” Communications on Pure and Applied Mathematics (2019)

    2019

  31. [38]

    Generalisation error in learning with random features and the hidden manifold model,

    Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc M´ ezard, and Lenka Zdeborov´ a, “Generalisation error in learning with random features and the hidden manifold model,” Journal of Statistical Mechanics: Theory and Experiment 2021, 124013 (2021)

    2021

  32. [39]

    Random features and polynomial rules,

    Fabi´ an Aguirre-L´ opez, Silvio Franz, and Mauro Pastore, “Random features and polynomial rules,” https://synthical.com/article/ 0a46daf2-150f-45d9-83f7-6bb4ecf92e3d (2024), arXiv:2402.10164 [cond-mat.dis-nn]

    2024 arXiv

  33. [40]

    Universal mean-field upper bound for the generalization gap of deep neural networks,

    S. Ariosto, R. Pacelli, F. Ginelli, M. Gherardi, and P. Rotondo, “Universal mean-field upper bound for the generalization gap of deep neural networks,” Phys. Rev. E 105, 064309 (2022)

    2022

  34. [41]

    Central limit theorems for non-linear functionals of gaussian fields,

    P´ eter Breuer and P´ eter Major, “Central limit theorems for non-linear functionals of gaussian fields,” Journal of Multivariate Analysis 13, 425–441 (1983)

    1983

  35. [42]

    Quantitative Breuer-Major theorems,

    Ivan Nourdin, Giovanni Peccati, and Mark Podolskij, “Quantitative Breuer-Major theorems,” (2010)

    2010

  36. [43]

    A smaller subset of 10 easily classified classes from imagenet,

    Jeremy et al Howard, “A smaller subset of 10 easily classified classes from imagenet,”

  37. [44]

    Finite versus infinite neural networks: an empirical study,

    Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein, “Finite versus infinite neural networks: an empirical study,” in Advances in Neural Information Processing Systems , Vol. 33, edited by H. Larochelle, M. Ranzato...

    2020

  38. [45]

    Representations and generalization in artificial and brain neural networks,

    Qianyi Li, Ben Sorscher, and Haim Sompolinsky, “Representations and generalization in artificial and brain neural networks,” Proceedings of the National Academy of Sciences 121, e2311805121 (2024), https://www.pnas.org/doi/pdf/10.1073/pnas.2311805121

    2024 doi

  39. [46]

    A self consistent theory of gaussian processes captures feature learning effects in finite cnns,

    Gadi Naveh and Zohar Ringel, “A self consistent theory of gaussian processes captures feature learning effects in finite cnns,” in Advances in Neural Information Processing Systems , Vol. 34, edited by M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan ...

    2021

  40. [47]

    Separation of scales and a thermodynamic description of feature learning in some cnns,

    Inbar Seroussi, Gadi Naveh, and Zohar Ringel, “Separation of scales and a thermodynamic description of feature learning in some cnns,” Nature Communications 7 14, 908 (2023)

    2023

  41. [48]

    Bayesian inference with deep weakly nonlinear networks,

    Boris Hanin and Alexander Zlokapa, “Bayesian inference with deep weakly nonlinear networks,” arXiv preprint arXiv:2405.16630 (2024)

    2024 arXiv

  42. [49]

    Data- driven emergence of convolutional structure in neural networks,

    Alessandro Ingrosso and Sebastian Goldt, “Data- driven emergence of convolutional structure in neural networks,” Proceedings of the National Academy of Sciences 119, e2201854119 (2022), https://www.pnas.org/doi/pdf/10.1073/pnas.2201854119

    2022 doi

  43. [50]

    Moment bounds and central limit theorems for gaussian subordinated arrays,

    Jean-Marc Bardet and Donatas Surgailis, “Moment bounds and central limit theorems for gaussian subordinated arrays,” Journal of Multivariate Analysis 114, 457–473 (2013)

    2013

  44. [51]

    Multivariate normal approximation using stein’s method and malliavin calculus,

    Ivan Nourdin, Giovanni Peccati, and Anthony R´ eveillac, “Multivariate normal approximation using stein’s method and malliavin calculus,” (2008)

    2008

  45. [52]

    An integral which occurs in statistics,

    A. E. Ingham, “An integral which occurs in statistics,” Mathematical Proceedings of the Cambridge Philosophical Society 29, 271–276 (1933)

    1933

  46. [54]

    Adrian Barbu and Song-Chun Zhu, Monte Carlo Methods (Springer Singapore, 2020)

    2020

  47. [55]

    Heermann, Monte Carlo Simulation in Statistical Physics (Springer Berlin Heidelberg, 2002)

    Kurt Binder and Dieter W. Heermann, Monte Carlo Simulation in Statistical Physics (Springer Berlin Heidelberg, 2002)

    2002

  48. [56]

    Monte carlo errors with less errors,

    Ulli Wolff, “Monte carlo errors with less errors,” Computer Physics Communications 156, 143–153 (2004)

    2004

  49. [57]

    EfficientNet: Rethinking model scaling for convolutional neural networks,

    Mingxing Tan and Quoc Le, “EfficientNet: Rethinking model scaling for convolutional neural networks,” in Proceedings of the 36th International Conference on Machine Learning , Proceedings of Machine Learning Research, Vol. 97, edited by Kamalika Chaudhuri and Ruslan Salakhutdi...

    2019

  50. [58]

    Kernel shape renormalization explains output-output correlations in finite Bayesian one-hidden layer networks

    Alex Krizhevsky and Geoffrey Hinton, Learning multiple layers of features from tiny images , Tech. Rep. 0 (University of Toronto, Toronto, Ontario, 2009). 8 Supplemental Material for “Kernel shape renormalization explains output-output correlations in finite Bayesian one-hidde...

    2009

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.