Pith. sign in

REVIEW 4 cited by

For a high-dimensional Gaussian sequence model, empirical risk minimization in single-head tied attention has exactly computable test error, interpolation and recovery thresholds, and a singular-value spectrum that becomes low-rank under weight decay.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 13:52 UTC pith:XZDXCK6W

arxiv 2509.24914 v2 pith:XZDXCK6W submitted 2025-09-29 stat.ML cond-mat.dis-nncs.ITcs.LGmath.IT

Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws

Fabrizio Boncoraglio , Vittorio Erba , Emanuele Troiani , Yizhou Xu , Florent Krzakala , Lenka Zdeborov\'a This is my paper
classification stat.ML cond-mat.dis-nncs.ITcs.LGmath.IT
keywords spectraltheorytrainedattentiongeneralizationhigh-dimensionalincludingisolated
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper studies a stripped-down version of an attention layer, the piece of a transformer that lets each token in a sequence look at the others. In this version, there is a single attention head, the key and query matrices are the same, and the value matrix is the identity. The authors generate synthetic training data from a known random target attention matrix and ask how well a learner can recover it from n examples, in the limit where the embedding dimension d is large and the number of examples grows like d squared.

The main output is a set of exact formulas for the test error as a function of the sample size, the location of thresholds where the model first interpolates the training data and then generalizes perfectly, and the distribution of singular values of the learned weight matrix. The formulas come from random matrix theory and a message-passing algorithm called AMP. A key finding is that weight decay on the key and query matrices is equivalent to a nuclear-norm penalty on their product, which automatically pushes the learned attention matrix to be low rank. This explains why overparameterized attention can still generalize: the regularization kills the unnecessary directions.

The theory is tested against gradient descent simulations on small systems (d=100) and matches well. One caveat: the central result is presented as a claim with a proof sketch, and several technical steps, including Gaussian universality for multiple tokens and the validity of the replica calculation, are assumed rather than proved. The abstract also promises a result about power-law scaling laws that the body of the paper explicitly leaves for future work.

Core claim

Claim 1 asserts that in the high-dimensional limit (d,n,m -> infinity, alpha=n/d^2=O(1), kappa=m/d=O(1)), any global minimizer W of the ERM loss in Eq.(2) has test error and training loss given exactly by the order parameters (Sigma, m, q, ...) at the global minimum of the variational functional Phi in Eq.(7), and that the empirical singular-value density of W/(md)^(1/4) converges to the shifted, cropped law in Eq.(13), under the replicon condition (Eq.10). If correct, this provides a complete asymptotic characterization of ERM in single-head tied attention.

Load-bearing premise

The derivation of Claim 1 relies on two unproven technical extensions to the multi-token softmax setting: (i) Gaussian universality, replacing the data matrices X_ab(x^mu) by independent GOE matrices (Appendix B, 'Gaussian universality'), and (ii) tightness of the AMP state evolution and absence of replica symmetry breaking for the non-convex loss (Appendix B, 'Analysis of the final ERM problem'). The paper states these 'should not pose any roadblock' and 'would require substantial additional work' (Section 3, Sketch of proof), but provides no proof. Also, the replicon condition (Eq.10) is assumed, with only empirical evidence from AMP convergence. If any of these fail, the predicted test-error curves and spectral law may not describe the actual global minima.

Share X LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The central prediction rests on the Gaussian data model and on three unproven technical extensions of existing results to the multi-token softmax setting. No free parameters are fitted: all model parameters (alpha, kappa, lambda, Delta, T, beta) are inputs, and simulations use the same values. No new entities are introduced.

axioms (4)
  • domain assumption Input tokens are i.i.d. Gaussian, x_in,a ~ N(0, I_d), and the limit d,n,m -> infinity is taken with alpha=n/d^2, kappa=m/d, kappa0=m0/d all O(1).
    Assumed in Section 2 to make the model analyzable; the authors argue universality extends beyond Gaussian, but do not prove it in this paper.
  • domain assumption Gaussian universality replaces each data matrix X_ab(x^mu) by an independent GOE(d) matrix without changing the asymptotics for T>1.
    Stated in Appendix B as 'a priori non-trivial'; the proof would require generalizing [48], [41], [28] to multiple tokens, which is not provided. The paper says 'We do not foresee any technical roadblock here'.
  • ad hoc to paper The replicon condition (Eq.10) is satisfied at the global minimum.
    Assumed in Claim 1. The paper provides only empirical evidence (AMP converged in experiments) and says a general proof is difficult.
  • domain assumption The AMP state evolution and the replica formula from [35] extend to the multi-token softmax loss; the global minimum is selected by the AMP fixed point with lowest loss.
    Required in Appendix B to complete the proof of Claim 1. The extension to multiple tokens is argued to be straightforward ('should not pose any roadblock') but not written down.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws." pith.science (2026). https://pith.science/paper/XZDXCK6W

@misc{pith2026250924914,
  author       = {Pith},
  title        = {Pith review of: Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XZDXCK6W}},
  note         = {Machine review of arXiv:2509.24914}
}
Share X LinkedIn Reddit HN
read the original abstract

Trained attention layers exhibit striking and reproducible spectral structure of the weights, including low-rank collapse, bulk deformation, and isolated spectral outliers, yet the origin of these phenomena and their implications for generalization remain poorly understood. We study empirical risk minimization in a single-head tied-attention layer trained on synthetic high-dimensional sequence tasks generated from the attention-indexed model. Using tools from random matrix theory, spin-glass theory, and approximate message passing, we obtain an exact high-dimensional characterization of training and test error, interpolation and recovery thresholds, and the spectrum of the key and query matrices. Our theory predicts the full singular-value distribution of the trained query-key map, including low-rank structure and isolated spectral outliers, in qualitative agreement with observations in more realistic transformers. Finally, for targets with power-law spectra, we show that learning proceeds through sequential spectral recovery, leading to the emergence of power-law scaling laws.

Figures

Figures reproduced from arXiv: 2509.24914 by Emanuele Troiani, Fabrizio Boncoraglio, Florent Krzakala, Lenka Zdeborov\'a, Vittorio Erba, Yizhou Xu.

Figure 1
Figure 1. Figure 1: (Bottom left) Test error of the ERM estimator Eq. [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Test error (left) and training loss (right) as functions of the number of samples [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (Left) Comparison between the standard factorized key and query training Eq. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison between the seq2seq model and the seq2lab model, both in Eq. [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: We show the test error for the model in Eq. [PITH_FULL_IMAGE:figures/full_fig_p028_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: We show the test error computed through Adam for the model in Eq. [PITH_FULL_IMAGE:figures/full_fig_p028_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

    cs.LG 2026-05 unverdicted novelty 8.0

    Neural LoFi models deep learning as layer-wise spectral filtering that selects maximal low-degree correlations, yielding a tractable surrogate for hierarchical representation learning beyond the lazy regime.

  2. High-Dimensional Theory of LoRA Fine-Tuning in a Solvable Attention Model

    cs.LG 2026-06 unverdicted novelty 7.0

    In a solvable attention model, pre-training followed by rank-one LoRA admits sharp asymptotic predictions for test errors and representation alignment via an effective noise term.

  3. How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Models

    stat.ML 2026-05 conditional novelty 7.0

    Attention pooling produces a free-multiplicative-convolution bulk spectrum and two phase transitions for signal recovery; optimal weights are the top eigenvector of the positional correlation matrix R.

  4. How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks

    cs.LG 2026-06 unverdicted novelty 5.0

    Quadratic two-layer networks exhibit data-dependent power-law generalization scaling with distinct regimes in width and sample size, including an interpolation transition whose location depends on target spectrum.

Reference graph

Works this paper leans on

57 extracted references · 6 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Attention is all you need.Advances in neural information processing systems, 30(1): 261–272, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, et al. Attention is all you need.Advances in neural information processing systems, 30(1): 261–272, 2017

    2017

  2. [2]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    2019

  3. [3]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Pith/arXiv arXiv 2010

  4. [4]

    Lan- guage models are few-shot learners.Advances in neural information processing systems, 33: 1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners.Advances in neural information processing systems, 33: 1877–1901, 2020

    1901

  5. [5]

    Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data.Nature Communications, 12(1):4122, 2021

    Charles H Martin, Tongsu Peng, and Michael W Mahoney. Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data.Nature Communications, 12(1):4122, 2021

    2021

  6. [6]

    Small singular values matter: A random matrix analysis of transformer models.arXiv preprint arXiv:2410.17770, 2024

    Max Staats, Matthias Thamm, and Bernd Rosenow. Small singular values matter: A random matrix analysis of transformer models.arXiv preprint arXiv:2410.17770, 2024. 11

    arXiv 2024

  7. [7]

    Traditional and heavy tailed self regularization in neural network models

    Michael Mahoney and Charles Martin. Traditional and heavy tailed self regularization in neural network models. InInternational Conference on Machine Learning, pages 4284–4293. PMLR, 2019

    2019

  8. [8]

    Weight spectra induced efficient model adaptation.arXiv preprint arXiv:2505.23099, 2025

    Chongjie Si, Xuankun Yang, Muqing Liu, Yadao Wang, Xiaokang Yang, Wenbo Su, Bo Zheng, and Wei Shen. Weight spectra induced efficient model adaptation.arXiv preprint arXiv:2505.23099, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Approaching deep learning through the spectral dynamics of weights.arXiv preprint arXiv:2408.11804, 2024

    David Yunis, Kumar Kshitij Patel, Samuel Wheeler, Pedro Savarese, Gal Vardi, Karen Livescu, Michael Maire, and Matthew R Walter. Approaching deep learning through the spectral dynamics of weights.arXiv preprint arXiv:2408.11804, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  11. [11]

    Weight decay induces low-rank attention layers.Advances in Neural Information Processing Systems, 37:4481–4510, 2024

    Seijin Kobayashi, Yassir Akram, and Johannes Von Oswald. Weight decay induces low-rank attention layers.Advances in Neural Information Processing Systems, 37:4481–4510, 2024

    2024

  12. [12]

    Implicit regularization in matrix factorization.Advances in neural information processing systems, 30, 2017

    Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization.Advances in neural information processing systems, 30, 2017

    2017

  13. [13]

    Characterizing implicit bias in terms of optimization geometry

    Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. InInternational Conference on Machine Learning, pages 1832–1841. PMLR, 2018

    2018

  14. [14]

    Implicit regularization in deep matrix factorization.Advances in neural information processing systems, 32, 2019

    Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization.Advances in neural information processing systems, 32, 2019

    2019

  15. [15]

    Sgd and weight decay secretly minimize the rank of your neural network

    Tomer Galanti, Zachary S Siegel, Aparna Gupte, and Tomaso A Poggio. Sgd and weight decay secretly minimize the rank of your neural network. InNeurIPS 2024 Workshop on Mathematics of Modern Machine Learning, 2024

    2024

  16. [16]

    High-dimensional data analysis: The curses and blessings of dimensionality.AMS math challenges lecture, 1(2000):32, 2000

    David L Donoho et al. High-dimensional data analysis: The curses and blessings of dimensionality.AMS math challenges lecture, 1(2000):32, 2000

    2000

  17. [17]

    Online stochastic gradient descent on non-convex losses from high-dimensional inference.Journal of Machine Learning Research, 22(106):1–51, 2021

    Gerard Ben Arous, Reza Gheissari, and Aukosh Jagannath. Online stochastic gradient descent on non-convex losses from high-dimensional inference.Journal of Machine Learning Research, 22(106):1–51, 2021

    2021

  18. [18]

    The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks

    Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. The merged-staircase property: a necessary and nearly sufficient condition for sgd learning of sparse functions on two-layer neural networks. InConference on Learning Theory, pages 4782–4887. PMLR, 2022

    2022

  19. [19]

    High-dimensional asymptotics of feature learning: How one gradient step improves the representation.Advances in Neural Information Processing Systems, 35:37932–37946, 2022

    Jimmy Ba, Murat A Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High-dimensional asymptotics of feature learning: How one gradient step improves the representation.Advances in Neural Information Processing Systems, 35:37932–37946, 2022

    2022

  20. [20]

    From high- dimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd in two-layers networks

    Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From high- dimensional & mean-field dynamics to dimensionless odes: A unifying approach to sgd in two-layers networks. InThe Thirty Sixth Annual Conference on Learning Theory, pages 1199–1227. PMLR, 2023. 12

    2023

  21. [21]

    Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models.Advances in Neural Information Processing Systems, 36:752–784, 2023

    Alex Damian, Eshaan Nichani, Rong Ge, and Jason D Lee. Smoothing the landscape boosts the signal for sgd: Optimal sample complexity for learning single index models.Advances in Neural Information Processing Systems, 36:752–784, 2023

    2023

  22. [22]

    On learning Gaussian multi- index models with gradient flow part i: General properties and two-timescale learning

    Alberto Bietti, Joan Bruna, and Loucas Pillaud-Vivien. On learning Gaussian multi- index models with gradient flow part i: General properties and two-timescale learning. Communications on Pure and Applied Mathematics, 2025

    2025

  23. [23]

    Learning time-scales in two-layers neural networks.Foundations of Computational Mathematics, pages 1–84, 2024

    Raphaël Berthier, Andrea Montanari, and Kangjie Zhou. Learning time-scales in two-layers neural networks.Foundations of Computational Mathematics, pages 1–84, 2024

    2024

  24. [24]

    A phase transition between positional and semantic learning in a solvable model of dot-product attention

    Hugo Cui, Freya Behrens, Florent Krzakala, and Lenka Zdeborová. A phase transition between positional and semantic learning in a solvable model of dot-product attention. Advances in Neural Information Processing Systems, 37:36342–36389, 2024

    2024

  25. [25]

    Fun- damental limits of learning in sequence multi-index models and deep attention networks: high-dimensional asymptotics and sharp thresholds

    Emanuele Troiani, Hugo Cui, Yatin Dandi, Florent Krzakala, and Lenka Zdeborová. Fun- damental limits of learning in sequence multi-index models and deep attention networks: high-dimensional asymptotics and sharp thresholds. InForty-second International Confer- ence on Machine Learning, 2025

    2025

  26. [26]

    Asymptotics of SGD in sequence-single index models and single-layer attention networks

    Luca Arnaboldi, Bruno Loureiro, Ludovic Stephan, Florent Krzakala, and Lenka Zdeborová. Asymptotics of SGD in sequence-single index models and single-layer attention networks. NeurIPS 2025, 2025. arXiv preprint arXiv:2506.02651

    arXiv 2025

  27. [27]

    Bayes optimal learning of attention-indexed models.NeurIPS 2025, 2025

    Fabrizio Boncoraglio, Emanuele Troiani, Vittorio Erba, and Lenka Zdeborová. Bayes optimal learning of attention-indexed models.NeurIPS 2025, 2025. arXiv:2506.01582

    arXiv 2025

  28. [28]

    The nuclear route: Sharp asymptotics of erm in overparameterized quadratic networks.NeurIPS 2025,

    Vittorio Erba, Emanuele Troiani, Lenka Zdeborová, and Florent Krzakala. The nuclear route: Sharp asymptotics of erm in overparameterized quadratic networks.NeurIPS 2025,

    2025

  29. [29]

    American Mathematical Soc., 1992

    Dan V Voiculescu, Ken J Dykema, and Alexandru Nica.Free random variables, volume 1. American Mathematical Soc., 1992

    1992

  30. [30]

    Universality for the global spectrum of random inner-product kernel matrices in the polynomial regime.arXiv preprint arXiv:2310.18280, 2023

    Sofiia Dubova, Yue M Lu, Benjamin McKenna, and Horng-Tzer Yau. Universality for the global spectrum of random inner-product kernel matrices in the polynomial regime.arXiv preprint arXiv:2310.18280, 2023

    Pith/arXiv arXiv 2023

  31. [31]

    Fitting an ellipsoid to random points: predictions using the replica method.IEEE Transactions on Information Theory, 70(10):7273–7296, 2024

    Antoine Maillard and Dmitriy Kunisky. Fitting an ellipsoid to random points: predictions using the replica method.IEEE Transactions on Information Theory, 70(10):7273–7296, 2024

    2024

  32. [32]

    Exact threshold for approximate ellipsoid fitting of random points.Electronic Journal of Probability, 30:1–46, 2025

    Afonso S Bandeira and Antoine Maillard. Exact threshold for approximate ellipsoid fitting of random points.Electronic Journal of Probability, 30:1–46, 2025

    2025

  33. [33]

    Springer Science & Business Media, 2003

    Michel Talagrand.Spin glasses: a challenge for mathematicians: cavity and mean field models, volume 46. Springer Science & Business Media, 2003

    2003

  34. [34]

    Theoretical insights into multiclass classification: A high-dimensional asymptotic view.Advances in Neural Information Processing Systems, 33:8907–8920, 2020

    Christos Thrampoulidis, Samet Oymak, and Mahdi Soltanolkotabi. Theoretical insights into multiclass classification: A high-dimensional asymptotic view.Advances in Neural Information Processing Systems, 33:8907–8920, 2020

    2020

  35. [35]

    Asymptotics of non-convex generalized linear models in high-dimensions: A proof of the replica formula

    Matteo Vilucchio, Yatin Dandi, Cedric Gerbelot, and Florent Krzakala. Asymptotics of non-convex generalized linear models in high-dimensions: A proof of the replica formula. arXiv preprint arXiv:2502.20003, 2025. 13

    arXiv 2025

  36. [36]

    The LASSO risk for Gaussian matrices.IEEE Transactions on Information Theory, 58(4):1997–2017, 2011

    Mohsen Bayati and Andrea Montanari. The LASSO risk for Gaussian matrices.IEEE Transactions on Information Theory, 58(4):1997–2017, 2011

    1997

  37. [37]

    State evolution for general approximate message passing algorithms, with applications to spatial coupling.Information and Inference: A Journal of the IMA, 2(2):115–144, 2013

    Adel Javanmard and Andrea Montanari. State evolution for general approximate message passing algorithms, with applications to spatial coupling.Information and Inference: A Journal of the IMA, 2(2):115–144, 2013

    2013

  38. [38]

    State evolution for approxi- mate message passing with non-separable functions.Information and Inference: A Journal of the IMA, 9(1):33–79, March 2020

    Raphaël Berthier, Andrea Montanari, and Phan-Minh Nguyen. State evolution for approxi- mate message passing with non-separable functions.Information and Inference: A Journal of the IMA, 9(1):33–79, March 2020. ISSN 2049-8764, 2049-8772

    2020

  39. [39]

    Graph-based approximate message passing iterations

    Cédric Gerbelot and Raphaël Berthier. Graph-based approximate message passing iterations. Information and Inference: A Journal of the IMA, 12(4):2562–2628, 2023

    2023

  40. [40]

    A convergence analysis of approximate message passing with non-separable functions and applications to multi-class classification

    Burak Çakmak, Yue M Lu, and Manfred Opper. A convergence analysis of approximate message passing with non-separable functions and applications to multi-class classification. In2024 IEEE International Symposium on Information Theory (ISIT), pages 747–752. IEEE, 2024

    2024

  41. [41]

    Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applications, March 2025

    Yizhou Xu, Antoine Maillard, Lenka Zdeborová, and Florent Krzakala. Fundamental Limits of Matrix Sensing: Exact Asymptotics, Universality, and Applications, March 2025. arXiv:2503.14121 [stat]

    arXiv 2025

  42. [42]

    Universality of empirical risk minimization

    Andrea Montanari and Basil N Saeed. Universality of empirical risk minimization. In Conference on Learning Theory, pages 4310–4312. PMLR, 2022

    2022

  43. [43]

    Lu, and Subhabrata Sen

    Rishabh Dudeja, Yue M. Lu, and Subhabrata Sen. Universality of approximate message passing with semirandom matrices.The Annals of Probability, 51(5):1616–1683, 2023

    2023

  44. [44]

    Universality laws for Gaussian mixtures in generalized linear models.Advances in Neural Information Processing Systems, 36:54754–54768, 2023

    Yatin Dandi, Ludovic Stephan, Florent Krzakala, Bruno Loureiro, and Lenka Zdeborová. Universality laws for Gaussian mixtures in generalized linear models.Advances in Neural Information Processing Systems, 36:54754–54768, 2023

    2023

  45. [45]

    Adam: A method for stochastic gradient descent

    Diederik P Kingma and Jimmy Lei Ba. Adam: A method for stochastic gradient descent. InICLR: international conference on learning representations, pages 1–15, 2015

    2015

  46. [46]

    Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning.Journal of Machine Learning Research, 22(165):1–73, 2021

    Charles H Martin and Michael W Mahoney. Implicit self-regularization in deep neural networks: Evidence from random matrix theory and implications for learning.Journal of Machine Learning Research, 22(165):1–73, 2021

    2021

  47. [47]

    Learning Gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions.Advances in Neural Information Processing Systems, 34: 10144–10157, 2021

    Bruno Loureiro, Gabriele Sicuro, Cédric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zdeborová. Learning Gaussian mixtures with generalized linear models: Precise asymptotics in high-dimensions.Advances in Neural Information Processing Systems, 34: 10144–10157, 2021

    2021

  48. [48]

    Bayes-optimal learning of an extensive-width neural network from quadratically many samples.Advances in Neural Information Processing Systems, 37:82085–82132, December 2024

    Antoine Maillard, Emanuele Troiani, Simon Martin, Lenka Zdeborová, and Florent Krzakala. Bayes-optimal learning of an extensive-width neural network from quadratically many samples.Advances in Neural Information Processing Systems, 37:82085–82132, December 2024

    2024

  49. [49]

    Various thresholds for ℓ1-optimization in compressed sensing.arXiv preprint arXiv:0907.3666, 2009

    Mihailo Stojnic. Various thresholds for ℓ1-optimization in compressed sensing.arXiv preprint arXiv:0907.3666, 2009

    Pith/arXiv arXiv 2009

  50. [50]

    The Gaussian min-max theorem in the presence of convexity.arXiv preprint arXiv:1408.4837, 2014

    Christos Thrampoulidis, Samet Oymak, and Babak Hassibi. The Gaussian min-max theorem in the presence of convexity.arXiv preprint arXiv:1408.4837, 2014. 14

    Pith/arXiv arXiv 2014

  51. [51]

    Message-passing algorithms for compressed sensing.Proceedings of the National Academy of Sciences, 106(45):18914–18919, 2009

    David L Donoho, Arian Maleki, and Andrea Montanari. Message-passing algorithms for compressed sensing.Proceedings of the National Academy of Sciences, 106(45):18914–18919, 2009

    2009

  52. [52]

    An iterative construction of solutions of the tap equations for the sherrington–kirkpatrick model.Communications in Mathematical Physics, 325(1):333–366, 2014

    Erwin Bolthausen. An iterative construction of solutions of the tap equations for the sherrington–kirkpatrick model.Communications in Mathematical Physics, 325(1):333–366, 2014

    2014

  53. [53]

    Distribution of eigenvalues for some sets of random matrices.Mat

    VA Marchenko and Leonid A Pastur. Distribution of eigenvalues for some sets of random matrices.Mat. Sb.(NS), 72(114):4, 1967

    1967

  54. [54]

    torch.optim.adam — pytorch (stable) documentation.https://docs

    PyTorch Core Team. torch.optim.adam — pytorch (stable) documentation.https://docs. pytorch.org/docs/stable/generated/torch.optim.Adam.html, 2025

    2025

  55. [55]

    Norm-based capacity control in neural networks

    Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. InConference on learning theory, pages 1376–1401. PMLR, 2015

    2015

  56. [56]

    The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19(70):1–57, 2018

    Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data.Journal of Machine Learning Research, 19(70):1–57, 2018

    2018

  57. [57]

    Saddle-to-saddle dynamics in diagonal linear networks

    Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. Advances in Neural Information Processing Systems, 36:7475–7505, 2023. 15 A Inductive Nuclear Norm Bias for Tied and Untied weights We briefly recall here the mathematical reasons behind the appearance of theinductive nuclear norm, following arguments that go back t...

    2023

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.