Pith. sign in

REVIEW 6 minor 2 cited by

Proportional infinite-width infinite-depth limit for deep linear neural networks

T0 review · 0 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read In the proportional infinite-width, infinite-depth limit, the output prior of a deep linear network is an explicit, non-Gaussian mixture of Gaussians, and the same structure survives in the posterior.

desk verdict Rigorous multi-output proportional depth-width limit for deep linear Bayesian networks, with an explicit label-dependent posterior; worth a serious referee once the Eq. (8) constant is checked. read the letter →

arxiv 2411.15267 v1 pith:ZCBQG2EU submitted 2024-11-22 stat.ML cond-mat.dis-nncs.LGmath.PR

classification stat.MLcond-mat.dis-nncs.LGmath.PR MSC 60F0560H0560B20
keywords deeplinearneuralnetworksproportionallimitinfinite-widthNetworkGaussianProcessmixtureofGaussiansmultiplestochasticintegralsWishartmatricesBayesianinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper studies fully connected deep linear networks whose random weights are Gaussian, taking both the number of layers L and the width N to infinity while their ratio L/N tends to a fixed constant a>0. It claims that in this proportional limit the prior law of the network outputs is not the Gaussian process found in the usual infinite-width regime, but a nontrivial mixture of Gaussians with an explicit mixing distribution built from multiple stochastic integrals. The same mixture structure holds for the posterior under a bounded continuous likelihood, and for a Gaussian likelihood the posterior and predictive distributions are explicit mixtures of normals whose mixing measure depends on the training labels. This matters because the infinite-width Gaussian process forces output components to be independent and makes their covariance blind to the labels, while this proportional limit keeps correlations and gives the covariance a label-dependent component.

What carries the argument

The central object is the random product $\bar{V}_{L,N}=V_L\cdots V_1$ of independent $D$-dimensional Bartlett matrices, i.e. lower-triangular Cholesky factors of Wishart matrices with $N$ degrees of freedom and scale matrix $N^{-1}1_D$; this product is exactly the non-Gaussian part of the prior representation. The proof machinery is a Skorokhod coupling: diagonal squared entries behave as random walks on log-Gamma increments and converge to the drifted Brownian processes $Z^{(k)}_t=\sqrt{a/2}\,W^{(k)}_t-(k/2)at$, while off-diagonal entries are coupled to independent Brownian increments of size $\sqrt{L/N}$. This turns each term in the expansion of $\bar{V}_{L,N}$ into a discrete Riemann sum that converges to a multiple stochastic integral $H(r)$, and the limit matrix $\bar{V}_\infty$ is assembled from those integrals.

What would settle it

Take $D=2$, set $a=1$, and simulate $\bar{V}_{L,N}=V_L\cdots V_1$ for large $N$ with $L=N$. Proposition 3.2 predicts the off-diagonal entry converges in distribution to $H(1,2)=e^{Z^{(2)}_1}\int_0^1 e^{Z^{(1)}_t-Z^{(2)}_t}\,dW^{(2,1)}_t$, with $Z^{(k)}_t=\sqrt{1/2}\,W^{(k)}_t-(k/2)t$; comparing the simulated empirical distribution of the off-diagonal entry to this law (in mean, variance, and tail quantiles) would settle the claim. A cheaper necessary check: the diagonal entry $\bar{V}^{1,1}_{L,N}$ should satisfy $E[\bar{V}^{1,1}_{L,N}]\to e^{-1/4}$ and $E[(\bar{V}^{1,1}_{L,N})^2]\to 1$ as $L/N\to 1$; any systematic mismatch beyond Monte Carlo error would disprove Proposition 3.2.

Watch

Extended reading notes

Core claim

Let $f_{L,N}(X|\theta)$ be the $D$-dimensional outputs of a deep linear network with $L$ hidden layers of width $N$, evaluated on $P$ inputs collected in $X$, with Gaussian weights of layer-wise inverse variance $\lambda_\ell$. The paper's starting point is the representation $f_{L,N}(X|\theta)\stackrel{d}{=}(N_0\lambda^*_L)^{-1/2}\bar{V}_{L,N}ZX$, where $Z$ is a $D\times N_0$ standard Gaussian matrix and $\bar{V}_{L,N}=V_L\cdots V_1$ is a product of independent Cholesky factors of Wishart matrices. Proposition 3.2 proves that if $L/N\to a$ with $0<a<\infty$, then $\bar{V}_{L,N}$ converges in distribution to an explicit lower-triangular matrix $\bar{V}_\infty$ whose diagonal entries are exponentials of independent drifted Brownian motions at time $1$ and whose off-diagonal entries are sums of multiple stochastic integrals $H(r)$. Consequently the output prior converges to the law of $(N_0\lambda^*_\infty)^{-1/2}\bar{V}_\infty ZX$, which is a genuine mixture of Gaussians. Propositions 3.3 and 3.4 carry this convergence to the posterior: with a bounded continuous likelihood the posterior converges weakly to the corresponding mixture, and under a Gaussian likelihood both the posterior and the predictive are mixtures of normals with mean and covariance functions of a label-dependent mixing measure $Q_\infty(dQ|\tilde{X},y_{1:P})$.

Load-bearing premise

The load-bearing premise is that the prior law of the network outputs equals the law of a product of independent random triangular matrices (the Cholesky square roots of Wishart matrices) acting on a Gaussian matrix; every asymptotic statement in the paper follows from that equality of laws, so if the representation failed the mixture limit would not describe the network.

Editorial extensions

If this is right

  • Whenever $L/N\to a>0$, the output prior is a nontrivial Gaussian mixture, so output components are dependent and the large-network prior is no longer the featureless Gaussian process of the infinite-width limit.
  • The same weak convergence holds for posteriors under any bounded continuous likelihood; with a Gaussian likelihood the asymptotic posterior and posterior predictive are explicit mixtures of normals, with formulas given in Proposition 3.4.
  • The asymptotic mixing measure for the posterior is the law of $\bar{Q}_\infty=\bar{V}_\infty\bar{V}_\infty^\top$ reweighted by a function $\Psi$ that depends on the training labels, so conditioning on data changes the covariance structure rather than only shifting the mean.
  • The boundary case $a=0$ recovers exactly the Neural Network Gaussian Process limit, showing that a strictly positive depth-to-width ratio is the minimal change that breaks Gaussianity for linear networks.
  • In the scalar-output case the limit reduces to a log-normal random variable, matching earlier single-input, single-output results; the multidimensional prior and posterior mixture statements are the new content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The explicit multiple-integral representation of $\bar{V}_\infty$ gives a route to finite-width corrections: expanding around the Brownian limit should yield quantitative rates of approach to the mixture, of the same kind that exist for the Gaussian infinite-width regime.
  • A testable prediction of the paper's posterior formulas is that, for fixed inputs, the posterior predictive variance on a test point changes when the training labels are permuted, and this label-dependence should disappear continuously as $a\to 0$.
  • The Bartlett/Wishart mechanism is specific to linear activations; whether a similar multivariate stochastic-integral mixture exists for nonlinear activations is left open, but the paper's Brownian-coupling strategy suggests the main obstacle is finding a tractable analogue of the product-of-Wishart representation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

0 major / 6 minor

Summary. The paper studies deep linear neural networks with Gaussian weights under the proportional scaling L/N -> a, where L is depth and N is width. Using the representation of the output as V_L ... V_1 Z X / sqrt(N0 lambda*_L), with V_l independent Cholesky factors of Wishart matrices, it proves that for a > 0 the product V_bar_{L,N} converges in distribution to a random lower-triangular matrix V_bar_infty whose entries are explicit iterated stochastic integrals driven by independent Brownian motions. This gives a non-Gaussian mixture-of-Gaussians prior for the network output, in contrast to the Gaussian NNGP limit recovered when a = 0. The paper also derives analogous mixture representations for posterior and predictive distributions under a Gaussian likelihood, and it proves that the posterior covariance depends on the labels, a feature absent in the NNGP regime.

Significance. If the result stands, it provides the first rigorous multi-input, multi-output characterization of the proportional infinite-width/infinite-depth limit for linear networks, extending Hanin's single-input/single-output analysis and complementing the physics literature on proportional limits. The proofs are substantial and mostly self-contained: exact finite-N moments, a Skorokhod coupling of the Bartlett entries, variance bounds, and convergence of discrete product sums to iterated stochastic integrals. No parameter is fitted, and the limiting objects are defined explicitly, which makes the claims falsifiable and checkable. The main external input is Proposition 2.1, cited from Bassetti et al. (2024); the representation is standard and appears sound. The paper is a solid contribution to the theory of non-Gaussian scaling limits for Bayesian neural networks.

minor comments (6)
  1. [Section 3.1, Eq. (8)] The displayed definition of H(r) contains the prefactor e^{(a/2) ln a}, whereas the proof in Section 4.4 concludes that the limiting term has prefactor a^{h/2}; since h varies with r, the printed formula would mis-specify the limit matrix unless corrected.
  2. [Section 4.4, definition of H_L(r)] The displayed expression for H_L(r) includes a summation over r in R^h_{k,i}; as written this would make H_L(r) depend on all r, but the subsequent lines treat r as fixed, so the redundant summation should be removed.
  3. [Lemma 4.5, proof] The definition of R_N is missing parentheses around the limiting linear combination; as written it can be read as subtracting Z^{(r_j)}_t rather than the difference Z^{(r_{j-1})}_t - Z^{(r_j)}_t.
  4. [Proposition 4.3, proof] The use of the strong approximation theorem from Kallenberg (1997, Thm 12.20) should be stated explicitly for the triangular-array setting, since the increments change with N; a brief justification or a reference to a triangular-array version would remove ambiguity.
  5. [Throughout] There are numerous typographical errors, including 'representaion' in the Introduction, 'Morevoer' in Proposition 3.4, 'rigth-continuous' in Appendix B, and the title line 'DEE P'; these should be cleaned up in a revision.
  6. [Section 3.3, Eqs. (15)-(16)] The text uses Moore-Penrose inverses in the definitions of Sigma* and m* without comment; Remark 1 clarifies the invertible case, but a brief sentence in the main text explaining the generalized inverse convention would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the non-Gaussian mixture limit is derived from an exact finite-N representation and stochastic-integral estimates, not from the claimed conclusion.

full rationale

The only author-overlap citation that enters the derivation is Proposition 2.1 from Bassetti et al. (2024), introduced as 'a result proved in Bassetti et al. (2024), which serves as the basis for our study.' This is a self-citation, but it is not a circular step: it is an exact, parameter-free finite-N distributional identity (Wishart/Bartlett representation of the output prior) whose hypotheses do not contain the asymptotic mixture limit. The rest of the paper is a genuine derivation from that starting point. Lemma 4.1 computes the log-Gamma moment generating function; Proposition 4.3 constructs a Skorohod coupling of the Bartlett variables to Brownian motions; Lemma 4.5 and Lemma B.3 establish uniform convergence of the discrete exponential integrands and the corresponding Riemann-sum stochastic integrals; Proposition 3.2 then follows by continuous mapping. The posterior statements (Propositions 3.3 and 3.4) are weak-convergence transfers through Bayes' rule under a bounded continuous likelihood, followed by explicit normal-mixture algebra in Appendix C; they do not assume the limiting mixture. There is no fitted parameter, no quantity is renamed as a prediction, and no ansatz is imported to force the claimed limit. A possible prefactor typo in Eq. (8) would be a correctness issue, not circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted constants appear in the paper; the only parameter a is the asymptotic depth-to-width ratio, not fitted to data. The central new construction is the limit matrix V-bar_infty defined by stochastic integrals, which is a mathematical object rather than a new physical entity.

assumptions (5)
  • domain assumption Exact prior representation of Bassetti et al. 2024, Proposition 2.1: f_L(X|theta) equals V_L...V_1 Z X / sqrt(N0 lambda*_L), with V_l Cholesky factors of independent Wishart(N_l, I/N_l) matrices.
    Load-bearing starting point for all asymptotics; stated but not reproved in this paper, and the cited paper shares authors with this one.
  • standard math Bartlett decomposition of Wishart Cholesky factors: diagonal squared entries are independent Gamma, off-diagonal entries are independent normal with variance 1/N.
    Used in Lemma 4.1 and in the Skorokhod construction in Section 4.2.
  • standard math Strong approximation of random walks by Brownian motion, Kallenberg 1997, Theorem 12.20.
    Needed to couple the diagonal random walks for all N and r to Brownian motions, Proposition 4.3(C).
  • standard math Existence, adaptedness, and L2-stability of Ito stochastic integrals, Karatzas and Shreve 1991, Proposition 1.3 and Proposition 2.26.
    Used to define the multiple stochastic integrals in equation (8) and to prove sup convergence in Lemma 4.5 and Appendix B.
  • domain assumption Layer variance products converge: lambda*_L -> lambda*_infty as N -> infinity.
    Required to state convergence of the output distribution; the matrix V-bar limit itself does not need it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Proportional infinite-width infinite-depth limit for deep linear neural networks." pith.science (2026). https://pith.science/paper/ZCBQG2EU

@misc{pith2026241115267,
  author       = {Pith},
  title        = {Pith review of: Proportional infinite-width infinite-depth limit for deep linear neural networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZCBQG2EU}},
  note         = {Machine review of arXiv:2411.15267}
}
read the original abstract

We study the distributional properties of linear neural networks with random parameters in the context of large networks, where the number of layers diverges in proportion to the number of neurons per layer. Prior works have shown that in the infinite-width regime, where the number of neurons per layer grows to infinity while the depth remains fixed, neural networks converge to a Gaussian process, known as the Neural Network Gaussian Process. However, this Gaussian limit sacrifices descriptive power, as it lacks the ability to learn dependent features and produce output correlations that reflect observed labels. Motivated by these limitations, we explore the joint proportional limit in which both depth and width diverge but maintain a constant ratio, yielding a non-Gaussian distribution that retains correlations between outputs. Our contribution extends previous works by rigorously characterizing, for linear activation functions, the limiting distribution as a nontrivial mixture of Gaussians.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Product

    math.PR 2026-06 unverdicted novelty 7.0 of 10

    In double asymptotic limits, the squared singular value process of non-square matrix products obeys geometric Dyson Brownian motion whose T-transform solves a Burgers equation, producing the free log-normal law via fr...

  2. Large deviation principles for convolutional Bayesian neural networks

    math.PR 2026-03 accept novelty 6.5 of 10

    Convolutional Bayesian NNs with Gaussian weights satisfy an LDP for their conditional covariance matrices (and posterior) in the infinite-channel limit, with an explicit good rate function built from layer-wise cumula...

Reference graph

Works this paper leans on

36 extracted references · 29 canonical work pages · cited by 2 Pith papers

  1. [1]

    Aiudi, R., Pacelli, R., Vezzani, A., Burioni, R., and Rotondo, P. (2023). Local kernel renormalization as a mechanism for feature learning in overparametrized convolutional neural networks. https://arxiv.org/abs/2307.11807

  2. [2]

    Baglioni, P., Pacelli, R., Aiudi, R., Di Renzo, F., Vezzani, A., Burioni, R., and Rotondo, P. (2024). Predictive power of a bayesian effective action for fully connected one hidden layer neural networks in the proportional limit. Phys. Rev. Lett. , 133:027301

  3. [3]

    Bassetti, F., Gherardi, M., Ingrosso, A., Pastore, M., and Rotondo, P. (2024). Feature learning in finite-width bayesian deep linear networks with multiple outputs and convolutional layers. https://arxiv.org/abs/2406.03260

  4. [4]

    Bordino, A., Favaro, S., and Fortini, S. (2023). Infinitely wide limits for deep stable neural networks: sub-linear, linear and super-linear activation functions. https://arxiv.org/abs/2304.04008

  5. [5]

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.,...

  6. [6]

    and Bach, F

    Chizat, L. and Bach, F. (2018). On the global convergence of gradient descent for over-parameterized models using optimal transport. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc

  7. [7]

    Chizat, L., Colombo, M., Fernández-Real, X., and Figalli, A. (2024). Infinite-width limit of deep linear neural networks. Communications on Pure and Applied Mathematics , 77(10):3958--4007

  8. [8]

    Chizat, L., Oyallon, E., and Bach, F. (2019). On lazy training in differentiable programming. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc

Show all 36 references
  1. [9]

    Matthews, A

    de G. Matthews, A. G., Hron, J., Rowland, M., Turner, R. E., and Ghahramani, Z. (2018). Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations

  2. [10]

    Eaton, M. L. (2007). Multivariate statistics: A vector space approach , volume 53 of Institute of Mathematical Statistics Lecture Notes - Monograph Series . Institute of Mathematical Statistics, Beachwood, OH

  3. [11]

    Favaro, S., Hanin, B., Marinucci, D., Nourdin, I., and Peccati, G. (2023a). Quantitative CLT s in deep neural networks. https://arxiv.org/abs/2307.06092

  4. [12]

    Favaro, S., Sandra, F., and Stefano, P. (2023b). Ilarge-width asymptotics for relu neural networks with -stable initializationss. https://arxiv.org/abs/2206.08065

  5. [13]

    E., and Aitchison, L

    Garriga-Alonso, A., Rasmussen, C. E., and Aitchison, L. (2019). Deep convolutional networks as shallow G aussian processes. In International Conference on Learning Representations

  6. [14]

    Gupta, A. K. and Nagar, D. K. (2000). Matrix variate distributions , volume 104 of Chapman & Hall/CRC Monographs and Surveys in Pure and Applied Mathematics . Chapman & Hall/CRC, Boca Raton, FL. eBook https://doi.org/10.1201/9780203749289

  7. [15]

    Hanin, B. (2023). Random neural networks in the infinite width limit as G aussian processes. Ann. Appl. Probab. , 33(6A):4798--4819

  8. [16]

    Hanin, B. (2024). Random fully connected neural networks as perturbatively solvable hierarchies. Journal of Machine Learning Research , 25(167):1--58

  9. [17]

    and Zlokapa, A

    Hanin, B. and Zlokapa, A. (2023). Bayesian interpolation with deep linear networks. Proceedings of the National Academy of Sciences , 120(23):e2301345120

  10. [18]

    Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , vo...

  11. [19]

    Kallenberg, O. (1997). Foundations of modern probability . Probability and its Applications (New York). Springer-Verlag, New York

  12. [20]

    and Shreve, S

    Karatzas, I. and Shreve, S. E. (1991). Brownian motion and stochastic calculus , volume 113 of Graduate Texts in Mathematics . Springer-Verlag, New York, second edition

  13. [21]

    Kshirsagar, A. M. (1959). Bartlett decomposition and W ishart distribution. The Annals of Mathematical Statistics , 30(1):239--241

  14. [22]

    Lee, J., Sohl-dickstein, J., Pennington, J., Novak, R., Schoenholz, S., and Bahri, Y. (2018). Deep neural networks as G aussian processes. In International Conference on Learning Representations

  15. [23]

    Lewkowycz, Aitor and Bahri, Yasaman and Dye, Ethanr and Sohl-Dickstein, Jascha and Gur-Ari, Guy. (2021). The large learning rate phase of deep learning . https://arxiv.org/abs/2003.02218

  16. [24]

    and Sompolinsky, H

    Li, Q. and Sompolinsky, H. (2021). Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization. Phys. Rev. X , 11:031059

  17. [25]

    Mei, S., Montanari, A., and Nguyen, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences , 115(33):E7665--E7671

  18. [26]

    Neal, R. M. (1996). Priors for Infinite Networks , pages 29--53. Springer New York, New York, NY

  19. [27]

    A., Pennington, J., and Sohl-dickstein, J

    Novak, R., Xiao, L., Bahri, Y., Lee, J., Yang, G., Abolafia, D. A., Pennington, J., and Sohl-dickstein, J. (2019). Bayesian deep convolutional networks with many channels are G aussian processes. In International Conference on Learning Representations

  20. [28]

    Pacelli, R., Ariosto, S., Pastore, M., Ginelli, F., Gherardi, M., and Rotondo, P. (2023). A statistical mechanics framework for B ayesian deep neural networks beyond the infinite-width limit. Nature Machine Intelligence , 5:1497--1507

  21. [29]

    and Vanden-Eijnden, E

    Rotskoff, G. and Vanden-Eijnden, E. (2022). Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics , 75(9):1889--1935

  22. [30]

    M., McClelland, J

    Saxe, A. M., McClelland, J. L., and Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Bengio, Y. and LeCun, Y., editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-...

  23. [31]

    M., McClelland, J

    Saxe, A. M., McClelland, J. L., and Ganguli, S. (2019). A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences , 116(23):11537--11546

  24. [32]

    and Spiliopoulos, K

    Sirignano, J. and Spiliopoulos, K. (2020). Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics , 80(2):725--752

  25. [33]

    Trevisan, D. (2023). Wide deep neural networks with G aussian weights are very close to G aussian processes. https://arxiv.org/abs/2312.11737

  26. [34]

    Tricomi, F. G. and Erd\'elyi, A. (1951). The asymptotic expansion of a ratio of gamma functions. Pacific J. Math. , 1:133--142

  27. [35]

    and Hu, E

    Yang, G. and Hu, E. J. (2021). Tensor programs IV : Feature learning in infinite-width neural networks. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11...

  28. [36]

    Zavatone-Veth, J. A. and Pehlevan, C. (2021). Exact marginal prior distributions of finite B ayesian neural networks. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.