REVIEW 6 minor 2 cited by
Proportional infinite-width infinite-depth limit for deep linear neural networks
T0 review · 0 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read In the proportional infinite-width, infinite-depth limit, the output prior of a deep linear network is an explicit, non-Gaussian mixture of Gaussians, and the same structure survives in the posterior.
desk verdict Rigorous multi-output proportional depth-width limit for deep linear Bayesian networks, with an explicit label-dependent posterior; worth a serious referee once the Eq. (8) constant is checked. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the random product $\bar{V}_{L,N}=V_L\cdots V_1$ of independent $D$-dimensional Bartlett matrices, i.e. lower-triangular Cholesky factors of Wishart matrices with $N$ degrees of freedom and scale matrix $N^{-1}1_D$; this product is exactly the non-Gaussian part of the prior representation. The proof machinery is a Skorokhod coupling: diagonal squared entries behave as random walks on log-Gamma increments and converge to the drifted Brownian processes $Z^{(k)}_t=\sqrt{a/2}\,W^{(k)}_t-(k/2)at$, while off-diagonal entries are coupled to independent Brownian increments of size $\sqrt{L/N}$. This turns each term in the expansion of $\bar{V}_{L,N}$ into a discrete Riemann sum that converges to a multiple stochastic integral $H(r)$, and the limit matrix $\bar{V}_\infty$ is assembled from those integrals.
What would settle it
Take $D=2$, set $a=1$, and simulate $\bar{V}_{L,N}=V_L\cdots V_1$ for large $N$ with $L=N$. Proposition 3.2 predicts the off-diagonal entry converges in distribution to $H(1,2)=e^{Z^{(2)}_1}\int_0^1 e^{Z^{(1)}_t-Z^{(2)}_t}\,dW^{(2,1)}_t$, with $Z^{(k)}_t=\sqrt{1/2}\,W^{(k)}_t-(k/2)t$; comparing the simulated empirical distribution of the off-diagonal entry to this law (in mean, variance, and tail quantiles) would settle the claim. A cheaper necessary check: the diagonal entry $\bar{V}^{1,1}_{L,N}$ should satisfy $E[\bar{V}^{1,1}_{L,N}]\to e^{-1/4}$ and $E[(\bar{V}^{1,1}_{L,N})^2]\to 1$ as $L/N\to 1$; any systematic mismatch beyond Monte Carlo error would disprove Proposition 3.2.
Extended reading notes
Core claim
Let $f_{L,N}(X|\theta)$ be the $D$-dimensional outputs of a deep linear network with $L$ hidden layers of width $N$, evaluated on $P$ inputs collected in $X$, with Gaussian weights of layer-wise inverse variance $\lambda_\ell$. The paper's starting point is the representation $f_{L,N}(X|\theta)\stackrel{d}{=}(N_0\lambda^*_L)^{-1/2}\bar{V}_{L,N}ZX$, where $Z$ is a $D\times N_0$ standard Gaussian matrix and $\bar{V}_{L,N}=V_L\cdots V_1$ is a product of independent Cholesky factors of Wishart matrices. Proposition 3.2 proves that if $L/N\to a$ with $0<a<\infty$, then $\bar{V}_{L,N}$ converges in distribution to an explicit lower-triangular matrix $\bar{V}_\infty$ whose diagonal entries are exponentials of independent drifted Brownian motions at time $1$ and whose off-diagonal entries are sums of multiple stochastic integrals $H(r)$. Consequently the output prior converges to the law of $(N_0\lambda^*_\infty)^{-1/2}\bar{V}_\infty ZX$, which is a genuine mixture of Gaussians. Propositions 3.3 and 3.4 carry this convergence to the posterior: with a bounded continuous likelihood the posterior converges weakly to the corresponding mixture, and under a Gaussian likelihood both the posterior and the predictive are mixtures of normals with mean and covariance functions of a label-dependent mixing measure $Q_\infty(dQ|\tilde{X},y_{1:P})$.
Load-bearing premise
The load-bearing premise is that the prior law of the network outputs equals the law of a product of independent random triangular matrices (the Cholesky square roots of Wishart matrices) acting on a Gaussian matrix; every asymptotic statement in the paper follows from that equality of laws, so if the representation failed the mixture limit would not describe the network.
Editorial extensions
If this is right
- Whenever $L/N\to a>0$, the output prior is a nontrivial Gaussian mixture, so output components are dependent and the large-network prior is no longer the featureless Gaussian process of the infinite-width limit.
- The same weak convergence holds for posteriors under any bounded continuous likelihood; with a Gaussian likelihood the asymptotic posterior and posterior predictive are explicit mixtures of normals, with formulas given in Proposition 3.4.
- The asymptotic mixing measure for the posterior is the law of $\bar{Q}_\infty=\bar{V}_\infty\bar{V}_\infty^\top$ reweighted by a function $\Psi$ that depends on the training labels, so conditioning on data changes the covariance structure rather than only shifting the mean.
- The boundary case $a=0$ recovers exactly the Neural Network Gaussian Process limit, showing that a strictly positive depth-to-width ratio is the minimal change that breaks Gaussianity for linear networks.
- In the scalar-output case the limit reduces to a log-normal random variable, matching earlier single-input, single-output results; the multidimensional prior and posterior mixture statements are the new content.
Reading between the lines
- The explicit multiple-integral representation of $\bar{V}_\infty$ gives a route to finite-width corrections: expanding around the Brownian limit should yield quantitative rates of approach to the mixture, of the same kind that exist for the Gaussian infinite-width regime.
- A testable prediction of the paper's posterior formulas is that, for fixed inputs, the posterior predictive variance on a test point changes when the training labels are permuted, and this label-dependence should disappear continuously as $a\to 0$.
- The Bartlett/Wishart mechanism is specific to linear activations; whether a similar multivariate stochastic-integral mixture exists for nonlinear activations is left open, but the paper's Brownian-coupling strategy suggests the main obstacle is finding a tractable analogue of the product-of-Wishart representation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies deep linear neural networks with Gaussian weights under the proportional scaling L/N -> a, where L is depth and N is width. Using the representation of the output as V_L ... V_1 Z X / sqrt(N0 lambda*_L), with V_l independent Cholesky factors of Wishart matrices, it proves that for a > 0 the product V_bar_{L,N} converges in distribution to a random lower-triangular matrix V_bar_infty whose entries are explicit iterated stochastic integrals driven by independent Brownian motions. This gives a non-Gaussian mixture-of-Gaussians prior for the network output, in contrast to the Gaussian NNGP limit recovered when a = 0. The paper also derives analogous mixture representations for posterior and predictive distributions under a Gaussian likelihood, and it proves that the posterior covariance depends on the labels, a feature absent in the NNGP regime.
Significance. If the result stands, it provides the first rigorous multi-input, multi-output characterization of the proportional infinite-width/infinite-depth limit for linear networks, extending Hanin's single-input/single-output analysis and complementing the physics literature on proportional limits. The proofs are substantial and mostly self-contained: exact finite-N moments, a Skorokhod coupling of the Bartlett entries, variance bounds, and convergence of discrete product sums to iterated stochastic integrals. No parameter is fitted, and the limiting objects are defined explicitly, which makes the claims falsifiable and checkable. The main external input is Proposition 2.1, cited from Bassetti et al. (2024); the representation is standard and appears sound. The paper is a solid contribution to the theory of non-Gaussian scaling limits for Bayesian neural networks.
minor comments (6)
- [Section 3.1, Eq. (8)] The displayed definition of H(r) contains the prefactor e^{(a/2) ln a}, whereas the proof in Section 4.4 concludes that the limiting term has prefactor a^{h/2}; since h varies with r, the printed formula would mis-specify the limit matrix unless corrected.
- [Section 4.4, definition of H_L(r)] The displayed expression for H_L(r) includes a summation over r in R^h_{k,i}; as written this would make H_L(r) depend on all r, but the subsequent lines treat r as fixed, so the redundant summation should be removed.
- [Lemma 4.5, proof] The definition of R_N is missing parentheses around the limiting linear combination; as written it can be read as subtracting Z^{(r_j)}_t rather than the difference Z^{(r_{j-1})}_t - Z^{(r_j)}_t.
- [Proposition 4.3, proof] The use of the strong approximation theorem from Kallenberg (1997, Thm 12.20) should be stated explicitly for the triangular-array setting, since the increments change with N; a brief justification or a reference to a triangular-array version would remove ambiguity.
- [Throughout] There are numerous typographical errors, including 'representaion' in the Introduction, 'Morevoer' in Proposition 3.4, 'rigth-continuous' in Appendix B, and the title line 'DEE P'; these should be cleaned up in a revision.
- [Section 3.3, Eqs. (15)-(16)] The text uses Moore-Penrose inverses in the definitions of Sigma* and m* without comment; Remark 1 clarifies the invertible case, but a brief sentence in the main text explaining the generalized inverse convention would improve readability.
Circularity Check
No circularity found: the non-Gaussian mixture limit is derived from an exact finite-N representation and stochastic-integral estimates, not from the claimed conclusion.
full rationale
The only author-overlap citation that enters the derivation is Proposition 2.1 from Bassetti et al. (2024), introduced as 'a result proved in Bassetti et al. (2024), which serves as the basis for our study.' This is a self-citation, but it is not a circular step: it is an exact, parameter-free finite-N distributional identity (Wishart/Bartlett representation of the output prior) whose hypotheses do not contain the asymptotic mixture limit. The rest of the paper is a genuine derivation from that starting point. Lemma 4.1 computes the log-Gamma moment generating function; Proposition 4.3 constructs a Skorohod coupling of the Bartlett variables to Brownian motions; Lemma 4.5 and Lemma B.3 establish uniform convergence of the discrete exponential integrands and the corresponding Riemann-sum stochastic integrals; Proposition 3.2 then follows by continuous mapping. The posterior statements (Propositions 3.3 and 3.4) are weak-convergence transfers through Bayes' rule under a bounded continuous likelihood, followed by explicit normal-mixture algebra in Appendix C; they do not assume the limiting mixture. There is no fitted parameter, no quantity is renamed as a prediction, and no ansatz is imported to force the claimed limit. A possible prefactor typo in Eq. (8) would be a correctness issue, not circularity.
Assumptions & free parameters
assumptions (5)
- domain assumption Exact prior representation of Bassetti et al. 2024, Proposition 2.1: f_L(X|theta) equals V_L...V_1 Z X / sqrt(N0 lambda*_L), with V_l Cholesky factors of independent Wishart(N_l, I/N_l) matrices.
- standard math Bartlett decomposition of Wishart Cholesky factors: diagonal squared entries are independent Gamma, off-diagonal entries are independent normal with variance 1/N.
- standard math Strong approximation of random walks by Brownian motion, Kallenberg 1997, Theorem 12.20.
- standard math Existence, adaptedness, and L2-stability of Ito stochastic integrals, Karatzas and Shreve 1991, Proposition 1.3 and Proposition 2.26.
- domain assumption Layer variance products converge: lambda*_L -> lambda*_infty as N -> infinity.
Cite this review
Pith. "Pith review of Proportional infinite-width infinite-depth limit for deep linear neural networks." pith.science (2026). https://pith.science/paper/ZCBQG2EU
@misc{pith2026241115267,
author = {Pith},
title = {Pith review of: Proportional infinite-width infinite-depth limit for deep linear neural networks},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZCBQG2EU}},
note = {Machine review of arXiv:2411.15267}
}
read the original abstract
We study the distributional properties of linear neural networks with random parameters in the context of large networks, where the number of layers diverges in proportion to the number of neurons per layer. Prior works have shown that in the infinite-width regime, where the number of neurons per layer grows to infinity while the depth remains fixed, neural networks converge to a Gaussian process, known as the Neural Network Gaussian Process. However, this Gaussian limit sacrifices descriptive power, as it lacks the ability to learn dependent features and produce output correlations that reflect observed labels. Motivated by these limitations, we explore the joint proportional limit in which both depth and width diverge but maintain a constant ratio, yielding a non-Gaussian distribution that retains correlations between outputs. Our contribution extends previous works by rigorously characterizing, for linear activation functions, the limiting distribution as a nontrivial mixture of Gaussians.
Forward citations
Cited by 2 Pith papers
-
Geometric Dyson Brownian Motions and the Free Log-Normal Limit for a Non-Square Gaussian Matrix Product
In double asymptotic limits, the squared singular value process of non-square matrix products obeys geometric Dyson Brownian motion whose T-transform solves a Burgers equation, producing the free log-normal law via fr...
-
Large deviation principles for convolutional Bayesian neural networks
Convolutional Bayesian NNs with Gaussian weights satisfy an LDP for their conditional covariance matrices (and posterior) in the infinite-channel limit, with an explicit good rate function built from layer-wise cumula...
Reference graph
Works this paper leans on
-
[1]
Aiudi, R., Pacelli, R., Vezzani, A., Burioni, R., and Rotondo, P. (2023). Local kernel renormalization as a mechanism for feature learning in overparametrized convolutional neural networks. https://arxiv.org/abs/2307.11807
arXiv 2023
-
[2]
Baglioni, P., Pacelli, R., Aiudi, R., Di Renzo, F., Vezzani, A., Burioni, R., and Rotondo, P. (2024). Predictive power of a bayesian effective action for fully connected one hidden layer neural networks in the proportional limit. Phys. Rev. Lett. , 133:027301
work page 2024
-
[3]
Bassetti, F., Gherardi, M., Ingrosso, A., Pastore, M., and Rotondo, P. (2024). Feature learning in finite-width bayesian deep linear networks with multiple outputs and convolutional layers. https://arxiv.org/abs/2406.03260
arXiv 2024
-
[4]
Bordino, A., Favaro, S., and Fortini, S. (2023). Infinitely wide limits for deep stable neural networks: sub-linear, linear and super-linear activation functions. https://arxiv.org/abs/2304.04008
work page Pith review arXiv 2023
-
[5]
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A.,...
work page 2020
-
[6]
Chizat, L. and Bach, F. (2018). On the global convergence of gradient descent for over-parameterized models using optimal transport. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc
work page 2018
-
[7]
Chizat, L., Colombo, M., Fernández-Real, X., and Figalli, A. (2024). Infinite-width limit of deep linear neural networks. Communications on Pure and Applied Mathematics , 77(10):3958--4007
work page 2024
-
[8]
Chizat, L., Oyallon, E., and Bach, F. (2019). On lazy training in differentiable programming. In Wallach, H., Larochelle, H., Beygelzimer, A., d Alch\' e -Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc
work page 2019
Show all 36 references
-
[9]
Matthews, A
de G. Matthews, A. G., Hron, J., Rowland, M., Turner, R. E., and Ghahramani, Z. (2018). Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations
2018
-
[10]
Eaton, M. L. (2007). Multivariate statistics: A vector space approach , volume 53 of Institute of Mathematical Statistics Lecture Notes - Monograph Series . Institute of Mathematical Statistics, Beachwood, OH
2007
-
[11]
Favaro, S., Hanin, B., Marinucci, D., Nourdin, I., and Peccati, G. (2023a). Quantitative CLT s in deep neural networks. https://arxiv.org/abs/2307.06092
2023 arXiv
-
[12]
Favaro, S., Sandra, F., and Stefano, P. (2023b). Ilarge-width asymptotics for relu neural networks with -stable initializationss. https://arxiv.org/abs/2206.08065
2023 arXiv
-
[13]
E., and Aitchison, L
Garriga-Alonso, A., Rasmussen, C. E., and Aitchison, L. (2019). Deep convolutional networks as shallow G aussian processes. In International Conference on Learning Representations
2019
-
[14]
Gupta, A. K. and Nagar, D. K. (2000). Matrix variate distributions , volume 104 of Chapman & Hall/CRC Monographs and Surveys in Pure and Applied Mathematics . Chapman & Hall/CRC, Boca Raton, FL. eBook https://doi.org/10.1201/9780203749289
2000 doi
-
[15]
Hanin, B. (2023). Random neural networks in the infinite width limit as G aussian processes. Ann. Appl. Probab. , 33(6A):4798--4819
2023
-
[16]
Hanin, B. (2024). Random fully connected neural networks as perturbatively solvable hierarchies. Journal of Machine Learning Research , 25(167):1--58
2024
-
[17]
and Zlokapa, A
Hanin, B. and Zlokapa, A. (2023). Bayesian interpolation with deep linear networks. Proceedings of the National Academy of Sciences , 120(23):e2301345120
2023
-
[18]
Jacot, A., Gabriel, F., and Hongler, C. (2018). Neural tangent kernel: Convergence and generalization in neural networks. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , vo...
2018
-
[19]
Kallenberg, O. (1997). Foundations of modern probability . Probability and its Applications (New York). Springer-Verlag, New York
1997
-
[20]
and Shreve, S
Karatzas, I. and Shreve, S. E. (1991). Brownian motion and stochastic calculus , volume 113 of Graduate Texts in Mathematics . Springer-Verlag, New York, second edition
1991
-
[21]
Kshirsagar, A. M. (1959). Bartlett decomposition and W ishart distribution. The Annals of Mathematical Statistics , 30(1):239--241
1959
-
[22]
Lee, J., Sohl-dickstein, J., Pennington, J., Novak, R., Schoenholz, S., and Bahri, Y. (2018). Deep neural networks as G aussian processes. In International Conference on Learning Representations
2018
-
[23]
Lewkowycz, Aitor and Bahri, Yasaman and Dye, Ethanr and Sohl-Dickstein, Jascha and Gur-Ari, Guy. (2021). The large learning rate phase of deep learning . https://arxiv.org/abs/2003.02218
2021 arXiv
-
[24]
and Sompolinsky, H
Li, Q. and Sompolinsky, H. (2021). Statistical mechanics of deep linear neural networks: The backpropagating kernel renormalization. Phys. Rev. X , 11:031059
2021
-
[25]
Mei, S., Montanari, A., and Nguyen, P.-M. (2018). A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences , 115(33):E7665--E7671
2018
-
[26]
Neal, R. M. (1996). Priors for Infinite Networks , pages 29--53. Springer New York, New York, NY
1996
-
[27]
A., Pennington, J., and Sohl-dickstein, J
Novak, R., Xiao, L., Bahri, Y., Lee, J., Yang, G., Abolafia, D. A., Pennington, J., and Sohl-dickstein, J. (2019). Bayesian deep convolutional networks with many channels are G aussian processes. In International Conference on Learning Representations
2019
-
[28]
Pacelli, R., Ariosto, S., Pastore, M., Ginelli, F., Gherardi, M., and Rotondo, P. (2023). A statistical mechanics framework for B ayesian deep neural networks beyond the infinite-width limit. Nature Machine Intelligence , 5:1497--1507
2023
-
[29]
and Vanden-Eijnden, E
Rotskoff, G. and Vanden-Eijnden, E. (2022). Trainability and accuracy of artificial neural networks: An interacting particle system approach. Communications on Pure and Applied Mathematics , 75(9):1889--1935
2022
-
[30]
M., McClelland, J
Saxe, A. M., McClelland, J. L., and Ganguli, S. (2014). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In Bengio, Y. and LeCun, Y., editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-...
2014
-
[31]
M., McClelland, J
Saxe, A. M., McClelland, J. L., and Ganguli, S. (2019). A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences , 116(23):11537--11546
2019
-
[32]
and Spiliopoulos, K
Sirignano, J. and Spiliopoulos, K. (2020). Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics , 80(2):725--752
2020
-
[33]
Trevisan, D. (2023). Wide deep neural networks with G aussian weights are very close to G aussian processes. https://arxiv.org/abs/2312.11737
2023 arXiv
-
[34]
Tricomi, F. G. and Erd\'elyi, A. (1951). The asymptotic expansion of a ratio of gamma functions. Pacific J. Math. , 1:133--142
1951
-
[35]
and Hu, E
Yang, G. and Hu, E. J. (2021). Tensor programs IV : Feature learning in infinite-width neural networks. In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 11...
2021
-
[36]
Zavatone-Veth, J. A. and Pehlevan, C. (2021). Exact marginal prior distributions of finite B ayesian neural networks. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems
2021
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.