{"id":"2dc2113a-8c1d-40d3-bf62-2d98b4e81fa6","arxiv_id":"2411.15267","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Deep linear neural networks in the proportional depth-width limit converge to a nontrivial mixture of Gaussians, with posterior output correlations that depend on the observed labels.","lead":"When depth and width of a random linear neural network grow together in proportion, the network's output distribution does not become Gaussian; it becomes a mixture of Gaussians. This gives a rare rigorous example in which a Bayesian network prior and posterior encode label-dependent output correlations, which the standard infinite-width Gaussian limit cannot.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The reader's identified weakest assumption is Proposition 2.1, and it is indeed the foundation of the paper. I examined it and find the representation credible, so the asymptotic results do not rest on a shaky base. The proof of Proposition 3.2 is intricate but internally consistent: the diagonal limit (Lemma 4.1) has the correct variance and drift, the off-diagonal sums converge to iterated Itô integrals rather than Stratonovich integrals, and the filtration construction in Section 4.3 makes the recursive stochastic integrals well-defined. The posterior transfer (Propositions 3.3 and 3.4) follows from weak convergence of priors and bounded continuous likelihoods. My only reservation is a possible typographical prefactor in Eq. (8), which does not change the verdict unless it is real in the original PDF. The paper would benefit from an independent numerical check, but I see no mathematical reason to reject or condition acceptance.","tokens_in":24043,"tokens_out":46082,"duration_ms":464101,"concrete_test":"Run a Monte Carlo check of Proposition 3.2 for D=2, N0=2, P=1, a=1: simulate V_{L,N} for N=400 and 800 from the Bartlett distribution, form V_{L,N} Z x, and compare the empirical distribution of a fixed coordinate with the theoretical limit obtained by Euler-Maruyama discretization of the stochastic integrals in Eq. (8). A KS test or QQ-plot will show whether the finite-N law approaches the claimed limit; the test also resolves the a^{h/2} versus e^{(a/2) ln a} prefactor question.","verdict_should_be":"UNCHANGED","load_bearing_attack":"I found no load-bearing flaw in the central argument. Proposition 2.1, though cited, is sound: the freedom to replace each V_l by any other square root of the same Wishart matrix is preserved in distribution because the orthogonal difference can be pushed through the product and absorbed by the independent Gaussian matrix at the right end, using the left-invariance of the accumulated factor. The Skorohod construction in Proposition 4.3 couples the Bartlett diagonals and off-diagonal Gaussians consistently, and the convergence of the discrete product sums to the iterated stochastic integrals in Eq. (8) follows from Lemma B.3 via the uniform convergence of the exponential integrands in Lemma 4.5. The only point I would check is the constant in Eq. (8): the proof concludes with a prefactor a^{h/2}, while the displayed formula in the provided text reads e^{(a/2) ln a}; this is almost certainly an OCR artifact, but if it is not, the limiting matrix in Proposition 3.2 would be mis-specified by a deterministic factor.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies deep linear neural networks with Gaussian weights under the proportional scaling L/N -> a, where L is depth and N is width. Using the representation of the output as V_L ... V_1 Z X / sqrt(N0 lambda*_L), with V_l independent Cholesky factors of Wishart matrices, it proves that for a > 0 the product V_bar_{L,N} converges in distribution to a random lower-triangular matrix V_bar_infty whose entries are explicit iterated stochastic integrals driven by independent Brownian motions. This gives a non-Gaussian mixture-of-Gaussians prior for the network output, in contrast to the Gaussian NNGP limit recovered when a = 0. The paper also derives analogous mixture representations for posterior and predictive distributions under a Gaussian likelihood, and it proves that the posterior covariance depends on the labels, a feature absent in the NNGP regime.","tokens_in":24214,"tokens_out":24948,"duration_ms":229867,"significance":"If the result stands, it provides the first rigorous multi-input, multi-output characterization of the proportional infinite-width/infinite-depth limit for linear networks, extending Hanin's single-input/single-output analysis and complementing the physics literature on proportional limits. The proofs are substantial and mostly self-contained: exact finite-N moments, a Skorokhod coupling of the Bartlett entries, variance bounds, and convergence of discrete product sums to iterated stochastic integrals. No parameter is fitted, and the limiting objects are defined explicitly, which makes the claims falsifiable and checkable. The main external input is Proposition 2.1, cited from Bassetti et al. (2024); the representation is standard and appears sound. The paper is a solid contribution to the theory of non-Gaussian scaling limits for Bayesian neural networks.","major_comments":[],"minor_comments":[{"comment":"The displayed definition of H(r) contains the prefactor e^{(a/2) ln a}, whereas the proof in Section 4.4 concludes that the limiting term has prefactor a^{h/2}; since h varies with r, the printed formula would mis-specify the limit matrix unless corrected.","section":"Section 3.1, Eq. (8)"},{"comment":"The displayed expression for H_L(r) includes a summation over r in R^h_{k,i}; as written this would make H_L(r) depend on all r, but the subsequent lines treat r as fixed, so the redundant summation should be removed.","section":"Section 4.4, definition of H_L(r)"},{"comment":"The definition of R_N is missing parentheses around the limiting linear combination; as written it can be read as subtracting Z^{(r_j)}_t rather than the difference Z^{(r_{j-1})}_t - Z^{(r_j)}_t.","section":"Lemma 4.5, proof"},{"comment":"The use of the strong approximation theorem from Kallenberg (1997, Thm 12.20) should be stated explicitly for the triangular-array setting, since the increments change with N; a brief justification or a reference to a triangular-array version would remove ambiguity.","section":"Proposition 4.3, proof"},{"comment":"There are numerous typographical errors, including 'representaion' in the Introduction, 'Morevoer' in Proposition 3.4, 'rigth-continuous' in Appendix B, and the title line 'DEE P'; these should be cleaned up in a revision.","section":"Throughout"},{"comment":"The text uses Moore-Penrose inverses in the definitions of Sigma* and m* without comment; Remark 1 clarifies the invertible case, but a brief sentence in the main text explaining the generalized inverse convention would improve readability.","section":"Section 3.3, Eqs. (15)-(16)"}],"recommendation":"minor_revision","confidential_remarks":"The only point I would ask the authors to verify is the prefactor in Eq. (8); if the intended exponent is h/2 rather than a/2, the correction is trivial but important because the printed definition is part of the main theorem statement. I see no other correctness concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plain take: this paper gives the first rigorous multi-output proportional depth-width limit for deep linear Bayesian networks and shows the posterior covariance becomes label-dependent. The math is detailed and the central argument holds up.\n\nWhat is actually new: Hanin 2024 had the D=1 prior limit. The extension to D>1 prior via multiple stochastic integrals, and the posterior characterization, are genuinely new. The proof is substantial: exact moment computations, a Skorohod coupling, and convergence of discrete multiple sums to stochastic integrals. The posterior formula in Proposition 3.4 is explicit, and the qualitative claim about label-dependent correlations is supported by the equations.\n\nSoft spots: the definition of H(r) in Eq. (8) is displayed in a way that seems to conflict with the proof's prefactor a^{h/2}; the proof is clear that the limit is a^{h/2} e^{Z} times the integral, so I suspect the displayed e^{(a/2) ln a} is a rendering error, but it should be fixed before publication. The paper leans on Proposition 2.1 from the authors' companion paper; it is an exact representation, and the stress-test confirms it is sound, but readers should know the self-citation is load-bearing. The posterior asymptotics are derived for bounded continuous likelihood and then specialized to Gaussian; that is fine. Minor: the arXiv OCR has garbled formulas, which is not the authors' fault but could confuse readers.\n\nFor whom: anyone working on Bayesian neural networks, NNGP limits, and feature learning in linear nets. It is a rigorous theory paper; no experiments, but it does not need them.\n\nRecommendation: send to peer review. It deserves a serious referee. The main things to check are the constant in Eq. (8) and the clarity of the filtration construction.","headline":"Rigorous multi-output proportional depth-width limit for deep linear Bayesian networks, with an explicit label-dependent posterior; worth a serious referee once the Eq. (8) constant is checked.","tokens_in":24694,"tokens_out":3173,"would_cite":true,"duration_ms":29752,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["60F05","60H05","60B20"],"pacs":[],"model":"deepseek-v4-flash","headline":"In the proportional infinite-width, infinite-depth limit, the output prior of a deep linear network is an explicit, non-Gaussian mixture of Gaussians, and the same structure survives in the posterior.","keywords":["deep linear neural networks","proportional limit","infinite-width limit","Neural Network Gaussian Process","mixture of Gaussians","multiple stochastic integrals","Wishart matrices","Bayesian inference"],"falsifier":"Take $D=2$, set $a=1$, and simulate $\\bar{V}_{L,N}=V_L\\cdots V_1$ for large $N$ with $L=N$. Proposition 3.2 predicts the off-diagonal entry converges in distribution to $H(1,2)=e^{Z^{(2)}_1}\\int_0^1 e^{Z^{(1)}_t-Z^{(2)}_t}\\,dW^{(2,1)}_t$, with $Z^{(k)}_t=\\sqrt{1/2}\\,W^{(k)}_t-(k/2)t$; comparing the simulated empirical distribution of the off-diagonal entry to this law (in mean, variance, and tail quantiles) would settle the claim. A cheaper necessary check: the diagonal entry $\\bar{V}^{1,1}_{L,N}$ should satisfy $E[\\bar{V}^{1,1}_{L,N}]\\to e^{-1/4}$ and $E[(\\bar{V}^{1,1}_{L,N})^2]\\to 1$ as $L/N\\to 1$; any systematic mismatch beyond Monte Carlo error would disprove Proposition 3.2.","tokens_in":2241,"feed_emoji":"🧠","tokens_out":2528,"duration_ms":139224,"temperature":0.7,"pith_summary":"This paper studies fully connected deep linear networks whose random weights are Gaussian, taking both the number of layers L and the width N to infinity while their ratio L/N tends to a fixed constant a>0. It claims that in this proportional limit the prior law of the network outputs is not the Gaussian process found in the usual infinite-width regime, but a nontrivial mixture of Gaussians with an explicit mixing distribution built from multiple stochastic integrals. The same mixture structure holds for the posterior under a bounded continuous likelihood, and for a Gaussian likelihood the posterior and predictive distributions are explicit mixtures of normals whose mixing measure depends on the training labels. This matters because the infinite-width Gaussian process forces output components to be independent and makes their covariance blind to the labels, while this proportional limit keeps correlations and gives the covariance a label-dependent component.","feed_headline":"Deep networks stop being Gaussian when depth grows with width","feed_subtitle":"New limit law is an explicit mixture of Gaussians, so output correlations can depend on the training labels","key_machinery":"The central object is the random product $\\bar{V}_{L,N}=V_L\\cdots V_1$ of independent $D$-dimensional Bartlett matrices, i.e. lower-triangular Cholesky factors of Wishart matrices with $N$ degrees of freedom and scale matrix $N^{-1}1_D$; this product is exactly the non-Gaussian part of the prior representation. The proof machinery is a Skorokhod coupling: diagonal squared entries behave as random walks on log-Gamma increments and converge to the drifted Brownian processes $Z^{(k)}_t=\\sqrt{a/2}\\,W^{(k)}_t-(k/2)at$, while off-diagonal entries are coupled to independent Brownian increments of size $\\sqrt{L/N}$. This turns each term in the expansion of $\\bar{V}_{L,N}$ into a discrete Riemann sum that converges to a multiple stochastic integral $H(r)$, and the limit matrix $\\bar{V}_\\infty$ is assembled from those integrals.","core_discovery":"Let $f_{L,N}(X|\\theta)$ be the $D$-dimensional outputs of a deep linear network with $L$ hidden layers of width $N$, evaluated on $P$ inputs collected in $X$, with Gaussian weights of layer-wise inverse variance $\\lambda_\\ell$. The paper's starting point is the representation $f_{L,N}(X|\\theta)\\stackrel{d}{=}(N_0\\lambda^*_L)^{-1/2}\\bar{V}_{L,N}ZX$, where $Z$ is a $D\\times N_0$ standard Gaussian matrix and $\\bar{V}_{L,N}=V_L\\cdots V_1$ is a product of independent Cholesky factors of Wishart matrices. Proposition 3.2 proves that if $L/N\\to a$ with $0<a<\\infty$, then $\\bar{V}_{L,N}$ converges in distribution to an explicit lower-triangular matrix $\\bar{V}_\\infty$ whose diagonal entries are exponentials of independent drifted Brownian motions at time $1$ and whose off-diagonal entries are sums of multiple stochastic integrals $H(r)$. Consequently the output prior converges to the law of $(N_0\\lambda^*_\\infty)^{-1/2}\\bar{V}_\\infty ZX$, which is a genuine mixture of Gaussians. Propositions 3.3 and 3.4 carry this convergence to the posterior: with a bounded continuous likelihood the posterior converges weakly to the corresponding mixture, and under a Gaussian likelihood both the posterior and the predictive are mixtures of normals with mean and covariance functions of a label-dependent mixing measure $Q_\\infty(dQ|\\tilde{X},y_{1:P})$.","pith_inferences":["The explicit multiple-integral representation of $\\bar{V}_\\infty$ gives a route to finite-width corrections: expanding around the Brownian limit should yield quantitative rates of approach to the mixture, of the same kind that exist for the Gaussian infinite-width regime.","A testable prediction of the paper's posterior formulas is that, for fixed inputs, the posterior predictive variance on a test point changes when the training labels are permuted, and this label-dependence should disappear continuously as $a\\to 0$.","The Bartlett/Wishart mechanism is specific to linear activations; whether a similar multivariate stochastic-integral mixture exists for nonlinear activations is left open, but the paper's Brownian-coupling strategy suggests the main obstacle is finding a tractable analogue of the product-of-Wishart representation."],"forward_implications":["Whenever $L/N\\to a>0$, the output prior is a nontrivial Gaussian mixture, so output components are dependent and the large-network prior is no longer the featureless Gaussian process of the infinite-width limit.","The same weak convergence holds for posteriors under any bounded continuous likelihood; with a Gaussian likelihood the asymptotic posterior and posterior predictive are explicit mixtures of normals, with formulas given in Proposition 3.4.","The asymptotic mixing measure for the posterior is the law of $\\bar{Q}_\\infty=\\bar{V}_\\infty\\bar{V}_\\infty^\\top$ reweighted by a function $\\Psi$ that depends on the training labels, so conditioning on data changes the covariance structure rather than only shifting the mean.","The boundary case $a=0$ recovers exactly the Neural Network Gaussian Process limit, showing that a strictly positive depth-to-width ratio is the minimal change that breaks Gaussianity for linear networks.","In the scalar-output case the limit reduces to a log-normal random variable, matching earlier single-input, single-output results; the multidimensional prior and posterior mixture statements are the new content."],"supporting_citations":[{"why":"Gives the prior representation as a product of Cholesky factors of Wishart matrices times a Gaussian matrix, from which all asymptotic statements follow.","marker":"Bassetti et al. (2024)"},{"why":"Introduces the effective-depth proportional limit and the non-Gaussian mixture phenomenon for one input/output, which this paper generalizes.","marker":"Hanin (2024)"},{"why":"Establishes the infinite-width NNGP limit that the paper recovers when a=0 and uses as the comparison baseline.","marker":"Lee et al. (2018)"},{"why":"Provides the Bartlett decomposition used to approximate the Cholesky factors by Brownian increments.","marker":"Kshirsagar (1959)"},{"why":"Gives the gamma-ratio asymptotics used to identify the diagonal log-normal limit.","marker":"Tricomi and Erdélyi (1951)"},{"why":"Supplies stochastic-integral stability results used to replace discrete sums by multiple integrals.","marker":"Karatzas and Shreve (1991)"}],"fun_headline_variants":["Depth and width scaling turns net priors into Gaussian mixtures","When depth matches width, neural nets leave Gaussian regime","Proportional depth-width limit: non-Gaussian mixtures appear","Deep linear nets become mixtures as depth and width grow together"],"cache_read_input_tokens":27008,"weakest_assumption_plain":"The load-bearing premise is that the prior law of the network outputs equals the law of a product of independent random triangular matrices (the Cholesky square roots of Wishart matrices) acting on a Gaussian matrix; every asymptotic statement in the paper follows from that equality of laws, so if the representation failed the mixture limit would not describe the network.","fun_headline_variants_meta":{"raw":{"variants":["Depth and width scaling turns net priors into Gaussian mixtures","When depth matches width, neural nets leave Gaussian regime","Proportional depth-width limit: non-Gaussian mixtures appear","Deep linear nets become mixtures as depth and width grow together"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00083,"raw_usage":{"total_tokens":3663,"prompt_tokens":1019,"completion_tokens":2644,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":2578}},"tokens_in":635,"tokens_out":2644,"duration_ms":19433,"temperature":1.0,"reasoning_tokens":2578,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T14:49:50.453490+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take $D=2$, set $a=1$, and simulate $\\bar{V}_{L,N}=V_L\\cdots V_1$ for large $N$ with $L=N$. Proposition 3.2 predicts the off-diagonal entry converges in distribution to $H(1,2)=e^{Z^{(2)}_1}\\int_0^1 e^{Z^{(1)}_t-Z^{(2)}_t}\\,dW^{(2,1)}_t$, with $Z^{(k)}_t=\\sqrt{1/2}\\,W^{(k)}_t-(k/2)t$; comparing the simulated empirical distribution of the off-diagonal entry to this law (in mean, variance, and tail quantiles) would settle the claim. A cheaper necessary check: the diagonal entry $\\bar{V}^{1,1}_{L,N}$ should satisfy $E[\\bar{V}^{1,1}_{L,N}]\\to e^{-1/4}$ and $E[(\\bar{V}^{1,1}_{L,N})^2]\\to 1$ as $L/N\\to 1$; any systematic mismatch beyond Monte Carlo error would disprove Proposition 3.2.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the effective-depth proportional limit and the non-Gaussian mixture phenomenon for one input/output, which this paper generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the infinite-width NNGP limit that the paper recovers when a=0 and uses as the comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Bartlett decomposition used to approximate the Cholesky factors by Brownian increments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the gamma-ratio asymptotics used to identify the diagonal log-normal limit."},{"cited_title":"and Shreve, S","cited_arxiv_id":null,"evidence_quote":"Supplies stochastic-integral stability results used to replace discrete sums by multiple integrals."}],"review_version":1}