{"id":"1efc3727-4746-44b4-bd8e-91c6e7884914","arxiv_id":"2506.08764","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A general random matrix stability theorem shows when sparse or correlated neural-network weights still preserve stable Jacobians, and derives the required weight-rescaling.","lead":"This paper proves mathematical conditions under which deep networks keep their gradients from exploding or vanishing after weight pruning or after correlations are introduced between weights. It gives the required rescaling of the kept weights and a correlation threshold, using recent random matrix theory.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1 is proved for exogenous Bernoulli ReLU signs; in the actual network the signs are functions of the same weights, so the free-probability step does not cover the real Jacobian, and the exact limit equalities in Sections 4-5 rest on an unverified decoupling.","rationale":"The reader's weakest-assumption analysis and mine converge on the same load-bearing point: Theorem 3.1 is rigorous for an idealized model where ReLU sign matrices are exogenous Bernoulli coins, but the paper applies it to actual ReLU networks by invoking Approximation 2.1. My reading sharpens the concern: the proof structure of Theorem 3.1 requires independence of the H_k matrices, which fails structurally whenever D_k is a function of W_k, and the empirical section does not test the needed independence in the sparse or correlated regimes. This is not a disagreement with the random matrix theory used inside Theorem 3.1; the gap is in the transfer from the theorem's assumptions to the neural network. The paper is explicit about the approximation, so a conditional verdict is the right level: the theorem itself can stand, and the applications are plausible, but the exact equality of limits in Sections 4-5 is not yet justified. I therefore keep the reader's CONDITIONAL verdict unchanged rather than moving to accept or reject. The proposed L=2 computational comparison is a concrete, finite-width test that would distinguish the actual Jacobian from the exogenous-D surrogate and would settle whether the approximation preserves the spectral norm limit, not just the exponential growth rate.","tokens_in":18794,"tokens_out":17704,"duration_ms":228177,"concrete_test":"Fix L=2 (or L=3), a fixed nonzero input x, and i.i.d. critical Gaussian weights. For widths n=256,512,1024,2048, estimate E||W_2 D_1 W_1|| with D_1 = diag(1_{W_1 x>0}) and compare it with E||W_2 Δ W_1|| where Δ is an independent i.i.d. Bernoulli(1/2) matrix. If the extrapolated n→∞ limits differ, Approximation 2.1 is not asymptotically exact even for i.i.d. weights. Repeat the same comparison for magnitude-pruned Gaussian weights with the Theorem 4.2 scaling and for correlated Gaussian weights of Theorem 5.1. As a direct check of Assumption (A3), compute the chi-square statistic between entries of D_1 and entries of B_1⊙W_1 in each setting; a nonvanishing dependence would show that the exogenous-D model used in Theorem 3.1 is not the actual network model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof of Theorem 3.1 invokes [BvH24, Theorem 3.29] on independent copies H_k,...,H_L, where H_k is built from M_k = B_k⊙W_k D_{k-1}. That independence is guaranteed only by Assumptions (A2)-(A3), which make each D_k an exogenous i.i.d. Bernoulli(1/2) diagonal matrix independent of all weights and masks. In the actual network, D_k = diag(1_{(B_k⊙W_k)φ(Y_{k-1})>0}) is a deterministic function of B_k and W_k, so H_k and H_{k+1} share W_k through D_k and are not independent. Approximation 2.1 is the sole bridge between the theorem's model and the real Jacobian. Section 6.1 verifies this approximation only for i.i.d. Gaussian weights, using scatter plots and a single chi-square test on D entries; it does not test dependence between D_k and (B_k,W_k), nor any sparse or correlated setting. For magnitude pruning, B_k is a threshold function of |W_k|, the surviving weights are truncated Gaussians, and the pre-activations are not Gaussian, so no argument shows that the independent-coin model is asymptotically exact even in distribution, let alone for the spectral norm of the product. Consequently the equalities lim_n ||J^B_1|| = lim_n ||J^iid_1|| claimed in Theorems 4.1, 4.2, 4.3, and 5.1 are not established for the actual Jacobian. If this decoupling assumption fails, the central stability conclusions hold only for a surrogate model with exogenous activation signs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the spectral norm of the input-output Jacobian of deep MLPs with ReLU activations, aiming to extend classical critical-initialization results beyond i.i.d. Gaussian weights. The central result (Theorem 3.1) is a stability theorem for products of masked random weight matrices, proved by invoking strong asymptotic freeness results of Brailovskaya--van Handel. The theorem assumes that the ReLU sign matrices D_1,...,D_L are exogenous i.i.d. Bernoulli(1/2) matrices independent of the weights and masks. The paper then applies this theorem to randomly pruned networks, magnitude-pruned networks with deterministic or random thresholds, and Gaussian weights with weak correlations, deriving explicit scaling rules that make the Jacobian norm asymptotically equal to that of a critically initialized i.i.d. network. The theoretical results are complemented by numerical experiments on pruning, training, and correlation strength.","tokens_in":19132,"tokens_out":32640,"duration_ms":313187,"significance":"If the main results were fully established, the paper would extend the initialization-theory toolbox to two practically relevant settings: pruned networks and weakly correlated weights. The proof strategy is modern and the paper makes no attempt to fit constants: all scaling factors are derived from moment conditions, and the theory is benchmarked against external random matrix theory results. The random-pruning scaling and the correlated-Gaussian threshold are crisp, falsifiable predictions. However, the central theorem is conditional on a decoupling approximation (Approximation 2.1) that is verified empirically only in the i.i.d. Gaussian case, and the magnitude-pruning scaling contains a substantial algebraic error. These issues currently prevent the paper from delivering its advertised 'rigorous guarantees' for the actual Jacobian of pruned or correlated networks.","major_comments":[{"comment":"The proof of Theorem 3.1 requires the sign matrices D_1,...,D_L to be independent of the weights and masks (Assumptions A2--A3). In the actual network, however, D_k is a deterministic function of W_k, B_k, and the previous activations. The paper's only bridge from the theorem's model to the real Jacobian is Approximation 2.1, but Section 6.1 verifies this approximation only for i.i.d. Gaussian weights, through scatter plots and a single pairwise chi-square test on entries of D_l; it does not test dependence between D_k and (W_k,B_k), nor does it cover sparse or correlated weights. For magnitude pruning (Theorems 4.2 and 4.3), B is a threshold function of |W|, and the surviving weights are truncated Gaussians, so the pre-activations are not Gaussian and no argument is given for the independent-coin model even in distribution. Consequently the equalities lim_n ||J^B_1|| = lim_n ||J^iid_1|| claimed in Theorems 4.1, 4.2, 4.3, and 5.1 are not established for the actual Jacobian; they hold only for a surrogate model with exogenous sign matrices. This is the central gap of the paper and should be addressed directly, either by proving the decoupling in each new setting or by clearly rephrasing the theorems as results for the surrogate model and providing explicit empirical validation for pruned and correlated networks.","section":"Section 3, Approximation 2.1, and Sections 4--5"},{"comment":"The mask scaling in Theorem 4.2 appears to be incorrect. The proof claims that E|w_{i,k}|^2 1_{|w_{i,k}|>t} ~ e^{-nt^2/2} sqrt(2nt^2)/pi, but this expression is not the correct asymptotic. With w ~ N(0,2/n) and N ~ N(0,1), we have w^2 = (2/n) N^2, so the event |w|>t is N^2 > nt^2/2, not N^2 > nt^2, and E|w|^2 1_{|w|>t} = (2/n) E[N^2 1_{N^2>nt^2/2}] ~ (2t/sqrt(n pi)) e^{-nt^2/4}. Using the mask b defined in the theorem, one obtains E|b w|^2 ~ 2 sqrt(2) e^{nt^2/4}/n^2. Under the theorem's scaling condition nt^2 <= 4 log n - c log log n, this tends to 0, not to 2/n. Condition (ii) of Theorem 3.1 therefore fails, and the claimed equality with ||J^iid_1|| is not justified. The constant in the mask must be rederived; with a corrected constant satisfying condition (ii), the proof of condition (i) will also need to be revisited, as the present c>7 may no longer suffice.","section":"Proof of Theorem 4.2"},{"comment":"There is an inconsistency between the statement of Theorem 3.1 in the Gaussian case and its proof. The theorem states condition (ii') as |(2/n) sum_{k=1}^n |b_{i,k}|^2 - 1| -> 0, but the proof of the Gaussian case bounds ||E[H^2|D] - 1|| using the term |(1/n) sum_k |b_{i,k}|^2 - 1|, not the factor 2/n. For the unpruned case B=1, the stated condition gives |2-1| = 1, so Theorem 3.1 as stated does not apply to the setting of Theorem 5.1. This appears to be a typo, but since Theorem 5.1 depends on it, the statement must be corrected and the proof aligned with the corrected condition.","section":"Theorem 3.1, condition (ii')"}],"minor_comments":[{"comment":"The Jacobian product J^B_k uses D_{l-1} for l=k,...,L, but Assumptions (A2) define D_1,...,D_L; the indexing of D_0 is never specified. Please clarify whether D_0 is another independent copy or D_1.","section":"Section 3, notation"},{"comment":"The numerical claim 'log^10(256)/256 ~ 0.009' is unclear; log^4(256)/256 is about 3.69, so the stated asymptotic condition 1-s_n >> log^4(n)/n is not satisfied at the experimental width n=256. Please check the displayed threshold and its relation to the phase transition shown in Figure 2.","section":"Section 4.1, page 9"},{"comment":"The independence verification is limited to two randomly selected entries of D_l and to scalar aggregates T_W and T_D. This does not establish independence of the full matrix D_l from W_l, nor does it cover the sparse or correlated settings used in later theorems. The text should state this limitation explicitly.","section":"Section 6.1"},{"comment":"The conclusion states that 'stability at initialization implies stability throughout training (under some approximations)', but the paper only analyzes the Jacobian at initialization. This claim is not supported by the results and should be removed or substantially qualified.","section":"Section 7, Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is interesting and the use of strong asymptotic freeness is appropriate, but the load-bearing issues are real: the decoupling approximation is only verified for the i.i.d. Gaussian case, and the magnitude-pruning scaling contains a concrete algebraic error. These are fixable in principle, but they are not merely presentation issues. I would also flag that the title and abstract promise 'rigorous guarantees' for sparse and correlated networks, which the current manuscript does not deliver; a revision should either provide the missing support for Approximation 2.1 in those settings or sharply delimit the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here’s the short version: this is worth engaging, but read Theorem 3.1 as a theorem about a surrogate model where the ReLU sign matrices are exogenous coins, not about the actual Jacobian of a real MLP. The applications inherit that caveat. If you need guaranteed statements about pruned or correlated networks, they aren’t there yet; if you want clean scaling heuristics with an honest theoretical backbone, this is a good paper.\n\nWhat’s new: it extends the edge-of-chaos program from i.i.d. Gaussian weights to sparse masks and weakly dependent Gaussian weights, using strong asymptotic freeness results from Bandeira–Boedihardjo–van Handel and Brailovskaya–van Handel. The checks of the free-probability hypotheses are careful, the mask-scaling formulas are explicit, and there is no curve-fitting: the scaling factors come from moment conditions. Theorem 4.1 and the deterministic-threshold Theorem 4.2 are clean applications. The paper is also honest about scope: MLPs, pruning at initialization, fixed input.\n\nThe soft spot is the load-bearing one. Approximation 2.1 says D_k is an i.i.d. Bernoulli diagonal independent of all weights and masks. That is exactly what makes the H_k independent so the free-probability theorem applies. In the real network, D_k = diag(1_{(B_k⊙W_k)φ(Y_{k-1})>0}) is a function of the same weights, so the independence is only asymptotic, and only in the i.i.d. Gaussian case is any evidence supplied. Section 6.1 checks marginal Bernoulli-ness and pairwise independence of entries for i.i.d. weights, not independence from W and B, and not sparse or correlated weights. For magnitude pruning the surviving weights are truncated Gaussians and pre-activations are not Gaussian, so the surrogate gap is widest there. The exact equalities in Theorems 4.1–4.3 and 5.1 are therefore not established for the actual Jacobian; they are established for the exogenous-sign model, with only indirect empirical support.\n\nMinor: Theorem 4.3 proves single-layer spectral closeness between random-threshold and deterministic-threshold masks, then jumps to Jacobian product convergence; for fixed depth and bounded factor norms this is repairable, but the argument is missing. Also the conclusion line “stability at initialization implies stability throughout training” is a stretch even with the “under some approximations” hedge.\n\nWho it’s for: people working on initialization and pruning theory, and anyone applying free-probability tools to neural networks. It deserves a serious referee: the conditional theorem and the scaling recipes are worth having, but the referee should push for either a proof of Approximation 2.1 in the sparse/correlated settings or a reformulation of the theorems as surrogate-model results.","headline":"A solid, clean application of recent free-probability results to Jacobian stability, with genuinely useful scaling prescriptions; the main caveat is that the central theorem is proved for exogenous ReLU signs, and the bridge to real networks is only verified for i.i.d. Gaussian weights.","tokens_in":19675,"tokens_out":5440,"would_cite":true,"duration_ms":65430,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["60B20","15B52","46L54","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes a general stability theorem: for a deep ReLU network, the spectral norm of the input–output Jacobian converges, as width grows, to the norm of a product of free semicircular elements, and this universal limit is…","keywords":["Jacobian spectral norm","deep neural networks","exploding and vanishing gradients","random matrix theory","free probability","network pruning","weight correlation","edge of chaos"],"falsifier":"Take a randomly pruned MLP (e.g., width 256, sparsity 0.5, mask scaling $(1-s)^{-1/2}$) and test the joint law of the pre-activation sign diagonals $D_l$ against the weights and mask: if a chi-square test of independence between entries of $D_l$ and mask or weight values, or between pairs of entries of $D_l$, rejects at the asymptotic level as width grows, or if the empirical distribution of signs visibly departs from Bernoulli(1/2), then Approximation 2.1 fails and the universal limit in Theorem 3.1 loses its justification. A direct quantitative check is to compute $\\|J^B_1\\|$ for increasing widths at fixed depth and compare the values to the predicted free-probability limit $\\|s_1 \\cdots s_L\\|$; a systematic gap that does not close as $n$ grows would indicate non-universal behavior.","tokens_in":18606,"feed_emoji":"📉","tokens_out":6777,"duration_ms":74215,"temperature":0.7,"pith_summary":"The paper tries to establish that Jacobian stability—the spectral norm of the input–output Jacobian growing only sub-exponentially with depth—holds far beyond the textbook setting of i.i.d. Gaussian weights. Its central theorem shows that, under a fair-coin model of ReLU sign patterns, the Jacobian norm of an MLP converges as width grows to the norm of a product of free semicircular elements, a universal quantity independent of the weight distribution. From this it derives concrete scaling prescriptions for pruned networks: random pruning needs a $(1-s)^{-1/2}$ rescaling of the surviving weights, while magnitude-based pruning needs a different, threshold-dependent scaling; in both cases the pruned network's Jacobian norm matches the critically initialized i.i.d. network. For correlated Gaussian weights, the same limit holds as long as correlations decay faster than $n^{-3/2}/\\log n$. If correct, this extends edge-of-chaos initialization theory to the structured and dependent weight distributions that arise in pruning, fine-tuning, and data-dependent initialization.","feed_headline":"Rescaling pruned weights restores stable gradients in deep ReLU nets","feed_subtitle":"A universality theorem gives scaling rules that keep Jacobian norms from exploding or vanishing with depth after pruning or correlation.","key_machinery":"The engine is strong asymptotic freeness for products of structured random matrices. The Jacobian is expressed through trace moments of a symmetrized matrix $H = \\begin{pmatrix} 0 & M \\\\ M^* & 0 \\end{pmatrix}$ with $M = B \\odot W D$; the theorem verifies the conditions under which trace moments of powers of $H$ converge to those of a freely independent product of semicircular elements $s_k \\cdots s_L$. This free-probability limit is the universal baseline against which all the pruning and correlation results are compared.","core_discovery":"The paper's central claim is Theorem 3.1: under Assumptions (A1)–(A3), which require subgaussian weights, bounded masks, and ReLU sign diagonals that are i.i.d. Bernoulli(1/2) independent of weights and masks, together with conditions on the masked weights (vanishing means, row-wise second moments converging to 1, and a vanishing mask-size-to-width ratio), the spectral norm of the Jacobian $J^B_k = \\prod_{l=k}^L (B_l \\odot W_l) D_{l-1}$ converges as $n \\to \\infty$ to $\\|s_k \\cdots s_L\\|$, the norm of a product of freely independent semicircular elements. A Gaussian variant under Assumption (A$'$1) yields the same convergence in expectation for centered Gaussian entries with variance $2/n$ and covariances $O(n^{-5/2}/\\log n)$. The point is universality: the limit does not depend on the fine structure of the weights or masks, so stability of the Jacobian is preserved whenever the masked weights are close enough to critical i.i.d. weights.","pith_inferences":["If the universality theorem transfers to convolutional or residual architectures with the same sign-independence, the scaling prescriptions here give a principled way to renormalize weights after structured pruning, not just unstructured MLP pruning.","The $O(n^{-3/2}/\\log n)$ correlation threshold gives a quantitative target for how much shared noise a fine-tuning or meta-learning procedure can inject before gradient propagation degrades; this could be tested by measuring Jacobian norms of networks fine-tuned from a common pretrained initialization with increasing shared-noise components.","For magnitude pruning, using the random-pruning scaling $(1-s)^{-1/2}$ is provably suboptimal: it leaves the Jacobian unstable in the infinite-width limit, as the paper's numerical comparison of scaling factors illustrates, so pruning algorithms should report their scaling rule along with the mask."],"forward_implications":["Randomly pruned networks: with mask entries $(1-s_n)^{-1/2} \\mathrm{Bernoulli}(1-s_n)$, the Jacobian norm has the same limit as the i.i.d. critical network, provided $1-s_n \\gg \\log^4 n / n$; without rescaling, the norm grows exponentially in depth.","Magnitude-pruned networks: keeping weights above a threshold $t$ yields stability after rescaling by $\\left(\\sqrt{2\\pi}\\, e^{n t^2/2} / (t n^{3/2})\\right)^{1/2}$ (Theorem 4.2), and keeping the $r = \\lceil n \\log^c n\\rceil$ largest weights needs the random-threshold variant (Theorem 4.3) with $c > 7/4$.","Correlated Gaussian weights with $\\mathrm{corr}(W_{ij}, W_{ml}) = O(n^{-3/2}/\\log n)$ for distinct entries give the same expected Jacobian norm limit as i.i.d. weights, so mild training-induced correlations do not break stability.","An edge of stability in sparsity appears near $1-s_n \\sim n^{-1}$ up to logarithmic factors; empirical results show the scaling restores trainability of pruned MLPs on Fashion-MNIST."],"supporting_citations":[{"why":"Supplies the strong asymptotic freeness theorem that the paper's Theorem 3.1 verifies conditions for; it is the engine of the universality result.","marker":"[BvH24]"},{"why":"Gives the matrix concentration and free probability theorem used in the Gaussian (A$'$1) part of the proof to obtain convergence in expectation.","marker":"[BBvH23]"},{"why":"Establishes the i.i.d. baseline $\\|J^{\\mathrm{iid}}_1\\| \\approx \\Theta_L((\\sigma_w^2/2)^{L/2})$ that all stability comparisons are made against, along with the critical value $\\sigma_w^2 = 2$.","marker":"[PSG17]"},{"why":"Provides the edge-of-chaos and activation-function analysis that motivates the stability definition and the sub-exponential growth criterion.","marker":"[HDR19]"},{"why":"Underwrites Approximation 2.1: wide-layer pre-activations are asymptotically i.i.d. Gaussian and independent of the weights, so ReLU signs behave like independent fair coins.","marker":"[LBN+18]"},{"why":"Defines magnitude-based pruning, the score-based method analyzed in Theorems 4.2 and 4.3.","marker":"[HPTD15]"},{"why":"Documents that unstructured sparsity disrupts gradient flow, providing the motivation for the pruning-stability analysis.","marker":"[EIKD22]"},{"why":"Gives the expected-norm bound for random matrices used in the proof of Theorem 4.3 to compare random-threshold and deterministic-threshold pruning.","marker":"[Seg00]"}],"fun_headline_variants":["Universal stability rule for Jacobians in deep nets with pruning","Spectral stability proven for sparse and correlated deep nets","Jacobian stability extends beyond i.i.d. weights in deep nets","Universality ensures Jacobian stability for non-i.i.d. deep nets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the ReLU sign pattern in each layer is an independent fair coin that is independent of the weights and the pruning mask—a fact verified empirically only for i.i.d. Gaussian weights, and potentially false for sparse or correlated weights.","fun_headline_variants_meta":{"raw":{"variants":["Universal stability rule for Jacobians in deep nets with pruning","Spectral stability proven for sparse and correlated deep nets","Jacobian stability extends beyond i.i.d. weights in deep nets","Universality ensures Jacobian stability for non-i.i.d. deep nets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000793,"raw_usage":{"total_tokens":3477,"prompt_tokens":912,"completion_tokens":2565,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":2494}},"tokens_in":528,"tokens_out":2565,"duration_ms":22611,"temperature":1.0,"reasoning_tokens":2494,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:02:35.158146+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a randomly pruned MLP (e.g., width 256, sparsity 0.5, mask scaling $(1-s)^{-1/2}$) and test the joint law of the pre-activation sign diagonals $D_l$ against the weights and mask: if a chi-square test of independence between entries of $D_l$ and mask or weight values, or between pairs of entries of $D_l$, rejects at the asymptotic level as width grows, or if the empirical distribution of signs visibly departs from Bernoulli(1/2), then Approximation 2.1 fails and the universal limit in Theorem 3.1 loses its justification. A direct quantitative check is to compute $\\|J^B_1\\|$ for increasing widths at fixed depth and compare the values to the predicted free-probability limit $\\|s_1 \\cdots s_L\\|$; a systematic gap that does not close as $n$ grows would indicate non-universal behavior.","supporting_citations":[],"review_version":1}