Pith. sign in

REVIEW 4 major objections 3 minor 46 references

The paper proposes Stiefel-optimized semi-orthogonal weight matrices that map the all-ones vector to itself, giving a growing rectification parameter that prevents dying ReLU and stabilizes 100-layer ReLU networks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

An initialization that maximizes a semi-orthogonal weight matrix's alignment with the all-ones vector prevents dying ReLU and keeps 100-layer ReLU networks trainable.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A clean, correct semi-orthogonal initialization with real empirical value, but the headline dying-ReLU guarantee is conditional on positive input mean, and the paper overstates it in the abstract. the 4 major comments →

arxiv 2509.00362 v1 pith:4FQ7GX63 submitted 2025-08-30 cs.LG

Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks

Hyungu Lee , Taehyeong Kim , Hayoung Choi This is my paper
classification cs.LG MSC 68T0760F05
keywords weight initializationStiefel manifoldsemi-orthogonal matrixReLUdying ReLUvariance propagationdeep feedforward networksrectified Gaussian
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a weight-initialization scheme for deep ReLU networks: each weight matrix is chosen from the semi-orthogonal matrices that maximize the sum of all entries, equivalently those that map the normalized all-ones vector to itself. This one geometric constraint does two things at once: it preserves isotropic variance through every layer, and it amplifies any constant positive input mean by the square-root width ratio each layer, so the pre-activation mean-to-spread ratio grows with depth. A growing ratio means ReLU's negative region is rarely hit, so dying ReLU is prevented, variance is not progressively cut by the activation, and gradients do not vanish. The paper derives the closed-form optimum and an efficient sampling algorithm, proves convergence of pre-activations to Gaussianity in wide layers, and reports stable training of 100-layer ReLU networks on image, few-shot, and tabular benchmarks. If correct, it shows that a parameter-free initialization alone can stabilize very deep ReLU training, provided the inputs carry a positive, roughly coordinate-constant mean.

Core claim

On the paper's own terms, the central claim is that eO_{m,n} — the semi-orthogonal matrices W satisfying ξ_m^T W ξ_n = 1, equivalently W ξ_n = ξ_m — is the right initialization family for ReLU networks. Such a W preserves isotropic covariance and turns a coordinate-constant input mean μ into μ√(n/m)1_m. With non-increasing widths, √(n/m) ≥ 1, so after ReLU the rectification parameter α = μ/σ grows with depth; the paper argues that as α grows, ReLU acts nearly as the identity, the inactive-neuron probability Φ(−α) decays, and the per-layer variance ratio tends to 1, preserving signal and gradients. The paper also proves a Gaussian-approximation theorem for pre-activations and supports the the

What carries the argument

The central object is the Stiefel manifold O_{m,n} of semi-orthogonal matrices and its subset eO_{m,n} of optima for maximizing tr(J^T W), i.e., matrices whose total entry sum is maximal, where J denotes the all-ones matrix. The load-bearing identity is the equivalence W ξ_n = ξ_m from Lemma 1: it turns 'maximally aligned with all-ones' into a linear eigenvector condition, which makes the mean update E[W x] = μ√(n/m)1_m exact and drives the growing rectification parameter in Proposition 4. The efficient construction factorizes any such W as LQ + (1/√mn)J_{m×n}, where L is an explicit closed-form Cholesky factor of I − (1/m)J and Q is a semi-orthogonal frame with last row ξ_n^T; this gives ne

Load-bearing premise

The whole mechanism assumes the network input has a positive, roughly coordinate-constant mean; if the data are standardized to zero mean, the method needs a manually imposed mean shift, and the paper's own tabular results show negative shifts make it perform at or below baselines.

What would settle it

Take a 100-layer ReLU MLP with non-increasing widths, initialize every layer from eO_{m,n}, and feed inputs with a positive coordinate-constant mean (α0 = 2 on standardized data). Before any training, record the per-layer fraction of active ReLUs and the per-layer variance ratio. The paper's mechanism predicts the active fraction approaches 1 and the variance ratio approaches 1 from below as depth grows; observing instead a decreasing active fraction or a variance ratio consistently below 1 − ε would refute the mechanism on its own terms.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • For non-increasing width architectures and inputs with a positive approximate coordinate-constant mean, deep ReLU networks initialized this way keep pre-activation variance within a (1 − ε) lower bound per layer once α is large enough, so signal does not collapse with depth.
  • The inactive-neuron probability at initialization is Φ(−α_ℓ), which decays as α grows; with the width ratio √(n/m) ≥ 1 amplifying the mean each layer, dying ReLU is mitigated in very deep nets.
  • Gradient backpropagation is stabilized because the expected diagonal of the ReLU Jacobian tends to the identity as α grows and the weights are semi-orthogonal, limiting Jacobian contraction.
  • A single closed-form matrix family serves all layers without hyperparameter tuning; the only data-dependent choice is the input mean shift, which the paper reports should be moderate (α0 = 2) on standardized tabular data.
  • Because the construction costs one QR decomposition per layer, it is practical for large widths and adds little overhead over existing orthogonal initialization.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same 'align with all-ones' principle could be tested as a regularizer or post-hoc correction for pretrained layers, not just an initialization: projecting a trained weight matrix toward eO_{m,n} might reduce neuron death during fine-tuning.
  • The mechanism is sensitive to the sign and shape of the input mean. For zero-mean or negative-mean data, α stays at zero or becomes negative, so the paper's own tabular results imply the method needs a per-dataset mean shift; a principled automatic choice of that shift would be a natural next step.
  • The theory is developed for fully connected non-expanding architectures; applying the same construction to convolutional layers, where the all-ones alignment would target a spatial pattern, or to expanding architectures would require a different mean-scaling factor and may not inherit the variance guarantee.
  • The Gaussian-approximation results suggest the method could plug into mean-field analyses of deep networks, potentially yielding depth-scaling laws that depend explicitly on the input mean.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes a new weight initialization for deep ReLU feed-forward networks, obtained by solving max_{W in O_{m,n}} 1_m^T W 1_n on the Stiefel manifold. The optimal set is characterized as those semi-orthogonal matrices satisfying W ξ_n = ξ_m (Lemma 1), and efficient sampling algorithms are given, including a one-QR scheme (Algorithm 2). The theoretical analysis shows that such W preserves isotropic covariance, rescales a constant input mean by sqrt(n/m), drives the pre-activation distribution toward Gaussian in large input dimension (Theorem 4), and the authors argue that the resulting rectification parameter α = μ/σ grows with depth, thereby preventing dying ReLU, bounding variance decay, and mitigating gradient vanishing. Experiments on MNIST, Fashion-MNIST, few-shot settings, and tabular datasets report stable training at depth 100 and gains over He, Xavier, orthogonal, and Lee initializations.

Significance. If the central claim holds, the paper offers a simple, computationally cheap initialization that stabilizes very deep ReLU MLPs without normalization or residual connections, and the closed-form solution to the Stiefel optimization is a useful contribution in its own right. The explicit Cholesky construction in Theorem 2 and the use of Raic's Berry-Esseen bound in Theorem 4 are strengths, as are the breadth of experiments (depth, activations, few-shot, tabular). However, the headline theoretical mechanism is conditional on a positive, roughly coordinate-constant input mean, which the paper's own tabular experiments show is a real limitation. The main theoretical proposition is heuristic in exactly the place where a rigorous statement is needed, and one equation in the central characterization is misstated. These issues are fixable, but they affect the paper's principal claims.

major comments (4)
  1. [§V.B, Proposition 4] The proof of Proposition 4, case (ii), invokes 'the rectification parameter α_{ℓ-1} = sqrt(N_{ℓ-2}/N_{ℓ-1}) μ_{ℓ-2}/σ_{ℓ-2} grows with depth' to justify approximating the rectified Gaussian by a Gaussian. This is precisely the behavior the proposition is meant to establish; no independent bound on α_{ℓ} is derived. Moreover, the statement does not assume μ0 > 0, yet if μ0 ≤ 0 the same mean amplification drives pre-activations negative and α does not grow. Table V confirms the limitation: at α0 = -2 the proposed method is at or below several baselines (Adult depth-50: 76.07 vs Lee 85.33/He 85.58; Cancer: 83.33 vs Lee 96.49; Ionosphere: 70.42 vs Lee 88.73; Diabetes RMSE: 70.53 vs He 62.63). Section VI.D admits that choosing α0 = 2 is critical. The abstract and the bullet on dying ReLU present the conclusion unconditionally; the theoretical claim should be restated with an explicit positivi
  2. [§IV.B, Eq. (17)] The characterization is stated as {L Q + (1/sqrt(mn)) J_{m×n} : Q ∈ O_{m,n}, Q^T e1 = ξ_n}. The proof and Algorithm 2 (line 11) require the last row of Q to be ξ_n^T, i.e. Q e_m = ξ_n. With Q^T e1 = ξ_n, the first row of Q would be fixed to ξ_n^T, which does not yield W ξ_n = ξ_m. The proof's own argument uses 'its m-th row is ξ_n^T' and `Q ξ_n = e_m`. Thus Eq. (17) contains a genuine orientation error in the central characterization and must be corrected.
  3. [§V.B] Proposition 4's induction is only heuristic for the architectures used in the main experiments. Theorem 4 is an asymptotic statement in n for fixed m; it justifies approximate Gaussianity for the first layer when N0 is large, but the subsequent layers in the experiments have Nℓ = 64, which is not 'large'. Case (i) of the proof relies on 'N_{ℓ-1} large', and case (ii) relies on the unproved growth of α. No quantitative or finite-width version is given, so the claim that pre-activations are approximately Gaussian in deep and narrow networks is not established. The empirical results may support the conclusion, but the proof as written is not a theorem in that regime.
  4. [§V.B] The statements 'µℓ nondecreasing and σℓ nonincreasing' and 'α grows with depth' are asserted rather than derived. Lemma 5 gives the per-layer update for a rectified Gaussian, but one must verify that the map (μ,σ) → (E[max(0,x)], Var[max(0,x)]) increases α for all relevant parameter regimes, starting from a positive mean. Without this check, bullet (a) 'P(x^{(ℓ)}_j > 0) = Φ(αℓ)' and the variance lower bound in bullet (b) are not justified. This is a load-bearing part of the claimed dying-ReLU prevention and should be either proved with precise assumptions or explicitly labeled as a heuristic.
minor comments (3)
  1. [§VI.D] The text says features are standardized to unit variance (σ² = 1) and the mean is shifted 'to control α = μ/σ²', while the rest of the paper defines α = μ/σ. Since σ² = 1 in those experiments the distinction is harmless, but the notation should be made consistent.
  2. [Example 1] The displayed entries of the 2×3 matrix W do not appear to match the given U and V factors; for instance, the first entry computed from the displayed U and V is (√6+√3)/6 ≈ 0.697, not (√6−√3)/6. Please verify the example or correct the typesetting.
  3. [Algorithm 2] Line 3 writes '0 otherwise for j ≤ i'; the condition is already covered by the cases above. This is a readability issue rather than an error.

Circularity Check

1 steps flagged

The main optimization and statistical derivation is self-contained, but the narrow-layer proof of Proposition 4 assumes the very depth-wise growth of α that it is meant to establish; the unconditional dying-ReLU wording also outruns the paper's own positive-mean condition.

specific steps
  1. other [Section V.B, Proposition 4, proof, case (ii)]
    "(ii) If Nℓ−1 is not large, we exploit the fact that the rectification parameter αℓ−1 = q Nℓ−2 Nℓ−1 µℓ−2 σℓ−2 grows with depth. ... For a sufficiently large αℓ−1, the rectified Gaussian distribution of x(ℓ−1) j becomes almost indistinguishable from a true Gaussian distribution, as the mass at zero vanishes."

    The proposition is supposed to justify the layer-wise Gaussian/rectified-Gaussian approximation and the depth-wise increase of α=μ/σ, which is then used to claim dying-ReLU prevention, bounded variance decay, and gradient non-vanishing. In the narrow-layer case (ii), the proof justifies the Gaussian approximation by saying αℓ−1 'grows with depth' and becomes sufficiently large. That is exactly the conclusion being proved: the later statements (a)-(c) all rely on Φ(αℓ)→1 following from α growth. The base case only handles a large input layer N0, so for the 'deep and narrow' regime the paper explicitly targets, the induction assumes its own target. The empirical benchmarks give independent support, but the formal derivation chain is not self-contained at this step.

full rationale

The paper's core construction (Theorem 1), the characterization of eO_{m,n} as semi-orthogonal matrices preserving the all-ones vector (Lemma 1), and the linear mean/variance propagation (Proposition 3) are self-contained linear algebra; the design objective is not disguised as an independent prediction. The UMAP/Pearson empirical motivation (r=0.8178) and the depth, few-shot, and tabular benchmarks are external evidence. The single genuine circularity is in Proposition 4's proof case (ii), where the approximation needed for narrow layers is justified by assuming that α grows with depth, which is the very property the proposition and its downstream conclusions are meant to establish. I do not count the positive-mean condition (μ0>0) as circular: Proposition 3 explicitly assumes it, and Section VI.D transparently reports that negative mean shifts (α0=-2) degrade performance; that is a scope limitation and an overstatement in the abstract, not a derivation that collapses into its own input. The self-citation to Lee et al. [14] is used as motivation and is backed by new correlation experiments, so it is not load-bearing. Overall the central theoretical derivation is partially circular in the narrow-layer regime, while the empirical portion remains independently supportive.

Axiom & Free-Parameter Ledger

1 free parameters · 7 axioms · 0 invented entities

The central claim rests on two domain assumptions not stated as conditions in the abstract: a positive, coordinate-constant input mean (μ0 > 0) and a mean-field Gaussian closure across depth. The remaining pillars are standard matrix-analysis and probability tools. No invented entities appear. One empirical parameter, the tabular input mean shift α0, is selected by hand after testing. The e1/em typo in Theorem 2 and the unproven monotonicity in Proposition 4 are the main blemishes in the derivation chain.

free parameters (1)
  • Input mean shift α0 (tabular experiments) = 2 (chosen among -2, 0, 2, 50)
    Tabular inputs are standardized and then mean-shifted to achieve a target rectification parameter; α0 = 2 yields the reported best scores and is selected after testing, so it is a hand-chosen parameter of the empirical claim.
axioms (7)
  • domain assumption Inputs have coordinate-constant positive mean: E[x_j^(0)] = μ0 > 0 for all j, with finite third moments (Proposition 4)
    The dying-ReLU-prevention and α-growth arguments start from E[x^(0)_j] = μ0 > 0; when μ0 ≤ 0 the mean amplification inverts, matching the paper's own degraded tabular results at α0 = -2.
  • domain assumption Mean-field closure: pre-activations remain approximately independent Gaussian with isotropic covariance at every depth (Proposition 4 induction)
    The layer-to-layer recursion assumes the previous layer's rectified Gaussian is approximately Gaussian. Case (ii) of the proof asserts that α grows with depth without proving the moment-recursion map is monotone for all α ≥ 0; case (i) iterates an approximate CLT without tracking errors.
  • domain assumption Non-expanding widths m ≤ n with W W^T = I_m throughout (Section III.A)
    All theorems assume m ≤ n (rows no more than columns, W W^T = I). The output layer and any expanding configurations are outside the theory.
  • domain assumption Zero biases: b^(ℓ) = 0 for all layers (Proposition 4)
    The mean-calibration computation E[W x] = μ sqrt(n/m) 1_m holds only with zero biases; adding bias would break Lemma 1's implications for pre-activation statistics.
  • standard math Classical tools: thin SVD, Horn-Johnson 7.3.11 factorization M = LQ, Cholesky uniqueness, QR-of-Gaussian yields Haar frames (Sections IV-V, [25], [26], [31])
    Routinely used results, mostly proved in-text or quoted from Horn and Johnson; acceptable as standard background.
  • standard math Multivariate Berry-Esseen with explicit constants (Lemma 4, Raic 2019)
    Quoted externally (Raic, Bernoulli 2019); the application in Theorem 4 is a direct check of its hypotheses.
  • standard math Rectified-Gaussian moment formulas (Lemma 5, [36])
    Used to compute the α-recursion and the variance-ratio bound in Section V.B.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks." pith.science (2026). https://pith.science/paper/4FQ7GX63

@misc{pith2026250900362,
  author       = {Pith},
  title        = {Pith review of: Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4FQ7GX63}},
  note         = {Machine review of arXiv:2509.00362}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Stable and efficient training of ReLU networks with large depth is highly sensitive to weight initialization. Improper initialization can cause permanent neuron inactivation dying ReLU and exacerbate gradient instability as network depth increases. Methods such as He, Xavier, and orthogonal initialization preserve variance or promote approximate isometry. However, they do not necessarily regulate the pre-activation mean or control activation sparsity, and their effectiveness often diminishes in very deep architectures. This work introduces an orthogonal initialization specifically optimized for ReLU by solving an optimization problem on the Stiefel manifold, thereby preserving scale and calibrating the pre-activation statistics from the outset. A family of closed-form solutions and an efficient sampling scheme are derived. Theoretical analysis at initialization shows that prevention of the dying ReLU problem, slower decay of activation variance, and mitigation of gradient vanishing, which together stabilize signal and gradient flow in deep architectures. Empirically, across MNIST, Fashion-MNIST, multiple tabular datasets, few-shot settings, and ReLU-family activations, our method outperforms previous initializations and enables stable training in deep networks.

Figures

Figures reproduced from arXiv: 2509.00362 by Hayoung Choi, Hyungu Lee, Taehyeong Kim.

Figure 1
Figure 1. Figure 1: A two-dimensional UMAP projection of vectors [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Convergence of W x to normal distribution as dimension n increases. Since ∥yj∥2 = |zj |∥wj∥2, Pn i=1 E[∥yi∥ 3 2 ] = β3 Pn i=1 E[∥wj∥ 3 2 ]. And Pn i=1 ∥wj∥ 3 2 can be bounded as follows: Xn i=1 ∥wj∥ 3 2 ≤  max 1≤j≤n ∥wj∥2 Xn i=1 ∥wj∥ 2 2 =  max 1≤j≤n ∥wj∥2  tr Xn i=1 wjwT j ! = m · max 1≤j≤n ∥wj∥2 ≤ m3/2 max i,j |Wij |. It implies that Pn i=1 E[∥wj∥ 3 2 ] ≤ m3/2E[maxi,j |Wij |]. Since maxi,j |Wij | < 1… view at source ↗
Figure 3
Figure 3. Figure 3: 4-shot classification accuracy for various initialization methods and network depths. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 33 canonical work pages

  1. [1]

    Hanin, D

    B. Hanin, D. A. Roberts, and S. Yaida, The Principles of Deep Learning Theory: An Effective Theory Approach to Understanding Neural Networks. Cambridge University Press, 2022, arXiv:2106.10165

    Pith/arXiv arXiv 2022

  2. [2]

    Goodfellow, Y

    I. Goodfellow, Y . Bengio, and A. Courville, Deep learning. MIT press Cambridge, 2016

    work page 2016

  3. [3]

    Learning deep architectures for ai,

    Y . Bengioet al., “Learning deep architectures for ai,” Foundations and trends® in Machine Learning , vol. 2, no. 1, pp. 1–127, 2009

    work page 2009

  4. [4]

    Deep learning,

    Y . LeCun, Y . Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, pp. 436–444, 2015

    2015

  5. [5]

    Understanding the difficulty of training deep feedforward neural networks,

    X. Glorot and Y . Bengio, “Understanding the difficulty of training deep feedforward neural networks,” in Proceedings of the thirteenth international conference on artificial intelligence and statistics . JMLR Workshop and Conference Proceedings, 2010, pp. 249–256

    2010

  6. [6]

    Rectified linear units improve restricted boltzmann machines,

    V . Nair and G. E. Hinton, “Rectified linear units improve restricted boltzmann machines,” in Proceedings of the 27th international conference on machine learning (ICML-10) , 2010, pp. 807–814

    work page 2010

  7. [7]

    Rectifier nonlinearities improve neural network acoustic models,

    A. L. Maas, A. Y . Hannun, and A. Y . Ng, “Rectifier nonlinearities improve neural network acoustic models,” in Proc. icml. Atlanta, GA, 2013

    work page 2013

  8. [8]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,

    K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in Proceedings of the IEEE international conference on computer vision , 2015, pp. 1026–1034

    2015

  9. [9]

    Fast and accurate deep network learning by exponential linear units (elus),

    D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (elus),” arXiv preprint arXiv:1511.07289, 2015

    Pith/arXiv arXiv 2015

  10. [10]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778

    2016

  11. [11]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift,

    S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in International conference on machine learning . pmlr, 2015, pp. 448–456

    work page 2015

  12. [12]

    Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice,

    J. Pennington, S. Schoenholz, and S. Ganguli, “Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice,” Advances in neural information processing systems , vol. 30, 2017

    work page 2017

  13. [13]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks,

    A. M. Saxe, J. L. McClelland, and S. Ganguli, “Exact solutions to the nonlinear dynamics of learning in deep linear neural networks,” arXiv preprint arXiv:1312.6120, 2013

    Pith/arXiv arXiv 2013

  14. [14]

    Improved weight initialization for deep and narrow feedforward neural network,

    H. Lee, Y . Kim, S. Y . Yang, and H. Choi, “Improved weight initialization for deep and narrow feedforward neural network,” Neural Networks, vol. 176, p. 106362, 2024

    work page 2024

  15. [15]

    Gradient-based learning applied to document recognition,

    Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE , vol. 86, no. 11, pp. 2278–2324, 1998

    1998

  16. [16]

    Greedy layer-wise training of deep networks,

    Y . Bengio, P. Lamblin, D. Popovici, and H. Larochelle, “Greedy layer-wise training of deep networks,” Advances in neural information processing systems, vol. 19, 2006

    work page 2006

  17. [17]

    Learning repre- sentations by back-propagating errors,

    D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning repre- sentations by back-propagating errors,” nature, vol. 323, no. 6088, pp. 533–536, 1986

    work page 1986

  18. [18]

    Backpropagation applied to handwritten zip code recognition,

    Y . LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, “Backpropagation applied to handwritten zip code recognition,” Neural computation, vol. 1, no. 4, pp. 541–551, 1989

    work page 1989

  19. [19]

    Dying relu and initializa- tion: Theory and numerical examples,

    L. Lu, Y . Shin, Y . Su, and G. E. Karniadakis, “Dying relu and initializa- tion: Theory and numerical examples,” arXiv preprint arXiv:1903.06733 , 2019

    Pith/arXiv arXiv 1903

  20. [20]

    Robust weight initialization for tanh neural networks with fixed point analysis,

    H. woo Lee, H. Choi, and H. Kim, “Robust weight initialization for tanh neural networks with fixed point analysis,” in The Thirteenth International Conference on Learning Representations, ICLR 2025 , 2025

    work page 2025

  21. [21]

    Revisiting weight initialization of deep neural networks,

    M. Skorski, A. Temperoni, and M. Theobald, “Revisiting weight initialization of deep neural networks,” in Proceedings of ACML, 2021. 16

    work page 2021

  22. [22]

    Provable benefit of orthogonal initialization in optimizing deep linearnetworks,

    W. Hu, L. Xiao, and J. Pennington, “Provable benefit of orthogonal initialization in optimizing deep linearnetworks,” in 8th International Conference on Learning Representations, ICLR 2020 , 2020

    work page 2020

  23. [23]

    Umap: Uniform manifold approximation and projection for dimension reduction,

    L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018

    Pith/arXiv arXiv 2018

  24. [24]

    Computational optimal transport: With applications to data science,

    G. Peyr ´e, M. Cuturi et al. , “Computational optimal transport: With applications to data science,” Foundations and Trends ® in Machine Learning, vol. 11, no. 5-6, pp. 355–607, 2019

    work page 2019

  25. [25]

    R. A. Horn and C. R. Johnson, Matrix analysis. Cambridge university press, 2012

    2012

  26. [26]

    How to generate random matrices from the classical compact groups,

    F. Mezzadri, “How to generate random matrices from the classical compact groups,” arXiv preprint math-ph/0609050 , 2006

    Pith/arXiv arXiv 2006

  27. [27]

    N. J. Higham, Accuracy and stability of numerical algorithms . SIAM, 2002

    work page 2002

  28. [28]

    Mean field residual networks: On the edge of chaos,

    G. Yang and S. Schoenholz, “Mean field residual networks: On the edge of chaos,” Advances in neural information processing systems , vol. 30, 2017

    work page 2017

  29. [29]

    Skewness and kurtosis in real data samples,

    M. J. Blanca, J. Arnau, D. L ´opez-Montiel, R. Bono, and R. Bendayan, “Skewness and kurtosis in real data samples,” Methodology, 2013

    work page 2013

  30. [30]

    Transforming variables to central normality,

    J. Raymaekers and P. J. Rousseeuw, “Transforming variables to central normality,” Machine Learning, vol. 113, no. 8, pp. 4953–4975, 2024

    work page 2024

  31. [31]

    A comparison principle for functions of a uniformly random subspace,

    J. A. Tropp, “A comparison principle for functions of a uniformly random subspace,” Probability Theory and Related Fields , vol. 153, no. 3, pp. 759–769, 2012

    work page 2012

  32. [32]

    Maxima of entries of haar distributed matrices,

    T. Jiang, “Maxima of entries of haar distributed matrices,” Probability Theory and Related Fields , vol. 131, no. 1, pp. 121–144, 2005

    work page 2005

  33. [33]

    A multivariate berry–esseen theorem with explicit constants,

    M. Rai ˇc, “A multivariate berry–esseen theorem with explicit constants,” Bernoulli, vol. 25, no. 4A, pp. 2824–2853, 2019

    work page 2019

  34. [34]

    A. W. Van der Vaart, Asymptotic statistics. Cambridge university press, 2000, vol. 3

    work page 2000

  35. [35]

    Deep information propagation,

    S. S. Schoenholz, J. Gilmer, S. Ganguli, and J. Sohl-Dickstein, “Deep information propagation,” arXiv preprint arXiv:1611.01232 , 2016

    Pith/arXiv arXiv 2016

  36. [36]

    On numerical computation for the distribution of the convolution of N independent rectified Gaussian variables,

    M. Beauchamp, “On numerical computation for the distribution of the convolution of N independent rectified Gaussian variables,” Journal de la soci ´et´e franc ¸aise de statistique, vol. 159, no. 1, pp. 88–111, 2018

    work page 2018

  37. [37]

    An analytic solution to covariance propagation in neural networks,

    O. Wright, Y . Nakahira, and J. M. Moura, “An analytic solution to covariance propagation in neural networks,” in International Conference on Artificial Intelligence and Statistics . PMLR, 2024, pp. 4087–4095

    work page 2024

  38. [38]

    How to start training: The effect of initialization and architecture,

    B. Hanin and D. Rolnick, “How to start training: The effect of initialization and architecture,” in NeurIPS, 2018

    work page 2018

  39. [39]

    Tensor programs VI: Feature learning in infinite depth neural networks,

    G. Yang, D. Yu, C. Zhu, and S. Hayou, “Tensor programs VI: Feature learning in infinite depth neural networks,” in The Twelfth International Conference on Learning Representations , 2024. [Online]. Available: https://openreview.net/forum?id=17pVDnpwwl

    work page 2024

  40. [40]

    Self- normalizing neural networks,

    G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self- normalizing neural networks,” Advances in neural information processing systems, vol. 30, 2017

    work page 2017

  41. [41]

    Becker and R

    B. Becker and R. Kohavi, “Adult,” https://archive.ics.uci.edu/ml/datasets/ adult, 1996, UCI Machine Learning Repository

    work page 1996

  42. [42]

    Breast cancer wisconsin (diagnostic),

    W. Wolberg, O. Mangasarian, N. Street, and W. Street, “Breast cancer wisconsin (diagnostic),” https://archive.ics.uci.edu/ml/datasets/breast+ cancer+wisconsin+(diagnostic), 1993, UCI Machine Learning Repository

    work page 1993

  43. [43]

    Using the adap learning algorithm to forecast the onset of diabetes mellitus,

    J. W. Smith, J. E. Everhart, W. C. Dickson, W. C. Knowler, and R. S. Johannes, “Using the adap learning algorithm to forecast the onset of diabetes mellitus,” in Proceedings of the Annual Symposium on Computer Application in Medical Care , 1988, p. 261

    work page 1988

  44. [44]

    Classification of radar returns from the ionosphere using neural networks,

    V . G. Sigillito, S. P. Wing, L. V . Hutton, and K. B. Baker, “Classification of radar returns from the ionosphere using neural networks,” Johns Hopkins APL Technical Digest , vol. 10, no. 3, pp. 262–266, 1989

    work page 1989

  45. [45]

    Aeberhard and M

    S. Aeberhard and M. Forina, “Wine,” https://archive.ics.uci.edu/ml/ datasets/wine, 1992, UCI Machine Learning Repository

    work page 1992

  46. [46]

    Least angle regression,

    B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, “Least angle regression,” The Annals of Statistics , vol. 32, no. 2, pp. 407–499, 2004

    work page 2004

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.