Derives layer-wise recursions for finite-width tensors under orthogonal initialization that reproduce the observed large-depth stability of nonlinear networks.
All you need is a good init
5 Pith papers cite this work. Polarity classification is still indexing.
5
Pith papers citing it
abstract
Layer-sequential unit-variance (LSUV) initialization - a simple method for weight initialization for deep net learning - is proposed. The method consists of the two steps. First, pre-initialize weights of each convolution or inner-product layer with orthonormal matrices. Second, proceed from the first to the final layer, normalizing the variance of the output of each layer to be equal to one. Experiment with different activation functions (maxout, ReLU-family, tanh) show that the proposed initialization leads to learning of very deep nets that (i) produces networks with test accuracy better or equal to standard methods and (ii) is at least as fast as the complex schemes proposed specifically for very deep nets such as FitNets (Romero et al. (2015)) and Highway (Srivastava et al. (2015)). Performance is evaluated on GoogLeNet, CaffeNet, FitNets and Residual nets and the state-of-the-art, or very close to it, is achieved on the MNIST, CIFAR-10/100 and ImageNet datasets.
citation-role summary
background 1
citation-polarity summary
years
2026 5roles
background 1polarities
background 1representative citing papers
XPERT extracts and reuses cross-domain expert knowledge from pre-trained MoE LLMs via inference analysis and tensor decomposition to improve performance and convergence in downstream language model training.
Introduces a margin-adaptive confidence ranking method that learns an estimator from simulated diversity and derives margin-dependent generalization bounds for use in fixed-sequence testing of LLM-human agreement.
CDLinear layers achieve population Hessian condition number exactly 1 under pre-whitening, deliver 3.8x parameter reduction versus dense layers at 0.65% accuracy cost, and show 310x better empirical conditioning on an MLP.
A compact neural network surrogate maps weather features to grid voltages on a 6717-bus Texas system, enabling grid-aware weather forecasting that prioritizes operationally critical conditions like wind drops.
citing papers explorer
-
Criticality and Saturation in Orthogonal Neural Networks
Derives layer-wise recursions for finite-width tensors under orthogonal initialization that reproduce the observed large-depth stability of nonlinear networks.
-
XPERT: Expert Knowledge Transfer for Effective Training of Language Models
XPERT extracts and reuses cross-domain expert knowledge from pre-trained MoE LLMs via inference analysis and tensor decomposition to improve performance and convergence in downstream language model training.
-
Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
Introduces a margin-adaptive confidence ranking method that learns an estimator from simulated diversity and derives margin-dependent generalization bounds for use in fixed-sequence testing of LLM-human agreement.
-
Communication Dynamics Neural Networks: FFT-Diagonalized Layers for Improved Hessian Conditioning at Reduced Parameter Count
CDLinear layers achieve population Hessian condition number exactly 1 under pre-whitening, deliver 3.8x parameter reduction versus dense layers at 0.65% accuracy cost, and show 310x better empirical conditioning on an MLP.
-
Learning the Weather-Grid Nexus via Weather-to-Voltage (W2V) Predictive Modeling
A compact neural network surrogate maps weather features to grid voltages on a 6717-bus Texas system, enabling grid-aware weather forecasting that prioritizes operationally critical conditions like wind drops.