REVIEW 8 cited by
Dying ReLU and Initialization: Theory and Numerical Examples
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The dying ReLU refers to the problem when ReLU neurons become inactive and only output 0 for any input. There are many empirical and heuristic explanations of why ReLU neurons die. However, little is known about its theoretical analysis. In this paper, we rigorously prove that a deep ReLU network will eventually die in probability as the depth goes to infinite. Several methods have been proposed to alleviate the dying ReLU. Perhaps, one of the simplest treatments is to modify the initialization procedure. One common way of initializing weights and biases uses symmetric probability distributions, which suffers from the dying ReLU. We thus propose a new initialization procedure, namely, a randomized asymmetric initialization. We prove that the new initialization can effectively prevent the dying ReLU. All parameters required for the new initialization are theoretically designed. Numerical examples are provided to demonstrate the effectiveness of the new initialization procedure.
Forward citations
Cited by 8 Pith papers
-
DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators
DeepONet learns nonlinear operators for differential equations via branch and trunk sub-networks, achieving high-order error convergence on small datasets.
-
The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System
The edge of stability is the first bifurcation of the finite-step gradient map; residual oscillations then drive balancing and representation selection beyond that edge.
-
Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes
Standard losses induce negative weight drift with positive-biased activations, producing up to 90% sparsity in GPT-nano and an accuracy cliff above ~70% sparsity; clipped ReLU² and GELU² improve the tradeoff.
-
Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes
The paper proves negative weight drift at initialization under MSE or cross-entropy with asymmetric activations, links it to up to 90% sparsity in GPT-nano, maps the sparsity-accuracy cliff across 79 configurations, a...
-
NASDAQ: Normalized Observation Space Dynamics-Augmented Q-Learning
NASDAQ normalizes observations in an online RL setting so that dynamics prediction losses are balanced across dimensions, yielding competitive performance with lower wall-time than prior model-based and self-predictiv...
-
Preserving Plasticity in Continual Learning via Dynamical Isometry
Dynamical isometry (Jacobian singular values near 1) preserves plasticity in continual learning; an isometry-promoting regularizer and decoupled AdamO optimizer match or beat prior methods on supervised and RL benchmarks.
-
Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks
An initialization that maximizes a semi-orthogonal weight matrix's alignment with the all-ones vector prevents dying ReLU and keeps 100-layer ReLU networks trainable.
-
Interpretable Material Spatial Intelligence for Discovery of Governing Microstructural Features
MSI is a multimodal representation learning framework that identifies key microstructural features governing mechanical behavior in structural alloys from spatial observations.
Discussion (0). Sign in to comment.