RNC-LM extends geodesic-accelerated Levenberg-Marquardt to arbitrary-order Riemann normal coordinate corrections, reusing the LM matrix factorization for all orders and achieving large speedups on PINN and potential-fitting benchmarks.
Fixup Initialization: Residual Learning Without Normalization
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
abstract
Normalization layers are a staple in state-of-the-art deep neural network architectures. They are widely believed to stabilize training, enable higher learning rate, accelerate convergence and improve generalization, though the reason for their effectiveness is still an active research topic. In this work, we challenge the commonly-held beliefs by showing that none of the perceived benefits is unique to normalization. Specifically, we propose fixed-update initialization (Fixup), an initialization motivated by solving the exploding and vanishing gradient problem at the beginning of training via properly rescaling a standard initialization. We find training residual networks with Fixup to be as stable as training with normalization -- even for networks with 10,000 layers. Furthermore, with proper regularization, Fixup enables residual networks without normalization to achieve state-of-the-art performance in image classification and machine translation.
years
2026 7representative citing papers
Stealth Pretraining Seeding plants persistent unsafe behaviors in LLMs via diffuse poisoned web content that activates on precise triggers and evades standard evaluation.
SteeringDRL identifies two optimization regimes in diffusion autoencoders and uses gated residual U-Nets with a log SNR curriculum to steer training toward disentangled representations, improving performance across multiple benchmarks.
TaperNorm gradually removes internal normalization in pre-norm transformers via learned gates that reach zero, revealing final norm as a scale anchor and enabling up to 1.18x faster KV-cached decoding with small loss increases.
Depth expansion in normalized residual networks yields provable test-risk improvement through representational, optimization, and generalization gains under first-order descent and norm-control conditions.
Review Residuals add an update-conditioned gate to transformer residual connections, yielding depth-stable training and performance gains that emerge and grow with model size from 590M parameters upward.
A hybrid neural network and finite element strategy computes Ginzburg-Landau energy minimizers across varying kappa values, using the network output either directly or as a starting guess for reliable classical optimization.
citing papers explorer
-
Higher-Order Geometric Updates for Levenberg-Marquardt Method via Riemann Normal Coordinates
RNC-LM extends geodesic-accelerated Levenberg-Marquardt to arbitrary-order Riemann normal coordinate corrections, reusing the LM matrix factorization for all orders and achieving large speedups on PINN and potential-fitting benchmarks.
-
PermaFrost-Attack: Stealth Pretraining Seeding(SPS) for planting Logic Landmines During LLM Training
Stealth Pretraining Seeding plants persistent unsafe behaviors in LLMs via diffuse poisoned web content that activates on precise triggers and evades standard evaluation.
-
Steering Optimisation Trajectories in Diffusion Representation Learning
SteeringDRL identifies two optimization regimes in diffusion autoencoders and uses gated residual U-Nets with a log SNR curriculum to steer training toward disentangled representations, improving performance across multiple benchmarks.
-
Gated Normalization Removal and Scale Anchoring in Pre-Norm Transformers
TaperNorm gradually removes internal normalization in pre-norm transformers via learned gates that reach zero, revealing final norm as a scale anchor and enabling up to 1.18x faster KV-cached decoding with small loss increases.
-
A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks
Depth expansion in normalized residual networks yields provable test-risk improvement through representational, optimization, and generalization gains under first-order descent and norm-control conditions.
-
Review Residuals: Update-Conditioned Residual Gating for Transformers
Review Residuals add an update-conditioned gate to transformer residual connections, yielding depth-stable training and performance gains that emerge and grow with model size from 590M parameters upward.
-
GLENN: Neural network-enhanced computation of Ginzburg-Landau energy minimizers
A hybrid neural network and finite element strategy computes Ginzburg-Landau energy minimizers across varying kappa values, using the network output either directly or as a starting guess for reliable classical optimization.