REVIEW 7 cited by
An Exponential Learning Rate Schedule for Deep Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Intriguing empirical evidence exists that deep learning can work well with exoticschedules for varying the learning rate. This paper suggests that the phenomenon may be due to Batch Normalization or BN, which is ubiquitous and provides benefits in optimization and generalization across all standard architectures. The following new results are shown about BN with weight decay and momentum (in other words, the typical use case which was not considered in earlier theoretical analyses of stand-alone BN. 1. Training can be done using SGD with momentum and an exponentially increasing learning rate schedule, i.e., learning rate increases by some $(1 +\alpha)$ factor in every epoch for some $\alpha >0$. (Precise statement in the paper.) To the best of our knowledge this is the first time such a rate schedule has been successfully used, let alone for highly successful architectures. As expected, such training rapidly blows up network weights, but the net stays well-behaved due to normalization. 2. Mathematical explanation of the success of the above rate schedule: a rigorous proof that it is equivalent to the standard setting of BN + SGD + StandardRate Tuning + Weight Decay + Momentum. This equivalence holds for other normalization layers as well, Group Normalization, LayerNormalization, Instance Norm, etc. 3. A worked-out toy example illustrating the above linkage of hyper-parameters. Using either weight decay or BN alone reaches global minimum, but convergence fails when both are used.
Forward citations
Cited by 7 Pith papers
-
Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors
Splitting weight matrices into a fixed-norm direction and learnable per-row/column magnitudes improves LLM training over AdamW/Muon, removes weight decay and warmup, and transfers the optimal LR across width.
-
Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
Weight decay on scale-invariant weights creates a norm-dependent sharpness boundary; crossing it predicts loss spikes in normalized networks.
-
Path optimization method for the sign problem: Insights from random matrix models
Path optimization improves the average phase factor in the Stephanov model at high chemical potential but not at low chemical potential or in the chiral random matrix model, pointing to the global sign problem as the ...
-
Data-Driven Adaptive Gradient Recovery for Unstructured Finite Volume Computations
A neural-network gradient corrector for unstructured finite volume solvers reports 20-60 percent accuracy gains on 2D Euler benchmarks with improved mesh convergence.
-
Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
OBLR-PO combines an SNR-based learning rate and a gradient-weighted baseline for RLVR, but the main theory is undermined by a flawed smoothness proof.
-
HGCN(O): A Self-Tuning GCN HyperModel Toolkit for Outcome Prediction in Event-Sequence Data
A self-tuning toolkit of four GCN input architectures (O-, T-, TP-, TE-GCN) is applied to outcome prediction; the claimed advantage over baselines is unsupported, and the balanced-data perfect scores are likely leakag...
-
Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification
DLRS adjusts the learning rate per epoch from the normalized first-to-last batch loss slope, and the authors report faster convergence on PINN and image classification benchmarks.
Discussion (0). Sign in to comment.