REVIEW 9 cited by
A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Stochastic Gradient Descent (SGD) and its variants are mainstream methods for training deep networks in practice. SGD is known to find a flat minimum that often generalizes well. However, it is mathematically unclear how deep learning can select a flat minimum among so many minima. To answer the question quantitatively, we develop a density diffusion theory (DDT) to reveal how minima selection quantitatively depends on the minima sharpness and the hyperparameters. To the best of our knowledge, we are the first to theoretically and empirically prove that, benefited from the Hessian-dependent covariance of stochastic gradient noise, SGD favors flat minima exponentially more than sharp minima, while Gradient Descent (GD) with injected white noise favors flat minima only polynomially more than sharp minima. We also reveal that either a small learning rate or large-batch training requires exponentially many iterations to escape from minima in terms of the ratio of the batch size and learning rate. Thus, large-batch training cannot search flat minima efficiently in a realistic computational time.
Forward citations
Cited by 9 Pith papers
-
Absorbing state dynamics of stochastic gradient descent
SGD on linearly repulsive particles is mapped onto biased random organization, sharing its critical packing fraction of about 0.64 and its Manna universality class.
-
On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds
SAM's largest Hessian eigenvalue is bounded by the cube root of bGamma/(2*rho*eta^2), so larger radius, smaller batch, or larger learning rate restrict linearly stable minima to flatter regions.
-
Seesaw: Accelerating Training by Balancing Learning Rate and Batch Size Scheduling
When a cosine schedule would halve the learning rate, Seesaw cuts it by √2 and doubles the batch, matching loss curves with ~36% fewer serial steps.
-
Neural Thermodynamic Laws for Large Language Model Training
Under a river-valley model of the loss landscape, the paper shows that valley fluctuations behave like heat, with learning rate as temperature, and derives a 1/t optimal decay schedule.
-
Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimization
Eigen-SAM improves Sharpness-Aware Minimization by explicitly aligning the perturbation with the top Hessian eigenvector, supported by a third-order SDE analysis and consistent small accuracy gains on CIFAR, SVHN, and...
-
Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation
Training the teacher a small number of steps ahead of the student and freezing it during distillation improves student generalization by up to 3.4% on image benchmarks.
-
Reasoning Bias of Next Token Prediction Training
Training on all tokens (next token prediction) beats training only on answer tokens (critical token prediction) on small-scale reasoning benchmarks, an effect the authors attribute to noise-induced regularization.
-
Generalization Capability for Imitation Learning
The paper argues that imitation learning generalization is governed by representation compression and encoder-data dependence, and that high conditional entropy in actions tightens the bound, but the key new bound is ...
-
Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impact
A broad survey arguing that AGI requires modular, memory-augmented, embodied architectures rather than scaled-up token prediction, with a brief proposal to decompose intelligence into five components.
Discussion (0). Continue with ORCID to comment.