REVIEW 12 cited by
Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
In this paper, we study the implicit regularization of the gradient descent algorithm in homogeneous neural networks, including fully-connected and convolutional neural networks with ReLU or LeakyReLU activations. In particular, we study the gradient descent or gradient flow (i.e., gradient descent with infinitesimal step size) optimizing the logistic loss or cross-entropy loss of any homogeneous model (possibly non-smooth), and show that if the training loss decreases below a certain threshold, then we can define a smoothed version of the normalized margin which increases over time. We also formulate a natural constrained optimization problem related to margin maximization, and prove that both the normalized margin and its smoothed version converge to the objective value at a KKT point of the optimization problem. Our results generalize the previous results for logistic regression with one-layer or multi-layer linear networks, and provide more quantitative convergence results with weaker assumptions than previous results for homogeneous smooth neural networks. We conduct several experiments to justify our theoretical finding on MNIST and CIFAR-10 datasets. Finally, as margin is closely related to robustness, we discuss potential benefits of training longer for improving the robustness of the model.
Forward citations
Cited by 12 Pith papers
-
Multiclass Loss Geometry Matters for Generalization of Gradient Descent in Separable Classification
In separable multiclass classification, the risk of gradient descent scales as k^{2/p} for losses with ℓ_p-smooth templates, giving logarithmic k-dependence for p=∞ and linear k-dependence for p=2 (provably unavoidable).
-
Querying Kernel Methods Suffices for Reconstructing their Training Data
Query-only access to kernel regression, SVM and KDE models suffices to reconstruct their exact training points, via a measure-theoretic proof and image experiments.
-
A Counterexample to Fourier Alignment in Single-Neuron Modular Addition
For every prime p at least 5, a single ReLU neuron can, with positive Gaussian probability, train to a frozen or diverging state whose normalized Fourier spectrum flatly distributes energy, contradicting single-freque...
-
A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuning
In diagonal linear networks, fine-tuning generalization is governed by a tunable per-dimension penalty whose sparsity and pretraining dependence define four regimes and a trade-off between feature reuse and new-featur...
-
Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
In a linear softmax memory model, Muon equalizes learning across frequency tiers and gives exponential (noiseless) or T^{-2} (noisy power-law) convergence, versus polynomial or T^{-(1-1/β)} for gradient descent.
-
Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge
Identity-bridge regularization, rephrased into an out-of-context reasoning form, yields ~40% reversal accuracy in a 1B LLM and provably fixes reversal in an idealized one-layer transformer.
-
Unveiling the Mechanisms of Multi-Hop Reasoning in Transformers via Identity Bridge
Adding identity supervision on bridge tokens enables out-of-distribution two-hop reasoning in simple transformers, with a nuclear-norm theory explaining the benefit.
-
PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learning
PADAM runs K differently averaged Adam trajectories in parallel, selects the one with the smallest test error, and achieves the best optimization error in nearly all of 13 tested scientific machine learning problems w...
-
Understanding Nonlinear Implicit Bias via Region Counts in Input Space
Region count, the number of connected same-label regions along random input-space lines, correlates strongly with the generalization gap and is proposed as a reparameterization-invariant measure of nonlinear implicit bias.
-
Neural Thermodynamic Laws for Large Language Model Training
Under a river-valley model of the loss landscape, the paper shows that valley fluctuations behave like heat, with learning rate as temperature, and derives a 1/t optimal decay schedule.
-
The Features at Convergence Theorem: a first-principles alternative to the Neural Feature Ansatz for how networks learn representations
FACT is a first-order stationarity identity for weight matrices that matches or beats the Neural Feature Ansatz as a description of learned features at convergence.
-
Theoretical Issues in Deep Networks: Approximation, Optimization and Generalization
A synthesis of approximation, optimization, and generalization theory arguing that gradient descent's implicit norm control on weight directions explains why overparameterized deep networks generalize.
Discussion (0). Continue with ORCID to comment.