REVIEW 2 cited by
(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
In this paper, we investigate the impact of stochasticity and large stepsizes on the implicit regularisation of gradient descent (GD) and stochastic gradient descent (SGD) over diagonal linear networks. We prove the convergence of GD and SGD with macroscopic stepsizes in an overparametrised regression setting and characterise their solutions through an implicit regularisation problem. Our crisp characterisation leads to qualitative insights about the impact of stochasticity and stepsizes on the recovered solution. Specifically, we show that large stepsizes consistently benefit SGD for sparse regression problems, while they can hinder the recovery of sparse solutions for GD. These effects are magnified for stepsizes in a tight window just below the divergence threshold, in the "edge of stability" regime. Our findings are supported by experimental results.
Forward citations
Cited by 2 Pith papers
-
Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regime
Under specific conditions, gradient descent converges in the unstable edge-of-stability regime for a quadratic loss on a depth-2 diagonal linear network, with a bias bound depending on step size and initialization.
-
Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits
An exponential moving average of weights consistently improves generalization, label-noise robustness, prediction consistency, calibration, and transfer learning for image classifiers.
Discussion (0). Continue with ORCID to comment.