The instabilities of large learning rate training: a loss landscape view

Lawrence Wang; Stephen Roberts

The instabilities of large learning rate training: a loss landscape view

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.11948 v1 pith:NWWVGYGT submitted 2023-07-22 cs.LG

Lawrence Wang , Stephen Roberts This is my paper

classification cs.LG

keywords landscapeinstabilitieslosstraininglargelearningtextitaffect

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Modern neural networks are undeniably successful. Numerous works study how the curvature of loss landscapes can affect the quality of solutions. In this work we study the loss landscape by considering the Hessian matrix during network training with large learning rates - an attractive regime that is (in)famously unstable. We characterise the instabilities of gradient descent, and we observe the striking phenomena of \textit{landscape flattening} and \textit{landscape shift}, both of which are intimately connected to the instabilities of training.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

An Analysis of Layer-Freezing Strategies for Enhanced Transfer Learning in YOLO Architectures
cs.CV 2025-09 conditional novelty 5.0 of 10

Freezing the first four blocks or the whole backbone of YOLOv8/YOLOv10 can match or beat full fine-tuning while using less GPU memory, but aggressive freezing fails on heavily augmented single-class data.

Pith tools