REVIEW 4 cited by
Provable and Practical Online Learning Rate Adaptation with Hypergradient Descent
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
This paper investigates the convergence properties of the hypergradient descent method (HDM), a 25-year-old heuristic originally proposed for adaptive stepsize selection in stochastic first-order methods. We provide the first rigorous convergence analysis of HDM using the online learning framework of [Gao24] and apply this analysis to develop new state-of-the-art adaptive gradient methods with empirical and theoretical support. Notably, HDM automatically identifies the optimal stepsize for the local optimization landscape and achieves local superlinear convergence. Our analysis explains the instability of HDM reported in the literature and proposes efficient strategies to address it. We also develop two HDM variants with heavy-ball and Nesterov momentum. Experiments on deterministic convex problems show HDM with heavy-ball momentum (HDM-HB) exhibits robust performance and significantly outperforms other adaptive first-order methods. Moreover, HDM-HB often matches the performance of L-BFGS, an efficient and practical quasi-Newton method, using less memory and cheaper iterations.
Forward citations
Cited by 4 Pith papers
-
Gradient Methods with Online Scaling Part I. Theoretical Foundations
Online scaled gradient methods adapt matrix step sizes via online learning, match the best fixed step size asymptotically, and achieve non-asymptotic superlinear convergence on smooth strongly convex problems.
-
Enhanced PDHG for Linear Programming with Online Preconditioning
Online preconditioning for a GPU LP solver cuts iteration counts by roughly 10-30% on Netlib and MIPLIB benchmarks, with the learning rate tuned per instance.
-
Online Learning-guided Learning Rate Adaptation via Gradient Alignment
GALA adapts the learning rate through online learning on gradient alignment, achieves a data-adaptive convergence rate for normalized SGD, and shows robust empirical performance across initial learning rates.
-
Accelerating Optimization via Differentiable Stopping Time
A discrete stopping-time sensitivity defined from an ODE discretization approximates the continuous hitting-time gradient with O(h) error, enabling gradient-based optimization of iteration counts.
Discussion (0). Sign in to comment.