Stochastic mirror ascent provably finds an approximately least-favorable distribution and minimax estimator for the Bounded Normal Mean problem, yielding 6–18% risk improvements over the minimax linear estimator.
Modified Loss of Momentum Gradient Descent: Fine-Grained Analysis
3 Pith papers cite this work. Polarity classification is still indexing.
3
Pith papers citing it
abstract
We analyze gradient descent with Polyak heavy-ball momentum (HB) whose fixed momentum parameter $\beta \in (0, 1)$ provides exponential decay of memory. Building on Kovachki and Stuart (2021), we prove that on an exponentially attractive invariant manifold the algorithm is exactly plain gradient descent with a modified loss, provided that the step size $h$ is small enough. Although the modified loss does not admit a closed-form expression, we describe it with arbitrary precision and prove global (finite "time" horizon) approximation bounds $O(h^{R})$ for any finite order $R \geq 2$. We then conduct a fine-grained analysis of the combinatorics underlying the memoryless approximations of HB, in particular, finding a rich family of polynomials in $\beta$ hidden inside which contains Eulerian and Narayana polynomials. We derive continuous modified equations of arbitrary approximation order (with rigorous bounds) and the principal flow that approximates the HB dynamics, generalizing Rosca et al. (2023). Approximation theorems cover both full-batch and mini-batch HB. Our theoretical results shed new light on the main features of gradient descent with heavy-ball momentum, and outline a road-map for similar analysis of other optimization algorithms.
years
2026 3representative citing papers
Trained ResNets on CIFAR-10 retain measurable dependence on initialization scale under low-LR SGD (26.5 pp test accuracy spread) but not under Adam, indicating that practical inductive bias is shaped by the forgetting time scale of the optimizer and regularizers.
Mini-batch noise reverses how Adam's β2 controls anti-regularization, making default momentum values suitable for small batches but requiring β1 closer to β2 for large batches to favor flatter minima.
citing papers explorer
-
Approximate Minimax Estimation of a Bounded Normal Mean via Stochastic Mirror Ascent
Stochastic mirror ascent provably finds an approximately least-favorable distribution and minimax estimator for the Bounded Normal Mean problem, yielding 6–18% risk improvements over the minimax linear estimator.
-
Do Deep Networks Forget Initialization? A Forgetting-Time View of Practical Inductive Bias
Trained ResNets on CIFAR-10 retain measurable dependence on initialization scale under low-LR SGD (26.5 pp test accuracy spread) but not under Adam, indicating that practical inductive bias is shaped by the forgetting time scale of the optimizer and regularizers.
-
The Effect of Mini-Batch Noise on the Implicit Bias of Adam
Mini-batch noise reverses how Adam's β2 controls anti-regularization, making default momentum values suitable for small batches but requiring β1 closer to β2 for large batches to favor flatter minima.