REVIEW 6 cited by
LaProp: Separating Momentum and Adaptivity in Adam
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
LaProp: Separating Momentum and Adaptivity in Adam
read the original abstract
We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and adaptivity in the Adam-style methods. We show that the decoupling leads to greater flexibility in the hyperparameters and allows for a straightforward interpolation between the signed gradient methods and the adaptive gradient methods. We experimentally show that Laprop has consistently improved speed and stability over Adam on a variety of tasks. We also bound the regret of Laprop on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.
Forward citations
Cited by 6 Pith papers
-
On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization
Momentum SGD incurs a provable drift-amplification penalty in nonstationary stochastic optimization that makes it worse than vanilla SGD in drift-dominated regimes, confirmed by finite-time upper bounds and minimax lo...
-
Mastering Diverse Domains through World Models
DreamerV3 uses world models and robustness techniques to solve over 150 tasks across domains with a single configuration, including Minecraft diamond collection from scratch.
-
UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction
UniRank is an open benchmark that standardizes chronological autoregressive supervision, multi-task evaluation, and capacity controls for 15 unified ranking models on five large datasets.
-
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
SOAP and Muon, stabilized by per-step QR eigenbasis updates and KL-Shampoo covariance accumulation, beat AdamW on large-batch LLM pretraining up to 100M-token batches.
-
Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss
Double preconditioning (DoPr) improves downstream task performance in test-time feedback settings without consistent gains in validation loss.
-
On the Provable Suboptimality of Momentum SGD in Nonstationary Stochastic Optimization
Momentum SGD pays a provable tracking penalty under distribution shift — lag growing as (1−β)⁻¹ and tracking floors worse than vanilla SGD in drift-dominated regimes.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.