REVIEW 11 cited by
A New Perspective on Shampoo's Preconditioner
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
A New Perspective on Shampoo's Preconditioner
abstract
Shampoo, a second-order optimization algorithm which uses a Kronecker product preconditioner, has recently garnered increasing attention from the machine learning community. The preconditioner used by Shampoo can be viewed either as an approximation of the Gauss--Newton component of the Hessian or the covariance matrix of the gradients maintained by Adagrad. We provide an explicit and novel connection between the $\textit{optimal}$ Kronecker product approximation of these matrices and the approximation made by Shampoo. Our connection highlights a subtle but common misconception about Shampoo's approximation. In particular, the $\textit{square}$ of the approximation used by the Shampoo optimizer is equivalent to a single step of the power iteration algorithm for computing the aforementioned optimal Kronecker product approximation. Across a variety of datasets and architectures we empirically demonstrate that this is close to the optimal Kronecker product approximation. Additionally, for the Hessian approximation viewpoint, we empirically study the impact of various practical tricks to make Shampoo more computationally efficient (such as using the batch gradient and the empirical Fisher) on the quality of Hessian approximation.
Forward citations
Cited by 11 Pith papers
-
Optimization Geometrodynamics: Variational Reduction and Interaction Curvature
Determinant-one SPD action fibers form a global analytic totally geodesic bundle whose AIRM nearest controller is computable by a residual-gradient iteration with posterior certificates and recovers inverse-Hessian sh...
-
Why Muon Outperforms Adam: A Curvature Perspective
Muon outperforms Adam by reducing curvature penalty via lower Normalized Directional Sharpness, as shown via Taylor approximation on LLM training and proven on stylized quadratic problems with heterogeneous curvature.
-
Old Optimizer, New Norm: An Anthology
Optimizers like Adam reduce to steepest descent under particular norms, opening a design space of norm assignments tailored to layer roles.
-
SOAP, Muon, and Beyond: Pushing LLM Pretraining Scales
SOAP and Muon, stabilized by per-step QR eigenbasis updates and KL-Shampoo covariance accumulation, beat AdamW on large-batch LLM pretraining up to 100M-token batches.
-
Causal Optimizer Interaction Calculus: Hidden Geometric Relaxation and Identifiable Interventions
Under fixed innovation coupling, finite-horizon optimizers admit minimal pathwise realizations and incidence-identifiable Möbius effects, with a five-term readout transfer from hidden relaxation and a closed reduced-v...
-
Optimization Geometrodynamics: Variational Reduction and Interaction Curvature
Dynamic geometric complexity of reducing condition number under full SPD metric control equals the affine-invariant distance from the relative log-spectrum to a low-width set.
-
Pro-KLShampoo: Projected KL-Shampoo with Whitening Recovered by Orthogonalization
Pro-KLShampoo projects KL-Shampoo preconditioners to a spike-and-flat parametric form on an r-dimensional subspace and recovers the full algebraic preconditioner via orthogonalization, outperforming KL-Shampoo on GPT-...
-
Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization
DeVA_S8 reweights Muon's matrix-sign update in the matrix's eigenbasis with a singular-value signal-to-noise ratio, reaching target LLM validation perplexity with ~6.6% fewer tokens than Muon.
-
Harnessing Optimization Dynamics for Curvature-Informed Model Merging
Optimization Trajectory Aware merging uses Adam second moments as a curvature proxy, first pruning task-vector edits with Fast Fisher Grafting, then reweighting survivors with a compressed curvature preconditioner.
-
Causal Optimizer Interaction Calculus: Hidden Geometric Relaxation and Identifiable Interventions
A modular calculus decomposes optimizer updates into geometric preconditioning plus structured nongeometric mechanisms, with a direction-expressivity theorem showing full SPD geometry captures exactly strict descent d...
-
On the Convergence Analysis of Muon
Convergence analysis shows Muon outperforms gradient descent by exploiting low-rank structure in neural network Hessians.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.