Spectral clipping of leading singular values in gradient matrices stabilizes SGD for non-convex problems with heavy-tailed noise and achieves the optimal convergence rate O(K^{(2-2α)/(3α-2)}).
From gradient clipping to normalization for heavy tailed sgd
7 Pith papers cite this work. Polarity classification is still indexing.
7
Pith papers citing it
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
Muon with Nesterov momentum and inexact polar decomposition achieves optimal convergence rates of O(ε^(-(3α-2)/(α-1))) under heavy-tailed noise for ε-stationary points in non-convex settings.
GT-NSGDm achieves the optimal non-asymptotic convergence rate O(1/T^{(p-1)/(3p-2)}) for decentralized nonconvex stochastic optimization under zero-mean heavy-tailed noise with p-th moment.
LionMuon alternates Lion and Muon steps with shared dual-EMA buffer to Pareto-dominate existing optimizers in loss and compute on models up to 720M parameters.
RSC-ZO achieves high-probability ε-stationary points for stochastic ZO optimization under weak-L_p heavy-tailed noise with Õ(d^{p/2(p-1)} ε^{-(3p-2)/(p-1)}) function queries.
Entry-wise clipping achieves spectral control of gradients via localization under heavy-tailed contamination, with O(ε^{-4}) convergence and empirical savings on NanoGPT pretraining.
Proposes a clipped two-point zeroth-order algorithm achieving O(d^{p/2(p-1)} δ^{-1} ε^{-(2p-1)/p-1}) complexity for (δ, ε)-Goldstein stationary points in nonconvex nonsmooth problems with heavy-tailed noise.
citing papers explorer
-
Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters
Spectral clipping of leading singular values in gradient matrices stabilizes SGD for non-convex problems with heavy-tailed noise and achieves the optimal convergence rate O(K^{(2-2α)/(3α-2)}).
-
Muon with Nesterov Momentum: Heavy-Tailed Noise and (Randomized) Inexact Polar Decomposition
Muon with Nesterov momentum and inexact polar decomposition achieves optimal convergence rates of O(ε^(-(3α-2)/(α-1))) under heavy-tailed noise for ε-stationary points in non-convex settings.
-
LionMuon: Alternating Spectral and Sign Descent for Efficient Training
LionMuon alternates Lion and Muon steps with shared dual-EMA buffer to Pareto-dominate existing optimizers in loss and compute on models up to 720M parameters.
-
Stochastic Zeroth-Order Optimization Under Heavy-Tailed Noise
RSC-ZO achieves high-probability ε-stationary points for stochastic ZO optimization under weak-L_p heavy-tailed noise with Õ(d^{p/2(p-1)} ε^{-(3p-2)/(p-1)}) function queries.
-
Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?
Entry-wise clipping achieves spectral control of gradients via localization under heavy-tailed contamination, with O(ε^{-4}) convergence and empirical savings on NanoGPT pretraining.
-
Zeroth-Order Nonconvex Nonsmooth Optimization with Heavy-Tailed Noise
Proposes a clipped two-point zeroth-order algorithm achieving O(d^{p/2(p-1)} δ^{-1} ε^{-(2p-1)/p-1}) complexity for (δ, ε)-Goldstein stationary points in nonconvex nonsmooth problems with heavy-tailed noise.