Spectral clipping of leading singular values in gradient matrices stabilizes SGD for non-convex problems with heavy-tailed noise and achieves the optimal convergence rate O(K^{(2-2α)/(3α-2)}).
Linear attention is (maybe) all you need (to understand transformer optimization).arXiv preprint arXiv:2310.01082, 2023
2 Pith papers cite this work. Polarity classification is still indexing.
2
Pith papers citing it
citation-role summary
background 1
citation-polarity summary
fields
cs.LG 2years
2026 2verdicts
UNVERDICTED 2roles
background 1polarities
background 1representative citing papers
Proposes a clipped two-point zeroth-order algorithm achieving O(d^{p/2(p-1)} δ^{-1} ε^{-(2p-1)/p-1}) complexity for (δ, ε)-Goldstein stationary points in nonconvex nonsmooth problems with heavy-tailed noise.
citing papers explorer
-
Gradient Clipping Beyond Vector Norms: A Spectral Approach for Matrix-Valued Parameters
Spectral clipping of leading singular values in gradient matrices stabilizes SGD for non-convex problems with heavy-tailed noise and achieves the optimal convergence rate O(K^{(2-2α)/(3α-2)}).
-
Zeroth-Order Nonconvex Nonsmooth Optimization with Heavy-Tailed Noise
Proposes a clipped two-point zeroth-order algorithm achieving O(d^{p/2(p-1)} δ^{-1} ε^{-(2p-1)/p-1}) complexity for (δ, ε)-Goldstein stationary points in nonconvex nonsmooth problems with heavy-tailed noise.