For a one-dimensional quadratic stochastic optimization problem, MUON with Newton-Schulz steps provably fails to converge to the minimizer for all sufficiently large mini-batch sizes when the data is skewed, while a new error analysis bounds its mean-square error by C(M^{-1} + gamma_n).
Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In stochastic optimization, a common tool to deal sequentially with large sample is to consider the well-known stochastic gradient algorithm. Nevertheless, since the stepsequence is the same for each direction, this can lead to bad results in practice in case of ill-conditionned problem. To overcome this, adaptive gradient algorithms such that Adagrad or Stochastic Newton algorithms should be prefered. This paper is devoted to the non asymptotic analyis of these adaptive gradient algorithms for strongly convex objective. All the theoretical results will be adapted to linear regression and regularized generalized linear model for both Adagrad and Stochastic Newton algorithms.
fields
math.OC 1years
2026 1verdicts
ACCEPT 1representative citing papers
citing papers explorer
-
On MUON optimization: From non-convergence to an error analysis with Polar Express and the Newton-Schulz polynomial from implementations
For a one-dimensional quadratic stochastic optimization problem, MUON with Newton-Schulz steps provably fails to converge to the minimizer for all sufficiently large mini-batch sizes when the data is skewed, while a new error analysis bounds its mean-square error by C(M^{-1} + gamma_n).