Pith. sign in

REVIEW 3 major objections 4 minor 61 references

Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Stochastic steepest descent with a general ℓp norm reaches ε-approximate stationarity in O(ε^-4) gradient queries, and the accelerated variant STACEY outperforms SGD, AdamW, and Lion when p is tuned to the task.

desk verdict Load-bearing Jensen error invalidates the headline convergence rate, but the algorithm and the ImageNet/LLM experiments are real enough to warrant a major-revision review. read the letter →

arxiv 2506.06606 v1 pith:4BFOULSH submitted 2025-06-07 cs.LG

Xinyu Luo , Cedar Site Bai , Bolian Li , Petros Drineas , Ruqi Zhang , Brian Bullins This is my paper
classification cs.LG
keywords lp-smoothstochasticoptimizationsteepestdescentnonconvexprimal-dualinterpolationsignSGDgeneralizationoptimizeraccelerationLLMpretraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the norm used in gradient-based training is a tunable part of the optimization problem, not a fixed Euclidean default, and that moving to a general $\ell_p$ norm with $p \in (2,\infty)$ can provably and practically improve nonconvex stochastic optimization. Its main theoretical result is that stochastic $\ell_p$ steepest descent, the update that signSGD uses at $p=\infty$, finds an $\varepsilon$-approximate stationary point in $O(\varepsilon^{-4})$ gradient queries under $\ell_p$-smoothness, bounded variance, and bounded gradients. The paper then builds STACEY by accelerating this descent through an interpolation of two iterate sequences, a primal steepest-descent sequence and a dual mirror-descent sequence, and reports that it converges faster and reaches higher accuracy than SGD, AdamW, and Lion on CIFAR, ImageNet, and a 100-million-parameter language model. The message a reader can take from the paper is that matching the norm to the loss geometry is a genuine design choice, with the best $p$ varying across tasks.

What carries the argument

The load-bearing object is the $\ell_p$ steepest-descent map $s_i(x) = x_i |x_i|^{(p-2)/(p-1)}$, which at $p=\infty$ collapses to the sign function (signSGD) and at $p=2$ to the gradient itself, so the algorithm interpolates between Euclidean and sign-based updates. What carries the acceleration is the two-sequence primal-dual interpolation: a primal sequence $y_t$ moves along the $\ell_p$ steepest-descent direction, a dual sequence $z_t$ moves along a gradient (or mirror) descent direction, and the output iterate is $\theta_t = \tau z_t + (1-\tau) y_t - \eta_t \lambda \theta_t$. The convergence proof works by trading the nonlinear rescaling bias for two error terms, $B_1$ and $B_2$, which are bounded with Markov's inequality and a zeroth-order Taylor expansion, respectively, and then telescoping the smoothness inequality across iterations.

What would settle it

A concrete check is to simulate Algorithm 3 on a nonconvex $\ell_p$-smooth objective whose stochastic gradients have coordinates that are zero with high probability, and record the empirical values of $\mathbb{E}[|\tilde g_i|^{(2-p)/(p-1)}]$ along with the average $\|\nabla f\|_{p_*}^{p_*}$; if the moment diverges or the gradient norm does not decay as $O(T^{-1/2})$ with batch size $n_t=T$, the missing bound in Lemma 3 is the cause.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that stochastic $\ell_p$ steepest descent with coordinate rescaling $s_i(x)=x_i |x_i|^{(p-2)/(p-1)}$ drives the average dual-norm gradient below $\varepsilon$ at the same $O(\varepsilon^{-4})$ query rate that signSGD enjoys at $p=\infty$, despite the bias introduced by coordinate-wise rescaling. The proof controls the bias by splitting the error into a sign-mismatch term and a same-sign magnitude term, bounding the latter through a zeroth-order Taylor expansion with Lagrange remainder, and then telescoping the $\ell_p$-smoothness inequality over $T$ steps. STACEY is the accelerated version: it couples the $\ell_p$ steepest-descent update with a mirror-descent update on a dual variable, producing the next iterate as a convex combination of the two sequences, and the experiments show that this coupling, with $p$ chosen per task, improves both convergence speed and final test accuracy relative to SGD, AdamW, and Lion.

Load-bearing premise

The load-bearing premise is that, for $p>2$, the negative-moment expectations $\mathbb{E}[|\tilde g_i|^{(2-p)/(p-1)}]$ used in Lemma 3 are finite; the proof applies Jensen's inequality as if they were bounded, but no such bound is assumed, and sparse or near-zero gradient coordinates would violate it.

Editorial extensions

If this is right

  • Setting $p=2$ recovers the Euclidean regime and $p \to \infty$ recovers sign-based updates, so the theory gives a continuous bridge between SGD-style and signSGD-style guarantees.
  • Because the $O(\varepsilon^{-4})$ rate is stated in the dual norm $\|\cdot\|_{p_*}$, the stationarity measure is matched to the update geometry, which is the natural comparison for non-Euclidean methods.
  • The empirical protocol makes $p$ a hyperparameter: $p \approx 2$ was best for CIFAR with ResNet18 while $p=3$ was best for 100M-parameter LLM pretraining, indicating that optimal geometry is task-dependent.
  • STACEY carries a $2d$ memory footprint, the same order as Adam, so the acceleration does not require storing more than the standard two-vector optimizer state.
  • For any fixed $p>2$, the query complexity matches that of signSGD inside the same stationarity measure, so intermediate $\ell_p$ norms need not pay a worse dependence on $\varepsilon$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a natural extension is to make $\alpha$ and $\tau$ adaptive or parameter-free, since the acceleration comes from the two-sequence interpolation rather than from a fixed momentum schedule.
  • Editorial inference: intermediate $p$ should help most when gradient coordinates are heavy-tailed or have skewed scales, because the rescaling dampens large coordinates less than sign but more than Euclidean updates.
  • Editorial inference: a diagnostic experiment the authors do not run is to measure the gradient-coordinate distribution during training and correlate its tail index with the best-performing $p$, which could turn norm selection into a data-driven procedure.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes STACEY, a family of stochastic steepest-descent optimizers that interpolate between ℓ2 and ℓp geometries, with two variants (STACEY(p,2) and STACEY(p,p)). The central theoretical claim is Theorem 1, which states that the unaccelerated stochastic ℓp descent (Algorithm 3) converges at rate O(ε^{-4}) in gradient complexity for nonconvex, ℓp-smooth objectives under standard unbiasedness, variance, and bounded-gradient assumptions, strictly generalizing signSGD guarantees. The paper also presents empirical comparisons on CIFAR, ImageNet, and LLM pretraining, reporting faster convergence and higher accuracy than SGD, Adam, AdamW, and Lion. No convergence analysis is provided for the actual STACEY algorithms (Algorithms 1 and 2).

Significance. If the theoretical claims were valid, the paper would contribute a useful non-Euclidean alternative to existing optimizers, with a plausible generalization of signSGD to general ℓp norms. The empirical study covers both vision and language benchmarks and includes code. However, the main theorem is not established due to a reversed Jensen inequality in a key lemma, and the accelerated algorithm itself is never analyzed. The paper correctly identifies an interesting research direction and situates it in the convex optimization literature, but the current theoretical and algorithmic contributions fall short of the claims.

major comments (3)
  1. [Appendix A.1, Lemma 3 (case 2)] The proof of Lemma 3, case (2), bounds E[|\tilde g|^{2(2-p)/(p-1)}] by (E|\tilde g|^2)^{(2-p)/(p-1)} citing Jensen's inequality. For p > 2, the exponent q = (2-p)/(p-1) is negative, and the map x ↦ x^q is convex on (0,∞); Jensen gives E[X^q] ≥ (EX)^q, not ≤. The displayed inequality is therefore reversed, and the resulting bound ηG^{1/(p-1)}∥σ∥₁/((p-1)√n_t) is unsupported. The lemma is in fact false: for d=1, p=4, g=1, G=1.99, σ=0.99, and a stochastic gradient X taking values 0.01 and 1.99 with equal probability, all Assumptions 2–4 hold, but the left side of Lemma 3 equals η·0.521 while the claimed right side is η·0.415. Since Lemma 3 is the only step that controls the bias term B2, the telescoping proof of Theorem 1 collapses, and the O(ε^{-4}) rate is not established under Assumptions 1–4.
  2. [Section 4 (STACEY algorithms)] Theorem 1 analyzes only Algorithm 3, the unaccelerated stochastic ℓp descent. No convergence guarantee is provided for the proposed STACEY methods, Algorithms 1 and 2. The abstract and introduction claim an 'accelerated ℓp-based method' with 'novel theoretical guarantees', but the acceleration mechanism is never analyzed. The cited convex acceleration results (Allen-Zhu & Orecchia 2017; Nemirovskii & Nesterov 1985) apply to deterministic convex problems and are not extended to the stochastic nonconvex setting. Thus the paper's central algorithmic contribution is without theoretical support, and the empirical results alone cannot substantiate the acceleration claim.
  3. [Section 5 (Experiments)] The empirical comparisons do not consistently support the claim of 'faster convergence'. In Table 2 (ImageNet), STACEY's training NLL is higher than AdamW at every reported epoch (e.g., 1.0324 vs. 0.7395 at 60 epochs), so the advantage appears only in test accuracy, not in training loss. Additionally, the hyperparameter tables list sharply different learning rates across methods (e.g., 0.1 for STACEY vs. 0.01 for AdamW on CIFAR) and the paper does not report the number of hyperparameter configurations tried per method, so the improvements may reflect unequal tuning effort. The paper should either provide a more controlled comparison or temper the 'consistently outperforms' claims in the abstract and Section 5.3.
minor comments (4)
  1. [Abstract] The abstract contains a typo: 'newaccelerated' should be 'new accelerated'.
  2. [Section 4.1, Theorem 1] The expression for η is ambiguous; it should be typeset clearly, e.g., η = 1/(L^{1/2} G^{1/(p-1)} T^{1/2}), to avoid confusion with a product in the denominator.
  3. [References] The reference to Krizhevsky (2009) lists 'University of Tront'; this should be 'University of Toronto'.
  4. [Table 2 caption] The caption states 'demonstrates superior test accuracy and faster convergence', but the training NLL values in the same table are higher for STACEY than for AdamW at all epochs; the caption should be revised to describe only the test accuracy advantage.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: Theorem 1 is derived self-contained from Assumptions 1-4; the acceleration design is heuristic and externally motivated, and self-citations are not load-bearing.

full rationale

Theorem 1 is a self-contained derivation: Algorithm 3's update and Assumptions 1-4 are the inputs, and the proof telescopes f(θ_{t+1}) using bounds on the terms A, B1, B2, and C obtained from Markov's inequality, Cauchy-Schwarz, and the stated variance/bounded-gradient assumptions. None of the constants (L, G, σ⃗, η) is fitted to the quantity being bounded, and the final O(ε^{-4}) gradient-query count follows by algebra from the displayed bound. The p=∞ limit (Algorithm 3 reduces exactly to signSGD) is an identification, not a circular reduction: the bound is proved for the general ℓp update and then specialized. The acceleration portion of STACEY is not claimed to be proven by a theorem in this paper; it is motivated by external convex optimization results (Allen-Zhu & Orecchia, Nemirovskii & Nesterov, Diakonikolas & Guzmán), so no prediction is derived from an input that already contains it. The self-citations (Bai & Bullins 2024a,b, 2025; Bullins 2020) are contextual references to related acceleration, lower-bound, and federated analyses and do not carry the proof of Theorem 1. The possible reverse-Jensen step in Lemma 3 is a correctness concern, not a circularity: even if the lemma is false, the claimed rate would be unproved rather than assumed-as-input. Hence no circular step is exhibited.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central theoretical result rests on Assumptions 1-4, plus an unstated negative-moment assumption required by Lemma 3. The empirical method adds several grid-searched hyperparameters (p, η, α, τ, β1, β2, ε) that are not covered by the theory. No new physical entities are introduced.

free parameters (6)
  • p = 2 (CIFAR), 3 (LLM), 3 or 2.8 (ImageNet)
    The ℓ_p norm order is grid-searched per dataset; the method is only beneficial when p is chosen to match the task geometry, and no a priori rule is provided.
  • learning rate η = 0.1 (CIFAR), 0.01 (ImageNet/LLM) for STACEY; baselines use different values
    Step size chosen by grid search; the theoretical step size depends on unknown L, G, T and is not used in practice.
  • α = 0.1 (CIFAR/LLM), 0.001 (ImageNet)
    Dual descent step size, tuned by grid search; no analysis guides the choice.
  • τ = 0.001 everywhere
    Interpolation weight between primal and dual iterates; chosen by hand; the 'acceleration' effect is small at this value.
  • β1, β2 = 0.9, 0.99 or 0.999
    Momentum hyperparameters, tuned as in Adam and Lion.
  • ε = 1e-12 to 1e-8
    Regularization in the steepest descent denominator; required for numerical stability and to prevent blow-up of s^ε; the analyzed Algorithm 3 does not include ε.
assumptions (6)
  • domain assumption Assumption 1: f satisfies the ℓ_p-smoothness descent inequality |f(y)-f(x)-∇f(x)^T(y-x)| ≤ L/2 ∥y-x∥_p² for p≥2
    This is stronger than Lipschitz gradient for nonconvex functions; deep network losses are not known to satisfy it globally with a finite L.
  • domain assumption Assumption 2: stochastic gradient is unbiased, E[ĝ]=g
    Standard but may not hold with data augmentation or shuffled batches.
  • domain assumption Assumption 3: per-coordinate variance bound E[|ĝ_i - g_i|²] ≤ σ_i²
    Coordinate-wise bounded variance is strong; it does not follow from a global variance bound.
  • domain assumption Assumption 4: ∥ĝ∥_{p*} ≤ G
    Bounded stochastic gradient in the dual norm; for p>2 this is stronger than a Euclidean bound.
  • ad hoc to paper Implicit negative-moment bound: E[|ĝ_i|^{(2-p)/(p-1)}] is finite and bounded above
    Needed in Lemma 3 case (2) but not stated; the claimed Jensen step would require the reverse inequality, and without this the lemma fails for small coordinates.
  • ad hoc to paper The primal-dual interpolation in Algorithm 1 preserves or improves on the convergence of Algorithm 3
    No theorem supports the acceleration; the paper relies on convex analogies (Allen-Zhu & Orecchia, Nemirovskii & Nesterov) that do not transfer to the stochastic nonconvex setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization." pith.science (2026). https://pith.science/paper/4BFOULSH

@misc{pith2026250606606,
  author       = {Pith},
  title        = {Pith review of: Stacey: Promoting Stochastic Steepest Descent via Accelerated $\ell_p$-Smooth Nonconvex Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4BFOULSH}},
  note         = {Machine review of arXiv:2506.06606}
}
abstract

While popular optimization methods such as SGD, AdamW, and Lion depend on steepest descent updates in either $\ell_2$ or $\ell_\infty$ norms, there remains a critical gap in handling the non-Euclidean structure observed in modern deep networks training. In this work, we address this need by introducing a new accelerated $\ell_p$ steepest descent algorithm, called Stacey, which uses interpolated primal-dual iterate sequences to effectively navigate non-Euclidean smooth optimization tasks. In addition to providing novel theoretical guarantees for the foundations of our algorithm, we empirically compare our approach against these popular methods on tasks including image classification and language model (LLM) pretraining, demonstrating both faster convergence and higher final accuracy. We further evaluate different values of $p$ across various models and datasets, underscoring the importance and efficiency of non-Euclidean approaches over standard Euclidean methods. Code can be found at https://github.com/xinyuluo8561/Stacey .

Figures

Figures reproduced from arXiv: 2506.06606 by the authors.

Figure 1
Figure 1. Learning curves of CIFAR classification with varying ℓp-norm. pretraining (Section 5.2). The hyperparameter choices and tuning are summarized in Appendix C. In all experiments, we underscore the efficiency of the STACEY optimizer by comparing it against other optimiz￾ers as baselines including SGD (with momentum) (Nes￾terov, 1983; Polyak, 1964), Adam (Kingma & Ba, 2015), AdamW (Loshchilov & Hutter, 2019), and Lion (… view at source ↗
Figure 3
Figure 3. Learning curves of LLM pretraining at the first 30K iterations with varying ℓp-norm. pretraining (Section 5.2). These results highlight the impor￾tance of developing non-Euclidean optimizers and adjusting the choice of ℓp-norm to enhance performance across dif￾ferent tasks, and we would note this choice may further benefit from, e.g., parameter-free approaches (Jacobsen & Cutkosky, 2022). 5.1. Image Classification W… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 52 canonical work pages

  1. [1]

    Convex optimization with p -norm oracles

    Adil, D., Bullins, B., Jambulapati, A., and Sidford, A. Convex optimization with p -norm oracles. arXiv preprint arXiv:2410.24158, 2024

    arXiv 2024

  2. [2]

    Adaptive norms for deep learning with regularized Newton methods

    Adolphs, L., Kohler, J., and Lucchi, A. Ellipsoidal trust region methods and the marginal value of hessian information for neural network training. arXiv preprint arXiv:1905.09201, 2019

    work page Pith review arXiv 1905

  3. [3]

    J., Bartlett, P., and Ravikumar, P

    Agarwal, A., Wainwright, M. J., Bartlett, P., and Ravikumar, P. Information-theoretic lower bounds on the oracle complexity of convex optimization. Advances in Neural Information Processing Systems, 22, 2009

    work page 2009

  4. [4]

    and Orecchia, L

    Allen-Zhu, Z. and Orecchia, L. Linear coupling: An ultimate unification of gradient and mirror descent. In 8th Innovations in Theoretical Computer Science Conference. Schloss Dagstuhl--Leibniz-Zentrum f \"u r Informatik, 2017

    work page 2017

  5. [5]

    C., Foster, D

    Arjevani, Y., Carmon, Y., Duchi, J. C., Foster, D. J., Srebro, N., and Woodworth, B. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199 0 (1): 0 165--214, 2023

    work page 2023

  6. [6]

    Faster Acceleration for Steepest Descent

    Bai, S. and Bullins, B. Faster acceleration for steepest descent. arXiv preprint arXiv:2409.19200, 2024 a

    work page Pith review arXiv 2024

  7. [7]

    and Bullins, B

    Bai, S. and Bullins, B. Local composite saddle point optimization. In International Conference on Learning Representations, 2024 b

    work page 2024

  8. [8]

    and Bullins, B

    Bai, S. and Bullins, B. Tight lower bounds under asymmetric high-order H\"older smoothness and uniform convexity. In International Conference on Learning Representations, 2025

    work page 2025

Show all 61 references
  1. [9]

    Balles, L., Pedregosa, F., and Roux, N. L. The geometry of sign gradient descent. arXiv preprint arXiv:2002.08056, 2020

    2002 arXiv

  2. [10]

    signsgd: Compressed optimisation for non-convex problems

    Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anandkumar, A. signsgd: Compressed optimisation for non-convex problems. In International Conference on Machine Learning, pp.\ 560--569. PMLR, 2018

    2018

  3. [11]

    Highly smooth minimization of non-smooth problems

    Bullins, B. Highly smooth minimization of non-smooth problems. In Conference on Learning Theory, pp.\ 988--1030. PMLR, 2020

    2020

  4. [12]

    Convex until proven guilty

    Carmon, Y., Duchi, J. C., Hinder, O., and Sidford, A. “ Convex until proven guilty”: dimension-free acceleration of gradient descent on non-convex functions. In International Conference on Machine Learning, pp.\ 654--663. PMLR, 2017

    2017

  5. [13]

    Lion secretly solves a constrained optimization: As lyapunov predicts

    Chen, L., Liu, B., Liang, K., and Liu, Q. Lion secretly solves a constrained optimization: As lyapunov predicts. In International Conference on Learning Representations, 2024

    2024

  6. [14]

    Symbolic discovery of optimization algorithms

    Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., et al. Symbolic discovery of optimization algorithms. Advances in Neural Information Processing Systems, 36, 2023

    2023

  7. [15]

    Z., and Talwalkar, A

    Cohen, J., Kaur, S., Li, Y., Kolter, J. Z., and Talwalkar, A. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2021

    2021

  8. [16]

    P., Guzm \'a n, C., and Mart \' nez-Rubio, D

    Contreras, J. P., Guzm \'a n, C., and Mart \' nez-Rubio, D. Non-euclidean high-order smooth convex optimization. arXiv preprint arXiv:2411.08987, 2024

    2024 arXiv

  9. [17]

    The road less scheduled

    Defazio, A., Yang, X., Khaled, A., Mishchenko, K., Mehta, H., and Cutkosky, A. The road less scheduled. Advances in Neural Information Processing Systems, 37: 0 9974--10007, 2024

    2024

  10. [18]

    A guide through the zoo of biased SGD

    Demidovich, Y., Malinovsky, G., Sokolov, I., and Richt \'a rik, P. A guide through the zoo of biased SGD . In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=OCtv4NyahI

    2023

  11. [19]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 248--255. Ieee, 2009

    2009

  12. [20]

    and Guzm \'a n, C

    Diakonikolas, J. and Guzm \'a n, C. Complementary composite minimization, small gradients in general norms, and applications. Mathematical Programming, pp.\ 1--45, 2024

    2024

  13. [21]

    and Orecchia, L

    Diakonikolas, J. and Orecchia, L. The approximate duality gap technique: A unified theory of first-order methods. SIAM Journal on Optimization, 29 0 (1): 0 660--689, 2019

    2019

  14. [22]

    Adaptive subgradient methods for online learning and stochastic optimization

    Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12 0 (7), 2011 a

    2011

  15. [23]

    C., Agarwal, A., and Wainwright, M

    Duchi, J. C., Agarwal, A., and Wainwright, M. J. Dual averaging for distributed optimization: Convergence analysis and network scaling. IEEE Transactions on Automatic control, 57 0 (3): 0 592--606, 2011 b

    2011

  16. [24]

    and Lan, G

    Ghadimi, S. and Lan, G. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization, 23 0 (4): 0 2341--2368, 2013

    2013

  17. [25]

    An investigation into neural net optimization via Hessian eigenvalue density

    Ghorbani, B., Krishnan, S., and Xiao, Y. An investigation into neural net optimization via Hessian eigenvalue density. In International Conference on Machine Learning, pp.\ 2232--2241. PMLR, 2019

    2019

  18. [26]

    Shampoo: Preconditioned stochastic tensor optimization

    Gupta, V., Koren, T., and Singer, Y. Shampoo: Preconditioned stochastic tensor optimization. In International Conference on Machine Learning, pp.\ 1842--1850. PMLR, 2018

    2018

  19. [27]

    and Nemirovski, A

    Guzm \'a n, C. and Nemirovski, A. On lower complexity bounds for large-scale smooth convex optimization. Journal of Complexity, 31 0 (1): 0 1--14, 2015

    2015

  20. [28]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 770--778, 2016

    2016

  21. [29]

    and Cutkosky, A

    Jacobsen, A. and Cutkosky, A. Parameter-free mirror descent. In Conference on Learning Theory, pp.\ 4160--4211. PMLR, 2022

    2022

  22. [30]

    A direct O (1/ ) iteration parallel algorithm for optimal transport

    Jambulapati, A., Sidford, A., and Tian, K. A direct O (1/ ) iteration parallel algorithm for optimal transport. Advances in Neural Information Processing Systems, 32, 2019

    2019

  23. [31]

    How does adaptive optimization impact local neural network geometry? Advances in Neural Information Processing Systems, 36, 2024

    Jiang, K., Malik, D., and Li, Y. How does adaptive optimization impact local neural network geometry? Advances in Neural Information Processing Systems, 36, 2024

    2024

  24. [32]

    M., and Jordan, M

    Jin, C., Ge, R., Netrapalli, P., Kakade, S. M., and Jordan, M. I. How to escape saddle points efficiently. In International Conference on Machine Learning, pp.\ 1724--1732. PMLR, 2017

    2017

  25. [33]

    Linear convergence of gradient and proximal-gradient methods under the Polyak- ojasiewicz condition

    Karimi, H., Nutini, J., and Schmidt, M. Linear convergence of gradient and proximal-gradient methods under the Polyak- ojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2016, Riva del Garda, Italy, September 19-23, 2...

    2016

  26. [34]

    A., Lee, Y

    Kelner, J. A., Lee, Y. T., Orecchia, L., and Sidford, A. An almost-linear-time algorithm for approximate max flow in undirected graphs, and its multicommodity generalizations. In Proceedings of the Twenty-Fifth Annual ACM-SIAM Symposium on Discrete Algorithms, pp.\ 217--226. S...

    2014

  27. [35]

    Kingma, D. P. and Ba, J. Adam : A method for stochastic optimization. In International Conference on Learning Representations, 2015

    2015

  28. [36]

    Learning multiple layers of features from tiny images

    Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009

    2009

  29. [37]

    Hessian based analysis of sgd for deep nets: Dynamics and generalization

    Li, X., Gu, Q., Zhou, Y., Chen, T., and Banerjee, A. Hessian based analysis of sgd for deep nets: Dynamics and generalization. In Proceedings of the 2020 SIAM International Conference on Data Mining, pp.\ 190--198. SIAM, 2020 a

    2020

  30. [38]

    On the convergence of fedavg on non-iid data

    Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2020 b

    2020

  31. [39]

    Liu, H., Li, Z., Hall, D. L. W., Liang, P., and Ma, T. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In International Conference on Learning Representations, 2024

    2024

  32. [40]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  33. [41]

    and Grosse, R

    Martens, J. and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. In International conference on machine learning, pp.\ 2408--2417. PMLR, 2015

    2015

  34. [42]

    M., and Janson, L

    Morwani, D., Shapira, I., Vyas, N., eran malach, Kakade, S. M., and Janson, L. A new perspective on shampoo's preconditioner. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=c6zI3Cp8c6

    2025

  35. [43]

    Nemirovskii, A. S. and Nesterov, Y. E. Optimal methods of smooth convex minimization. USSR Computational Mathematics and Mathematical Physics, 25 0 (2): 0 21--30, 1985

    1985

  36. [44]

    Nemirovskij, A. S. and Yudin, D. B. Problem Complexity and Method Efficiency in Optimization. A Wiley-Interscience publication. Wiley, 1983. ISBN 9780471103455

    1983

  37. [45]

    A method for solving the convex programming problem with convergence rate O (1/k2)

    Nesterov, Y. A method for solving the convex programming problem with convergence rate O (1/k2). In Dokl. Akad. Nauk SSSR, volume 269, pp.\ 543, 1983

    1983

  38. [46]

    Smooth minimization of non-smooth functions

    Nesterov, Y. Smooth minimization of non-smooth functions. Mathematical programming, 103: 0 127--152, 2005

    2005

  39. [47]

    Lectures on convex optimization, volume 137

    Nesterov, Y. Lectures on convex optimization, volume 137. Springer, 2018

    2018

  40. [48]

    The full spectrum of deepnet Hessians at scale: Dynamics with sgd training and sample size

    Papyan, V. The full spectrum of deepnet Hessians at scale: Dynamics with sgd training and sample size. arXiv preprint arXiv:1811.07062, 2018

    2018 arXiv

  41. [49]

    Polyak, B. T. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4 0 (5): 0 1--17, 1964

    1964

  42. [50]

    and Monro, S

    Robbins, H. and Monro, S. A stochastic approximation method. The Annals of Mathematical Statistics, pp.\ 400--407, 1951

    1951

  43. [51]

    Area-convexity, _ regularization, and undirected multicommodity flow

    Sherman, J. Area-convexity, _ regularization, and undirected multicommodity flow. In Proceedings of the 49th Annual ACM SIGACT Symposium on Theory of Computing, pp.\ 452--460, 2017

    2017

  44. [52]

    and Tian, K

    Sidford, A. and Tian, K. Coordinate methods for accelerating _ regression and faster approximate maximum flow. In 2018 IEEE 59th Annual Symposium on Foundations of Computer Science, pp.\ 922--933. IEEE, 2018

    2018

  45. [53]

    Unified acceleration of high-order algorithms under general holder continuity

    Song, C., Jiang, Y., and Ma, Y. Unified acceleration of high-order algorithms under general holder continuity. SIAM Journal on Optimization, 31 0 (3): 0 1797--1826, 2021

    2021

  46. [54]

    Stich, S. U. and Ajalloeian, A. Analysis of sgd with biased gradient estimators. arXiv preprint arXiv:2008.00051, 2020

    2008 arXiv

  47. [55]

    On the importance of initialization and momentum in deep learning

    Sutskever, I., Martens, J., Dahl, G., and Hinton, G. On the importance of initialization and momentum in deep learning. In International Conference on Machine Learning, pp.\ 1139--1147. PMLR, 2013

    2013

  48. [56]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  49. [57]

    Vyas, N., Morwani, D., Zhao, R., Shapira, I., Brandfonbrener, D., Janson, L., and Kakade, S. M. SOAP : Improving and stabilizing shampoo using adam for language modeling. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/f...

    2025

  50. [58]

    X., and Liu, H

    Wang, M., Fang, E. X., and Liu, H. Stochastic compositional gradient descent: algorithms for minimizing compositions of expected-value functions. Mathematical Programming, 161: 0 419--449, 2017

    2017

  51. [59]

    Online composite optimization between stochastic and adversarial environments

    Wang, Y., Chen, S., Jiang, W., Yang, W., Wan, Y., and Zhang, L. Online composite optimization between stochastic and adversarial environments. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=MbEB5aKmMK

    2024

  52. [60]

    Federated composite optimization

    Yuan, H., Zaheer, M., and Reddi, S. Federated composite optimization. In International Conference on Machine Learning, pp.\ 12253--12266. PMLR, 2021

    2021

  53. [61]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.