Pith. sign in

REVIEW 3 major objections 4 minor 33 references

Improved Analysis for Sign-based Methods with Momentum Updates

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Under standard L2-smoothness, signSGD with momentum converges at rate $O(d^{1/2} T^{-1/4})$, a $\sqrt{d}$ improvement over prior momentum signSGD, and under $L_\infty$-smoothness it reaches the dimension-free $O(T^{-1/4})$ with constant…

desk verdict The single-node rates are a real improvement, but the paper is not ready as written: Lemma 1 is false and the distributed proof has an algebra slip that leaves a non-vanishing floor. read the letter →

arxiv 2507.12091 v1 pith:644ZUBDV submitted 2025-07-16 math.OC cs.LG

Wei Jiang , Dingzhi Yu , Sifan Yang , Wenhao Yang , Lijun Zhang This is my paper · ORCID
classification math.OCcs.LG MSC 90C2690C1568W40
keywords signSGDmomentumupdatesnon-convexoptimizationstochasticgradientmethodscommunication-efficientdistributedmajorityvoteconvergenceanalysissign-basedcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that sign-based gradient methods with momentum can recover the $O(T^{-1/4})$ convergence rate of ordinary stochastic gradient methods without the two crutches earlier signSGD proofs leaned on: large batch sizes and the assumption that the gradient noise is unimodal and symmetric. The main quantitative claim is that under standard $L_2$-smoothness the method achieves $\mathbb{E}[\|\nabla f(x_\tau)\|_1] \leq O(d^{1/2}T^{-1/4})$, a factor $\sqrt{d}$ better than the previous momentum signSGD bound, and that under the weaker $L_\infty$-smoothness assumption the same algorithm achieves the dimension-free rate $O(T^{-1/4})$ with constant batch size and no noise-shape assumptions. A second set of results covers distributed majority-vote settings, where an unbiased sign operation plus momentum improves the best known rates from $O(d T^{-1/4} + d n^{-1/2})$ to $O(d^{1/2}T^{-1/2} + d n^{-1/2})$ and from $O(d^{3/8}T^{-1/8})$ to $O(\max\{d^{1/4}T^{-1/4}, d^{1/10}T^{-1/5}\})$. Why a reader should care: sign-based methods transmit only one bit per coordinate, and these results say that compression no longer has to cost a worse rate or a special noise model.

What carries the argument

The paper's engine is a momentum-corrected sign descent. Each step forms the exponential moving average $v_t = (1-\beta)v_{t-1} + \beta \nabla f(x_t;\xi_t)$ and updates $x_{t+1} = x_t - \eta\,\mathrm{sign}(v_t)$. The analysis rests on the sign discrepancy bound $\langle \nabla f(x_t), \mathrm{sign}(\nabla f(x_t)) - \mathrm{sign}(v_t)\rangle \leq 2\|\nabla f(x_t) - v_t\|_1$, which converts the loss decrease into a descent term $-\eta\|\nabla f(x_t)\|_1$ plus a term controlled by the momentum error $\|\nabla f(x_t) - v_t\|$. That error follows a geometric recursion whose steady-state square root $(\sigma^2/(\beta T) + \eta^2 L^2 d/\beta^2 + \beta\sigma^2)^{1/2}$ is what produces the $\sqrt{d}$ factor under $L_2$-smoothness, and a coordinate-weighted variant under $L_\infty$-smoothness produces the dimension-free rate. For the distributed theorems the load-bearing object is the unbiased sign operator $S_R(v)$, defined componentwise by $[S_R(v)]_k = 1$ with probability $(R+[v]_k)/(2R)$ and $-1$ otherwise, so that $\mathbb{E}[S_R(v)] = v/R$; this is what lets each node send one bit while keeping the server-side aggregate unbiased under the uniform bound $\|v\|_\infty \leq G$.

What would settle it

Run SMM on $f(x) = \frac{1}{2}\|x\|^2$ in dimension $d$ with coordinate noise drawn from a $t$-distribution with 3 degrees of freedom (finite variance, heavy tails, neither unimodal-symmetric nor bounded) and constant batch size 1; if $\mathbb{E}[\|\nabla f(x_\tau)\|_1]$ does not follow $O(T^{-1/4})$ independently of $d$, Theorem 2's dimension-free claim is refuted. For the distributed theorems, the direct test is to let a single sampled gradient exceed the claimed uniform bound $G$ on one coordinate and check whether the iterate sequence still converges at the promised rate despite the failure of $\mathbb{E}[S_G(v)] = v/G$.

Watch

Extended reading notes

Core claim

SignSGD with momentum updates — the SMM algorithm, which tracks $v_t = (1-\beta)v_{t-1} + \beta \nabla f(x_t;\xi_t)$ and updates $x_{t+1} = x_t - \eta\,\mathrm{sign}(v_t)$ — converges with $\mathbb{E}[\|\nabla f(x_\tau)\|_1] \leq O(d^{1/2}T^{-1/4})$ under the standard $L_2$-smoothness assumption, improving the previous $O(d T^{-1/4})$ bound of momentum signSGD by a factor of $d^{1/2}$ and reducing sample complexity from $O(d^4\epsilon^{-4})$ to $O(d^2\epsilon^{-4})$. Under the strictly weaker $L_\infty$-smoothness assumption the same method achieves the dimension-free rate $O(T^{-1/4})$ with a constant batch size, without assuming unimodal symmetric noise. In the distributed majority-vote setting, the MVSM algorithm uses the unbiased sign operator $S_R$ and obtains rates $O(d^{1/2}T^{-1/2} + dn^{-1/2})$, $O(n^{1/2}T^{-1} + dn^{-1/2})$, and $O(\max\{d^{1/4}T^{-1/4}, d^{1/10}T^{-1/5}\})$, the latter two improving on the previous $O(dT^{-1/4} + dn^{-1/2})$ and $O(d^{3/8}T^{-1/8})$ results.

Load-bearing premise

For the distributed results, the proofs assume a known constant $G$ such that every sampled gradient coordinate at every node lies in $[-G, G]$; the unbiased sign estimator is unbiased only inside that box, so a single coordinate exceeding $G$ breaks the identity $\mathbb{E}[S_G(v)] = v/G$ that the proof's central step depends on.

Editorial extensions

If this is right

  • SMM matches the $O(T^{-1/4})$ rate of SGD under $L_\infty$-smoothness with constant batch size, removing both the large-batch requirement and the unimodal-symmetric-noise assumption of earlier signSGD analyses.
  • Under $L_2$-smoothness, sample complexity drops from $O(d^4\epsilon^{-4})$ to $O(d^2\epsilon^{-4})$, so the method becomes quadratically cheaper in the problem dimension.
  • MVSM keeps 1-bit communication in both directions while improving the distributed rates to $O(d^{1/2}T^{-1/2}+dn^{-1/2})$ and $O(\max\{d^{1/4}T^{-1/4}, d^{1/10}T^{-1/5}\})$, beating both Sto-signSGD and MV-sto-signSGD-SIM on their own assumptions.
  • Theorem 4's rate $O(n^{1/2}T^{-1}+dn^{-1/2})$ is better than Theorem 3's whenever $T \geq n/d$, a condition that holds for large $d$; the two rates trade off the number of nodes against the dimension.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the $\sqrt{d}$ improvement suggests the earlier $O(d)$ factor was an artifact of how the momentum error was measured, and the same second-moment square-root technique could sharpen rates for other compressed-gradient methods with bounded compressors.
  • Editorial inference: the unbiased sign operator $S_R$ is a general recipe for de-biasing any vector quantizer with a known bound; the distributed analysis may transfer to unbiased random-$k$ sparsification under the same uniform-bound assumption on the gradients.
  • Editorial inference: a testable consequence is that the dimension dependence should be directly visible — at fixed $T$, doubling $d$ should multiply the expected $\ell^1$ gradient norm by about $\sqrt{2}$ under $L_2$-smoothness but leave it unchanged under $L_\infty$-smoothness; the paper's CIFAR experiments do not isolate this scaling.
  • Editorial note: the manuscript states that all proofs are deferred to the supplementary material; until those proofs are checked, Theorems 1-5 should be read as claimed rather than verified results.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript studies sign-based stochastic optimization with momentum updates. In the centralized setting it proposes SMM and claims an O(d^{1/2}T^{-1/4}) l1-gradient convergence rate under l2-smoothness and bounded noise, and an O(T^{-1/4}) rate under l∞-smoothness with separable bounded noise and constant batch size. In the distributed setting it proposes two variants of MVSM using unbiased sign operations and claims rates O(d^{1/2}T^{-1/2}+dn^{-1/2}), O(n^{1/2}T^{-1}+dn^{-1/2}), and O(max{d^{1/4}T^{-1/4}, d^{1/10}T^{-1/5}}) under node-wise smoothness, bounded noise, and uniform gradient bounds. The proofs are contained in Appendices A-E.

Significance. The claimed rates are potentially valuable: Theorem 1 improves the dimension dependence of momentum-based signSGD under standard l2-smoothness from O(d) to O(d^{1/2}) for the l1 norm, and Theorem 5 offers a substantially better distributed l2-norm rate than the earlier O(d^{3/8}T^{-1/8}) of Jin et al. The derivations are explicit, the parameter settings are stated, and I see no circularity or rate-fitting. However, the paper contains a false assumption-ordering claim around l∞-smoothness and an algebraic error in the distributed proof. These issues are load-bearing but appear fixable, so the underlying ideas remain worth considering after revision.

major comments (3)
  1. [Lemma 1 / Appendix A] Lemma 1 is false as stated, and its proof does not establish Assumption 3. The proof derives only the quadratic upper bound f(y) <= f(x) + <grad f(x), y-x> + (1/2)(sum_i L_i)||y-x||_infty^2, which is not the gradient Lipschitz condition ||grad f(x)-grad f(y)||_1 <= L_infty ||x-y||_infty. A one-dimensional counterexample is f(x) = -x^4 + (L/2)x^2: it is separable smooth with L_1 = L because f(x) - (L/2)x^2 = -x^4 is concave, but its second derivative is unbounded below, so no finite L_infty satisfies Assumption 3. In fact Assumption 3 implies Assumption 1 with L_i = L_infty by the standard descent lemma, so the paper's claim that l∞-smoothness is weaker than separable smoothness is backwards. This affects the Introduction, the remark after Assumption 3, and Table 1. Theorem 2 itself is valid under Assumption 3, but it should not be advertised as a weakening of prior smoothness assumptions.
  2. [Appendix D, Eq. (19) and final displayed bound] The proof of Theorems 3 and 4 contains a factor error. In the chain after Eq. (19), the term 2 eta G sqrt(d) sqrt(n^{-2} sum_j E||S_G(v_j^t)||^2) equals 2 eta G sqrt(d) sqrt(d/n) = 2 eta G d / sqrt(n), not 2 eta d G sqrt(n). In addition, when the per-step sum is formed, the factor eta is dropped: the displayed 2 d G sqrt(n) should still carry the factor eta. With the stated eta = O(T^{-1/2} d^{-1/2}), the printed nonvanishing O(d sqrt(n)) term is incorrect and does not lead to the claimed rates. The theorems can likely be repaired by correcting this factor and by using sigma = O(G sqrt(d)) from Assumption 9, but the proof as submitted does not establish them.
  3. [Theorem 3 and Eq. (20)] Even after the algebraic correction, the d n^{-1/2} term in Theorem 3 is not visible in the displayed bound without an additional estimate. From Eq. (20), the term 2 sqrt(d) sqrt(sigma^2 beta / n) with beta = 1/2 contributes O(sigma sqrt(d) / sqrt(n)), which is O(d / sqrt(n)) only if sigma = O(G sqrt(d)). The proof never states or uses this consequence of Assumption 9. Since ||grad f_j(x; xi)||_infty <= G indeed implies sigma <= 2 G sqrt(d), the theorem is salvageable, but the derivation as printed is incomplete.
minor comments (4)
  1. [Assumptions 4/5 and Appendix C] The proof of Theorem 2 uses E[n_{s,i} n_{t,i}] = 0 for s != t; this requires an independence assumption across iterations that is not stated in Assumptions 4 or 5.
  2. [Notation throughout] The notation is inconsistent: sign vs. Sign, and S_R, S_G, S_1 are introduced without a unified convention.
  3. [Appendix D] There is a minor typo: 'forth inequality' should be 'fourth inequality'.
  4. [Distributed setting] The distributed rates depend on the strong uniform bound of Assumption 9 and require G to be known to the algorithm; this limitation should be stated explicitly in the introduction and in the experimental discussion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorems 1-5 follow from explicit assumptions by self-contained proofs; self-citations are contextual and not load-bearing.

full rationale

The convergence rates are derived directly from the stated smoothness, bounded-noise, and bounded-gradient assumptions using standard recursion inequalities; no fitted parameter is subsequently renamed as a prediction, and no claimed rate is an input to a proof by construction. The unbiased sign operation S_R is defined so that E[S_R(v)]=v/R, but this is a property of the estimator used inside the algorithm rather than the target convergence bound, so using it in the proof is not circular. The momentum error recursions in Appendices B-E are self-contained, and the final rates are obtained by summing these recurrences. Self-citations to Jiang et al. 2024 occur only in the related-work discussion and assumption remarks, and none of the theorems invokes a prior result from the same authors as a load-bearing premise. The reviewer-identified algebra issue in Appendix D concerns whether the displayed O(d sqrt(n)) term is correctly computed; even if this is a proof defect, it is a correctness problem rather than circularity, because it does not make the theorem equivalent to its own assumptions. No circular step can be quoted.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No parameters are fitted to data. The learning rate eta and momentum beta are chosen as functions of T and d inside the proofs; they are algorithm hyperparameters, not free parameters used to match data. The unbiased sign operator is a known compression primitive, not a new postulated entity. Assumptions 2-9 are domain assumptions on the objective and stochastic oracle, and the unstated conditional-mean-zero noise property is standard but should be made explicit.

assumptions (6)
  • domain assumption Assumption 2: f is L-smooth, ||grad f(x) - grad f(y)|| <= L||x-y||
    Used in Theorems 1, 3, 4, and 5; standard in stochastic nonconvex optimization.
  • domain assumption Assumption 3: f is L-infinity smooth, ||grad f(x) - grad f(y)||_1 <= L_infinity ||x-y||_infinity
    Theorem 2's dimension-free rate rests on this. The paper claims it is weaker than Assumption 1, but Lemma 1's proof of that implication is invalid.
  • domain assumption Assumption 4: E||grad f(x;xi) - grad f(x)||^2 <= sigma^2
    Bounded variance used in Theorem 1.
  • domain assumption Assumption 5: coordinate-wise noise variance E[(n_i)^2] <= sigma_i^2
    Theorem 2's l1 analysis needs coordinate-level variance bounds; the proof also implicitly assumes noise terms at different times are uncorrelated.
  • domain assumption Assumptions 7-9: per-node smoothness, bounded noise, and pathwise uniform bound ||grad f_j(x;xi)||_infinity <= G with known G
    Distributed Theorems 3-5 rely on a known G so the unbiased sign operator S_G is valid; if gradients exceed G, E[S_G(v)] = v/G fails.
  • standard math Noise terms n_t are conditionally mean-zero given history, so E[n_s n_t] = 0 for s < t
    Used in Appendix C to cancel cross terms in the momentum error bound; follows from independent sampling but is not stated as an assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improved Analysis for Sign-based Methods with Momentum Updates." pith.science (2026). https://pith.science/paper/644ZUBDV

@misc{pith2026250712091,
  author       = {Pith},
  title        = {Pith review of: Improved Analysis for Sign-based Methods with Momentum Updates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/644ZUBDV}},
  note         = {Machine review of arXiv:2507.12091}
}
abstract

In this paper, we present enhanced analysis for sign-based optimization algorithms with momentum updates. Traditional sign-based methods, under the separable smoothness assumption, guarantee a convergence rate of $\mathcal{O}(T^{-1/4})$, but they either require large batch sizes or assume unimodal symmetric stochastic noise. To address these limitations, we demonstrate that signSGD with momentum can achieve the same convergence rate using constant batch sizes without additional assumptions. Our analysis, under the standard $l_2$-smoothness condition, improves upon the result of the prior momentum-based signSGD method by a factor of $\mathcal{O}(d^{1/2})$, where $d$ is the problem dimension. Furthermore, we explore sign-based methods with majority vote in distributed settings and show that the proposed momentum-based method yields convergence rates of $\mathcal{O}\left( d^{1/2}T^{-1/2} + dn^{-1/2} \right)$ and $\mathcal{O}\left( \max \{ d^{1/4}T^{-1/4}, d^{1/10}T^{-1/5} \} \right)$, which outperform the previous results of $\mathcal{O}\left( dT^{-1/4} + dn^{-1/2} \right)$ and $\mathcal{O}\left( d^{3/8}T^{-1/8} \right)$, respectively. Numerical experiments further validate the effectiveness of the proposed methods.

Figures

Figures reproduced from arXiv: 2507.12091 by the authors.

Figure 1
Figure 1. Results for CIFAR-10 dataset in the centralized environment. [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Results for CIFAR-100 dataset in the distributed environment. [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 30 canonical work pages

  1. [1]

    Arjevani, Y

    Y. Arjevani, Y. Carmon, J. C. Duchi, D. J. Foster, N. Srebro, and B. E. Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming, 199 0 (1-2): 0 165–214, 2023

    work page 2023

  2. [2]

    Balles, F

    L. Balles, F. Pedregosa, and N. L. Roux. The geometry of sign gradient descent. ArXiv e-prints, arXiv:2002.08056, 2020

    arXiv 2002

  3. [3]

    Bernstein, Y.-X

    J. Bernstein, Y.-X. Wang, K. Azizzadenesheli, and A. Anandkumar. sign SGD : Compressed optimisation for non-convex problems. In Proceedings of the 35th International Conference on Machine Learning, pages 560--569, 2018

    work page 2018

  4. [4]

    Bernstein, J

    J. Bernstein, J. Zhao, K. Azizzadenesheli, and A. Anandkumar. sign SGD with majority vote is communication efficient and fault tolerant. In International Conference on Learning Representations, 2019

    work page 2019

  5. [5]

    X. Chen, T. Chen, H. Sun, S. Z. Wu, and M. Hong. Distributed training with heterogeneous data: Bridging median- and mean-based algorithms. In Advances in Neural Information Processing Systems 33, pages 21616--21626, 2020

    work page 2020

  6. [6]

    Chzhen and S

    E. Chzhen and S. Schechtman. Sign SVRG : fixing SignSGD via variance reduction. ArXiv e-prints, arXiv:2305.13187, 2023

    arXiv 2023

  7. [7]

    Crawshaw, M

    M. Crawshaw, M. Liu, F. Orabona, W. Zhang, and Z. Zhuang. Robustness to unbounded smoothness of generalized signsgd. In Advances in Neural Information Processing Systems, volume 35, pages 9955--9968, 2022

    work page 2022

  8. [8]

    Cutkosky and F

    A. Cutkosky and F. Orabona. Momentum-based variance reduction in non-convex SGD . In Advances in Neural Information Processing Systems 32, pages 15210--15219, 2019

    work page 2019

Show all 33 references
  1. [9]

    Duchi, E

    J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12 0 (61): 0 2121--2159, 2011

    2011

  2. [10]

    C. Fang, C. J. Li, Z. Lin, and T. Zhang. SPIDER : Near-optimal non-convex optimization via stochastic path-integrated differential estimator. In Advances in Neural Information Processing Systems 31, pages 689--699, 2018

    2018

  3. [11]

    Ghadimi and G

    S. Ghadimi and G. Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23 0 (4): 0 2341--2368, 2013

    2013

  4. [12]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770--778, 2016

    2016

  5. [13]

    K. He, Z. Qu, and X. Li. On linear convergence of adaptive sign-based gradient descent. In 2024 36th Chinese Control and Decision Conference, pages 449--454. IEEE, 2024

    2024

  6. [14]

    Jiang, S

    W. Jiang, S. Yang, W. Yang, and L. Zhang. Efficient sign-based optimization: Accelerating convergence via variance reduction. In Advances in Neural Information Processing Systems 37, pages 33891--33932, 2024

    2024

  7. [15]

    R. Jin, Y. Huang, X. He, H. Dai, and T. Wu. Stochastic- S ign SGD for federated learning with theoretical guarantees. ArXiv e-prints, arXiv:2002.10940, 2021

    2002 arXiv

  8. [16]

    S. P. Karimireddy, Q. Rebjock, S. Stich, and M. Jaggi. Error feedback fixes S ign SGD and other gradient compression schemes. In Proceedings of the 36th International Conference on Machine Learning, pages 3252--3261, 2019

    2019

  9. [17]

    D. P. Kingma and J. L. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015

    2015

  10. [18]

    Krizhevsky

    A. Krizhevsky. Learning multiple layers of features from tiny images. Masters Thesis, Deptartment of Computer Science, University of Toronto, 2009

    2009

  11. [19]

    Kunstner, J

    F. Kunstner, J. Chen, J. W. Lavington, and M. Schmidt. Noise is not the main factor behind the gap between sgd and adam on transformers, but sign descent might be. In International Conference on Learning Representations, 2023

    2023

  12. [20]

    Li, K.-Y

    X. Li, K.-Y. Lin, L. Li, Y. Hong, and J. Chen. On faster convergence of scaled sign gradient descent. IEEE Transactions on Industrial Informatics, 20 0 (2): 0 1732--1741, 2023

    2023

  13. [21]

    Liu, P.-Y

    S. Liu, P.-Y. Chen, X. Chen, and M. Hong. sign SGD via zeroth-order oracle. In International Conference on Learning Representations, 2019

    2019

  14. [22]

    Y. Liu, R. Pan, and T. Zhang. Adagrad under anisotropic smoothness. In International Conference on Learning Representations, 2025

    2025

  15. [23]

    Loshchilov and F

    I. Loshchilov and F. Hutter. SGDR : Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017

    2017

  16. [24]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019

    2019

  17. [25]

    L. M. Nguyen, J. Liu, K. Scheinberg, and M. Tak a c . SARAH: A novel method for machine learning problems using stochastic recursive gradient. In Proceedings of the 34th International Conference on Machine Learning, pages 2613--2621, 2017

    2017

  18. [26]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-per...

    2019

  19. [27]

    Z. Qin, Z. Liu, and P. Xu. Convergence of sign-based random reshuffling algorithms for nonconvex optimization. ArXiv e-prints, arXiv:2310.15976, 2023

    2023

  20. [28]

    Safaryan and P

    M. Safaryan and P. Richtarik. Stochastic sign descent methods: New algorithms and better theory. In Proceedings of the 38th International Conference on Machine Learning, pages 9224--9234, 2021

    2021

  21. [29]

    T. Sun, Q. Wang, D. Li, and B. Wang. Momentum ensures convergence of SIGNSGD under weaker assumptions. In Proceedings of the 40th International Conference on Machine Learning, pages 33077--33099, 2023

    2023

  22. [30]

    Sutskever, J

    I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In Proceedings of the 30th International Conference on Machine Learning, pages 1139--1147, 2013

    2013

  23. [31]

    Z. Wang, K. Ji, Y. Zhou, Y. Liang, and V. Tarokh. SpiderBoost and momentum: Faster variance reduction algorithms. In Advances in Neural Information Processing Systems 32, pages 2406--2416, 2019

    2019

  24. [32]

    Zhang, S

    J. Zhang, S. P. Karimireddy, A. Veit, S. Kim, S. Reddi, S. Kumar, and S. Sra. Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems 33, pages 15383--15393, 2020

    2020

  25. [33]

    Zhang, M

    L. Zhang, M. Mahdavi, and R. Jin. Linear convergence with condition number independent access of full gradients. In Advances in Neural Information Processing Systems 26, pages 980--988, 2013

    2013

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.