Pith. sign in

REVIEW 3 major objections 4 minor 2 cited by

A decentralized version of the Muon matrix optimizer converges as fast as centralized methods under heavy-tailed noise.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 12:56 UTC pith:5PMEUGTM

arxiv 2510.01377 v2 pith:5PMEUGTM submitted 2025-10-01 math.OC cs.AIcs.LGcs.MAcs.SYeess.SY

DeMuon: A Decentralized Muon for Matrix Optimization over Graphs

Chuan He , Shuyi Ren , Jingwei Mao , Erik G. Larsson This is my paper
classification math.OC cs.AIcs.LGcs.MAcs.SYeess.SY MSC 90C2690C1565K05
keywords decentralized optimizationmatrix optimizationMuon optimizergradient trackingheavy-tailed noisenuclear norm stationarityNewton-Schulz iterationstransformer pretraining
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

DeMuon is a decentralized algorithm for minimizing a finite sum of matrix-valued functions over a network of nodes that can only exchange messages with neighbors. The paper's central claim is that this peer-to-peer setup does not cost anything in theory: under heavy-tailed gradient noise with a finite α-th moment (1 < α ≤ 2), DeMuon reaches an ε-small nuclear-norm gradient in O(ε^(−(3α−2)/(α−1))) iterations, the same dependence on ε as the best-known centralized stochastic algorithms. The algorithm keeps Muon's core idea—replacing each gradient matrix by its matrix sign (U Vᵀ from a reduced SVD)—and adds a momentum buffer and gradient tracking so local estimates converge to the global gradient. The proof works through a potential function that monitors function value, momentum error, and consensus error simultaneously. The experiments on decentralized transformer pretraining indicate faster or comparable loss reduction than existing decentralized baselines on complete, directed exponential, and ring graphs.

Core claim

DeMuon's central claim is that matrix orthogonalization is compatible with decentralized consensus and gradient tracking. The update for each local iterate is X_i^{k+1} = Σ_j w_ij (X_j^k − η msgn(V_j^k)), where V_j^k is a tracked estimate of the global gradient and msgn is the reduced-SVD matrix sign. Under Assumption 1—Lipschitz smooth local objectives, heavy-tailed noise with E‖G − ∇f‖^α ≤ σ^α, and a doubly stochastic mixing matrix with mixing rate λ—Theorem 1 bounds the average nuclear norm of the stacked gradient by ε after O(ε^(−(3α−2)/(α−1))) iterations, with step sizes η = K^(−(2α−1)/(3α−2)) and θ = K^(−α/(3α−2)). This matches the best-known centralized heavy-tailed complexity, and th

What carries the argument

The load-bearing components are (i) the matrix sign msgn(M)=U Vᵀ obtained from the reduced SVD, which normalizes each gradient matrix to spectral norm one and gives Muon-type updates; (ii) a momentum estimator M_i^k plus a gradient-tracking variable V_i^k that is mixed across neighbors and keeps local gradients close to the global gradient; (iii) a doubly stochastic mixing matrix W whose mixing rate λ < 1 controls consensus; and (iv) the potential function P^k = f(X̄^k) + p‖∇F − M^k‖_F^α + q‖V^k − 1⊗V̄^k‖_* that couples the three sources of error and yields the descent recurrence in Lemma 7.

Load-bearing premise

The proof's descent and consensus inequalities rest on the exact matrix sign msgn(V) from a reduced SVD having spectral norm exactly 1; the abstract promises Newton–Schulz iterations, but no error bound for a finite Newton–Schulz approximation is given, so the stated guarantee does not cover the algorithm as implemented with approximate orthogonalization.

What would settle it

Fix a small budget of Newton–Schulz iterations (e.g., 1 or 2) in place of the exact reduced-SVD sign on a matrix with rapidly decaying singular values, run DeMuon on a simple decentralized quadratic or logistic problem, and measure the spectral norm of the approximate sign and the consensus error at each round. If the approximate sign exceeds 1 often enough to break the inequality ‖msgn(V)‖ ≤ 1 used in Lemmas 1 and 4, the O(ε^(−(3α−2)/(α−1))) guarantee cannot be extrapolated to that implementation; one should observe the gap between the predicted and actual convergence rates.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Decentralized training with Muon-style matrix orthogonalization has the same ε-dependence as centralized normalized methods under heavy-tailed noise, so decentralization does not worsen the rate.
  • The spectral-norm consensus bound O(λη/(1−λ)) is tighter than Frobenius-norm bounds used for vector-variate decentralized methods.
  • The explicit step-size choices η ~ K^(−(2α−1)/(3α−2)), θ ~ K^(−α/(3α−2)) give a concrete recipe for running DeMuon without tuning.
  • On transformer pretraining over graphs, DeMuon reduces validation loss faster than DSGD and DSGD Clip on complete, directed exponential, and ring topologies, and stays close to the stronger GT-NSGDm baseline.
  • The theorem supplies the first convergence guarantee for a direct decentralized extension of Muon, filling a gap between Muon's empirical success and decentralized optimization theory.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The abstract advertises Newton–Schulz iterations for orthogonalization, but the algorithm and proofs use exact reduced-SVD matrix signs. A finite Newton–Schulz approximation changes the spectral norm of the update, so a practical implementation needs an error bound to inherit Theorem 1; that bound is not given.
  • Since the complexity depends on ε only through the same exponent as centralized heavy-tailed methods, one can conjecture that the same technique would extend to other matrix-variate updates, such as low-rank or polar-gradient variants, in decentralized settings.
  • A testable extension is to replace exact SVD with a fixed number of Newton–Schulz steps on ill-conditioned matrices and measure the resulting consensus error; if the approximate sign exceeds spectral norm 1, the descent inequality in Lemma 4 begins to fail.
  • The proof's parameter choices depend on problem constants through U_dm; deriving adaptive or parameter-free step sizes is an open direction suggested by the paper's own closing remark.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes DeMuon, a decentralized matrix optimization algorithm that combines momentum, gradient tracking, and matrix orthogonalization via the matrix sign of local gradient estimates. Under heavy-tailed noise with finite α-th moment (α∈(1,2]), it claims an O(ε^{-(3α-2)/(α-1)}) iteration complexity for reaching an ε-nuclear-norm stochastic stationary point, matching the best-known centralized complexity. The proof uses a potential function built from the objective, a momentum-error term, and a consensus-error term. Numerical experiments on a 3M-parameter GPT model compare DeMuon with DSGD, DSGD Clip, and GT NSGDm over complete, directed exponential, and ring graphs.

Significance. If the main theorem is correct, this is a useful contribution: it appears to be the first decentralized extension of Muon with a provable complexity guarantee, and it addresses the realistic heavy-tailed noise regime. The proof is systematic and the rate calculation is consistent with the stated potential construction; the paper also gives explicit constants and does not introduce ad-hoc parameters. However, the proof has a load-bearing initialization error that invalidates Theorem 1 as stated, and the advertised Newton–Schulz implementation is not the algorithm analyzed. These issues are likely repairable without changing the rate, but they must be fixed before the claims can be accepted.

major comments (3)
  1. [Section IV, proof of Theorem 1, Eq. (28)] The proof uses the initial bound E∥M^0_N − ∇F(X^0_N)∥_F^α ≤ E∥G(X^0_N;ξ^0)−∇F(X^0_N)∥_*^α ≤ (Nσ)^α. But Algorithm 1 initializes M^{-1}_i=0, so update (5) gives M^0_i = θ G_i(X^0_i;ξ^0_i), not G_i(X^0_i;ξ^0_i). The claimed inequality is therefore false in general. For a one-node quadratic f(x)=½∥x∥² with σ=0, it would assert E∥θ∇f(x^0)−∇f(x^0)∥^α = (1−θ)^α∥∇f(x^0)∥^α ≤ 0 for x^0≠0. The correct bound contains an additional (1−θ)^α∥∇F(X^0_N)∥_*^α term. After dividing by Kη this contributes a term with the same K^{-(α−1)/(3α−2)} dependence as the claimed rate, so the rate may survive after enlarging U_dm, but Theorem 1 as stated with the displayed U_dm does not follow from the proof.
  2. [Abstract and Algorithm 1] The abstract and introduction state that DeMuon incorporates matrix orthogonalization via Newton–Schulz iterations, but Algorithm 1 defines msgn(M) by the reduced SVD of M, and Lemmas 1 and 4 rely on the exact unit spectral-norm bound ∥msgn(V_i)∥≤1. No error bound is given for a finite Newton–Schulz approximation. If a practical implementation uses a few Newton–Schulz steps, the consensus and descent inequalities in the proof no longer hold exactly. The authors should either analyze an inexact sign oracle with an explicit perturbation bound, or change the abstract and intro to state that the theoretical guarantees are for the exact SVD-based orthogonalization.
  3. [Section IV, Lemma 5 and Theorem 1] The statement of Lemma 5, Eq. (19), omits the factor N on the noise term: the proof derives E∥V^{k+1}_N−1_N⊗V^{k+1}∥_* ≤ λ∥V^k_N−1_N⊗V^k∥_* + (λθ/(1−θ)) E∥∇F−M∥_* + Nσλθ/(1−θ), but the displayed statement has σ instead of Nσ. This error propagates into the initial bound for E∥V^0_N−1_N⊗V^0∥_* in the proof of Theorem 1, where the final constant becomes (N+1)λσθ/(1−θ) instead of the N-dependent expression implied by the proof. This affects the exact constant U_dm but not the iteration rate; it should be corrected.
minor comments (4)
  1. [Assumption 1(d)] The primitivity condition is written as 'W j >0 for some positive integer j'; it should read 'W^j > 0' componentwise for some positive integer j.
  2. [Section I-C] The organization paragraph says 'Sections III and IV present the simulation results and the proof', but the main algorithm and results are in Section II. This is slightly misleading and should be reworded.
  3. [Section III, numerical experiments] The experiments report only training/validation loss curves without error bars, seed information, or a description of whether the orthogonalization in DeMuon is computed by exact SVD or by Newton–Schulz. Since the theory concerns the exact SVD algorithm, the experimental setup should state which operation is used.
  4. [Remark 2] The remark says the spectral-norm consensus error is 'always bounded above by the Frobenius norm' and therefore tighter than the Frobenius-norm bound. This is true but would benefit from explicitly saying that a bound on the spectral norm implies a stronger statement, since ∥·∥ ≤ ∥·∥_F.

Circularity Check

0 steps flagged

No circularity: Theorem 1 is derived by telescoping an explicit potential; self-citations are ancillary.

full rationale

DeMuon's claimed complexity is not equivalent to an input by construction. The proof chain (Lemmas 1-7, Theorem 1) starts from Assumption 1, defines the potential P_k in (9), and obtains Lemma 7's one-step descent by substituting the recurrence bounds (16), (19), (21)-(22); the final rate follows by summing over K, choosing p,q in (27) and eta,theta in Theorem 1, and absorbing all constants into U_dm in (8). No parameter is fitted to data and no quantity is renamed as a 'prediction' after being used as an input. Self-citations [15] and [38] supply only a standard inequality (Lemma 2, stated in full) and the centralized complexity baseline used for comparison; neither carries the derivation, and Lemma 2 is parameter-free and independently checkable. Two correctness concerns were examined and are not circularity: (i) the abstract advertises Newton-Schulz orthogonalization while Algorithm 1 and the proof use the exact reduced-SVD sign msgn(.), so no finite-iteration Newton-Schulz error bound is proved; and (ii) in the proof of Theorem 1, the bound on E[||M^0 - grad F(X^0)||^alpha] replaces M^0 = theta G(X^0; xi^0) with G(X^0; xi^0), which is false for theta<1 and would require an extra (1-theta)^alpha ||grad F(X^0)||^alpha term in U_dm. These are technical gaps with the same rate dependence; they do not make the claimed rate equal by construction to its assumptions.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters are fitted; step sizes/momentum are chosen analytically from K and problem constants. The central proof relies on the exact polar-factor sign operator, heavy-tailed noise moment bounds, and standard mixing-matrix assumptions.

axioms (4)
  • domain assumption Assumption 1(a)-(d): lower bounded objective, L*-smoothness in nuclear/spectral norms, alpha-th moment noise bound, and primitive doubly stochastic mixing matrix.
    These are the standard smoothness, noise, and consensus assumptions invoked throughout the proof (Section I-B).
  • domain assumption Exact matrix sign (msgn) computation via reduced SVD in the update (7).
    Lemma 1 and Lemma 4 rely on the exact unit spectral norm of msgn(V_i); a finite Newton-Schulz approximation would not satisfy this exactly.
  • domain assumption Equal initialization X_i^0 = X_j^0 for all i,j.
    Used in Lemma 1 to get the consensus bound with zero initial error; unequal initialization would add a term λ^k to the bound.
  • standard math Lemma 2 (Young-type norm inequality) from [38].
    Used in Lemma 6 to bound the alpha-th power of the momentum error; cited from the authors' prior work.

pith-pipeline@v1.3.0-alltime-deepseek · 15710 in / 32272 out tokens · 222152 ms · 2026-08-04T12:56:27.979536+00:00 · methodology

0 comments
read the original abstract

In this paper, we propose DeMuon, a method for decentralized matrix optimization over a given communication topology. DeMuon incorporates matrix orthogonalization via Newton-Schulz iterations-a technique inherited from its centralized predecessor, Muon-and employs gradient tracking to mitigate heterogeneity among local functions. Under heavy-tailed noise conditions and additional mild assumptions, we establish the iteration complexity of DeMuon for reaching an approximate stochastic stationary point. This complexity result matches the best-known complexity bounds of centralized algorithms in terms of dependence on the target tolerance. To the best of our knowledge, DeMuon is the first direct extension of Muon to decentralized optimization over graphs with provable complexity guarantees. We conduct preliminary numerical experiments on decentralized transformer pretraining over graphs with varying degrees of connectivity. Our numerical results demonstrate a clear margin of improvement of DeMuon over other popular decentralized algorithms across different network topologies.

Figures

Figures reproduced from arXiv: 2510.01377 by Chuan He, Erik G. Larsson, Jingwei Mao, Shuyi Ren.

Figure 1
Figure 1. Figure 1: Training losses in decentralized training of Transformer models over complete, directed exponential, and ring graphs. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Validation losses in decentralized training of Transformer models over complete, directed exponential, and ring graphs. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SUDA-Muon: Structural Design Principles and Boundaries for Fully Decentralized Muon

    math.OC 2026-04 unverdicted novelty 6.0

    SUDA-Muon modularizes decentralized Muon via the SUDA template, proving a topology-separated convergence rate of O((1+σ/√N)K^{-1/4}) in nuclear-norm geometry while establishing that tracking-before-polarization is req...

  2. Non-Euclidean SGD for Structured Optimization: Unified Analysis and Improved Rates

    math.OC 2025-11 conditional novelty 5.0

    Non-Euclidean SGD variants (SignSGD, Muon) provably match adaptive optimizers' convergence rates under structured smoothness and noise assumptions.

Reference graph

Works this paper leans on

41 extracted references · 13 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Muon: An optimizer for hidden layers in neural networks,

    K. Jordan, Y . Jin, V . Boza, J. You, F. Cesista, L. Newhouse, and J. Bernstein, “Muon: An optimizer for hidden layers in neural networks,” 2024. [Online]. Available: https://kellerjordan.github.io/posts/muon/

    2024

  2. [2]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” inInternational Conference for Learning Representations, May 2015

    2015

  3. [3]

    ASGO: Adaptive structured gradient optimization,

    K. An, Y . Liu, R. Pan, S. Ma, D. Goldfarb, and T. Zhang, “ASGO: Adaptive structured gradient optimization,”arXiv preprint arXiv:2503.20762, Mar. 2025

    arXiv 2025

  4. [4]

    Muon optimizes under spectral norm constraints,

    L. Chen, J. Li, and Q. Liu, “Muon optimizes under spectral norm constraints,”arXiv preprint arXiv:2506.15054, Jun. 2025

    arXiv 2025

  5. [5]

    Understanding gradient orthogonalization for deep learning via non-Euclidean trust-region optimization,

    D. Kovalev, “Understanding gradient orthogonalization for deep learning via non-Euclidean trust-region optimization,”arXiv preprint arXiv:2503.12645, Apr. 2025

    Pith/arXiv arXiv 2025

  6. [6]

    A note on the convergence of Muon and further,

    J. Li and M. Hong, “A note on the convergence of Muon and further,”arXiv preprint arXiv:2502.02900, pp. arXiv–2502, Jun. 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Gluon: Making Muon & Scion great again! (bridging theory and practice of LMO-based optimizers for LLMs),

    A. Riabinin, E. Shulgin, K. Gruntkowska, and P. Richt ´arik, “Gluon: Making Muon & Scion great again! (bridging theory and practice of LMO-based optimizers for LLMs),”arXiv preprint arXiv:2505.13416, May 2025

    Pith/arXiv arXiv 2025

  8. [8]

    Analysis of Muon’s convergence and critical batch size,

    N. Sato, H. Naganuma, and H. Iiduka, “Analysis of Muon’s convergence and critical batch size,”arXiv preprint arXiv:2507.01598, Aug. 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Lions and Muons: Optimization via stochastic Frank-Wolfe,

    M.-E. Sfyraki and J.-K. Wang, “Lions and Muons: Optimization via stochastic Frank-Wolfe,”arXiv preprint arXiv:2506.04192, Jun. 2025

    Pith/arXiv arXiv 2025

  10. [10]

    On the convergence analysis of Muon,

    W. Shen, R. Huang, M. Huang, C. Shen, and J. Zhang, “On the convergence analysis of Muon,”arXiv preprint arXiv:2505.23737, May 2025

    Pith/arXiv arXiv 2025

  11. [11]

    SWAN: Preprocessing SGD enables Adam-level performance on LLM training with significant memory reduction,

    C. Ma, W. Gong, M. Scetbon, and E. Meeds, “SWAN: Preprocessing SGD enables Adam-level performance on LLM training with significant memory reduction,”arXiv e-prints, pp. arXiv–2412, Feb. 2024

    2024

  12. [12]

    Training deep learning models with norm-constrained LMOs,

    T. Pethick, W. Xie, K. Antonakopoulos, Z. Zhu, A. Silveti-Falls, and V . Cevher, “Training deep learning models with norm-constrained LMOs,”arXiv preprint arXiv:2502.07529, Jun. 2025

    Pith/arXiv arXiv 2025

  13. [13]

    PolarGrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective,

    T. T.-K. Lau, Q. Long, and W. Su, “PolarGrad: A class of matrix-gradient optimizers from a unifying preconditioning perspective,”arXiv preprint arXiv:2505.21799, Aug. 2025

    arXiv 2025

  14. [14]

    Dion: Distributed orthonormalized updates,

    K. Ahn, B. Xu, N. Abreu, and J. Langford, “Dion: Distributed orthonormalized updates,”arXiv preprint arXiv:2504.05295, Apr. 2025

    arXiv 2025

  15. [15]

    Low-rank orthogonalization for large-scale matrix optimization with applications to foundation model training,

    C. He, Z. Deng, and Z. Lu, “Low-rank orthogonalization for large-scale matrix optimization with applications to foundation model training,”arXiv preprint arXiv:2509.11983, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    A survey of distributed optimization,

    T. Yang, X. Yi, J. Wu, Y . Yuan, D. Wu, Z. Meng, Y . Hong, H. Wang, Z. Lin, and K. H. Johansson, “A survey of distributed optimization,”Annual Reviews in Control, vol. 47, pp. 278–305, Jan. 2019

    2019

  17. [17]

    Multi-agent optimization,

    A. Nedi ´c, J.-S. Pang, G. Scutari, and Y . Sun, “Multi-agent optimization,”Cham, Switzerland: Springer, 2018

    2018

  18. [18]

    Decentralized training of foundation models in heterogeneous environments,

    B. Yuan, Y . He, J. Davis, T. Zhang, T. Dao, B. Chen, P. S. Liang, C. Re, and C. Zhang, “Decentralized training of foundation models in heterogeneous environments,” inAdvances in Neural Information Processing Systems, vol. 35, Dec. 2022

    2022

  19. [19]

    d2: Decentralized training over decentralized data,

    H. Tang, X. Lian, M. Yan, C. Zhang, and J. Liu, “ d2: Decentralized training over decentralized data,” inInternational Conference on Machine Learning. PMLR, Jul. 2018

    2018

  20. [20]

    Large-scale matrix factorization with distributed stochastic gradient descent,

    R. Gemulla, E. Nijkamp, P. J. Haas, and Y . Sismanis, “Large-scale matrix factorization with distributed stochastic gradient descent,” inThe 17th ACM SIGKDD international conference on Knowledge discovery and data mining, Aug. 2011

    2011

  21. [21]

    Deepca: Decentralized exact pca with linear convergence rate,

    H. Ye and T. Zhang, “Deepca: Decentralized exact pca with linear convergence rate,”Journal of Machine Learning Research, vol. 22, no. 238, pp. 1–27, Feb. 2021

    2021

  22. [22]

    Matrix completion on graphs,

    V . Kalofolias, X. Bresson, M. Bronstein, and P. Vandergheynst, “Matrix completion on graphs,”arXiv preprint arXiv:1408.1717, Aug. 2014

    Pith/arXiv arXiv 2014

  23. [23]

    Decentralized low-rank matrix completion,

    Q. Ling, Y . Xu, W. Yin, and Z. Wen, “Decentralized low-rank matrix completion,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Aug. 2012

    2012

  24. [24]

    Decentralized and privacy-preserving low-rank matrix completion,

    A.-Y . Lin and Q. Ling, “Decentralized and privacy-preserving low-rank matrix completion,”Journal of the Operations Research Society of China, vol. 3, no. 2, pp. 189–205, Jun. 2015

    2015

  25. [25]

    Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent,

    X. Lian, C. Zhang, H. Zhang, C.-J. Hsieh, W. Zhang, and J. Liu, “Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent,” inAdvances in Neural Information Processing Systems, vol. 30, Dec. 2017

    2017

  26. [26]

    A unified theory of decentralized SGD with changing topology and local updates,

    A. Koloskova, N. Loizou, S. Boreiri, M. Jaggi, and S. Stich, “A unified theory of decentralized SGD with changing topology and local updates,” inInternational Conference on Machine Learning. PMLR, Nov. 2020

    2020

  27. [27]

    Extra: An exact first-order algorithm for decentralized consensus optimization,

    W. Shi, Q. Ling, G. Wu, and W. Yin, “Extra: An exact first-order algorithm for decentralized consensus optimization,” SIAM Journal on Optimization, vol. 25, no. 2, pp. 944–966, Nov. 2015

    2015

  28. [28]

    Distributed stochastic gradient tracking methods,

    S. Pu and A. Nedi ´c, “Distributed stochastic gradient tracking methods,”Mathematical Programming, vol. 187, no. 1, pp. 409–457, May 2021

    2021

  29. [29]

    High probability convergence of distributed clipped stochastic gradient descent with heavy-tailed noise,

    Y . Yang, K. Lu, and L. Wang, “High probability convergence of distributed clipped stochastic gradient descent with heavy-tailed noise,”arXiv preprint arXiv:2506.11647, Jun. 2025. 15

    Pith/arXiv arXiv 2025

  30. [30]

    High- probability convergence for composite and distributed stochastic minimization and variational inequalities with heavy-tailed noise,

    E. Gorbunov, A. Sadiev, M. Danilova, S. Horv ´ath, G. Gidel, P. Dvurechensky, A. Gasnikov, and P. Richt ´arik, “High- probability convergence for composite and distributed stochastic minimization and variational inequalities with heavy-tailed noise,” inInternational Conference on Machine Learning, Jul. 2024

    2024

  31. [31]

    High probability convergence of clipped distributed dual averaging with heavy-tailed noises,

    Y . Qin, K. Lu, H. Xu, and X. Chen, “High probability convergence of clipped distributed dual averaging with heavy-tailed noises,”IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 55, no. 4, pp. 2624–2632, Jan. 2025

    2025

  32. [32]

    Decentralized nonconvex optimization under heavy-tailed noise: Normalization and optimal convergence,

    S. Yu, D. Jakovetic, and S. Kar, “Decentralized nonconvex optimization under heavy-tailed noise: Normalization and optimal convergence,”arXiv preprint arXiv:2505.03736, May 2025

    Pith/arXiv arXiv 2025

  33. [33]

    Sign operator for coping with heavy-tailed noise: High probability convergence bounds with extensions to distributed optimization and comparison oracle,

    N. M. Kornilov, P. Zmushko, A. Semenov, A. Gasnikov, and A. Beznosikov, “Sign operator for coping with heavy-tailed noise: High probability convergence bounds with extensions to distributed optimization and comparison oracle,”arXiv preprint arXiv:2502.07923, May 2025

    Pith/arXiv arXiv 2025

  34. [34]

    Efficient distributed optimization under heavy-tailed noise,

    S. H. Lee, M. Zaheer, and T. Li, “Efficient distributed optimization under heavy-tailed noise,”arXiv preprint arXiv:2502.04164, Aug. 2025

    Pith/arXiv arXiv 2025

  35. [35]

    Fast linear iterations for distributed averaging,

    L. Xiao and S. Boyd, “Fast linear iterations for distributed averaging,”Systems & Control Letters, vol. 53, no. 1, pp. 65–78, Sep. 2004

    2004

  36. [36]

    NEXT: In-network nonconvex optimization,

    P. Di Lorenzo and G. Scutari, “NEXT: In-network nonconvex optimization,”IEEE Transactions on Signal and Information Processing over Networks, vol. 2, no. 2, pp. 120–136, Jun. 2016

    2016

  37. [37]

    On the convergence analysis of muon,

    W. Shen, R. Huang, M. Huang, C. Shen, and J. Zhang, “On the convergence analysis of muon,” 2025

    2025

  38. [38]

    Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise,

    C. He, Z. Lu, D. Sun, and Z. Deng, “Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise,”arXiv preprint arXiv:2506.11214, Jun. 2025

    arXiv 2025

  39. [39]

    Improving language understanding by generative pre-training,

    A. Radford, K. Narasimhan, T. Salimans, I. Sutskeveret al., “Improving language understanding by generative pre-training,” Open AI, 2018

    2018

  40. [40]

    Distributed subgradient methods for multi-agent optimization,

    A. Nedic and A. Ozdaglar, “Distributed subgradient methods for multi-agent optimization,”IEEE Transactions on Automatic Control, vol. 54, no. 1, pp. 48–61, Jan. 2009

    2009

  41. [41]

    Distributed stochastic strongly convex optimization under heavy-tailed noises,

    C. Sun and B. Chen, “Distributed stochastic strongly convex optimization under heavy-tailed noises,” in2024 IEEE International Conference on Cybernetics and Intelligent Systems (CIS) and IEEE International Conference on Robotics, Automation and Mechatronics (RAM), Aug. 2024

    2024