Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

High Probability Convergence of Distributed Clipped Stochastic Gradient Descent with Heavy-tailed Noise

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper proves that a distributed clipped stochastic gradient descent algorithm converges, with probability at least 1-\delta, for convex smooth distributed optimization when each agent's gradient noise is heavy-tailed with a bounded…

desk verdict Plausible high-probability convergence rate for distributed clipped SGD, but Theorem 1's proof misses a union bound and a step to handle the z_t weight. read the letter →

arxiv 2506.11647 v2 pith:2ASBVTSV submitted 2025-06-13 math.OC

Yuchen Yang , Kaihong Lu , Long Wang This is my paper
classification math.OC MSC 90C2590C15
keywords distributedoptimizationstochasticgradientdescentheavy-tailednoiseclippinghigh-probabilityconvergencetime-varyinggraphsconvexconsensus
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Distributed optimization typically assumes light-tailed gradient noise to get high-probability guarantees; this paper removes that restriction for convex smooth problems. The authors study a network of agents where each agent sees an unbiased gradient corrupted by heavy-tailed noise with a bounded p-th moment, p in (1,2], and can receive messages through a time-varying B-strongly connected directed graph. Their distributed clipped SGD algorithm clips each noisy gradient at level \lambda_t before the consensus step, and they prove that with probability at least 1-\delta the cumulative suboptimality satisfies \sum_{t=1}^T (f(\bar{x}_t)-f(x^*)) \le \frac{9}{2\eta_T}\$Delta_1^{2}$ + \frac{1}{\eta_T}\$log^{2}$(1/\delta), with \bar{x}_t the average agent state and \Delta_1 its initial distance to the optimum. Corollary 1 converts this into the rate O((N\$rho^{{5/3}}$+B^*\$rho^{{2/3}}$)$T^{{1/(2p)-1/2}}$\$log^{2}$(T/\delta)), attained without knowing the horizon T in advance. This matters because applications such as target tracking must run for a limited number of rounds, where expectation-based convergence is not enough.

What carries the argument

The argument rests on three pieces. The clipping operator \mathrm{clip}_{\$\lambda$}(y)=\min\{1,\$\lambda$/\|y\|\}y is applied to each agent's stochastic gradient, turning an unbounded-variance noise into one whose bad tail is truncated at \lambda_t. Lemma 5 (from [29]) splits the resulting error \theta_{i,t} into a martingale-difference part \$\theta$^u_{i,t}, bounded by 2\lambda_t, and a bias part \$\theta$^b_{i,t}; the exponential-martingale concentration inequality in Lemma 7 (from [32]) bounds the martingale part, while the p-th moment assumption bounds the bias. Lemma 1 (from [38]) supplies the consensus contraction \|[\Phi(k,s)]_{ij}-1/N\| \le \gamma\$beta^{{k-s}}$ for doubly stochastic B-strongly connected time-varying graphs, which controls the network error \|x_{i,t}-\bar{x}_t\|. The decreasing weight z_t = \frac{1}{a_0+4\max_{i\le t}\|\bar{x}_i-x^*\|} closes an induction that keeps the iterates within a tube around the optimum, and condition (7) ties \lambda_t, \eta_t, m, and the graph parameters together.

What would settle it

Fix a large number of agents N and a small \delta, run the algorithm many times with Student-t noise, and record how often the Theorem 1 inequality fails; if the proof really treated the N per-agent concentration events as a single 1-\delta event, the empirical failure probability should be near \delta, not near 1-(1-\delta)^N, so a rate of roughly N\delta would falsify the claimed confidence bound.

Watch

Extended reading notes

Core claim

The paper's central claim is that clipping makes heavy-tailed noise compatible with high-probability consensus-based optimization. For a convex L-smooth global objective f(x)=\frac{1}{N}\sum_{i=1}^N f_i(x), where each agent i observes an unbiased estimator of \nabla f_i with \mathbb{E}[\|\nabla f_i(x)-\nabla f_i(x)\|^p \mid x] \le \$\sigma$^p for p in (1,2], algorithm (3) with \eta_t = \frac{1}{m t^\kappa (\log t + b_1)^2}, \lambda_t = \$\lambda$ t^\$\alpha$, and parameters satisfying condition (7) satisfies Theorem 1: with probability at least 1-\delta, \sum_{t=1}^T (f(\bar{x}_t)-f(x^*)) \le \frac{9}{2\eta_T}\$Delta_1^{2}$ + \frac{1}{\eta_T}\$log^{2}$(1/\delta). The authors then choose \$\alpha$=\frac{1}{2p}, \kappa=\frac{1}{2}+\frac{1}{2p}, and obtain the high-probability rate of Corollary 1, O((N\$rho^{{5/3}}$+B^*\$rho^{{2/3}}$)$T^{{1/(2p)-1/2}}$\$log^{2}$(T/\delta)), where \rho=(1-\$\beta$)^{-1} and B^*=\max_i\|\nabla f_i(x^*)\|. The rate is stated without prior knowledge of the total iteration number T.

Load-bearing premise

The load-bearing premise is that the per-agent martingale tail events in Lemma 6 happen simultaneously with one probability 1-\delta; the proof as written bounds each agent's noise separately and then sums over agents without a union bound, so the stated global confidence needs an additional \delta/N argument.

Editorial extensions

If this is right

  • A single run of the algorithm over T iterations carries a 1-\delta certificate: the cumulative optimality gap is at most \frac{9}{2\eta_T}\Delta_1^2 + \frac{1}{\eta_T}\log^2(1/\delta), so repeated sampling is unnecessary.
  • The advertised rate O((N\rho^{5/3}+B^*\rho^{2/3})T^{1/(2p)-1/2}\log^2(T/\delta)) holds for diminishing step sizes and clipping thresholds, so the protocol needs no foreknowledge of the stopping time T.
  • The fastest high-probability rate occurs at \alpha=\frac{1}{2p} and \kappa=\frac{1}{2}+\frac{1}{2p}; when the noise has finite variance (p=2), the rate becomes O(T^{-1/4}\log^2(T/\delta)).
  • The guarantee applies to the average state \bar{x}_t; individual agents approach this average up to a network-error term controlled by the consensus contraction in Lemma 2.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: a clean way to make the per-agent concentration events global is to replace \delta by \delta/N in Lemma 6 and then union-bound over the N agents; this would multiply the stated rate by a \log N factor that Theorem 1 and Corollary 1 do not currently display.
  • Editorial inference: because only smoothness and the p-th moment bound are used, the same bias-variance split of the clipped gradient could plausibly carry over to nonconvex objectives or to gradient-tracking decentralized methods, although the paper only proves the convex smooth case.
  • Editorial inference: the empirical failure rate of the Theorem 1 bound under Student-t noise should be measured as a function of N; if the per-agent events were union-bounded, the failure rate would scale like \delta rather than like N\delta, giving a direct experimental check of the confidence statement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies distributed optimization over time-varying directed graphs when each agent observes only a heavy-tailed noisy gradient of its local function. It proposes a distributed clipped stochastic gradient descent algorithm (3), states Assumptions 1-4, and introduces a set of parameter conditions (7). The main theoretical claim, Theorem 1, asserts that with probability at least 1-delta the average regret satisfies sum_{t=1}^T (f(bar{x}_t)-f(x*)) <= (9/(2 eta_T)) Delta_1^2 + (1/eta_T) log^2(1/delta). Corollary 1 converts this into the rate O((N rho^{5/3} + B* rho^{2/3}) T^{1/(2p)-1/2} log^2(T/delta)). The proof is deferred to the appendix, and two numerical experiments on a9a and MNIST are reported. The central claim is an extension of high-probability clipped-SGD guarantees from the centralized heavy-tailed setting to the distributed setting without prior knowledge of T.

Significance. If the theorem were correct, the paper would make a useful contribution: it combines clipping for heavy-tailed noise with consensus over B-strongly connected time-varying graphs, and it claims a rate that does not require the iteration horizon T as prior knowledge. The algorithm is natural and the simulation results are encouraging. The paper also explicitly identifies the network-error term and proposes a condition on the step sizes and clipping parameters. However, the proof of the main theorem contains load-bearing gaps: the high-probability noise bound in Lemma 6 is per-agent and carries an extra weight z_t, while Theorem 1 needs an unweighted all-agent bound; no union bound over agents or over time is supplied. These are not merely cosmetic issues, because the final rates and confidence levels depend on how these factors are handled. As written, the central claim is not established, so the contribution remains conditional on a successful repair of the proof.

major comments (3)
  1. [Appendix 6.5 / Lemma 6] Lemma 6 is a per-agent statement: for a fixed i it bounds sum_{t=1}^n 2 z_t eta_t <theta_{i,t}, bar{x}_t - x*> by 5 Delta_1/4 + log(1/delta). In the proof of Theorem 1 the same lemma is used to control the all-agent, unweighted sum sum_{t=1}^T sum_{i=1}^N eta_t <theta_{i,t}, bar{x}_t - x*>. No union bound over i in [N] is given, so the stated global confidence 1-delta does not follow for all agents simultaneously. Repairing this requires replacing delta by delta/N, which introduces a log N term absent from Theorem 1 and Corollary 1.
  2. [Appendix 6.5 / Lemma 6] The proof also drops the factor z_t without justification. Lemma 6 controls the weighted sum sum 2 z_t eta_t <theta_{i,t}, bar{x}_t - x*>, but Theorem 1 needs the unweighted sum sum eta_t <theta_{i,t}, bar{x}_t - x*>. The ratio between the two is 1/z_t. Under the paper's own bound Delta_k <= 9 Delta_1 + 5 a_0, derived in the proof of Lemma 6, z_t can be as small as 1/(36 Delta_1 + 21 a_0), so the unweighted sum can exceed the controlled weighted sum by a factor as large as 36 Delta_1 + 21 a_0. No separate bound for the unweighted noise sum is provided, and the additional terms O(Delta_1 log(1/delta)) that such a bound would introduce do not appear in Theorem 1. Thus the main inequality is not derived from the stated lemmas.
  3. [Appendix 6.4, proof of Lemma 6] The exponential-martingale argument bounds the event S_n >= log(1/delta) for each fixed n. The lemma, however, states the bound holds for any n in [T], and the induction over k uses the partial sums for all k <= n under a single high-probability event. The proof does not provide a union bound over n or a maximal inequality for the supermartingale exp(S_n). This is another source of a log T factor that is missing from the statements of Lemma 6 and Theorem 1.
minor comments (5)
  1. [Assumption 1] Assumption 1 states a condition for all i in [n], but the agent set is [N]; this should be [N].
  2. [Lemma 6] The index i in Lemma 6 is not quantified. If the lemma is intended for every i in [N], this should be stated explicitly, since the subsequent proof of Theorem 1 depends on whether the bound is uniform over agents.
  3. [Appendix 6.3, proof of Lemma 4] The displayed derivation after equation (5) appears to mix factors of 2 and to replace z_t by z_1. Specifically, the telescoping of the terms z_t(||bar{x}_t - x*||^2 - ||bar{x}_{t+1} - x*||^2) produces a factor 1/2 and a z_1 Delta_1^2 term, while the proof writes z_1||bar{x}_1 - x*||^2 + sum_t z_t eta_t(f(bar{x}_t)-f(x*)) without those factors. Please check the algebra and the resulting constants.
  4. [Corollary 1, proof] The proof of Corollary 1 is only a scaling argument ('m ~ ...', 'lambda ~ ...'). It should show explicitly how the coefficients N, rho, and B* in the stated rate are obtained from condition (7) and from the bound in Theorem 1.
  5. [Section 4, Simulation] The simulations do not report the confidence level delta, nor do they compare the observed convergence to the theoretical rate. Adding error bars or multiple runs would also strengthen the empirical claim.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the central high-probability bound is not fitted or defined in terms of its own conclusion; the main weakness is a proof gap, not a circular derivation.

full rationale

The paper's derivation is not circular in the sense of reducing to its own inputs. The constants m and lambda in condition (7) are chosen to satisfy sufficient inequalities, not fitted to reproduce the final bound; no parameter is calibrated on the target rate. Lemma 5 and Lemma 7 are attributed to the external works [29] and [32], and the clipping construction is inherited from external centralized analyses [28,30,32], so the core technical lemmas are independent support rather than self-citation. The only overlapping-author reference is [19] (K. Lu and L. Wang), but it is used as a light-tailed baseline and as a simulation comparator; Theorem 1 and Corollary 1 do not rest on it, so that self-citation is not load-bearing. The proof does contain real rigor gaps: Lemma 6 is a per-agent, z_t-weighted statement, while Theorem 1's proof applies it to the unweighted all-agent sum without a union bound over agents or a bound on dropping the z_t factor, so Theorem 1 does not follow from Lemma 6 as written. This is a correctness/rigor defect, not a circularity: no equation in the paper is defined in terms of the target result, and no prediction is statistically forced by a fit. The central high-probability rate claim has independent content relative to its inputs. Score 2 reflects the minor non-load-bearing self-citation and the independence of the central claim, not a circular derivation.

Assumptions & free parameters 3 free parameters · 8 assumptions · 0 invented entities

The central claim rests on standard assumptions in distributed stochastic optimization: unbiased gradients, bounded p-th moment, smoothness, and connectivity. No new physical or mathematical entities are postulated. The algorithm parameters m, lambda, alpha, and kappa are design variables chosen to satisfy a sufficient condition, not fitted to data, though the simulation values are chosen by hand and do not match the theoretical conditions for the noise used.

free parameters (3)
  • m = a9a: 1/5, MNIST: 35
    Scale of the diminishing step size in eta_t = 1/(m t^kappa (log t + b1)^2). The theorem requires m to satisfy lower bounds in condition (7) involving Delta_1, sigma, lambda, and beta. In the simulations m is chosen by hand, and the chosen values imply p=2, which is inconsistent with the t2 noise used.
  • lambda = a9a: 2, MNIST: 100
    Scale of the clipping parameter lambda_t = lambda t^alpha. Condition (7) requires lambda to be at least 2L(9Delta_1+5a0)+2LD+2B*, so it is a problem-dependent constant. The simulation values are hand-picked and do not guarantee the theorem's conditions for the actual noise distribution.
  • alpha, kappa = 0.25 and 0.75 in simulations
    Exponents of the clipping parameter and step size. The theory sets alpha=1/(2p) and kappa=1/2+1/(2p), but the simulation values correspond to p=2, which is incompatible with the unbounded variance of t2 noise.
assumptions (8)
  • domain assumption Unbiased noisy gradients (Assumption 1)
    Each agent's gradient estimator is conditionally unbiased. Used throughout, starting in the algorithm formulation in Section 2.2.
  • domain assumption Bounded p-th moment of noise (Assumption 2)
    The heavy-tailed noise has finite p-th moment for some p in (1,2]. This is the core heavy-tailed noise model from [34,35] and is required for Lemma 5 and all subsequent bounds.
  • domain assumption L-smoothness and convexity of each f_i
    The problem statement in Section 2.1 says each f_i is convex, and Assumption 3 adds L-smoothness. These are used in Lemma 3 and throughout the regret analysis.
  • domain assumption B-strong connectivity and doubly stochastic weights (Assumption 4)
    The time-varying graph must be B-strongly connected and the weight matrices doubly stochastic with positive lower bound eta. This enables the consensus contraction bound in Lemma 1.
  • standard math Consensus contraction bound (Lemma 1, from [38])
    The paper uses the known bound on the deviation of products of doubly stochastic matrices from the uniform matrix, stated as Lemma 1 and proved in the cited literature.
  • standard math Clipped estimator bias and variance bounds (Lemma 5, from [29])
    The paper imports a bound on the bias and variance of the clipped gradient estimator under a bounded p-th moment assumption. This lemma is central to the proof of Lemma 6.
  • standard math Exponential concentration inequality (Lemma 7, from [32])
    Used to construct the exponential supermartingale that controls the heavy-tailed noise terms in Lemma 6.
  • standard math Markov's inequality and supermartingale convergence
    Standard probabilistic tools used in the proof of Lemma 6 to convert the exponential supermartingale bound into a high-probability statement.

how reviews work

0 comments
Cite this review

Pith. "Pith review of High Probability Convergence of Distributed Clipped Stochastic Gradient Descent with Heavy-tailed Noise." pith.science (2026). https://pith.science/paper/2ASBVTSV

@misc{pith2026250611647,
  author       = {Pith},
  title        = {Pith review of: High Probability Convergence of Distributed Clipped Stochastic Gradient Descent with Heavy-tailed Noise},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ASBVTSV}},
  note         = {Machine review of arXiv:2506.11647}
}
read the original abstract

In this paper, the problem of distributed optimization is studied via a network of agents. Each agent only has access to a noisy gradient of its own objective function, and can communicate with its neighbors via a network. To handle this problem, a distributed clipped stochastic gradient descent algorithm is proposed, and the high probability convergence of the algorithm is studied. Existing works on distributed algorithms involving stochastic gradients only consider the light-tailed noises. Different from them, we study the case with heavy-tailed settings. Under mild assumptions on the graph connectivity, we prove that the algorithm converges in high probability under a certain clipping operator. Finally, a simulation is provided to demonstrate the effectiveness of our theoretical results

Figures

Figures reproduced from arXiv: 2506.11647 by the authors.

Figure 1
Figure 1. The Time-varying Graph Corollary 1 Under the the same conditions stated in Theorem 1, if λt = λt 1 2p and ηt = 1 mtt 1 2 + 1 2p , then with probability at least 1 − δ, PT t=1 f(¯xt) − f(x ∗ ) T ≤O((N ρ 5 3 + B ∗ ρ 2 3 )T 1 2p − 1 2 log2 ( T δ )). where B∗ = maxi∈[N] ∥∇fi(x ∗ )∥ and ρ = 1 1−β . Proof: By the first line in condition (7), κ ≥ max{α+ 1 2 , 1−(p−1)α}. Note that min max{α+ 1 2 , 1 − (p − 1)α} = 1 2p + 1 2… view at source ↗
Figure 2
Figure 2. The Average Convergence of the Distributed Clipped SGD and the Distributed SGD on a9a 0 200 400 600 800 1000 1200 1400 1600 1800 2000 Iteration 0 50 100 150 200 250 300 350 Average Error of SGD and Clipped SGD distributed clipped SGD distributed SGD [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. The Average Convergence of the Distributed Clipped SGD and the Distributed SGD on MNIST and d = 784. The parameters are given by ηt = 1 35t 0.75(4 + log t) 2 , λt = 100t 0.25 . Similarly, we run the algorithm in a single round and compare the performance to the tradi￾tional distributed SGD algorithm in [19] for our problem with the same step size. The average convergence of the algorithms mentioned is depicted in Fi… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DeMuon: A Decentralized Muon for Matrix Optimization over Graphs

    math.OC 2025-10 conditional novelty 6.0 of 10

    A decentralized Muon optimizer with gradient tracking reaches a stochastic stationary point at the same iteration complexity as centralized heavy-tailed algorithms.

Reference graph

Works this paper leans on

40 extracted references · 36 canonical work pages · cited by 1 Pith paper

  1. [1]

    Distributed subgradient methods for multi-agent optimiza- tion,

    A. Nedic and A. Ozdaglar, “Distributed subgradient methods for multi-agent optimiza- tion,”IEEE Transactions on Automatic Control, vol. 54, no. 1, pp. 48–61, 2009

    work page 2009

  2. [2]

    A new approach to consensus problems in discrete-time multiagent 18 systems with time-delays,

    L. Wang and F. Xiao, “A new approach to consensus problems in discrete-time multiagent 18 systems with time-delays,”Science in China Series F: Information Sciences, vol. 50, no. 4, pp. 625–635, 2007

    work page 2007

  3. [3]

    Controllability of multi-agent systems based on agreement protocols,

    L. Wang, F. Jiang, G. Xie, and Z. Ji, “Controllability of multi-agent systems based on agreement protocols,”Science in China Series F: Information Sciences, vol. 52, no. 11, pp. 2074–2088, 2009

    work page 2009

  4. [4]

    Finite-time consensus problems for networks of dynamic agents,

    L. Wang and F. Xiao, “Finite-time consensus problems for networks of dynamic agents,” IEEE Transactions on Automatic Control, vol. 55, no. 4, pp. 950–955, 2010

    work page 2010

  5. [5]

    Distributed optimization over time-varying directed graphs,

    A. Nedi ´c and A. Olshevsky, “Distributed optimization over time-varying directed graphs,” IEEE Transactions on Automatic Control, vol. 60, no. 3, pp. 601–615, 2015

    work page 2015

  6. [6]

    Distributed continuous-time convex optimization with time- varying cost functions,

    S. Rahili and W. Ren, “Distributed continuous-time convex optimization with time- varying cost functions,”IEEE Transactions on Automatic Control, vol. 62, no. 4, pp. 1590–1605, 2017

    work page 2017

  7. [7]

    Large-scale distributed dedicated- and non-dedicated smart city sensing systems,

    H. Habibzadeh, Z. Qin, T. Soyata, and B. Kantarci, “Large-scale distributed dedicated- and non-dedicated smart city sensing systems,”IEEE Sensors Journal, vol. 17, no. 23, pp. 7649–7658, 2017

    work page 2017

  8. [8]

    Zhu and S

    M. Zhu and S. Martnez,Distributed Optimization-Based Control of Multi-Agent Networks in Complex Environments. Cham, Switzerland: Springer Publishing Company, Incorpo- rated, 2015

    work page 2015

Show all 40 references
  1. [9]

    Initialization-free distributed fixed-time convergent algorithms for optimal resource allocation,

    G. Chen and Z. Guo, “Initialization-free distributed fixed-time convergent algorithms for optimal resource allocation,”IEEE Transactions on Systems, Man, and Cybernetics: Sys- tems, vol. 52, no. 2, pp. 845–854, 2022

    2022

  2. [10]

    Wireless sensor networks for environmental monitoring: The sensorscope experience,

    G. Barrenetxea, F. Ingelrest, G. Schaefer, and M. Vetterli, “Wireless sensor networks for environmental monitoring: The sensorscope experience,”2008 IEEE International Zurich Seminar on Communications, pp. 98–101, 2008

    2008

  3. [11]

    Network topology and communication- computation tradeoffs in decentralized optimization,

    A. Nedi ´c, A. Olshevsky, and M. G. Rabbat, “Network topology and communication- computation tradeoffs in decentralized optimization,”Proceedings of the IEEE, vol. 106, no. 5, pp. 953–976, 2018

    2018

  4. [12]

    A general framework for decentralized opti- mization with first-order methods,

    R. Xin, S. Pu, A. Nedi ´c, and U. A. Khan, “A general framework for decentralized opti- mization with first-order methods,”Proceedings of the IEEE, vol. 108, no. 11, pp. 1869– 1889, 2020

    2020

  5. [13]

    Asymptotic network independence in dis- tributed stochastic optimization for machine learning: Examining distributed and central- ized stochastic gradient descent,

    S. Pu, A. Olshevsky, and I. C. Paschalidis, “Asymptotic network independence in dis- tributed stochastic optimization for machine learning: Examining distributed and central- ized stochastic gradient descent,”IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 114–122, 2020. 19

    2020

  6. [14]

    Gradient-tracking-based distributed optimization with guaranteed optimality under noisy information sharing,

    Y . Wang and T. Bas ¸ar, “Gradient-tracking-based distributed optimization with guaranteed optimality under noisy information sharing,”IEEE Transactions on Automatic Control, vol. 68, no. 8, pp. 4796–4811, 2023

    2023

  7. [15]

    Event-triggered distributed stochastic mirror descent for convex optimization,

    M. Xiong, B. Zhang, D. W. C. Ho, D. Yuan, and S. Xu, “Event-triggered distributed stochastic mirror descent for convex optimization,”IEEE Transactions on Neural Net- works and Learning Systems, pp. 1–12, 2022

    2022

  8. [16]

    High-probability convergence bounds for non- convex stochastic gradient descent,

    L. Madden, E. Dall’Anese, and S. Becker, “High-probability convergence bounds for non- convex stochastic gradient descent,”arXiv preprint arXiv:2006.05610, 2020

    2006

  9. [17]

    High probability convergence of adam under unbounded gradients and affine variance noise,

    Y . Hong and J. Lin, “High probability convergence of adam under unbounded gradients and affine variance noise,”arXiv preprint arXiv:2311.02000, 2023

    2023 arXiv

  10. [18]

    High probability conver- gence of stochastic gradient methods,

    Z. Liu, T. D. Nguyen, T. H. Nguyen, A. Ene, and H. Nguyen, “High probability conver- gence of stochastic gradient methods,”Proceedings of the 40th International Conference on Machine Learning, vol. 202, pp. 21 884–21 914, 2023

    2023

  11. [19]

    Convergence in high probability of distributed stochastic gradient descent algorithms,

    K. Lu, H. Wang, H. Zhang, and L. Wang, “Convergence in high probability of distributed stochastic gradient descent algorithms,”IEEE Transactions on Automatic Control, pp. 1– 16, 2023

    2023

  12. [20]

    Distributed (atc) gradient descent for high dimension sparse regression,

    Y . Ji, G. Scutari, Y . Sun, and H. Honnappa, “Distributed (atc) gradient descent for high dimension sparse regression,”IEEE Transactions on Information Theory, vol. 69, no. 8, pp. 5253–5276, 2023

    2023

  13. [21]

    L´evy flights in evolutionary ecology,

    B. Jourdain, S. M ´el´eard, and W. A. Woyczynski, “L´evy flights in evolutionary ecology,” Journal of Mathematical Biology, vol. 65, pp. 677–707, 2012

    2012

  14. [22]

    Generalized wiener filtering with fractional power spectro- grams,

    A. Liutkus and R. Badeau, “Generalized wiener filtering with fractional power spectro- grams,”2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 266–270, 2015

    2015

  15. [23]

    Fractals and scaling in finance,

    B. B. Mandelbrot, “Fractals and scaling in finance,”Springer,, 1997

    1997

  16. [24]

    A tail-index analysis of stochastic gradi- ent noise in deep neural networks,

    U. Simsekli, L. Sagun, and M. Gurbuzbalaban, “A tail-index analysis of stochastic gradi- ent noise in deep neural networks,”Proceedings of the 36th International Conference on Machine Learning, vol. 97, pp. 5827–5837, 2019

    2019

  17. [25]

    On the difficulty of training recurrent neural networks,

    R. Pascanu, T. Mikolov, and Y . Bengio, “On the difficulty of training recurrent neural networks,”Proceedings of the 30th International Conference on Machine Learning, vol. 28, no. 3, pp. 1310–1318, 2013. 20

    2013

  18. [26]

    Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework,

    S. Ghadimi and G. Lan, “Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework,”SIAM Journal on Optimization, vol. 22, no. 4, pp. 1469–1492, 2012

    2012

  19. [27]

    An ac- celerated method for decentralized distributed stochastic optimization over time-varying graphs,

    A. Rogozin, M. Bochko, P. Dvurechensky, A. Gasnikov, and V . Lukoshkin, “An ac- celerated method for decentralized distributed stochastic optimization over time-varying graphs,”2021 60th IEEE Conference on Decision and Control (CDC), pp. 3367–3373, 2021

    2021

  20. [28]

    High probability convergence of clipped-sgd under heavy-tailed noise,

    T. D. Nguyen, T. H. Nguyen, A. Ene, and H. L. Nguyen, “High probability convergence of clipped-sgd under heavy-tailed noise,”arXiv preprint arXiv:2302.05437, 2023

    2023 arXiv

  21. [29]

    High-probability bounds for stochastic optimization and varia- tional inequalities: the case of unbounded variance,

    A. Sadiev, M. Danilova, E. Gorbunov, S. Horv ´ath, G. Gidel, P. Dvurechensky, A. Gas- nikov, and P. Richt´arik, “High-probability bounds for stochastic optimization and varia- tional inequalities: the case of unbounded variance,”Proceedings of the 40th International Conference...

    2023

  22. [30]

    High-probability convergence for composite and distributed stochastic minimization and variational inequalities with heavy-tailed noise,

    E. Gorbunov, A. Sadiev, M. Danilova, S. Horv ´ath, G. Gidel, P. Dvurechensky, A. Gasnikov, and P. Richt ´arik, “High-probability convergence for composite and distributed stochastic minimization and variational inequalities with heavy-tailed noise,” Proceedings of the 41st Int...

    2024

  23. [31]

    Distributed online optimization in dynamic environ- ments using mirror descent,

    S. Shahrampour and A. Jadbabaie, “Distributed online optimization in dynamic environ- ments using mirror descent,”IEEE Transactions on Automatic Control, vol. 63, pp. 714– 725, 2018

    2018

  24. [32]

    Improved convergence in high probability of clipped gradient methods with heavy tails,

    T. D. Nguyen, A. Ene, and H. L. Nguyen, “Improved convergence in high probability of clipped gradient methods with heavy tails,”arXiv preprint arXiv:2304.01119, 2023

    2023 arXiv

  25. [33]

    Convergence and privacy of decentralized nonconvex optimization with gradient clipping and communication compression,

    B. Li and Y . Chi, “Convergence and privacy of decentralized nonconvex optimization with gradient clipping and communication compression,”arXiv preprint arXiv:2305.09896, 2023

    2023 arXiv

  26. [34]

    Problem complexity and method efficiency in opti- mization,

    A. S. Nemirovsky and D. B. Yudin, “Problem complexity and method efficiency in opti- mization,” 1983

    1983

  27. [35]

    Why are adaptive methods good for attention models?

    J. Zhang, S. P. Karimireddy, A. Veit, S. Kim, S. Reddi, S. Kumar, and S. Sra, “Why are adaptive methods good for attention models?”Advances in Neural Information Processing Systems, vol. 33, pp. 15 383–15 393, 2020

    2020

  28. [36]

    Asynchronous consensus in continuous-time multi-agent systems with switching topology and time-varying delays,

    F. Xiao and L. Wang, “Asynchronous consensus in continuous-time multi-agent systems with switching topology and time-varying delays,”IEEE Transactions on Automatic Con- trol, vol. 53, no. 8, pp. 1804–1816, 2008. 21

    2008

  29. [37]

    Group consensus in multi-agent systems with switching topologies,

    J. Yu and L. Wang, “Group consensus in multi-agent systems with switching topologies,” Proceedings of the 48h IEEE Conference on Decision and Control (CDC) held jointly with 2009 28th Chinese Control Conference, pp. 2652–2657, 2009

    2009

  30. [38]

    A new class of distributed optimiza- tion algorithms: Application to regression of distributed data,

    S. Sundhar Ram, A. Nedi ´c, and V . V . Veeravalli, “A new class of distributed optimiza- tion algorithms: Application to regression of distributed data,”Optimization Methods and Software, vol. 27, no. 1, pp. 71–88, 2012

    2012

  31. [39]

    On the difficulty of training recurrent neural networks,

    R. Pascanu, T. Mikolov, and Y . Bengio, “On the difficulty of training recurrent neural networks,”Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28, p. III–1310–III–1318, 2013

    2013

  32. [40]

    Stochastic optimization with heavy-tailed noise via accelerated gradient clipping,

    E. Gorbunov, M. Danilova, and A. Gasnikov, “Stochastic optimization with heavy-tailed noise via accelerated gradient clipping,”Advances in Neural Information Processing Sys- tems, vol. 33, pp. 15 042–15 053, 2020. 22

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.