Pith. sign in

REVIEW 1 major objections 3 minor 47 references

Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression

T0 review · 1 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For heterogeneous distributed logistic regression, a two-stage Local GD converges at $O(1/(KR))$, so more local steps $K$ provably reduce communication rounds—the first such guarantee for Local GD.

desk verdict Genuinely new K-improving rate claim for a Local GD variant, but the central proof has a load-bearing gap at Lemma 26's margin condition, so Theorem 1 is not established as written. read the letter →

arxiv 2501.13790 v2 pith:ZUZ3UDE5 submitted 2025-01-23 cs.LG

Michael Crawshaw , Blake Woodworth , Mingrui Liu This is my paper · ORCID
classification cs.LG MSC 90C2568Q25
keywords localgradientdescentfederatedlearningdistributedlogisticregressionseparabledatacommunicationcomplexitylargerateupdatesconvergence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to prove that local model updates can reduce the number of communication rounds in a concrete distributed learning problem: logistic regression on heterogeneous but linearly separable data. Its main result, Theorem 1, analyzes a two-stage Local GD that first runs a small-learning-rate warmup for $r_0$ rounds and then switches to a large learning rate $\eta_2 \le 4$; after warmup the objective satisfies $F(\hat w_2) \le 2/(\eta_2 \gamma^2 K (R-r_0))$, which translates to $\widetilde{O}(KM/\gamma^4 + 1/(\gamma^2 K \epsilon))$ rounds to reach error $\epsilon$. Because $\eta_2$ is not forced to shrink with $K$, more local steps $K$ strictly improve the rate, in contrast to every prior Local GD guarantee, which is at least $\Omega(1/R)$ and therefore shows no benefit from local steps. A second result gives a fixed-learning-rate guarantee for a gradient-flow variant in the special case of two clients with one sample each. If the main theorem is correct, it is the first proof that local steps speed up Local GD on any problem, and the speedup comes from the loss landscape rather than from similarity of client objectives.

What carries the argument

The carrying mechanism is the self-bounding structure of the logistic loss: for $\ell(z)=\log(1+e^{-z})$, the inequalities $0<\ell''(z)<|\ell'(z)|<\ell(z)$ imply $\|\nabla F_m(w)\| \le F_m(w)$ and $\|\nabla^2 F_m(w)\| \le F_m(w)$ for every client objective. This converts 'small objective value' into 'small local curvature,' which is what allows the second phase to use a large learning rate $\eta_2 \le 4$ (the smoothness constant is $H=1/4$) without instability: each local iterate stays within $\eta K F(\bar w_r)$ of the global model, and the round update is a biased gradient step whose bias is $O(\eta^2 K^2 F(\cdot)^2)$. The descent recursion is closed by a margin-based lower bound (Lemma 26): whenever every sample has nonnegative margin, meaning $y_{mi}\langle w, x_{mi}\rangle \ge 0$ for all $m,i$, then $\|\nabla F(w)\| \ge (\gamma/2) F(w)$, which turns objective descent into the $O(1/(\eta K R))$ rate. For the gradient-flow variant, the key object is the Lyapunov function $L_r = \max_m \rho_m^r$, where each surrogate loss $\rho_m^r$ is defined through the Lambert $W$ function and tracks the implicitly prioritized client; $L_r$ decreases after every two rounds by an amount proportional to $L_r^2$.

What would settle it

Run Two-Stage Local GD on separable data constructed so that, at the warmup output, the average logistic loss is below $\gamma^2/(42\eta_2 K M)$ while at least one sample still has a negative margin (possible when $n$ is large); with $\eta_2=4$, if Lemma 26's margin condition is load-bearing, the first second-stage round should violate $F(\bar w_{r+1}) \le 2/(\eta_2 \gamma^2 K (r+1-r_0))$, because the inner-product bound $\langle \nabla F(w), w^*\rangle \ge \gamma F(w)/2$ no longer holds.

Watch

Extended reading notes

Core claim

The paper's central claim is that separable distributed logistic regression has enough structure to break the $\eta \le 1/K$ barrier that makes local steps useless in worst-case analyses. The logistic loss $\ell(z)=\log(1+e^{-z})$ satisfies $0<\ell''(z)<|\ell'(z)|<\ell(z)$, which lifts to the objective inequalities $\|\nabla F(w)\| \le F(w)$ and $\|\nabla^2 F(w)\| \le F(w)$; once the warmup stage reaches $F(\hat w_1) \le \gamma^2/(42\eta_2 K M)$, local curvature is small, the heterogeneity bias of each round update is $O(\eta^2 K^2 F(\cdot)^2)$, and the global objective descends at rate $O(1/(\eta_2 \gamma^2 K R))$. Theorem 1 states this for Two-Stage Local GD, and Corollary 3 shows that with $K \approx \gamma/\sqrt{M\epsilon}$ the round complexity can be as small as $\widetilde{O}(\sqrt{M}/(\gamma^3 \sqrt{\epsilon}))$. Theorem 2 separately shows that Local Gradient Flow with fixed $\eta$ reaches $\widetilde{O}(1/(KR))$ in the $M=2,n=1$ case, using a Lyapunov function built from Lambert-$W$ surrogate losses. The proof also strengthens the baseline guarantees on which the warmup bound rests by removing the assumption that a global minimizer exists.

Load-bearing premise

The second stage assumes that after warmup every training sample has a nonnegative margin at every later global iterate, and the paper does not show that the stated warmup loss bound guarantees this when the number of samples per client is large.

Editorial extensions

If this is right

  • Local GD has its first regime in which communication rounds decrease as $K$ grows: setting $\eta_2=\Theta(1)$ and $K \approx \gamma/\sqrt{M\epsilon}$ gives round complexity $\widetilde{O}(\sqrt{M}/(\gamma^3\sqrt{\epsilon}))$.
  • The result moves the local-step speedup debate from worst-case heterogeneity assumptions to concrete problem structure: any separable objective whose loss satisfies the same self-bounding derivative chain should admit an analogous two-stage analysis.
  • The second stage is a last-iterate guarantee, not just an average-iterate one, which is stronger than the baseline guarantees it is compared with.
  • The warmup length $r_0 = \widetilde{\Theta}(KM/\gamma^4 + (KM)^{3/4}/\gamma^{5/2})$ grows with $K$, so the benefit of large $K$ appears after a communication cost that is amortized over the second phase.
  • The fixed-learning-rate Local Gradient Flow result in the $M=2,n=1$ case suggests that the warmup may be a proof device rather than an algorithmic necessity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Theorem 1's proof invokes Lemma 26, whose margin condition $y_{mi}\langle w, x_{mi}\rangle \ge 0$ must hold at every second-stage iterate; the warmup guarantee $F(\hat w_1)\le \gamma^2/(42\eta_2 K M)$ does not by itself imply nonnegative margins when $n$ is large, so a strengthened warmup or an explicit margin-certifying phase would be needed to close that step.
  • Editorial inference: If the self-bounding Hessian is the real driver, the same two-stage argument should extend to other losses with $\ell''(z)\to 0$ as $\ell(z)\to 0$, such as exponential-tail losses; rerunning the proof with $\ell(z)=e^{-z}$ is a direct test.
  • Editorial inference: The $M=2,n=1$ Lyapunov construction is likely to generalize through the Gram matrix $G$ and the surrogate losses $\rho_m^r$ to more clients and more samples per client, which would turn the fixed-learning-rate result into a proof for vanilla Local GD.
  • Editorial inference: The observed non-monotone loss under fixed large $\eta K$ parallels edge-of-stability analyses of gradient descent, suggesting that a non-monotone convergence proof for vanilla Local GD might remove the need for the warmup stage altogether.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 3 minor

Summary. The paper analyzes Local Gradient Descent (Local GD) applied to distributed logistic regression on separable, heterogeneous data. It proposes Two-Stage Local GD, which first runs Local GD with a small step size for r0 rounds and then continues with a large step size, and claims a last-iterate bound F(w_hat2) <= 2/(eta2 gamma^2 K (R - r0)) after O~(KM/gamma^4 + (KM)^{3/4}/gamma^{5/2}) warmup rounds. It also presents a Lyapunov-function analysis of a continuous-time Local Gradient Flow variant in a special M=2, n=1 case, extends two existing Local SGD analyses to objectives without minimizers, and reports experiments on synthetic and MNIST data.

Significance. If Theorem 1 were fully established, it would be the first convergence guarantee for Local GD whose rate improves with the number of local steps K, a conceptually significant result for distributed optimization. The paper is largely self-contained, provides careful extensions of baseline analyses, and includes reproducible experiments. The Local Gradient Flow result is an independent contribution. However, the main theorem has a load-bearing proof gap (see Major Comment 1), so the significance is conditional on repairing that argument.

major comments (1)
  1. [Appendix A, Lemma 10 proof step (v); Appendix D.1, Lemma 26] The descent inequality in Lemma 10 uses Lemma 26 at step (v), but Lemma 26 is proved only under the condition that y_mi <wbar_r, x_mi> >= 0 for every sample at each global iterate. The warmup condition F(wbar_r) <= gamma^2/(42 eta K M) (Eq. (63)) does not imply this condition. Concretely, take M=1, eta=1, K=1, n=3, with two copies of x1=(-1,10) and one copy of x2=(1,0.1), all labels 1. This dataset is separable with gamma=1, and at w=(-0.1,0.5) the margin on x2 is -0.05, while F(w) = (2 ell(5.1) + ell(-0.05))/3 ≈ 0.0207 <= 1/42, so the warmup bound holds but Lemma 26 does not apply. Moreover, Lemma 2 only bounds the distance from wbar_r to the local models, not the location of the next global iterate relative to the nonnegative-margin cone, so the condition is not preserved along the trajectory. Therefore the step marked '(v) uses Lemma 26' is not justified, and the O(1/(eta2 gamma^2 K R)) second-stage rate of Theorem 1 is not established by the proof as written.
minor comments (3)
  1. [Section 6.1] The experiments set r0 = lambda K with lambda tuned per dataset, whereas Theorem 1's r0 depends on gamma, M, eta2, and logarithmic factors; the experimental schedule is therefore a heuristic specialization, and for the synthetic dataset with K=1024 the chosen lambda=4 gives r0=4096 > R=2048, so the second stage never runs in that configuration. The text should note this explicitly when interpreting the K=1024 two-stage curve.
  2. [Theorem 1 and Theorem 3] The definition of eta1 in Theorem 1 uses a tilde-O inside a min expression; it would be clearer to state the explicit expression used in the proof of Theorem 3, including how the universal constants are absorbed.
  3. [Section 4.2 and Appendix A] The main-text Lemma 4 and the appendix's Lemma 10 have different hypotheses and constants (r0 vs 2r0, and factor 2 vs 4 in the bound), so aligning the numbering and hypotheses between the main text and the appendix would help readers verify the proof.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the O(1/KR) bound follows from explicit logistic-loss inequalities and an independently derived warmup, with the Lemma 26 margin-condition issue being a proof gap rather than a circular step.

full rationale

The central claim is not circular. Theorem 1 is obtained by combining Lemma 4/10, whose proof uses the logistic-loss derivative inequalities (Lemmas 24-25), local drift bounds (Lemmas 2-3), and the PL-type inequality Lemma 26, with a warmup guarantee Corollary 2. Corollary 2 is not assumed from prior work; it is proven in Appendix C by extending the Woodworth et al. and Koloskova et al. analyses with fully specified comparators and constants, so it provides independent evidence rather than a self-citation chain. The quantities r0 and eta1 are explicit closed-form choices satisfying the warmup condition; no fitted parameter is renamed as a prediction, and the experimental tuning of lambda in Appendix E is presented as an implementation detail, not as evidence for the theorem. The only substantive issue found is a correctness gap, not circularity: Lemma 10 invokes Lemma 26 at step (v), but Lemma 26 requires every sample margin y_mi <w, x_mi> to be nonnegative at each global iterate, while the warmup condition F(w_hat1) <= gamma^2/(42 eta2 K M) is an average-loss bound that does not imply per-sample nonnegative margins when n is large, and Lemma 2 does not show that the nonnegative-margin cone is preserved under second-stage updates. This gap means the proof of Theorem 1 is incomplete as written, but it does not make the conclusion equivalent to its assumptions or reduce to a fitted input, so the circularity score remains low.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The theoretical rates introduce no fitted constants; the only fitted number is an experimental warmup multiplier. The main non-standard object is the surrogate Lyapunov function rho^m_r for Local Gradient Flow, which is a mathematical construction rather than a physical entity. One unproved condition, nonnegative margins, is load-bearing in the written proof.

free parameters (1)
  • lambda (two-stage warmup multiplier in experiments) = 4 (synthetic), 1/16 (MNIST)
    Tuned in Appendix E.3 by grid search over powers of two; this is an empirical scheduling choice, not part of the theorem's hyperparameter formulas.
assumptions (6)
  • domain assumption Distribution is linearly separable with maximum margin gamma > 0, and all data norms are at most 1.
    Section 3.1, Eq (2); central to the rates via gamma.
  • domain assumption Each client's samples have labels normalized to +1 by transforming features.
    Section 3.1; used without loss of generality throughout.
  • domain assumption Deterministic full-gradient local updates (Local GD), no stochastic noise.
    Algorithms 1-3; the analysis is deterministic apart from baseline extensions.
  • standard math Logistic derivative relations: 0 < ell'' <= |ell'| <= ell and ||grad F_m(w)|| <= F_m(w), ||Hess F_m(w)|| <= F_m(w).
    Lemmas 24-25; the key structural property enabling large stepsizes.
  • standard math A Gronwall-type integral inequality (Lemma 27) and standard properties of the Lambert W function.
    Used in Lemma 1 and Section 5 proofs.
  • ad hoc to paper At each second-stage iterate, all sample margins y_mi <w, x_mi> are nonnegative so Lemma 26 applies.
    Lemma 4 proof invokes Lemma 26, but the paper does not prove this condition follows from the warmup bound for general n; this is the main proof gap.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression." pith.science (2026). https://pith.science/paper/ZUZ3UDE5

@misc{pith2026250113790,
  author       = {Pith},
  title        = {Pith review of: Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZUZ3UDE5}},
  note         = {Machine review of arXiv:2501.13790}
}
abstract

We analyze two variants of Local Gradient Descent applied to distributed logistic regression with heterogeneous, separable data and show convergence at the rate $O(1/KR)$ for $K$ local steps and sufficiently large $R$ communication rounds. In contrast, all existing convergence guarantees for Local GD applied to any problem are at least $\Omega(1/R)$, meaning they fail to show the benefit of local updates. The key to our improved guarantee is showing progress on the logistic regression objective when using a large stepsize $\eta \gg 1/K$, whereas prior analysis depends on $\eta \leq 1/K$.

Figures

Figures reproduced from arXiv: 2501.13790 by the authors.

Figure 1
Figure 1. Train loss of Local GD for a synthetic dataset and MN [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. First, Local SGD is significantly faster when using a larger number of local steps K, up to a thresh￾old. The final training loss of Local SGD improves steadily as K increases from K = 1 to K = 8. When the number of local steps is large (K = 16), training becomes less stable, although the train￾ing loss is still smaller than that reached by every K ≤ 4. These results suggest that our theoretical results about the op… view at source ↗
Figure 2
Figure 2. Train loss and testing accuracy for heterogeneous [PITH_FULL_IMAGE:figures/full_fig_p062_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 28 canonical work pages

  1. [1]

    Communication complexity of distributed convex learning and optimization

    Yossi Arjevani and Ohad Shamir. Communication complexity of distributed convex learning and optimization. Advances in neural information processing systems, 28, 2015

    work page 2015

  2. [2]

    Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression

    Francis Bach. Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression. Journal of Machine Learning Research, 15 0 (19): 0 595--627, 2014. URL http://jmlr.org/papers/v15/bach14a.html

    work page 2014

  3. [3]

    Distributed learning, communication complexity and privacy

    Maria Florina Balcan, Avrim Blum, Shai Fine, and Yishay Mansour. Distributed learning, communication complexity and privacy. In Conference on Learning Theory, pp.\ 26--1. JMLR Workshop and Conference Proceedings, 2012

    work page 2012

  4. [4]

    Gradient descent on neural networks typically occurs at the edge of stability

    Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations, 2021

    work page 2021

  5. [5]

    Optimal distributed online prediction using mini-batches

    Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research, 13 0 (1), 2012

    work page 2012

  6. [6]

    Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework

    Saeed Ghadimi and Guanghui Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework. SIAM Journal on Optimization, 22 0 (4): 0 1469--1492, 2012

    2012

  7. [7]

    Sharp bounds for federated averaging (local sgd) and continuous perspective

    Margalit R Glasgow, Honglin Yuan, and Tengyu Ma. Sharp bounds for federated averaging (local sgd) and continuous perspective. In International Conference on Artificial Intelligence and Statistics, pp.\ 9050--9090. PMLR, 2022

    work page 2022

  8. [8]

    Characterizing implicit bias in terms of optimization geometry

    Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pp.\ 1832--1841. PMLR, 2018

    work page 2018

Show all 47 references
  1. [9]

    On the convergence of local descent methods in federated learning

    Farzin Haddadpour and Mehrdad Mahdavi. On the convergence of local descent methods in federated learning. arXiv preprint arXiv:1910.14425, 2019

    1910 arXiv

  2. [10]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  3. [11]

    Risk and parameter convergence of logistic regression

    Ziwei Ji and Matus Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300, 2018

    2018 arXiv

  4. [12]

    Fast margin maximization via dual acceleration

    Ziwei Ji, Nathan Srebro, and Matus Telgarsky. Fast margin maximization via dual acceleration. In International Conference on Machine Learning, pp.\ 4860--4869. PMLR, 2021

    2021

  5. [13]

    Advances and open problems in federated learning

    Peter Kairouz, H Brendan McMahan, Brendan Avent, Aur \'e lien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977, 2019

    1912 arXiv

  6. [14]

    Advances and open problems in federated learning

    Peter Kairouz, H Brendan McMahan, Brendan Avent, Aur \'e lien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and trends in machine learning , 1...

    2021

  7. [15]

    Scaffold: Stochastic controlled averaging for federated learning

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pp.\ 5132--5143. PMLR, 2020

    2020

  8. [16]

    Tighter theory for local sgd on identical and heterogeneous data

    Ahmed Khaled, Konstantin Mishchenko, and Peter Richt \'a rik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics, pp.\ 4519--4529. PMLR, 2020

    2020

  9. [17]

    A unified theory of decentralized sgd with changing topology and local updates

    Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pp.\ 5381--5393. PMLR, 2020

    2020

  10. [18]

    Slowcal-sgd: Slow query points improve local-sgd for stochastic convex optimization

    Kfir Y Levy. Slowcal-sgd: Slow query points improve local-sgd for stochastic convex optimization. arXiv preprint arXiv:2304.04169, 2023

    2023 arXiv

  11. [19]

    Don't use large mini-batches, use local sgd

    Tao Lin, Sebastian Urban Stich, Kumar Kshitij Patel, and Martin Jaggi. Don't use large mini-batches, use local sgd. In Proceedings of the 8th International Conference on Learning Representations, 2019

    2019

  12. [20]

    Efficient large-scale distributed training of conditional maximum entropy models

    Ryan Mcdonald, Mehryar Mohri, Nathan Silberman, Dan Walker, and Gideon Mann. Efficient large-scale distributed training of conditional maximum entropy models. Advances in neural information processing systems, 22, 2009

    2009

  13. [21]

    Distributed training strategies for the structured perceptron

    Ryan McDonald, Keith Hall, and Gideon Mann. Distributed training strategies for the structured perceptron. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pp.\ 456--464. Association for ...

    2010

  14. [22]

    Communication-Efficient Learning of Deep Networks from Decentralized Data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data . In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference on Artificial Intelligence an...

    2017

  15. [23]

    Proximal and federated random reshuffling

    Konstantin Mishchenko, Ahmed Khaled, and Peter Richt \'a rik. Proximal and federated random reshuffling. In International Conference on Machine Learning, pp.\ 15718--15749. PMLR, 2022

    2022

  16. [24]

    Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate

    Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent on separable data: Exact convergence with a fixed learning rate. In The 22nd International Conference on Artificial Intelligence and Statistics, pp.\ 3051--3059. PMLR, 2019

    2019

  17. [25]

    Introductory lectures on convex optimization: A basic course, volume 87

    Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2013

    2013

  18. [26]

    A minimizer far, far away, 2024

    Francesco Orabona. A minimizer far, far away, 2024. URL https://parameterfree.com/2024/02/14/a-minimizer-far-far-away/

    2024

  19. [27]

    On the still unreasonable effectiveness of federated averaging for heterogeneous distributed learning

    Kumar Kshitij Patel, Margalit Glasgow, Lingxiao Wang, Nirmit Joshi, and Nathan Srebro. On the still unreasonable effectiveness of federated averaging for heterogeneous distributed learning. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and...

    2023

  20. [28]

    The limits and potentials of local sgd for distributed heterogeneous learning with intermittent communication

    Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari, Lingxiao Wang, Sebastian U Stich, Ziheng Cheng, Nirmit Joshi, and Nathan Srebro. The limits and potentials of local sgd for distributed heterogeneous learning with intermittent communication. In Shipra Agrawal and Aaron Roth ...

    2024

  21. [29]

    Distributed stochastic optimization and learning

    Ohad Shamir and Nathan Srebro. Distributed stochastic optimization and learning. In 2014 52nd Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp.\ 850--857. IEEE, 2014

    2014

  22. [30]

    The implicit bias of gradient descent on separable data

    Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19 0 (70): 0 1--57, 2018. URL http://jmlr.org/papers/v19/18-188.html

    2018

  23. [31]

    Local sgd converges fast and communicates little

    Sebastian U Stich. Local sgd converges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018

    2018 arXiv

  24. [32]

    Local sgd converges fast and communicates little

    Sebastian Urban Stich. Local sgd converges fast and communicates little. In ICLR 2019-International Conference on Learning Representations, 2019

    2019

  25. [33]

    A field guide to federated optimization

    Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917, 2021

    2021 arXiv

  26. [34]

    On the unreasonable effectiveness of federated averaging with heterogeneous data

    Jianyu Wang, Rudrajit Das, Gauri Joshi, Satyen Kale, Zheng Xu, and Tong Zhang. On the unreasonable effectiveness of federated averaging with heterogeneous data. arXiv preprint arXiv:2206.04723, 2022

    2022 arXiv

  27. [35]

    Is local sgd better than minibatch sgd? In International Conference on Machine Learning, pp.\ 10334--10343

    Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In International Conference on Machine Learning, pp.\ 10334--10343. PMLR, 2020 a

    2020

  28. [36]

    Graph oracle models, lower bounds, and gaps for parallel stochastic optimization

    Blake E Woodworth, Jialei Wang, Adam Smith, Brendan McMahan, and Nati Srebro. Graph oracle models, lower bounds, and gaps for parallel stochastic optimization. Advances in neural information processing systems, 31, 2018

    2018

  29. [37]

    Minibatch vs local sgd for heterogeneous distributed learning

    Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems, 33: 0 6281--6292, 2020 b

    2020

  30. [38]

    The min-max complexity of distributed stochastic convex optimization with intermittent communication

    Blake E Woodworth, Brian Bullins, Ohad Shamir, and Nathan Srebro. The min-max complexity of distributed stochastic convex optimization with intermittent communication. In Conference on Learning Theory, pp.\ 4386--4437. PMLR, 2021

    2021

  31. [39]

    Large stepsize gradient descent for logistic loss: Non-monotonicity of the loss improves optimization efficiency

    Jingfeng Wu, Peter L Bartlett, Matus Telgarsky, and Bin Yu. Large stepsize gradient descent for logistic loss: Non-monotonicity of the loss improves optimization efficiency. arXiv preprint arXiv:2402.15926, 2024 a

    2024 arXiv

  32. [40]

    Implicit bias of gradient descent for logistic regression at the edge of stability

    Jingfeng Wu, Vladimir Braverman, and Jason D Lee. Implicit bias of gradient descent for logistic regression at the edge of stability. Advances in Neural Information Processing Systems, 36, 2024 b

    2024

  33. [41]

    Federated accelerated stochastic gradient descent

    Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems, 33: 0 5332--5344, 2020

    2020

  34. [42]

    Information-theoretic lower bounds for distributed statistical estimation with communication constraints

    Yuchen Zhang, John Duchi, Michael I Jordan, and Martin J Wainwright. Information-theoretic lower bounds for distributed statistical estimation with communication constraints. Advances in Neural Information Processing Systems, 26, 2013

    2013

  35. [43]

    Parallelized stochastic gradient descent

    Martin Zinkevich, Markus Weimer, Lihong Li, and Alex J Smola. Parallelized stochastic gradient descent. In Advances in neural information processing systems, pp.\ 2595--2603, 2010

    2010

  36. [44]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  37. [45]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  38. [46]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  39. [47]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.