Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

What Makes Local Updates Effective: The Role of Data Heterogeneity and Smoothness

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This thesis proves that small second-order heterogeneity—the extent to which clients' objectives disagree in curvature—is both necessary and sufficient for local update algorithms to outperform centralized or mini-batch methods.

desk verdict A solid synthesis of prior work on second-order heterogeneity, but the 'necessary and sufficient' claim outstrips the theorems: sufficiency needs globally small tau, and the manuscript is unfinished. read the letter →

arxiv 2507.00195 v1 pith:IPOHDGXN submitted 2025-06-30 cs.LG cs.AIcs.MAmath.OCstat.ML

Kumar Kshitij Patel This is my paper
classification cs.LGcs.AIcs.MAmath.OCstat.ML MSC 68Q2568W4090C2590C60
keywords localSGDfederatedlearningdataheterogeneitysecond-orderconsensuserrorcommunicationcomplexitymin-maxoptimalitythird-ordersmoothness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The thesis argues that the success of local-update algorithms such as Local SGD in federated learning is governed by second-order data heterogeneity: the extent to which clients' objective functions disagree in curvature. The central claim is that a small second-order heterogeneity bound $\tau$ is both necessary and sufficient for local updates to strictly outperform centralized or mini-batch methods, in convex, non-convex, and online settings. A sympathetic reader should care because this replaces vague 'data heterogeneity' intuitions with a parameter that provably controls communication complexity, and it identifies when the dominant practice of local averaging is theoretically justified.

What carries the argument

The central object is second-order heterogeneity $\tau$ (Assumption 11), a global uniform bound on Hessian disagreement between any two clients. It carries the argument because it enters into sharper consensus-error recursions—quantities measuring how far local models drift apart between communication rounds—which are coupled to iterate sub-optimality and tracked in second and fourth moments. A second central identity is the fixed-point analysis for quadratics: Local SGD converges to $x_\infty = (1/M)\sum_m C^{-1}C_m x^\star_m$ with $C_m = I - (I - \eta A_m)^K$, and its discrepancy from the true optimum $x^\star$ is bounded by $\zeta_\star \tau/\mu$, which goes to zero only when $\tau$ or $\zeta_\star$ does.

What would settle it

Construct a smooth, strongly convex two-client problem whose Hessian disagreement is at most $\tau$ inside a small ball around the shared optimum but large (order $H$) outside it, run Local SGD from a distant initialization, and measure communication rounds to reach accuracy $\epsilon$. If the observed complexity fails to match the $\tau$-dependent upper bound predicted by the theory (for example, it behaves like the mini-batch rate), then the global-uniform $\tau$ assumption is doing the work and the stated necessity/sufficiency claim does not survive under local-only heterogeneity.

Watch

Extended reading notes

Core claim

The paper's central claim is that local update algorithms outperform centralized or mini-batch methods exactly when the Hessians of clients' objectives are close: $\sup_{m,n,x}\|\nabla^2 F_m(x) - \nabla^2 F_n(x)\| \le \tau$ with small $\tau$. It proves this by establishing tight lower bounds showing that under only first-order heterogeneity (small gradient disagreement or shared optimizers) no local algorithm beats accelerated mini-batch SGD, whose rate is independent of heterogeneity, and by constructing matching upper bounds showing that when $\tau$ is small, Local SGD's communication complexity improves, for example $R(\epsilon) = \tilde{O}(H\tau/\mu^2 + \tau H \phi_\star/(\mu^2\sqrt{\epsilon}) + H^2 \zeta_\star/(\mu^2\sqrt{\epsilon}))$ in the strongly convex regime. In the non-convex setting, the new algorithm CE-LSGD is shown to be minimax optimal under exact oracles and nearly optimal with stochastic oracles, with rates that improve as $\tau$ decreases.

Load-bearing premise

The load-bearing premise is Assumption 11: the Hessians of every pair of clients must differ by at most $\tau$ uniformly over the entire space, so that 'low heterogeneity' holds globally, not just near the optima where local updates actually operate.

Editorial extensions

If this is right

  • Communication complexity of Local SGD provably improves as $\tau$ decreases, interpolating to homogeneous problems where a constant number of communication rounds suffices for quadratic objectives.
  • Under first-order heterogeneity alone, accelerated mini-batch SGD is min-max optimal, so no local-update benefit exists without second-order structure.
  • In the non-convex setting, CE-LSGD achieves minimax optimal rates under exact oracles and nearly optimal rates with stochastic oracles, with improvements governed by $\tau$.
  • In online optimization, collaboration gives no worst-case benefit with full gradient feedback, but provably helps under bandit feedback in high-dimensional or low-heterogeneity regimes.
  • Third-order smoothness extends the known homogeneous benefits to heterogeneous regimes, yielding tighter finite-time bounds under small $Q$ and $\tau$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A practical diagnostic suggests itself: estimate $\tau$ (curvature disagreement across clients) to decide whether increasing local steps $K$ or reducing communication will help in a given federated deployment.
  • The fixed-point analysis implies an implicit spectral filtering: local updates down-weight high-curvature directions, potentially connecting Local SGD to regularization and generalization—an interpretation the author draws, and one that invites testing on overparameterized models.
  • The global-uniform nature of $\tau$ may be relaxable; if the theory can be rebuilt with local bounds near the optima, its applicability to real data, where Hessian disagreement is often large far from the optimum, would expand.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper is a Ph.D. thesis that develops a unified theoretical account of local update algorithms, principally Local SGD, under heterogeneity measures that separate first- and second-order effects. The central claim is that small second-order heterogeneity, measured by the parameter τ, is both necessary and sufficient for local updates to outperform centralized or mini-batch methods in convex and non-convex settings. The thesis contributes lower bounds (Theorems 1–3, 13, 15), consensus-error-based upper bounds (Theorems 5–12), a communication-efficient algorithm CE-LSGD (Theorem 14), and extensions to online and bandit convex optimization (Theorems 17–21). The technical core is a fine-grained consensus-error framework that couples iterate error with local drift and introduces fourth-moment bounds under third-order smoothness. All results are drawn from four of the author's own prior papers, as disclosed in Section 1.3.

Significance. If the main claims hold, the thesis provides a coherent, falsifiable theory of when local updates are provably advantageous, with communication-complexity predictions that scale with τ and are empirically supported on synthetic linear regression and CIFAR-10 experiments. The strength of the work is that the upper and lower bounds are derived from explicit assumptions rather than fitted to data, and the lower-bound constructions carefully decouple τ from the smoothness constant H. The consensus-error recursions with fourth-moment control are technically substantive. However, the central 'necessary and sufficient' claim is currently established only for globally small τ, and several upper bounds are admitted to be loose or unmatched by lower bounds, so the significance is conditional on closing or qualifying these gaps.

major comments (4)
  1. [Section 2.5.1, Assumption 11; Chapter 5, Lemmas 6–7] The sufficiency half of the central claim is established only for globally small τ. Lemma 6's consensus-error recursion bounds local drift using the global uniform bound sup_{m,n,x} ||∇²F_m(x) − ∇²F_n(x)|| ≤ τ, and the proofs of Theorems 5–12 inherit this global requirement. The lower-bound constructions in Theorems 1–3 and 15, however, are quadratics or near-quadratics on which global and local τ coincide. Consequently, the abstract and Section 1.1's statement that small second-order heterogeneity is 'necessary and sufficient' is not proven for the practically relevant interpretation of small τ only in the region visited by the iterates; a construction with small τ near the optima but large away from them would expose the gap, since the current proofs do not control drift during early rounds or under large step sizes.
  2. [Section 5.2–5.3, Theorem 9] The sufficiency of small τ for general convex objectives is not established. Theorem 9, the most general result, retains terms depending on ζ☆ and φ☆ that do not vanish as τ → 0, and the text explicitly admits that it does not recover the quadratic convergence rate when Q = 0 and that the bound may contain extraneous terms. This means the strong form of the central claim holds only for quadratics (Q = 0), for third-order-smooth cases under additional restrictions, or under the restrictive Assumption 12. The abstract's unqualified statement should be revised to reflect that small τ alone is not shown to suffice in general smooth convex settings.
  3. [Section 6.2.2, Theorems 14–15] The stochastic non-convex upper bound for CE-LSGD contains a ΔH/√(KR) term, while the lower bound in Theorem 15 has ΔH/(KR); the resulting gap is acknowledged in Section 6.2.2 but left open. Because Table 6.1 presents these rates side by side and the chapter claims 'nearly minimax optimal' performance, the near-optimality claim in the stochastic setting is stronger than what is proven. This is a load-bearing issue for the non-convex contribution, though it does not undermine the qualitative role of τ.
  4. [Section 3.3 and Chapter 5] The strongly convex upper bounds (Theorems 5–8) imply a communication complexity of the form R(ε) = O~(Hτ/μ² + τHφ☆/(μ²√ε) + H²ζ☆/(μ²√ε)), while the only matching-style lower bound (Theorem 3) is proved for non-strongly convex instances and gives R = Ω(τB²/ε) in the general convex class. Because these bounds are derived under different problem classes, the strongly convex min-max complexity is not actually characterized, contrary to the abstract's claim of 'min-max complexity of multiple problem classes'. The qualitative necessary/sufficient story survives, but the 'min-max' language overstates the degree of characterization.
minor comments (4)
  1. [Throughout main text] Many central theorems (e.g., Theorems 5–12) are presented as 'Informal' with precise statements only in appendices; for a self-contained article, the main text should include full theorem statements or clearly indicate that the appendix versions are the official ones.
  2. [Figure 6.1] The figure contains a leftover annotation 'TODO: change smoothness constant' that should be removed before publication.
  3. [Theorem 3 and Theorem 1] There are notational inconsistencies: Theorem 3 writes F(x_{KR}) although the iterate is denoted \bar{x}_R elsewhere, and Theorem 1 uses the constant name c6 that already appears in equation (3.4), which can confuse the reader.
  4. [Section 1.3] The thesis discloses that all results are drawn from four prior papers by the author; if this document is intended as a stand-alone journal submission, the overlap with those published papers should be addressed in a separate novelty statement, not only as a footnote.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the upper and lower bounds are independently derived from stated assumptions, and the self-citations are disclosed provenance rather than load-bearing argument.

full rationale

The thesis's central claim is that small second-order heterogeneity tau is necessary and sufficient for local-update algorithms to outperform centralized or mini-batch methods. This claim is supported by two logically independent components: parameter-free upper bounds (Theorems 5-9 and 14) derived from explicit assumptions such as Assumptions 2, 4-11, and lower bounds (Theorems 1-3, 13, and 15) built on separate hard instances, including quadratics and the Carmon et al. non-convex construction. The quantity tau enters as an assumption parameter, not as a fitted value, and communication-complexity statements such as R(epsilon) = O~(H tau / mu^2 + ...) follow from solving the stated consensus-error recursions rather than from defining tau in terms of the target rate. No equation in the paper defines tau, zeta_star, or phi_star so that a claimed rate is forced by construction; the lower-bound instances are designed to realize tau, while the upper bounds apply to all instances satisfying Assumption 11. The explicit statement in Section 1.3 that all results derive from four papers by the primary author is a statement of provenance, and the cited works include externally anchored results (e.g., Woodworth et al., Carmon et al., Arjevani et al.) with proofs carried out in the appendices. The concern about Assumption 11 being global while the lower-bound constructions are quadratic is a limitation of the sufficiency claim's scope, not a circularity: global tau being small is what the proofs require, and local tau being small is not established as sufficient. Empirical sections confirm the framework's predictions but are not used as inputs to the derivations. No load-bearing step reduces to a self-citation, a fitted input, or a definitional equivalence, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

No numerical constants are fitted to data; the parameters H, mu, tau, zeta_star, phi_star, sigma_2, and sigma_4 are assumed inputs from the problem class. Algorithm step sizes are chosen explicitly in the theorem statements. All listed axioms are stated assumptions of the theorems rather than hidden background facts.

assumptions (6)
  • domain assumption Assumption 10: for each machine there exists a global minimizer within phi_star of a local minimizer
    Used as the relaxed first-order heterogeneity condition in lower bounds and consensus-error upper bounds throughout Chapter 5.
  • domain assumption Assumption 11: sup_{m,n,x} ||nabla^2 F_m(x) - nabla^2 F_n(x)|| <= tau for all machine pairs
    The central second-order heterogeneity measure that drives all main upper and lower bound claims.
  • domain assumption Assumption 12: uniform gradient heterogeneity sup_{m,n,x} ||nabla F_m(x) - nabla F_n(x)|| <= H zeta
    Used in Theorems 10-12 and in non-convex lower bounds; Proposition 2 shows it restricts curvature to be equal across machines.
  • domain assumption Assumption 5: third-order smoothness, i.e., Hessians are Lipschitz with constant Q
    Enables improved rates in homogeneous and heterogeneous settings; Q appears in Theorems 9-12.
  • domain assumption Definition 3: lower bounds are restricted to distributed zero-respecting algorithms
    The hard-instance constructions only apply within A_ZR; min-max optimality claims such as Theorem 2 are relative to this restricted algorithm class.
  • domain assumption Assumptions 6 and 7: bounded fourth and second moments of stochastic gradients
    Used to control stochastic noise in both upper and lower bounds; the distinction between sigma_2 and sigma_4 is exploited in the rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of What Makes Local Updates Effective: The Role of Data Heterogeneity and Smoothness." pith.science (2026). https://pith.science/paper/IPOHDGXN

@misc{pith2026250700195,
  author       = {Pith},
  title        = {Pith review of: What Makes Local Updates Effective: The Role of Data Heterogeneity and Smoothness},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IPOHDGXN}},
  note         = {Machine review of arXiv:2507.00195}
}
read the original abstract

This thesis contributes to the theoretical understanding of local update algorithms, especially Local SGD, in distributed and federated optimization under realistic models of data heterogeneity. A central focus is on the bounded second-order heterogeneity assumption, which is shown to be both necessary and sufficient for local updates to outperform centralized or mini-batch methods in convex and non-convex settings. The thesis establishes tight upper and lower bounds in several regimes for various local update algorithms and characterizes the min-max complexity of multiple problem classes. At its core is a fine-grained consensus-error-based analysis framework that yields sharper finite-time convergence bounds under third-order smoothness and relaxed heterogeneity assumptions. The thesis also extends to online federated learning, providing fundamental regret bounds under both first-order and bandit feedback. Together, these results clarify when and why local updates offer provable advantages, and the thesis serves as a self-contained guide for analyzing Local SGD in heterogeneous environments.

Figures

Figures reproduced from arXiv: 2507.00195 by the authors.

Figure 2.1
Figure 2.1. Illustration of the intermittent communication setting. distributions {Dm ∈ ∆(Z)}m∈[M] and a differentiable loss function f(·; z ∈ Z) : R d → R and assume it belongs to some function class f ∈ F. With this we can denote the set of all problem instances by P ∈ ∆(Z) ⊗M × F. In the rest of this section, we will define different restrictions on F and distributions {D1, . . . , DM}, which would lead to interesting sub-pr… view at source ↗
Figure 2.2
Figure 2.2. Illustration of the functions in Proposition 1. Proof. Our construction uses square loss, which makes Fm’s quadratic functions. Note that assuming every data-point is z = (β, y) we can denote the hessian of machine m’s objective as, ∇2Fm(·) = E(β,y) [PITH_FULL_IMAGE:figures/full_fig_p022_2_2.png] view at source ↗
Figure 3.1
Figure 3.1. The class of convex and third-order smooth problems satisfying Assumptions 1 and 5 interpo￾lates between the class of problems with quadratic objective functions and convex-smooth functions satisfying Assumptions 1 and 4. Notably, the lower bound of Woodworth et al. [157] is a non-quadratic instance. this assumption, Yuan and Ma [160] derived the following convergence guarantee for Local SGD: E [PITH_FULL_IMAGE:fig… view at source ↗
Figures from the paper (7 more)
Figure 4.1
Figure 4.1. Figure 4.1: Illustration of a two-dimensional optimization problem with M = 5 machines, each with a 1-strongly convex and 6-smooth objective. On the left, we draw the contour lines for each machine’s objective as well as for the average objective. We also indicate the two releva…
Figure 4.2
Figure 4.2. Figure 4.2: Illustration of the same distributed problem as [PITH_FULL_IMAGE:figures/full_fig_p053_4_2.png]
Figure 4.3
Figure 4.3. Figure 4.3: The effect of having an outlier with a sharp curvature on Local SGD’s fixed point with progres￾sively higher local update steps. Thus the main difference with respect to Local SGD with K > 1, is a different geometry on each machine defined by Am as opposed to Cm of L…
Figure 5.1
Figure 5.1. Figure 5.1: Impact of First- and Second-Order Heterogeneity on Local SGD. In both figures, we use d = 5, M = 20 clients, K = 10 local steps, and a noise level of σnoise = 0.1. The step-size is tuned over a logarithmic grid in [10−3 , 10−1 ], and the error is averaged over multip…
Figure 6.1
Figure 6.1. Figure 6.1: Illustration of the best communication complexity R and oracle complexity N that our method can obtain for different ϵ and τ . Green regime: Our method can obtain optimal communication and oracle complexities. Orange regime: Our method achieves optimal communication …
Figure 6.2
Figure 6.2. Figure 6.2: Training loss of CE-LSGD and BVR-L-SGD on CIFAR-10 data-set versus the number of communication rounds in the intermittent communication setting with different local-updates K. We use M = 10 machines, and synthetically generate heterogeneous data-sets (see Section 6.3…
Figure 6.3
Figure 6.3. Figure 6.3: Comparing CE-LSGD to centralized and local-update methods, for fixed K = 32 and varying heterogeneity controlled by q on CIFAR-10 data-set. Like [PITH_FULL_IMAGE:figures/full_fig_p083_6_3.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. What's in a Smoothness Constant? Tighter Rates for Local SGD with Bounded Second-order Heterogeneity

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Local SGD provably improves over Mini-batch SGD under bounded second-order heterogeneity in the general convex setting, with nearly tight upper and lower bounds.

Reference graph

Works this paper leans on

164 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    The de-democratization of ai: Deep learning and the compute divide in artificial intelligence research

    Nur Ahmed and Muntasir Wahed. The de-democratization of ai: Deep learning and the compute divide in artificial intelligence research. arXiv preprint arXiv:2010.15581 , 2020

    arXiv 2010

  2. [2]

    Byzantine stochastic gradient descent

    Dan Alistarh, Zeyuan Allen-Zhu, and Jerry Li. Byzantine stochastic gradient descent. Advances in neural information processing systems , 31, 2018

    2018

  3. [3]

    The convergence of sparsified gradient methods

    Dan Alistarh, Torsten Hoefler, Mikael Johansson, Nikola Konstantinov, Sarit Khirirat, and Cedric Renggli. The convergence of sparsified gradient methods. In Proceedings of Advances in Neural Infor- mation Processing Systems, 2018

    2018

  4. [4]

    Is federated learning still alive in the foundation model era? In AAAI Spring Symposium, 2024

    Nathalie Baracaldo Angel. Is federated learning still alive in the foundation model era? In AAAI Spring Symposium, 2024

    2024

  5. [5]

    Generative ai has an intellectual property problem

    Gil Appel, Juliana Neelbauer, and David A Schweidel. Generative ai has an intellectual property problem. Harvard Business Review , 7, 2023

    2023

  6. [6]

    Designing for privacy - wwdc19 - videos, 2019

    Apple. Designing for privacy - wwdc19 - videos, 2019. URL https://developer.apple.com/videos/ play/wwdc2019/708

    2019

  7. [7]

    Communication complexity of distributed convex learning and optimization

    Yossi Arjevani and Ohad Shamir. Communication complexity of distributed convex learning and optimization. Advances in neural information processing systems , 28, 2015

    2015

  8. [8]

    Lower bounds for non-convex stochastic optimization

    Yossi Arjevani, Yair Carmon, John C Duchi, Dylan J Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. arXiv preprint arXiv:1912.02365 , 2019

    arXiv 1912

Show all 164 references
  1. [9]

    Self-concordant analysis for logistic regression

    Francis Bach. Self-concordant analysis for logistic regression. Electronic Journal of Statistics , 4:384– 414, 2010. ISSN 1935-7524. doi: 10.1214/09-[]EJS521. URL https://doi.org/10.1214/09-[]EJS521

    2010 doi

  2. [10]

    Implicit gradient alignment in distributed and federated learning

    Luis Barba, Martin Jaggi, and Yatin Dandi. Implicit gradient alignment in distributed and federated learning. In AAAI Conference on Artificial Intelligence, AAAI , volume 22, 2021. 214

    2021

  3. [11]

    A model of inductive bias learning

    Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research , 12: 149–198, 2000

    2000

  4. [12]

    On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 610–623, 2021

    Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 610–623, 2021

    2021

  5. [13]

    The compute divide in machine learning: A threat to academic contribution and scrutiny? arXiv preprint arXiv:2401.02452 , 2024

    Tamay Besiroglu, Sage Andrus Bergerson, Amelia Michael, Lennart Heim, Xueyun Luo, and Neil Thompson. The compute divide in machine learning: A threat to academic contribution and scrutiny? arXiv preprint arXiv:2401.02452 , 2024

    2024 arXiv

  6. [14]

    Collaborative pac learning

    Avrim Blum, Nika Haghtalab, Ariel D Procaccia, and Mingda Qiao. Collaborative pac learning. Advances in Neural Information Processing Systems , 30, 2017

    2017

  7. [15]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021

    2021 arXiv

  8. [16]

    Reinforcement learning, efficient coding, and the statistics of natural tasks

    Matthew Botvinick, Ari Weinstein, Alec Solway, and Andrew Barto. Reinforcement learning, efficient coding, and the statistics of natural tasks. Current opinion in behavioral sciences , 5:71–77, 2015

    2015

  9. [17]

    The computational and neural basis of cognitive control: charted territory and new frontiers

    Matthew M Botvinick and Jonathan D Cohen. The computational and neural basis of cognitive control: charted territory and new frontiers. Cognitive science, 38(6):1249–1285, 2014

    2014

  10. [18]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020

    2005 arXiv

  11. [19]

    Regret analysis of stochastic and nonstochastic multi- armed bandit problems

    S´ ebastien Bubeck, Nicolo Cesa-Bianchi, et al. Regret analysis of stochastic and nonstochastic multi- armed bandit problems. Foundations and Trends® in Machine Learning , 5(1):1–122, 2012

    2012

  12. [20]

    Convex optimization: Algorithms and complexity

    S´ ebastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends® in Machine Learning , 8(3-4):231–357, 2015

    2015

  13. [21]

    Highly smooth minimization of non-smooth problems

    Brian Bullins. Highly smooth minimization of non-smooth problems. In Conference on Learning Theory, pages 988–1030. PMLR, 2020

    2020

  14. [22]

    A stochastic newton algorithm for distributed convex optimization

    Brian Bullins, Kshitij Kumar Patel, Ohad Shamir, Nathan Srebro, and Blake E Woodworth. A stochastic newton algorithm for distributed convex optimization. Advances in Neural Information Processing Systems, 34, 2021

    2021

  15. [23]

    Lower bounds for finding stationary points i

    Yair Carmon, John C Duchi, Oliver Hinder, and Aaron Sidford. Lower bounds for finding stationary points i. Mathematical Programming, 184(1):71–120, 2020. 215

    2020

  16. [24]

    Acceleration with a ball optimization oracle

    Yair Carmon, Arun Jambulapati, Qijia Jiang, Yujia Jin, Yin Tat Lee, Aaron Sidford, and Kevin Tian. Acceleration with a ball optimization oracle. In Advances in Neural Information Processing Systems , volume 33, 2020

    2020

  17. [25]

    Multitask learning

    Rich Caruana. Multitask learning. Machine learning, 28:41–75, 1997

    1997

  18. [26]

    On the outsized importance of learning rates in local update methods

    Zachary Charles and Jakub Konecny. On the outsized importance of learning rates in local update methods. arXiv preprint arXiv:2007.00878 , 2020

    2007 arXiv

  19. [27]

    On large- cohort training for federated learning

    Zachary Charles, Zachary Garrett, Zhouyuan Huo, Sergei Shmulyian, and Virginia Smith. On large- cohort training for federated learning. Advances in neural information processing systems , 34:20461– 20475, 2021

    2021

  20. [28]

    Federated learning of out-of- vocabulary words

    Mingqing Chen, Rajiv Mathews, Tom Ouyang, and Francoise Beaufays. Federated learning of out-of- vocabulary words. arXiv preprint arXiv:1903.10635 , 2019

    1903 arXiv

  21. [29]

    Fl-qsar: a federated learning-based qsar prototype for collaborative drug discovery

    Shaoqi Chen, Dongyu Xue, Guohui Chuai, Qiang Yang, and Qi Liu. Fl-qsar: a federated learning-based qsar prototype for collaborative drug discovery. Bioinformatics, 36(22-23):5492–5498, 2020

    2020

  22. [30]

    Opportunities and obstacles for deep learning in biology and medicine

    Travers Ching, Daniel S Himmelstein, Brett K Beaulieu-Jones, Alexandr A Kalinin, Brian T Do, Gregory P Way, Enrico Ferrero, Paul-Michael Agapow, Michael Zietz, Michael M Hoffman, et al. Opportunities and obstacles for deep learning in biology and medicine. Journal of the Royal...

    2018

  23. [31]

    Machine learning needs big data to revolutionise drug discovery

    Matthew Clark. Machine learning needs big data to revolutionise drug discovery. Drug Discovery World, 2021. Accessed: 2024-11-10

    2021

  24. [32]

    Cognitive control over learning: creating, clustering, and generalizing task-set structure

    Anne GE Collins and Michael J Frank. Cognitive control over learning: creating, clustering, and generalizing task-set structure. Psychological review, 120(1):190, 2013

    2013

  25. [33]

    Momentum-based variance reduction in non-convex sgd

    Ashok Cutkosky and Francesco Orabona. Momentum-based variance reduction in non-convex sgd. Advances in neural information processing systems , 32, 2019

    2019

  26. [34]

    Addressing modern and practical challenges in machine learning: A survey of online federated and transfer learning

    Shuang Dai and Fanlin Meng. Addressing modern and practical challenges in machine learning: A survey of online federated and transfer learning. arXiv preprint arXiv:2202.03070 , 2022

    2022 arXiv

  27. [35]

    Federated learning for predicting clinical outcomes in patients with covid-19

    Ittai Dayan, Holger R Roth, Aoxiao Zhong, Ahmed Harouni, Amilcare Gentili, Anas Z Abidin, Andrew Liu, Anthony Beardsworth Costa, Bradford J Wood, Chien-Sung Tsai, et al. Federated learning for predicting clinical outcomes in patients with covid-19. Nature medicine, 27(10):1735...

    2021

  28. [36]

    Optimal distributed online prediction using mini-batches

    Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, and Lin Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research , 13(1), 2012

    2012

  29. [37]

    Communication trade-offs for local-sgd with large step size

    Aymeric Dieuleveut and Kumar Kshitij Patel. Communication trade-offs for local-sgd with large step size. Advances in Neural Information Processing Systems , 32, 2019

    2019

  30. [38]

    Differentially-private federated linear bandits

    Abhimanyu Dubey and AlexSandy’ Pentland. Differentially-private federated linear bandits. Advances in Neural Information Processing Systems , 33:6003–6014, 2020

    2020

  31. [39]

    Optimal rates for zero- order convex optimization: The power of two function evaluations

    John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero- order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, 61(5):2788–2806, 2015

    2015

  32. [40]

    The multiple-demand (md) system of the primate brain: mental programs for intelligent behaviour

    John Duncan. The multiple-demand (md) system of the primate brain: mental programs for intelligent behaviour. Trends in cognitive sciences, 14(4):172–179, 2010

    2010

  33. [41]

    Federated learning in vehicular networks

    Ahmet M Elbir, Burak Soner, and Sinem Coleri. Federated learning in vehicular networks. arXiv preprint arXiv:2006.01412, 2020

    2006 arXiv

  34. [42]

    European Parliament and Council of the European Union. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repeali...

    2016

  35. [43]

    Spider: Near-optimal non-convex op- timization via stochastic path-integrated differential estimator

    Cong Fang, Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non-convex op- timization via stochastic path-integrated differential estimator. Advances in Neural Information Pro- cessing Systems, 31, 2018

    2018

  36. [44]

    Model-agnostic meta-learning for fast adaptation of deep networks

    Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126–1135. PMLR, 2017

    2017

  37. [45]

    Online convex optimization in the bandit setting: gradient descent without a gradient

    Abraham D Flaxman, Adam Tauman Kalai, and H Brendan McMahan. Online convex optimization in the bandit setting: gradient descent without a gradient. arXiv preprint cs/0408007 , 2004

    2004 arXiv

  38. [46]

    EControl: Fast distributed optimization with compression and error control

    Yuan Gao, Rustem Islamov, and Sebastian U Stich. EControl: Fast distributed optimization with compression and error control. In The Twelfth International Conference on Learning Representations , 2024. 217

    2024

  39. [47]

    Resource-aware asynchronous online federated learning for nonlinear regression

    Francois Gauthier, Vinay Chakravarthi Gogineni, Stefan Werner, Yih-Fang Huang, and Anthony Kuh. Resource-aware asynchronous online federated learning for nonlinear regression. In ICC 2022-IEEE International Conference on Communications , pages 2828–2833. IEEE, 2022

    2022

  40. [48]

    Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework

    Saeed Ghadimi and Guanghui Lan. Optimal stochastic approximation algorithms for strongly convex stochastic composite optimization i: A generic algorithmic framework. SIAM Journal on Optimization , 22(4):1469–1492, 2012

    2012

  41. [49]

    Ai and memory wall

    Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer. Ai and memory wall. IEEE Micro, 2024

    2024

  42. [50]

    Sharp bounds for federated averaging (local sgd) and continuous perspective

    Margalit R Glasgow, Honglin Yuan, and Tengyu Ma. Sharp bounds for federated averaging (local sgd) and continuous perspective. In International Conference on Artificial Intelligence and Statistics , pages 9050–9090. PMLR, 2022

    2022

  43. [51]

    Communication- efficient online federated learning framework for nonlinear regression

    Vinay Chakravarthi Gogineni, Stefan Werner, Yih-Fang Huang, and Anthony Kuh. Communication- efficient online federated learning framework for nonlinear regression. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 522...

    2022

  44. [52]

    Gene H. Golub. Cme 302: Numerical linear algebra, fall 2005/06 — lecture 10, 2005. URL https: //www2.stat.duke.edu/~mukee002/Lek-[]Heng/Golub_notes/notes10.pdf. Lecture notes

    2005

  45. [53]

    Your voice amp; audio data stays private while google assistant improves, 2023

    Google. Your voice amp; audio data stays private while google assistant improves, 2023. URL https: //support.google.com/assistant/answer/10176224?hl=en

    2023

  46. [54]

    Why (and when) does local sgd generalize better than sgd? arXiv preprint arXiv:2303.01215 , 2023

    Xinran Gu, Kaifeng Lyu, Longbo Huang, and Sanjeev Arora. Why (and when) does local sgd generalize better than sgd? arXiv preprint arXiv:2303.01215 , 2023

    2023 arXiv

  47. [55]

    On-demand sampling: Learning optimally from multiple distributions

    Nika Haghtalab, Michael Jordan, and Eric Zhao. On-demand sampling: Learning optimally from multiple distributions. Advances in Neural Information Processing Systems , 35:406–419, 2022

    2022

  48. [56]

    On the effect of defections in federated learning and how to prevent them

    Minbiao Han, Kumar Kshitij Patel, Han Shao, and Lingxiao Wang. On the effect of defections in federated learning and how to prevent them. arXiv preprint arXiv:2311.16459 , 2023. Under review

    2023 arXiv

  49. [57]

    How apple personalizes siri without hoovering up your data

    Karen Hao. How apple personalizes siri without hoovering up your data. Technology Review, 2020

    2020

  50. [58]

    Federated learning for mobile keyboard prediction

    Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Francoise Beaufays, Sean Au- genstein, Hubert Eichner, Chloe Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction. arXiv preprint arXiv:1811.03604 , 2018. 218

    2018 arXiv

  51. [59]

    Predicting text selections with federated learning, Nov 2021

    Florian Hartmann. Predicting text selections with federated learning, Nov 2021. URL https://ai. googleblog.com/2021/11/predicting-[]text-[]selections-[]with.html

    2021

  52. [60]

    Introduction to online convex optimization

    Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimiza- tion, 2(3-4):157–325, 2016

    2016

  53. [61]

    A simple and provably efficient algorithm for asynchronous federated contextual linear bandits

    Jiafan He, Tianhao Wang, Yifei Min, and Quanquan Gu. A simple and provably efficient algorithm for asynchronous federated contextual linear bandits. In Advances in Neural Information Processing Systems, 2022

    2022

  54. [62]

    Benchmarking neural network robustness to common corrup- tions and perturbations

    Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corrup- tions and perturbations. arXiv preprint arXiv:1903.12261 , 2019

    1903 arXiv

  55. [63]

    Federated linear contextual bandits

    Ruiquan Huang, Weiqiang Wu, Jing Yang, and Cong Shen. Federated linear contextual bandits. Advances in Neural Information Processing Systems , 34:27057–27068, 2021

    2021

  56. [64]

    Stabilized proximal-point methods for federated optimization

    Xiaowen Jiang, Anton Rodomanov, and Sebastian U Stich. Stabilized proximal-point methods for federated optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  57. [65]

    Federated optimization with doubly reg- ularized drift correction

    Xiaowen Jiang, Anton Rodomanov, and Sebastian U Stich. Federated optimization with doubly reg- ularized drift correction. In Proceedings of the 41st International Conference on Machine Learning , volume 235 of Proceedings of Machine Learning Research, pages 21912–21945. PMLR, 7 2024

    2024

  58. [66]

    Advances and open problems in federated learning

    Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurelien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Keith Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. corr. arXiv preprint arXiv:1912.04977 , 2019

    1912 arXiv

  59. [67]

    End-to-end privacy preserving deep learning on multi-institutional medical imaging

    Georgios Kaissis, Alexander Ziller, Jonathan Passerat-Palmbach, Theo Ryffel, Dmitrii Usynin, An- drew Trask, Ionesio Lima Jr, Jason Mancuso, Friederike Jungmann, Marc-Matthias Steinborn, et al. End-to-end privacy preserving deep learning on multi-institutional medical imaging....

    2021

  60. [68]

    Functional specificity in the human brain: a window into the functional architecture of the mind

    Nancy Kanwisher. Functional specificity in the human brain: a window into the functional architecture of the mind. Proceedings of the national academy of sciences , 107(25):11163–11170, 2010

    2010

  61. [69]

    Scaling laws for neural language models

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. 219

    2001 arXiv

  62. [70]

    Error feedback fixes signsgd and other gradient compression schemes

    Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes signsgd and other gradient compression schemes. In Proceedings of the 36th International Conference on Machine Learning (ICML 2019) , 2019

    2019

  63. [71]

    Mime: Mimicking centralized stochastic algorithms in federated learning

    Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning. arXiv preprint arXiv:2008.03606 , 2020

    2008 arXiv

  64. [72]

    Scaffold: Stochastic controlled averaging for federated learning

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In In- ternational Conference on Machine Learning , pages 5132–5143. PMLR, 2020

    2020

  65. [73]

    Learning from history for byzantine robust optimization

    Sai Praneeth Karimireddy, Lie He, and Martin Jaggi. Learning from history for byzantine robust optimization. In International conference on machine learning , pages 5311–5319. PMLR, 2021

    2021

  66. [74]

    Tighter theory for local sgd on identical and heterogeneous data

    Ahmed Khaled, Konstantin Mishchenko, and Peter Richtarik. Tighter theory for local sgd on identical and heterogeneous data. In International Conference on Artificial Intelligence and Statistics , pages 4519–4529. PMLR, 2020

    2020

  67. [75]

    A payload optimization method for federated recommender systems

    Farwa K Khan, Adrian Flanagan, Kuan Eeik Tan, Zareen Alamgir, and Muhammad Ammad-Ud-Din. A payload optimization method for federated recommender systems. In Fifteenth ACM Conference on Recommender Systems, pages 432–442, 2021

    2021

  68. [76]

    Stem: A stochastic two-sided momentum algorithm achieving near-optimal sample and com- munication complexities for federated learning

    Prashant Khanduri, Pranay Sharma, Haibo Yang, Mingyi Hong, Jia Liu, Ketan Rajawat, and Pramod Varshney. Stem: A stochastic two-sided momentum algorithm achieving near-optimal sample and com- munication complexities for federated learning. Advances in Neural Information Process...

    2021

  69. [77]

    A unified theory of decentralized sgd with changing topology and local updates

    Anastasia Koloskova, Nicolas Loizou, Sadra Boreiri, Martin Jaggi, and Sebastian Stich. A unified theory of decentralized sgd with changing topology and local updates. In International Conference on Machine Learning, pages 5381–5393. PMLR, 2020

    2020

  70. [78]

    Federated optimization: Distributed machine learning for on-device intelligence

    Jakub Konecny, H Brendan McMahan, Daniel Ramage, and Peter Richtarik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527 , 2016

    2016 arXiv

  71. [79]

    Optimal gradient sliding and its application to optimal distributed optimization under similarity

    Dmitry Kovalev, Aleksandr Beznosikov, Ekaterina Dmitrievna Borodich, Alexander Gasnikov, and Gesualdo Scutari. Optimal gradient sliding and its application to optimal distributed optimization under similarity. In Advances in Neural Information Processing Systems , 2022. 220

    2022

  72. [80]

    Learning multiple layers of features from tiny images.Citeseer, 2009

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.Citeseer, 2009

    2009

  73. [81]

    Imagenet classification with deep convolu- tional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems , 25, 2012

    2012

  74. [82]

    Real time kernel learning for sensor networks using principles of federated learning

    Anthony Kuh. Real time kernel learning for sensor networks using principles of federated learning. In 2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), pages 2089–2093. IEEE, 2021

    2021

  75. [83]

    A simpler approach to obtaining an o (1/t) convergence rate for the projected stochastic subgradient method

    Simon Lacoste-Julien, Mark Schmidt, and Francis Bach. A simpler approach to obtaining an o (1/t) convergence rate for the projected stochastic subgradient method. arXiv preprint arXiv:1212.2002 , 2012

    2002 arXiv

  76. [84]

    Asynchronous upper confidence bound algorithms for federated linear bandits

    Chuanhao Li and Hongning Wang. Asynchronous upper confidence bound algorithms for federated linear bandits. In International Conference on Artificial Intelligence and Statistics , pages 6529–6553. PMLR, 2022

    2022

  77. [85]

    Privacy-preserving federated brain tumour segmentation

    Wenqi Li, Fausto Milletari, Daguang Xu, Nicola Rieke, Jonny Hancox, Wentao Zhu, Maximilian Baust, Yan Cheng, Sebastien Ourselin, M Jorge Cardoso, et al. Privacy-preserving federated brain tumour segmentation. In International workshop on machine learning in medical imaging , p...

    2019

  78. [86]

    Fedrec++: Lossless federated recommendation with explicit feedback

    Feng Liang, Weike Pan, and Zhong Ming. Fedrec++: Lossless federated recommendation with explicit feedback. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 4224–4231, 2021

    2021

  79. [87]

    Analyzing implicit regularization in federated learning, 2024

    Jinwoo Lim, Sangyoon Yu, Suhyun Kim, and Soo-Mook Moon. Analyzing implicit regularization in federated learning, 2024

    2024

  80. [88]

    Don’t use large mini-batches, use local sgd

    Tao Lin, Sebastian U Stich, Kumar Kshitij Patel, and Martin Jaggi. Don’t use large mini-batches, use local sgd. arXiv preprint arXiv:1808.07217 , 2018

    2018 arXiv

  81. [89]

    Threats, attacks and defenses to federated learning: issues, taxonomy and perspectives

    Pengrui Liu, Xiangrui Xu, and Wei Wang. Threats, attacks and defenses to federated learning: issues, taxonomy and perspectives. Cybersecurity, 5(1):1–19, 2022

    2022

  82. [90]

    Revisiting the last-iterate convergence of stochastic gradient methods

    Zijian Liu and Zhengyuan Zhou. Revisiting the last-iterate convergence of stochastic gradient methods. arXiv preprint arXiv:2312.08531 , 2023. 221

    2023

  83. [91]

    High probability convergence of stochastic gradient methods

    Zijian Liu, Ta Duy Nguyen, Thien Hang Nguyen, Alina Ene, and Huy Nguyen. High probability convergence of stochastic gradient methods. In International Conference on Machine Learning , pages 21884–21914. PMLR, 2023

    2023

  84. [92]

    Lohn and Micah Musser

    Andrew J. Lohn and Micah Musser. Ai and compute: How much longer can computing power drive artificial intelligence progress. White paper, Center for Security and Emerging Technology (CSET), 2022

    2022

  85. [93]

    On maintaining linear convergence of distributed learning and optimization under limited communication

    Sindri Magn´ usson, Hossein Shokri-Ghadikolaei, and Na Li. On maintaining linear convergence of distributed learning and optimization under limited communication. In 2019 53rd Asilomar Conference on Signals, Systems, and Computers , pages 432–436, 2019. doi: 10.1109/IEEECONF44...

    2019

  86. [94]

    From local sgd to local fixed-point methods for federated learning

    Grigory Malinovskiy, Dmitry Kovalev, Elnur Gasanov, Laurent Condat, and Peter Richtarik. From local sgd to local fixed-point methods for federated learning. In International Conference on Machine Learning, pages 6692–6701. PMLR, 2020

    2020

  87. [95]

    Efficient large- scale distributed training of conditional maximum entropy models

    Ryan Mcdonald, Mehryar Mohri, Nathan Silberman, Dan Walker, and Gideon Mann. Efficient large- scale distributed training of conditional maximum entropy models. Advances in neural information processing systems, 22, 2009

    2009

  88. [96]

    Federated learning: Collaborative machine learn- ing without centralized training data, 4 2017

    Brendan McMahan and Daniel Ramage. Federated learning: Collaborative machine learn- ing without centralized training data, 4 2017. URL https://ai.googleblog.com/2017/04/ federated-[]learning-[]collaborative.html

    2017

  89. [98]

    Federated learning of deep networks using model averaging

    H Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Aguera y Arcas. Federated learning of deep networks using model averaging. arXiv preprint arXiv:1602.05629 , 2016

    2016 arXiv

  90. [99]

    Steps toward artificial intelligence

    Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8–30, 1961

    1961

  91. [100]

    Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning , pages 15750–15769

    Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richtarik. Proxskip: Yes! local gradient steps provably lead to communication acceleration! finally! In International Conference on Machine Learning , pages 15750–15769. PMLR, 2022. 222

    2022

  92. [101]

    Online federated learning

    Aritra Mitra, Hamed Hassani, and George J Pappas. Online federated learning. In 2021 60th IEEE Conference on Decision and Control (CDC) , pages 4083–4090. IEEE, 2021

    2021

  93. [102]

    Bias-variance reduced local sgd for less heterogeneous federated learning

    Tomoya Murata and Taiji Suzuki. Bias-variance reduced local sgd for less heterogeneous federated learning. arXiv preprint arXiv:2102.03198 , 2021

    2021 arXiv

  94. [103]

    Efficient methods in convex programming

    Arkadi Nemirovski. Efficient methods in convex programming. Lecture notes, 1994

    1994

  95. [104]

    Primal-dual subgradient methods for convex problems

    Yurii Nesterov. Primal-dual subgradient methods for convex problems. Mathematical programming, 120(1):221–259, 2009

    2009

  96. [105]

    Implementable tensor methods in unconstrained convex optimization

    Yurii Nesterov. Implementable tensor methods in unconstrained convex optimization. Mathematical Programming, pages 1–27, 2019

    2019

  97. [106]

    Interior-point polynomial algorithms in convex programming

    Yurii Nesterov and Arkadii Nemirovskii. Interior-point polynomial algorithms in convex programming . SIAM, 1994

    1994

  98. [107]

    Cubic regularization of Newton method and its global performance

    Yurii Nesterov and Boris T Polyak. Cubic regularization of Newton method and its global performance. Mathematical Programming, 108(1):177–205, 2006

    2006

  99. [108]

    Lectures on convex optimization , volume 137

    Yurii Nesterov et al. Lectures on convex optimization , volume 137. Springer, 2018

    2018

  100. [109]

    Human problem solving

    Allen Newell. Human problem solving. Upper Saddle River/Prentive Hall , 1972

    1972

  101. [110]

    Deep federated learning for autonomous driving

    Anh Nguyen, Tuong Do, Minh Tran, Binh X Nguyen, Chien Duong, Tu Phan, Erman Tjiputra, and Quang D Tran. Deep federated learning for autonomous driving. In 2022 IEEE Intelligent Vehicles Symposium (IV), pages 1824–1830. IEEE, 2022

    2022

  102. [111]

    Improved algorithms for collaborative pac learning

    Huy Nguyen and Lydia Zakynthinou. Improved algorithms for collaborative pac learning. Advances in Neural Information Processing Systems , 31, 2018

    2018

  103. [112]

    Sarah: A novel method for machine learning problems using stochastic recursive gradient

    Lam M Nguyen, Jie Liu, Katya Scheinberg, and Martin Takac. Sarah: A novel method for machine learning problems using stochastic recursive gradient. In International Conference on Machine Learn- ing, pages 2613–2621. PMLR, 2017

    2017

  104. [113]

    Numerical optimization

    Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006

    2006

  105. [114]

    Towards optimal communication complexity in distributed non-convex optimization

    Kumar Kshitij Patel, Lingxiao Wang, Blake Woodworth, Brian Bullins, and Nathan Srebro. Towards optimal communication complexity in distributed non-convex optimization. In Advances in Neural Information Processing Systems, 2022. 223

    2022

  106. [115]

    On the still unreasonable effectiveness of federated averaging for heterogeneous distributed learning

    Kumar Kshitij Patel, Margalit Glasgow, Lingxiao Wang, Nirmit Joshi, and Nathan Srebro. On the still unreasonable effectiveness of federated averaging for heterogeneous distributed learning. In Federated Learning and Analytics in Practice: Algorithms, Systems, Applications, and...

    2023

  107. [116]

    Federated online and bandit convex optimization

    Kumar Kshitij Patel, Lingxiao Wang, Aadirupa Saha, and Nathan Srebro. Federated online and bandit convex optimization. In International Conference on Machine Learning , pages 27439–27460. PMLR, 2023

    2023

  108. [117]

    The limits and potentials of local sgd for distributed heterogeneous learning with intermittent communication

    Kumar Kshitij Patel, Margalit Glasgow, Ali Zindari, Lingxiao Wang, Sebastian U Stich, Ziheng Cheng, Nirmit Joshi, and Nathan Srebro. The limits and potentials of local sgd for distributed heterogeneous learning with intermittent communication. In Shipra Agrawal and Aaron Roth,...

    2024

  109. [118]

    Revisiting consensus error: A fine-grained analysis of local sgd under second-order data heterogeneity

    Kumar Kshitij Patel, Ali Zindari, Sebastian Stich, and Lingxiao Wang. Revisiting consensus error: A fine-grained analysis of local sgd under second-order data heterogeneity. arxiv, 2025

    2025

  110. [119]

    Federated evaluation and tuning for on- device personalization: System design & applications

    Matthias Paulik, Matt Seigel, Henry Mason, Dominic Telaar, Joris Kluivers, Rogier van Dalen, Chi Wai Lau, Luke Carlson, Filip Granqvist, Chris Vandevelde, et al. Federated evaluation and tuning for on- device personalization: System design & applications. arXiv preprint arXiv:...

    2021 arXiv

  111. [120]

    Nvidia clara federated learning to deliver ai to hospitals while protecting patient data

    Kimberly Powell. Nvidia clara federated learning to deliver ai to hospitals while protecting patient data. Nvidia Blog, 2019

    2019

  112. [121]

    Exploring new chemical space for the treatments of tomorrow.American Pharmaceuti- cal Review, 2023

    Adityo Prakash. Exploring new chemical space for the treatments of tomorrow.American Pharmaceuti- cal Review, 2023. URL https://www.americanpharmaceuticalreview.com/Featured-[]Articles/ 597596-[]Exploring-[]New-[]Chemical-[]Space-[]for-[]the-[]Treatments-[]of-[]Tomorrow/. Acce...

    2023

  113. [122]

    Federated learning for breast density clas- sification: A real-world implementation

    Holger R Roth, Ken Chang, Praveer Singh, Nir Neumark, Wenqi Li, Vikash Gupta, Sharut Gupta, Liangqiong Qu, Alvin Ihsani, Bernardo C Bizzo, et al. Federated learning for breast density clas- sification: A real-world implementation. In Domain Adaptation and Representation Transf...

    2020

  114. [123]

    An overview of multi-task learning in deep neural networks

    S Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017

    2017 arXiv

  115. [124]

    Some studies in machine learning using the game of checkers

    Arthur L Samuel. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):210–229, 1959

    1959

  116. [125]

    J¨ urgen Schmidhuber.Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-... hook . PhD thesis, Technische Universit¨ at M¨ unchen, 1987

    1987

  117. [126]

    Fairness and abstraction in sociotechnical systems

    Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. Fairness and abstraction in sociotechnical systems. In Proceedings of the conference on fairness, ac- countability, and transparency, pages 59–68, 2019

    2019

  118. [127]

    Measuring the effects of data parallelism on neural network training

    Christopher J Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03600, 2018

    2018 arXiv

  119. [128]

    An optimal algorithm for bandit and zero-order convex optimization with two-point feedback

    Ohad Shamir. An optimal algorithm for bandit and zero-order convex optimization with two-point feedback. The Journal of Machine Learning Research , 18(1):1703–1713, 2017

    2017

  120. [130]

    Communication-efficient distributed optimization using an approximate newton-type method

    Ohad Shamir, Nati Srebro, and Tong Zhang. Communication-efficient distributed optimization using an approximate newton-type method. In International conference on machine learning , pages 1000–

  121. [131]

    Federated multi-armed bandits with personalization

    Chengshuai Shi, Cong Shen, and Jing Yang. Federated multi-armed bandits with personalization. In International Conference on Artificial Intelligence and Statistics , pages 2917–2925. PMLR, 2021

    2021

  122. [132]

    Federated learning through revolutionary technology ” consilient, 2 2021

    Gary Shiffman, Juan Zarate, Nikhil Deshpande, Raghuram Yeluri, and Parviz Peiravi. Federated learning through revolutionary technology ” consilient, 2 2021. URL https://consilient.com/ white-[]paper/federated-[]learning-[]through-[]revolutionary-[]technology/

    2021

  123. [133]

    Neural representational geometry underlies few-shot concept learning

    Ben Sorscher, Surya Ganguli, and Haim Sompolinsky. Neural representational geometry underlies few-shot concept learning. Proceedings of the National Academy of Sciences , 119(43):e2200800119, 2022. 225

    2022

  124. [134]

    Optimistic rates for learning with a smooth loss

    Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Optimistic rates for learning with a smooth loss. arXiv preprint arXiv:1009.3896 , 2010

    2010 arXiv

  125. [135]

    Local sgd converges fast and communicates little

    Sebastian U Stich. Local sgd converges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018

    2018 arXiv

  126. [136]

    Unified optimal analysis of the (stochastic) gradient method

    Sebastian U Stich. Unified optimal analysis of the (stochastic) gradient method. arXiv preprint arXiv:1907.04232, 2019

    1907 arXiv

  127. [137]

    The error-feedback framework: Better rates for sgd with delayed gradients and compressed updates

    Sebastian U Stich and Sai Praneeth Karimireddy. The error-feedback framework: Better rates for sgd with delayed gradients and compressed updates. Journal of Machine Learning Research , 2020

    2020

  128. [138]

    Sparsified sgd with memory

    Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified sgd with memory. In Proceedings of Advances in Neural Information Processing Systems , 2018

    2018

  129. [139]

    Distributed optimization based on gradient tracking revisited: Enhancing convergence rate via surrogation

    Ying Sun, Gesualdo Scutari, and Amir Daneshmand. Distributed optimization based on gradient tracking revisited: Enhancing convergence rate via surrogation. In SIAM Journal on Optimization , 2022

    2022

  130. [140]

    The bitter lesson

    Richard Sutton. The bitter lesson. Incomplete Ideas (blog), 13(1):38, 2019

    2019

  131. [141]

    Intriguing properties of neural networks

    C Szegedy. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013

    2013 arXiv

  132. [142]

    Measuring robustness to natural distribution shifts in image classification

    Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Infor- mation Processing Systems, 33:18583–18599, 2020

    2020

  133. [143]

    The computational limits of deep learning

    Neil C Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F Manso. The computational limits of deep learning. arXiv preprint arXiv:2007.05558 , 10, 2020

    2007 arXiv

  134. [144]

    Is learning the n-th thing any easier than learning the first? Advances in neural information processing systems, 8, 1995

    Sebastian Thrun. Is learning the n-th thing any easier than learning the first? Advances in neural information processing systems, 8, 1995

    1995

  135. [145]

    Primate cognition

    M Tomasello. Primate cognition. Oxford University Press, 1997

    1997

  136. [146]

    Position: Will we run out of data? limits of llm scaling based on human-generated data

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: Will we run out of data? limits of llm scaling based on human-generated data. In Forty-first International Conference on Machine Learning , 2022

    2022

  137. [147]

    A field guide to federated optimization

    Jianyu Wang, Zachary Charles, Zheng Xu, Gauri Joshi, H Brendan McMahan, Maruan Al-Shedivat, Galen Andrew, Salman Avestimehr, Katharine Daly, Deepesh Data, et al. A field guide to federated optimization. arXiv preprint arXiv:2107.06917 , 2021. 226

    2021 arXiv

  138. [148]

    On the unreason- able effectiveness of federated averaging with heterogeneous data

    Jianyu Wang, Rudrajit Das, Gauri Joshi, Satyen Kale, Zheng Xu, and Tong Zhang. On the unreason- able effectiveness of federated averaging with heterogeneous data. arXiv preprint arXiv:2206.04723 , 2022

    2022 arXiv

  139. [149]

    Robust constrained consensus and inequality-constrained dis- tributed optimization with guaranteed differential privacy and accurate convergence

    Yongqiang Wang and Angelia Nedi´ c. Robust constrained consensus and inequality-constrained dis- tributed optimization with guaranteed differential privacy and accurate convergence. IEEE Transac- tions on Automatic Control , 69(11):7463–7478, 2024. doi: 10.1109/TAC.2024.3385546

    2024

  140. [150]

    Distributed bandit learning: Near-optimal regret with efficient communication

    Yuanhao Wang, Jiachen Hu, Xiaoyu Chen, and Liwei Wang. Distributed bandit learning: Near-optimal regret with efficient communication. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=SJxZnR4YvB

    2020

  141. [151]

    Federated learning with differential privacy: Algorithms and performance analysis

    Kang Wei, Jun Li, Ming Ding, Chuan Ma, Howard H Yang, Farhad Farokhi, Shi Jin, Tony QS Quek, and H Vincent Poor. Federated learning with differential privacy: Algorithms and performance analysis. IEEE transactions on information forensics and security , 15:3454–3469, 2020

    2020

  142. [152]

    The minimax complexity of distributed optimization

    Blake Woodworth. The minimax complexity of distributed optimization. arXiv preprint arXiv:2109.00534, 2021

    2021 arXiv

  143. [153]

    Is local sgd better than minibatch sgd? In International Conference on Machine Learning , pages 10334–10343

    Blake Woodworth, Kumar Kshitij Patel, Sebastian Stich, Zhen Dai, Brian Bullins, Brendan Mcmahan, Ohad Shamir, and Nathan Srebro. Is local sgd better than minibatch sgd? In International Conference on Machine Learning , pages 10334–10343. PMLR, 2020

    2020

  144. [154]

    An even more optimal stochastic optimization algorithm: minibatching and interpolation learning

    Blake E Woodworth and Nathan Srebro. An even more optimal stochastic optimization algorithm: minibatching and interpolation learning. Advances in neural information processing systems, 34:7333– 7345, 2021

    2021

  145. [155]

    Graph oracle models, lower bounds, and gaps for parallel stochastic optimization

    Blake E Woodworth, Jialei Wang, Adam Smith, Brendan McMahan, and Nati Srebro. Graph oracle models, lower bounds, and gaps for parallel stochastic optimization. Advances in neural information processing systems, 31, 2018

    2018

  146. [156]

    Minibatch vs local sgd for heterogeneous distributed learning

    Blake E Woodworth, Kumar Kshitij Patel, and Nati Srebro. Minibatch vs local sgd for heterogeneous distributed learning. Advances in Neural Information Processing Systems , 33:6281–6292, 2020

    2020

  147. [157]

    The min-max complexity of distributed stochastic convex optimization with intermittent communication

    Blake E Woodworth, Brian Bullins, Ohad Shamir, and Nathan Srebro. The min-max complexity of distributed stochastic convex optimization with intermittent communication. In Conference on Learning Theory, pages 4386–4437. PMLR, 2021. 227

    2021

  148. [158]

    Data station: delegated, trustworthy, and auditable computation to enable data-sharing consortia with a data escrow

    Siyuan Xia, Zhiru Zhu, Chris Zhu, Jinjin Zhao, Kyle Chard, Aaron J Elmore, Ian Foster, Michael Franklin, Sanjay Krishnan, and Raul Castro Fernandez. Data station: delegated, trustworthy, and auditable computation to enable data-sharing consortia with a data escrow. arXiv prepr...

    2023 arXiv

  149. [159]

    Communication-computation efficient gradient coding

    Min Ye and Emmanuel Abbe. Communication-computation efficient gradient coding. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 5610–5619. PMLR, 10–1...

    2018

  150. [160]

    Federated accelerated stochastic gradient descent

    Honglin Yuan and Tengyu Ma. Federated accelerated stochastic gradient descent. Advances in Neural Information Processing Systems, 33:5332–5344, 2020

    2020

  151. [161]

    Federated composite optimization

    Honglin Yuan, Manzil Zaheer, and Sashank Reddi. Federated composite optimization. In International Conference on Machine Learning , pages 12253–12266. PMLR, 2021

    2021

  152. [162]

    Parallel sgd: When does averaging help? arXiv preprint arXiv:1606.07365 , 2016

    Jian Zhang, Christopher De Sa, Ioannis Mitliagkas, and Christopher Re. Parallel sgd: When does averaging help? arXiv preprint arXiv:1606.07365 , 2016

    2016 arXiv

  153. [163]

    Fedpd: A federated learning framework with optimal rates and adaptivity to non-iid data

    Xinwei Zhang, Mingyi Hong, Sairaj Dhople, Wotao Yin, and Yang Liu. Fedpd: A federated learning framework with optimal rates and adaptivity to non-iid data. arXiv preprint arXiv:2005.11418 , 2020

    2005 arXiv

  154. [164]

    Fedpage: A fast local stochastic gradient method for communication-efficient federated learning

    Haoyu Zhao, Zhize Li, and Peter Richtarik. Fedpage: A fast local stochastic gradient method for communication-efficient federated learning. arXiv preprint arXiv:2108.04755 , 2021

    2021 arXiv

  155. [165]

    Parallelized stochastic gradient descent

    Martin Zinkevich, Markus Weimer, Lihong Li, and Alex Smola. Parallelized stochastic gradient descent. Advances in neural information processing systems , 23, 2010

    2010

  156. [166]

    The age of surveillance capitalism: The fight for a human future at the new frontier of power, edn

    Shoshana Zuboff. The age of surveillance capitalism: The fight for a human future at the new frontier of power, edn. PublicAffairs, New York , 2019. 228

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.