Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

FedSub confines client updates to low-dimensional subspaces and adds low-dimensional dual variables that offset client drift, cutting uplink communication from O(md) to O(rd) while retaining sublinear convergence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 05:29 UTC pith:KU7A2EVY

load-bearing objection A genuinely new subspace-FL algorithm with a plausible convergence proof, but the headline computation/memory savings rest on a complexity model that backprop cannot deliver. the 4 major comments →

arxiv 2509.05213 v1 pith:KU7A2EVY submitted 2025-09-05 cs.LG cs.DC

An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data

Jiaojiao Zhang , Yuqi Xu , Kun Yuan This is my paper
classification cs.LG cs.DC
keywords federated learningclient driftsubspace projectiondual variableprimal-dual methodnonconvex optimizationcommunication efficiencyheterogeneous data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes FedSub, an algorithm that makes federated learning on heterogeneous data cheaper by confining every client's local update to a randomly chosen low-dimensional subspace. The authors claim this reduces per-round uplink communication, gradient computation, and memory cost, and that adding low-dimensional dual variables prevents the client drift that otherwise plagues federated averaging. The central theoretical result is a nonconvex convergence bound whose only constant error floor comes from the variance of the random projections; that floor disappears when the projection is the identity or when each client takes a single local step. If correct, FedSub offers a concrete way to train large models across bandwidth-limited clients without sacrificing convergence.

Core claim

Under L_f-smoothness of each client loss and a bound on the average squared gradient norm, FedSub with step size η ≤ O(1/(θ_r θ_m² L_f² n τ)) achieves (1/K) Σ_k E‖∇f(x^k)‖² ≤ O(θ_m/(Kτη) + ητ θ_r² θ_m²(θ_m−1) B_g²/n). The first term is the usual sublinear nonconvex rate; the second is the price of random subspace projection and vanishes when P^k = I, because then θ_m − 1 = 0. The authors also show that the dual variables are equivalent to subtracting the previous round's average-minus-local gradient from each local step, so the drift correction is achieved without any extra uplink transmission.

What carries the argument

The load-bearing object is a random subspace projection matrix per layer and round, P^k ∈ R^{m×r}, with (P^k)^T P^k = (m/r)I and E[P^k(P^k)^T] = I, paired with dual variables Λ^k_i that live in the subspace. The dual variable recursively encodes the difference between the global average gradient and client i's local gradient from the previous round, turning local updates into a variance-reduced correction. The expectation identity keeps the projected gradient unbiased, and the quantity θ_m − 1 measures how far the projection is from full space; the proof's Lyapunov function tracks f(x^k) plus the spread of dual-variable terms around their mean, which produces the constant error floor.

Load-bearing premise

The proof's constant error term relies on Assumption 3, that the average squared norm of client gradients is always bounded by a fixed constant; the authors introduce it only to handle random-projection variance, and it is unnecessary when the projection is the identity.

What would settle it

In the Section 5.1 logistic-regression setup, run FedSub with coordinate-descent projection at ranks r = 2 and r = 18, using the same step size and τ, and measure the asymptotic error plateaus; the ratio of the plateaus should scale roughly as (m/2 − 1)/(m/18 − 1) if the Theorem 1 variance term is the dominant error. A plateau ratio far from that prediction would show that the bound's subspace-noise floor does not capture what actually limits convergence.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Per-client uplink communication drops from O(md) to O(rd), with gradient computation reduced to O(τ C_g(rd)) and memory to O(3rd + M_g(rd) + 2rm + md), making large-model federated training feasible on bandwidth- and memory-limited clients.
  • When P^k = I, FedSub becomes a full-space primal-dual method that recovers the convergence rate of full-space federated averaging with drift correction, matching the behavior of SCAFFOLD and FedLin without their extra communication.
  • The convergence bound makes the accuracy-efficiency trade-off explicit: larger subspace dimension r reduces the constant error floor, while larger local step count τ and smaller step size can amplify or shrink the projection-noise term.
  • With τ = 1 the rate reduces to O(1/(Kη)), recovering the standard single-machine nonconvex gradient-descent rate, so the subspace mechanism does not by itself slow convergence when local updates are not repeated.
  • The choice of projection matrix matters: coordinate-descent projections performed best in the paper's experiments, and the theory ties performance to θ_m, so projection schemes with lower variance relative to θ_m − 1 should inherit tighter guarantees.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the variance floor is the dominant error term, then for a fixed subspace dimension, increasing the number of clients n should shrink the error floor at a 1/n rate, so FedSub should approach full-space accuracy in very large federated deployments—a prediction the paper does not test.
  • The bounded-average-gradient assumption is the fragile point for deep networks; gradient clipping, which makes B_g finite, should make the theorem's subspace-noise floor observable in practice, yielding an experiment that separates the bound's behavior from other sources of error.
  • The paper broadcasts the full-space global model to every client, so the downlink remains O(md); a natural extension is to broadcast a compressed or factored global model, although the full model is needed to apply the projection at each client.
  • The dual-variable correction is structurally a variance-reduced gradient, so combining FedSub with momentum or adaptive step sizes may preserve the same convergence form with a similar subspace-noise floor, a direction the paper explicitly leaves for future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper introduces FedSub, a federated learning algorithm in which each client performs local updates in a low-dimensional random subspace of the model parameters, communicates only the low-dimensional update B_i^{k,\tau} to the server, and maintains dual variables in the same subspace to correct client drift. The server updates the global model as x^{k+1}=x^k+P^k \bar B^{k,\tau} and broadcasts the full model. Under smoothness (Assumption 2) and a uniform bound on average gradient norms (Assumption 3), Theorem 1 gives E||\nabla f(x^k)||^2 \le O(\theta_m/(K\tau\eta)+\eta\tau\theta_r^2\theta_m^2(\theta_m-1)B_g^2/n). The paper compares FedSub with FedAvg and full-space variants on logistic regression and CIFAR-100/ResNet, reporting accuracy. Table 1 claims O(rd) uplink, O(\tau(mrd+C_g(rd))+2mrd) computation, and O(3rd+M_g(rd)+2rm+md) memory.

Significance. The theoretical analysis is a step toward principled subspace FL: it identifies projection variance as the source of a non-vanishing error floor and recovers a full-space rate when P^k=I. The proof is largely self-contained; the main external ingredient is standard. The dual-variable formulation is elegant and gives a SCAFFOLD-like correction without extra uplink. However, the paper's headline efficiency claim is not supported: the complexity model in §8.3 assumes projected gradients can be evaluated in O(rd) and with O(rd) memory, which is not achievable in a deep network; and the experiments measure only accuracy, not runtime or memory. The communication reduction is real, but the computation/memory reductions are unsubstantiated. With those corrected, the contribution would be narrower but still of interest.

major comments (4)
  1. [§8.3, Table 1] The claims that the gradient projection (P^k)^T\nabla f_i can be computed with cost C_g(rd) and memory M_g(rd) 'without explicitly evaluating \nabla f_i' are not realizable for standard DNNs. The forward pass through x^k+P^kB_i^{k,t} has per-layer dimensions m_l×d_l; activations are independent of r_l, and backpropagation must produce the full layer gradient before it is projected. Re-using the forward pass of x^k still requires storing the full activations for each layer. Therefore the per-round computation is O(\tau C_g(md)+\tau mrd+2mrd) and the memory is O(M_g(md)+md+rd+2rm), not the entries in Table 1. Since the paper's central claim of simultaneous reduction rests on these columns, this is a load-bearing inconsistency.
  2. [Theorem 1, Assumption 3] The second term in Theorem 1, \eta\tau\theta_r^2\theta_m^2(\theta_m-1)B_g^2/n, does not vanish unless B_g=0 or P^k=I. The authors acknowledge Assumption 3 is 'introduced specifically to handle the variance of the random subspace projection matrices', but the abstract and conclusions state efficiency gains without this caveat. More importantly, substituting the stated upper bound \eta \le O(1/(\theta_r\theta_m^2L_f^2 n\tau)) into the first term gives O(\theta_r\theta_m^3L_f^2 n/K), so the rate degrades with the number of clients n; this dependence should be clarified and, if possible, removed.
  3. [Section 4, discussion after Theorem 1] The statement 'we can take \theta_r\theta_m=O(1)' is not generally true. With \theta_r=\max_l r_l/m_l and \theta_m=\max_l m_l/r_l, the product can be arbitrarily large when layer-wise ratios r_l/m_l vary (e.g., two layers with ratios close to 1 and close to 0). Since experiments use a fixed r for all layers, \theta_m may be large and the advertised rate hides this dependence. The claim needs a supporting assumption or should be removed.
  4. [Section 5] The experiments report only test accuracy; no wall-clock time, GPU/CPU memory, or communication volume is measured. Given that the paper's contribution is an efficiency method, this leaves the efficiency claim without direct experimental support. The plots show single trajectories without error bars or number of seeds; this should be reported.
minor comments (5)
  1. [Abstract, §3, §8.3] 'Reducing communication' is too broad; §8.3 acknowledges the server broadcasts the full model x^{k+1}, so downlink cost remains O(md) and total communication is O((m+r)d). Qualify the claim as 'uplink communication' or report total communication.
  2. [§8.1, after Eq. (11)] The phrase 'let r/m (P^k)^T y^k_i as \Lambda^k_i =' has a dangling '='. Also, r/m and m/r are used interchangeably across sections; layerwise subscripts would avoid confusion.
  3. [Figure 2, §5.2] Only three curves are reported, and no table with final accuracies, standard deviations, or training time is given. A concise table would improve reproducibility and comparability.
  4. [References] References [5] and [16] are duplicates; [5] should be removed or replaced.
  5. [§5.1] The constructions of CD, RD, and SS projection matrices are cited only collectively to [8,9]; a one-sentence definition of each would make the experiments self-contained.

Circularity Check

0 steps flagged

No circularity found: the convergence proof is self-contained given standard assumptions, and the one self-citation is for an elementary projection inequality that is not equivalent to the target result.

full rationale

The paper's derivation chain is self-contained. FedSub is derived from a primal-dual formulation, and Theorem 1 is proved through Lemmas 1–3 using only Assumptions 1–3 (random projection properties, smoothness, and bounded average gradient norm). The only citation to prior work by overlapping authors is in the proof of Lemma 2, where the paper says ‘we use similar derivations as in [8, Lemma 5]’ to bound E||(I − P^k(P^k)^T)∇f_i(x^k)||^2. That inequality follows directly from Assumption 1 and the definition of θ_m; it is a parameter-free elementary identity, not a restatement of the convergence rate. No parameter is fitted to the data or to the target quantity; no prediction is defined in terms of its own output; and no uniqueness claim is imported from the authors' prior work. Assumption 3 is explicitly acknowledged as introduced only to control the variance of the random projection matrices and is not needed when P^k = I; this weakens the theorem but is not circular. The Table 1 efficiency claims rely on a substantive modeling assumption about computing projected gradients without a full forward/backward pass, which may be unrealistic for DNNs, but that is a correctness/efficiency-model concern rather than circularity. Therefore the central derivation is not circular.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The central convergence claim rests on three stated assumptions: the random subspace projection property (Assumption 1), L_f-smoothness (Assumption 2), and a uniform bound on average squared gradient norms (Assumption 3). No constants are fitted to data; the algorithm's hyperparameters η, τ, r are design choices, not fit parameters.

axioms (3)
  • domain assumption Random subspace projection matrices satisfy (P^l_k)^T P^l_k = (m_l/r_l) I and E[P^l_k (P^l_k)^T] = I for all layers and rounds.
    This is the core structural assumption that makes the projection algebra and variance calculations in Appendix 8.4 work. It can be satisfied by constructions from [9], but the analysis depends on it.
  • domain assumption Each local loss f_i is L_f-smooth.
    Used throughout the descent lemmas and the bound on gradient differences in Lemma 1.
  • domain assumption The average squared gradient norm is uniformly bounded by B_g^2.
    Introduced solely to bound the variance of I - P(P)^T; the paper notes it is unnecessary when P = I. The error floor in Theorem 1 is proportional to B_g^2.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 14997 in / 17952 out tokens · 162278 ms · 2026-08-05T05:29:19.074874+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data." pith.science (2026). https://pith.science/paper/KU7A2EVY

@misc{pith2026250905213,
  author       = {Pith},
  title        = {Pith review of: An Efficient Subspace Algorithm for Federated Learning on Heterogeneous Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KU7A2EVY}},
  note         = {Machine review of arXiv:2509.05213}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This work addresses the key challenges of applying federated learning to large-scale deep neural networks, particularly the issue of client drift due to data heterogeneity across clients and the high costs of communication, computation, and memory. We propose FedSub, an efficient subspace algorithm for federated learning on heterogeneous data. Specifically, FedSub utilizes subspace projection to guarantee local updates of each client within low-dimensional subspaces, thereby reducing communication, computation, and memory costs. Additionally, it incorporates low-dimensional dual variables to mitigate client drift. We provide convergence analysis that reveals the impact of key factors such as step size and subspace projection matrices on convergence. Experimental results demonstrate its efficiency.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations

    cs.IT 2026-04 unverdicted novelty 6.0

    A correlation-based taxonomy unifies existing FL compression methods, experiments show correlation strengths vary by task and architecture, and adaptive mode-switching designs are proposed to exploit this.

  2. Subspace Optimization for Efficient Federated Learning under Heterogeneous Data

    cs.LG 2026-04 unverdicted novelty 5.0

    SSF enables efficient federated learning under heterogeneous data by optimizing in a low-dimensional subspace with projected corrections and backfill updates, achieving a non-asymptotic convergence rate of order O~(1/...

  3. FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection

    cs.LG 2026-04 unverdicted novelty 5.0

    FedSLoP reduces communication and memory costs in federated learning through stochastic low-rank gradient projections, with a nonconvex convergence rate of O(1/sqrt(NT)) and competitive accuracy on heterogeneous MNIST data.

  4. FedSLoP: Memory-Efficient Federated Learning with Low-Rank Gradient Projection

    cs.LG 2026-04 unverdicted novelty 4.0

    FedSLoP applies stochastic low-rank gradient projections in federated learning to reduce communication volume and client memory while proving O(1/sqrt(NT)) convergence to stationary points under standard assumptions a...

Reference graph

Works this paper leans on

28 extracted references · 22 canonical work pages · cited by 3 Pith papers

  1. [1]

    INTRODUCTION Federated Learning (FL) enables collaborative training across distributed clients while preserving data privacy [1,2]. Its ap- plication to large-scale tasks, such as training large language models (LLMs), is promising, as it leverages distributed data resources and parallel computation to accelerate training and improve performance [3]. Desp...

    work page

  2. [2]

    When the projection matrix becomes the identity, FedSub naturally reduces to a full-space primal-dual FL algorithm. (ii)We provide convergence analysis that characterizes the arXiv:2509.05213v1 [cs.LG] 5 Sep 2025 Table 1:O-notation is omitted.C g(rd)denotes the computation cost of evaluating a gradient with sizer×d.M g(rd)denotes the memory cost of comput...

    Pith/arXiv arXiv 2025

  3. [3]

    A server coordinatesnclients, where each clienti∈[n]holds a local loss functionf i(x)

    PROBLEM FORMULATION AND SUBSPACE We consider training a DNN withLlayers in the FL setting. A server coordinatesnclients, where each clienti∈[n]holds a local loss functionf i(x). The global objective is to minimize the average loss across all clients: min x={xl}L l=1 f(x) = 1 n nX i=1 fi(x),(1) wherex={x l}L l=1 denotes the collection of DNN weights across...

    work page

  4. [4]

    Our proposed algorithm is summarized in Algorithm 1

    ALGORITHM Proposed Algorithm.We propose an efficient subspace FL algorithm that leverages low-dimensional subspace projec- tions to reduce communication, computation, and memory overhead, while use dual variables to mitigate client drift caused by heterogeneous data. Our proposed algorithm is summarized in Algorithm 1. It is derived from the primal- dual ...

    work page

  5. [5]

    Assumption 2.Each functionf i(x)isL f -smooth such that ∥∇fi(x)− ∇fi(z)∥≤L f ∥x−z∥,∀i, x, z

    ANALYSIS In our analysis, we introduce the following assumptions. Assumption 2.Each functionf i(x)isL f -smooth such that ∥∇fi(x)− ∇fi(z)∥≤L f ∥x−z∥,∀i, x, z. Assumption 3.There exists a constantB g >0such that the average squared norm of the gradients is bounded: 1 n Pn i=1∥∇fi(x)∥2≤B 2 g ,∀x. Assumption 3 is introduced specifically to handle the vari- a...

    work page

  6. [6]

    In our experiments, we consider three commonly used examples: Coordinate Descent (CD), Ran- dom Generation (RD), and Spherical Smoothing (SS) [8, 9]

    NUMERICAL EXPERIMENTS There exist multiple approaches to generate a projection ma- trixP k ∈R m×r that exactly or approximately satisfies As- sumption 1 [9, 20]. In our experiments, we consider three commonly used examples: Coordinate Descent (CD), Ran- dom Generation (RD), and Spherical Smoothing (SS) [8, 9]. 5.1. Logistic Regression We generate60,000sam...

    work page 2000

  7. [7]

    CONCLUSIONS We propose an efficient subspace algorithm for FL, address- ing the challenges posed by data heterogeneity and large-scale model training. By making local updates of each client within low-dimensional subspaces, FedSub reduces communication, computation, and memory costs, while uses dual variable to mitigate client drift. Theoretical analysis ...

    work page

  8. [8]

    Federated learning: Strategies for im- proving communication efficiency,

    Jakub Kone ˇcn`y, H Brendan McMahan, Felix X Yu, Peter Richt ´arik, Ananda Theertha Suresh, and Dave Bacon, “Federated learning: Strategies for im- proving communication efficiency,”arXiv preprint arXiv:1610.05492, 2016

    Pith/arXiv arXiv 2016

  9. [9]

    Hybrid federated learning for feature & sample heterogeneity: Algorithms and implementation,

    Xinwei Zhang, Wotao Yin, Mingyi Hong, and Tianyi Chen, “Hybrid federated learning for feature & sample heterogeneity: Algorithms and implementation,”Trans- actions on Machine Learning Research, 2024

    work page 2024

  10. [10]

    Fedlion: Faster adaptive federated optimization with fewer communica- tion,

    Zhiwei Tang and Tsung-Hui Chang, “Fedlion: Faster adaptive federated optimization with fewer communica- tion,” inIEEE International Conference on Acoustics, Speech and Signal Processing, 2024, pp. 13316–13320

    work page 2024

  11. [11]

    Scaffold: Stochastic controlled averaging for federated learning,

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” inInternational Conference on Ma- chine Learning, 2020, pp. 5132–5143

    work page 2020

  12. [13]

    LoRA: Low-rank adaptation of large lan- guage models.,

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al., “LoRA: Low-rank adaptation of large lan- guage models.,”ICLR, vol. 1, no. 2, pp. 3, 2022

    work page 2022

  13. [14]

    Ga- lore: Memory-efficient llm training by gradient low- rank projection,

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian, “Ga- lore: Memory-efficient llm training by gradient low- rank projection,” inForty-first International Conference on Machine Learning

    work page

  14. [15]

    Subspace optimization for large language models with convergence guarantees,

    Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan, “Subspace optimization for large language models with convergence guarantees,” inForty-second International Conference on Machine Learning

    work page

  15. [16]

    A memory efficient randomized subspace optimization method for training large language mod- els,

    Yiming Chen, Yuan Zhang, Yin Liu, Kun Yuan, and Zai- wen Wen, “A memory efficient randomized subspace optimization method for training large language mod- els,” inForty-second International Conference on Ma- chine Learning

    work page

  16. [17]

    Improving LoRA in privacy-preserving feder- ated learning,

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding, “Improving LoRA in privacy-preserving feder- ated learning,”arXiv preprint arXiv:2403.12313, 2024

    Pith/arXiv arXiv 2024

  17. [18]

    Federated fine-tuning of large language models under heterogeneous tasks and client resources,

    Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li, “Federated fine-tuning of large language models under heterogeneous tasks and client resources,” arXiv preprint arXiv:2402.11505, 2024

    Pith/arXiv arXiv 2024

  18. [19]

    Communication- efficient federated low-rank update algorithm and its connection to implicit regularization,

    Haemin Park and Diego Klabjan, “Communication- efficient federated low-rank update algorithm and its connection to implicit regularization,”arXiv preprint arXiv:2409.12371, 2024

    work page arXiv 2024

  19. [20]

    FedSVD: Adaptive orthogonaliza- tion for private federated learning with LoRA,

    Seanie Lee, Sangwoo Park, Dong Bok Lee, Dominik Wagner, Haebin Seong, Tobias Bocklet, Juho Lee, and Sung Ju Hwang, “FedSVD: Adaptive orthogonaliza- tion for private federated learning with LoRA,”arXiv preprint arXiv:2505.12805, 2025

    arXiv 2025

  20. [21]

    FLoCoRA: Fed- erated learning compression with low-rank adaptation,

    Lucas Grativol, Mathieu Leonardon, Guillaume Muller, Virginie Fresse, and Matthieu Arzel, “FLoCoRA: Fed- erated learning compression with low-rank adaptation,” in2024 32nd European Signal Processing Conference (EUSIPCO). IEEE, 2024, pp. 1786–1790

    work page 2024

  21. [22]

    Communication-efficient personalized federated learn- ing for speech-to-text tasks,

    Yichao Du, Zhirui Zhang, Linan Yue, Xu Huang, Yuqing Zhang, Tong Xu, Linli Xu, and Enhong Chen, “Communication-efficient personalized federated learn- ing for speech-to-text tasks,” inIEEE International Conference on Acoustics, Speech and Signal Process- ing, 2024, pp. 10001–10005

    work page 2024

  22. [23]

    Exploring gradient subspaces: Ad- dressing and overcoming LoRA’s limitations in fed- erated fine-tuning of large language models,

    Navyansh Mahla, Kshitij Sharad Jadhav, and Ganesh Ramakrishnan, “Exploring gradient subspaces: Ad- dressing and overcoming LoRA’s limitations in fed- erated fine-tuning of large language models,”arXiv preprint arXiv:2410.23111, 2024

    Pith/arXiv arXiv 2024

  23. [24]

    A unified and refined convergence analysis for non-convex decentral- ized learning,

    Sulaiman A Alghunaim and Kun Yuan, “A unified and refined convergence analysis for non-convex decentral- ized learning,”IEEE Transactions on Signal Processing, vol. 70, pp. 3264–3279, 2022

    work page 2022

  24. [25]

    Linear convergence in federated learning: Tackling client heterogeneity and sparse gradi- ents,

    Aritra Mitra, Rayana Jaafar, George J Pappas, and Hamed Hassani, “Linear convergence in federated learning: Tackling client heterogeneity and sparse gradi- ents,”Advances in Neural Information Processing Sys- tems, vol. 34, pp. 14606–14619, 2021

    work page 2021

  25. [26]

    Stochastic variance reduction for nonconvex optimization,

    Sashank J Reddi, Ahmed Hefny, Suvrit Sra, Barnabas Poczos, and Alex Smola, “Stochastic variance reduction for nonconvex optimization,” inInternational confer- ence on machine learning. PMLR, 2016, pp. 314–323

    work page 2016

  26. [27]

    Zeroth-order optimization with orthogonal random directions,

    David Kozak, Cesare Molinari, Lorenzo Rosasco, Luis Tenorio, and Silvia Villa, “Zeroth-order optimization with orthogonal random directions,”Mathematical Pro- gramming, vol. 199, no. 1, pp. 1179–1219, 2023

    work page 2023

  27. [28]

    Deep residual learning for image recognition,

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE Conference on Computer Vi- sion and Pattern Recognition, 2016, pp. 770–778

    work page 2016

  28. [29]

    APPENDIX 8.1. Derivation of Algorithm 1 For a more compact representation, we definex={x i}n i=1 to aggregate all local weights and introduce a linear operator Lto encode the consensus constraint as: (Lx)i :=x i − 1 n nX i=1 xi,∀i∈[n].(5) Using (5) and expressing the objective in (2) asf(x), we rewrite (2) in a compact form: min x f(x),s.t.Lx= 0. (6) To r...

    work page