Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization

Kai Yi

REVIEW 4 major objections 5 minor 254 references

Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This dissertation claims that one framework unifies biased and unbiased gradient compression and that explicit personalization plus local training yields doubly accelerated communication in federated learning.

desk verdict A well-organized compilation of five published papers; EF-BV is a genuine contribution, but Scafflix's double-acceleration claim outruns its theorem, and SymWanda's 'theory' is equivalence by construction. read the letter →

arxiv 2509.08233 v1 pith:DEBXESNF submitted 2025-09-10 cs.LG cs.AI

Kai Yi This is my paper · ORCID

classification cs.LGcs.AI MSC 90C2590C2668T05

keywords federatedlearningcommunicationefficiencybiasedandunbiasedcompressorslocaltrainingpersonalizationnetworkpruningstochasticproximalpointmethodpost-training

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The dissertation tries to show that the three main levers for cutting communication in distributed and federated learning—compressing messages, doing more local work, and personalizing models—can be combined under provable guarantees rather than treated as separate heuristics. Its central theoretical contribution is a unified treatment of biased and unbiased compression operators, with an algorithm, EF-BV, that recovers the two prior state-of-the-art algorithms DIANA and EF21 as special cases. Its second contribution, Scafflix, fuses explicit personalization with accelerated local training and claims a double communication acceleration, with the personalization weight entering the convergence bound through the initial gap. The remaining chapters extend the same program to pruning, showing that layer-level personalized pruning, multiple intra-cohort communication rounds, and symmetric post-training pruning each improve accuracy, convergence, or communication in measured, sometimes provable, ways. A sympathetic reader would care because the dissertation offers parameterized trade-offs where communication cost, model accuracy, and convergence speed can be tuned together instead of being exchanged against each other blindly.

What carries the argument

The load-bearing object is the bias-variance compressor class C(η,ω): a randomized map whose expected output deviates from the input by at most η∥x∥ and whose variance is at most ω∥x∥², with an average variance ωran after parallel aggregation. EF-BV's two scaling parameters λ and ν separately control the control-variate error and the aggregated gradient estimate; this single mechanism reproduces EF21 when ν=λ and DIANA when ν=1, while allowing the stepsize to grow with the number of workers. Scafflix's machinery is the FLIX objective with explicit personalization factors α_i, plus per-client stepsizes γ_i and a control variate updated only with communication probability p; the Lyapunov funct

What would settle it

On a strongly convex logistic-regression problem with known per-client smoothness and strong-convexity constants, run Scafflix twice with α_i = 0.1: once with exact local minimizers and once with local minimizers approximated by a small fixed number of SGD steps. The theorem predicts the exact run's error contracts at a rate set by √κmax with an α_i²-scaled initial gap; if the approximate run shows no systematic slowdown, or if the gap between the two runs vanishes as the local solver improves, then the exact-optimum premise is not what carries the claimed acceleration.

Watch

Extended reading notes

Core claim

The central contention is that the three standard levers for cutting communication in distributed and federated learning—compressing the messages, training longer locally, and personalizing to each client—can be combined under provable guarantees rather than treated as heuristics. The first result, EF-BV, introduces a two-parameter compressor class C(η,ω) that separates bias from variance; with two scaling parameters λ and ν, the same algorithm recovers DIANA and EF21 as edge cases and converges linearly under a Kurdyka–Łojasiewicz condition. The second result, Scafflix, solves the FLIX personalized objective with individual stepsizes and randomized communication, achieving communication com

Load-bearing premise

Scafflix's double-acceleration guarantee assumes each client can compute the exact best model for its own data before training begins; the deep-learning experiments only approximate that model, so the proof does not by itself cover those measured speedups.

Editorial extensions

If this is right

If EF-BV's analysis holds, any compressor in C(η,ω) can be used with a linear-rate guarantee, and DIANA and EF21 analyses become corollaries, with the rate improving as the number of parallel workers grows.
If Scafflix's theorem holds, choosing smaller personalization factors α_i shrinks the initial Lyapunov gap and therefore the communication rounds to target accuracy, while local steps contribute a √κmax acceleration.
If FedP3's aggregation analysis holds, global pruning and personalized layer assignment reduce communication cost by a factor related to O(d/n) with nonconvex convergence to stationarity, and LDP-FedP3 gives a concrete privacy-utility-communication trade-off.
If Cohort-Squeeze's finding holds, increasing local communication rounds within a cohort lowers total communication cost to a target accuracy, with reported reductions up to 74%, and stratified sampling improves the convergence neighborhood over block and nice sampling.
If SymWanda holds, Wanda and RIA are endpoints of a broader symmetric reconstruction objective, guiding new pruning scores and training-free fine-tuning methods such as R2-DSnoT that improve high-sparsity LLM accuracy without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A consequence not developed in the dissertation: the unified compressor class turns compression design into a parameter-estimation problem—if a new hardware constraint produces a compressor, specifying its (η,ω,ωran) triple immediately yields convergence parameters, so compression choices become part of an optimization problem rather than a heuristic.
Scafflix makes personalization strength α_i a genuine communication knob. A testable extension would schedule α_i over time, using strong personalization early and tighter coupling later, to trade local fit against global coherence during training.
Cohort-Squeeze's result suggests that the stateless-client assumption of cross-device FL is not a barrier to variance reduction; restructuring state as intra-cohort communication replicates some of its benefits. Whether three-level hierarchies with hubs approximate the same savings is a natural next question.
SymWanda's symmetric objective invites output-side calibration data. Using downstream-task information to choose the output calibration matrix Y could yield task-adaptive pruning, a step beyond the calibration-set perplexity benchmarks reported in the dissertation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my take. The dissertation is exactly what it says it is: a compilation of five of Kai Yi's published papers, stitched together with a new intro, a comparison table, and shared notation. That means the 'new results' here are the synthesis, not the science. The underlying chapters are real, and two of them are strong. EF-BV's unified compressor class and convergence proof genuinely unify DIANA and EF21, and the thesis is careful about which chapter comes from which paper. So as a PhD thesis, it's an honest record.

The good parts: EF-BV has rigorous-looking theory, with linear convergence under PL/KL, and it clearly recovers EF21 and DIANA as special cases. The experimental chapters are extensive—pruning on ResNet, LLaMA2/3, and the Cohort-Squeeze observation about multi-round cohorts is empirically interesting. Writing is mostly clear, and the author explicitly cites the source papers.

The soft spots are real but proportioned. The biggest one is the stress-test concern: Scafflix's Theorem 3.2.3 requires each client to compute the exact local optimum (Algorithm 4, line 3). The deep-learning experiments replace that with a finite-SGD iterate, and the authors' own ablation uses thresholds as loose as 1e-1. So the 'double acceleration' seen in neural nets is not covered by the proof. The thesis flags this inexactness but doesn't supply the missing perturbation analysis. That's a genuine gap, central to the chapter's headline.

Second, SymWanda's recovery of Wanda and RIA is by construction. Theorem 6.3.5 chooses calibration matrices X and Y so the score equals the relative-importance formula. That's a valid algebraic observation, not an independent derivation of why Wanda works. The chapter oversells it as 'theoretical support.'

Third, FedP3 is called 'privacy-friendly,' but the base version sends selected layers, not a DP guarantee. The LDP variant has analysis; the base claim is overstated.

Minor: cross-reference typos (e.g., 'Figure 4.1' when meaning 'Figure 3.1') and text artifacts. Fixable.

Who is this for? A reader wanting a single-entry overview of Yi's PhD work, or someone citing EF-BV. It's not a new research contribution beyond the papers themselves.

Recommendation: if submitted as a new paper, it deserves peer review but with major revision expected—temper the overclaims, add the inexact-solve analysis or explicitly limit the claim, and clean up the presentation. As a dissertation, it's a fine record. I would not desk-reject it, but I would not accept it as new science as-is.

Referee Report

4 major / 5 minor

Summary. This dissertation (arXiv:2509.08233) collects five previously published or preprint research threads under the theme of improving communication efficiency in distributed and federated learning. Chapter 2 introduces a unified compressor class C(η,ω) parameterized by bias and variance, and an algorithm EF-BV that specializes to EF21 and DIANA, with linear convergence under PL/KL conditions and sublinear nonconvex rates. Chapter 3 proposes Scafflix, an individualized Scaffnew-type local-training scheme applied to the FLIX personalized objective, and proves accelerated communication complexity in the strongly convex setting. Chapter 4 presents FedP3, a federated pruning framework with client-specific layer selection and global/local pruning, plus an LDP variant with convergence and privacy guarantees. Chapter 5 develops SPPM-AS, a stochastic proximal point method with arbitrary cohort sampling and multiple local communication rounds, and reports large communication-cost reductions. Chapter 6 gives SymWanda, a symmetric input-output reconstruction formulation for post-training pruning that contains Wanda and RIA as special cases, and R2-DSnoT, a training-free fine-tuning procedure. The abstract claims a unified framework, favorable accuracy/communication trade-offs, and privacy-preserving pruning frameworks.

Significance. If the results hold, the dissertation makes several useful contributions. EF-BV's bias-variance compressor class is a nontrivial unification that yields new rates for DIANA with biased compressors and for EF21 with regularizers. Scafflix extends ProxSkip/Scaffnew-style acceleration to explicit personalization and gives a √κ communication rate depending on per-client condition numbers; the Lyapunov argument is clean. SPPM-AS provides a flexible arbitrary-sampling analysis for stochastic proximal point methods and demonstrates empirically that intra-cohort communication rounds can reduce total cost. SymWanda offers a compact algebraic framework in which Wanda, output-Wanda, and RIA appear as special scoring rules. Strengths include complete appendices with proofs, publication of core chapters in peer-reviewed venues, and extensive experiments across logistic regression, CNNs, and LLMs. The main caveat is that the strongest theoretical claims are proven under strong convexity and exact subproblem solves, while the headline practical results are obtained in nonconvex deep learning with approximate local solves; the dissertation also overstates the privacy guarantee of base FedP3 and th

major comments (4)

[§3.2, Algorithm 4, Theorem 3.2.3; §3.3.4] Algorithm 4 line 3 computes x_i^* := argmin f_i, and Theorem 3.2.3's Lyapunov function (3.3), unbiasedness Assumption 3.2.1, and all rates are stated with respect to this exact local optimum. The practical experiments in §3.3.2 replace x_i^* by finite-SGD iterates, and even the paper's own convex ablation (§3.3.4, Figs. 3.4, B.7, B.8) uses thresholds as loose as ε=1e-1. No theorem or lemma quantifies how an inexact x̂_i perturbs the fixed point of (FLIX), the gradient bias, or the Lyapunov decrease; the nonconvex CNN/RNN objectives lie outside the strong-convexity assumption altogether. The text acknowledges the issue in §3.3.4 but supplies no analysis. Consequently the advertised 'double communication acceleration' is proven only for exact, strongly convex local problems; the deep-learning results are not covered by Theorem 3.2.3. I request an inexact-local-solve perturbation result (or
[§4.1, Algorithm 5, Theorem 4.3.4] The base FedP3 is called 'privacy-friendly' because each client uploads only the layers it trained (Algorithm 5, line 12). This is a communication reduction, not a privacy mechanism: the selected-layer pattern and the values of the uploaded weights can carry information about local data, and no definition or bound is given for this disclosure. The only formal privacy statement is Theorem 4.3.4 for LDP-FedP3, which explicitly adds Gaussian noise and uses (ε,δ)-LDP. The abstract and §4.1's 'privacy-preserving pruning frameworks' therefore overstate what is established. Please either analyze the privacy of the layer-selection mechanism in FedP3 or clearly attribute all privacy guarantees to LDP-FedP3.
[§6.3.2, Lemma 6.3.1, Theorem 6.3.5] Theorem 6.3.5 constructs calibration matrices X and Y as explicit functions of W (inverse row/column ℓ1 norms) so that the SymWanda score ∥X:j∥₂+∥Yk:∥₂ equals the RIA relative-importance value αjk. This is a containment/representation result: RIA is shown to be a special case of the SymWanda scoring family. It is not an independent derivation of RIA from the reconstruction objective (Sym), nor does it prove that RIA or Wanda are optimal for the constrained pruning problem. Moreover, Lemma 6.3.1 derives the score for a single-weight elimination, and the transition to multi-weight pruning is an iterative greedy heuristic without an optimality guarantee. The 'theoretical support for post-training pruning' claim in §6.1 should be softened to a unification result, and the heuristic nature of the multi-weight extension should be stated.
[§5.3.2, Theorem 5.3.2; §5.4.2, Fig. 5.2] The abstract-level statement that Cohort-Squeeze 'significantly reduces overall communication overhead' rests mainly on experiments: the 74% figure is for logistic regression on a6a with BFGS and a particular hierarchical cost model (§5.4.2, Fig. 5.2), not a theorem. Theorem 5.3.2 proves convergence of exact SPPM-AS under strong convexity; it does not compare K>1 against K=1, and the experiments approximate the proximal operator with finite CG/BFGS/Adam iterations that are outside the theorem. Table 1.1 labels this a 'provable benefit'. Please clarify that the provable part is the convergence of SPPM-AS for arbitrary samplings, while the communication-reduction benefit is empirical and parameter/dataset-dependent.

minor comments (5)

[Remark 2.4.3] The remark refers to 'Theorems 2.4.1 and 3.2.3'; Theorem 3.2.3 belongs to Chapter 3. The intended reference is likely Theorem 2.4.2 (or 2.5.1).
[§2.1] Unresolved placeholders 'Sections ?? and ??' appear in the introduction to Chapter 2.
[§3.3.1, §3.3.3] The text refers to 'Figure 4.1' where Figure 3.1 is meant, and in §3.3.3 says results are for α=0.1 while the Figure 3.2 caption states α=0.5 (FEMNIST) and α=0.3 (Shakespeare).
[§4.4.2] The text says 'Results in Figure 4.1 demonstrate...' but Figure 4.1 is the pipeline diagram; the stated results are in Table 4.1 and Figure 4.3.
[§6.3.2] There is a duplicated sentence ('It is important to note that this formulation employs ...') and a missing space ('employsnon-squared'). Table 6.1's header notation is also difficult to parse.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: SymWanda's 'recovery' of RIA is an identity imposed by the choice of X and Y, and Scafflix's personalization acceleration is a definitional α^2 rescaling of the initial Lyapunov gap.

self definitional [Section 3.2, after Corollary 3.2.5 (p. 45)]
"the acceleration effect of personalization is not visible: it is “hidden” in Ψ0, because every client computes xt_i but what matters is its personalized model x~t_i, and ∥x~t_i − x~⋆_i∥^2 = α_i^2∥xt_i − x⋆∥^2. ... Thus, the smaller the αi, the smaller Ψ0 and the faster the convergence. This is why personalization is an acceleration mechanism in our setting."

In (FLIX), the personalized model is defined as x~ = αx + (1−α)x*_i, so the gap to the personalized optimum is α_i times the gap to the global optimum by construction. The theorem's convergence rate (3.4)–(3.5) is explicitly independent of α; the entire personalization benefit enters through Ψ0, which is weighted by α_i^2. Thus the claimed 'personalization acceleration' is a tautological rescaling of the initial error inherent to the objective, not an algorithmic effect. The paper itself acknowledges this by saying the effect is 'hidden' in Ψ0.
self definitional [Section 6.3.2, Theorem 6.3.5 and Section 6.3.3]
"Theorem 6.3.5. Assuming a = b and c = d, consider one of the following strategies: • X:j := tj(1; . . .; 1) ∈ Rb×1 and Yk: := sk(1, . . . ,1) ∈ R1×c, where tj = (√b∥Wj:∥1)−1 and sk = (√c∥W:k∥1)−1. • X = Diag(∥W1:∥−1 1 , . . . ,∥Wb:∥−1 1 ) and Y = Diag(∥W:1∥−1 1 , . . . ,∥W:c∥−1 1 ). For these configurations, the condition ∥X:j∥2 + ∥Yk:∥2 = αjk := ∥Wj:∥−1 1 + ∥W:k∥−1 1 holds for all j, k."

The theorem selects X and Y as explicit functions of W's row/column L1 norms, so ∥X:j∥2 = ∥Wj:∥1^{-1} and ∥Yk:∥2 = ∥W:k∥1^{-1} by direct algebra. Substituting into the SymWanda score Sjk = |Wjk|(∥X:j∥2 + ∥Yk:∥2) reproduces RIA's relative-importance score RIjk = |Wjk|(∥Wj:∥1^{-1} + ∥W:k∥1^{-1}) identically. Hence the 'recovery' of RIA is an equality imposed by the chosen calibration matrices, not an independent derivation or theoretical explanation. The claimed theoretical support for RIA is therefore circular: the target formula is reintroduced through the back door of X and Y.

full rationale

Chapters 2, 4, and 5 are essentially self-contained. EF-BV's recovery of DIANA and EF21 is an honest algebraic specialization with independent convergence proofs; FedP3 and Cohort-Squeeze provide their own theory and experiments without relying on the claimed results as inputs. The dissertation's heavy reliance on the author's own prior publications is normal and not load-bearing here because the technical content is reproduced in the text. The two clearest circularity problems are in the central claims of Chapters 3 and 6. Scafflix's 'double acceleration' has an independent local-training component, but the personalization component reduces to the definition of the personalized model: the Lyapunov gap is α_i^2 times the global gap, and the convergence rate is independent of α_i. SymWanda's 'theoretical support' for RIA is equivalence by construction: X and Y are chosen specifically so that the SymWanda score equals RIA's row/column relative importance, so the theorem does not derive RIA from a natural objective. The inexact-local-optimum gap in Scafflix's experiments is a correctness limitation, not a circularity, and is not counted here. Overall the dissertation is not wholly circular, but two load-bearing derivations reduce by construction, giving a partial circularity score of 6.

Assumptions & free parameters 4 free parameters · 6 assumptions · 2 invented entities

The central theoretical results rest on standard smoothness/convexity assumptions and, for Scafflix, on the strong assumption that each client can exactly solve its local objective. The free parameters are user-chosen hyperparameters, not fitted constants that make derivations work. The invented entities are mathematical definitions rather than empirical postulates.

free parameters (4)

personalization factor αi (Scafflix) = 0.3 for FEMNIST; grid-searched over {0.1,0.3,0.5,0.7,0.9}
Hand-chosen in experiments to balance global and personalized models; the theory allows any value but the initial gap depends on αi^2.
global pruning ratio p (FedP3) = 0.9 by default; ablated 0.5,0.7,0.9,1.0
User-set ratio of model parameters retained after server-to-client pruning; directly controls communication-accuracy trade-off.
SPPM-AS stepsize γ and local rounds K = γ up to 1000, K up to 10 in experiments
Hyperparameters controlling convergence-accuracy trade-off; tuned per dataset.
SymWanda norm α and sparsity ratio = α=1.0 default; sparsity 50-60% in experiments
Norm and sparsity level chosen via validation; the theoretical result does not determine their values.

assumptions (6)

standard math Strong convexity and L-smoothness of each fi (and PL/KL condition for f+R in Chapter 2)
Invoked in Theorems 2.4.1, 2.4.2, 3.2.3; Section 1.5.1. Standard regularity conditions for linear convergence.
domain assumption Unbiasedness and expected smoothness of stochastic gradients (Assumptions 3.2.1, 3.2.2)
Used in Theorem 3.2.3 for Scafflix; requires access to unbiased gradient estimates with bounded variance (Ci).
domain assumption Exact local minimization x*_i = argmin f_i per client (Algorithm 4 line 3)
Scafflix requires each client to solve its own ERM exactly; this is impractical for nonconvex deep learning and only approximated in the experiments.
domain assumption Differentiability and strong convexity of each fi, and proper nonvacuous sampling (Assumptions 5.3.1-5.3.3)
Foundations for SPPM-AS convergence in Chapter 5.
domain assumption Smoothness and bounded variance (Assumption C.3.1) for FedP3 analysis
Referenced in Theorem 4.3.3; full statement is in the appendix, not in the main text.
ad hoc to paper Single-weight elimination and row/column independence in pruning score (Lemma 6.3.1)
SymWanda's theory analyzes removing one weight at a time assuming the score decomposes as |Wjk|(||X:j||+||Yk:||), ignoring interactions between pruned weights.

invented entities (2)

C(η, ω) compressor class
purpose: Two-parameter bias-variance characterization of compressors generalizing U(ω) and B(α)
A mathematical definition introduced in Section 2.2.1; no empirical entity, so no independent evidence.
Symmetric calibration matrices X and Y in SymWanda
purpose: To express pruning error as input and output reconstruction loss and recover Wanda/RIA scores
Design choices in Section 6.3.2; the 'recovery' of RIA in Theorem 6.3.5 is by construction.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization." pith.science (2026). https://pith.science/paper/DEBXESNF

@misc{pith2026250908233,
  author       = {Pith},
  title        = {Pith review of: Strategies for Improving Communication Efficiency in Distributed and Federated Learning: Compression, Local Training, and Personalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DEBXESNF}},
  note         = {Machine review of arXiv:2509.08233}
}

read the original abstract

Distributed and federated learning are essential paradigms for training models across decentralized data sources while preserving privacy, yet communication overhead remains a major bottleneck. This dissertation explores strategies to improve communication efficiency, focusing on model compression, local training, and personalization. We establish a unified framework for biased and unbiased compression operators with convergence guarantees, then propose adaptive local training strategies that incorporate personalization to accelerate convergence and mitigate client drift. In particular, Scafflix balances global and personalized objectives, achieving superior performance under both IID and non-IID settings. We further introduce privacy-preserving pruning frameworks that optimize sparsity while minimizing communication costs, with Cohort-Squeeze leveraging hierarchical aggregation to reduce cross-device overhead. Finally, SymWanda, a symmetric post-training pruning method, enhances robustness under high sparsity and maintains accuracy without retraining. Extensive experiments on benchmarks and large-scale language models demonstrate favorable trade-offs among accuracy, convergence, and communication, offering theoretical and practical insights for scalable, efficient distributed learning.

Figures

Figures reproduced from arXiv: 2509.08233 by the authors.

**Figure 2.1.** In the three algorithms, g t+1 is an estimate of ∇f(x t ), the h t i are control variates converging to ∇fi(x ⋆ ), and their average h t = 1 n Pn i=1 h t i is maintained and updated by the master. EF21 is a particular case of EF-BV, when ν = λ = 1 and the compressors are in B(α); then g t+1 is simply equal to h t+1 for every t ≥ 0. DIANA is a particular case of EF-BV, when ν = 1 and the compressors are in U(ω); the… view at source ↗

**Figure 2.2.** Experimental results. We plot f(x t ) − f ⋆ with respect to the number of bits sent by each node during the learning process, which is proportional to tk. Top row: comp-(1, d/2), overlapping ξ = 1. Middle row: comp-(1, d/2), overlapping ξ = 2. Bottom row: comp-(2, d/2), overlapping ξ = 1 [PITH_FULL_IMAGE:figures/full_fig_p039_2_2.png] view at source ↗

**Figure 3.1.** The objective gap f(x k ) − f ⋆ and the squared gradient norm [PITH_FULL_IMAGE:figures/full_fig_p046_3_1.png] view at source ↗

Figures from the paper (15 more)

**Figure 3.2.** Figure 3.2: Comparative generalization analysis with baselines. We set the com [PITH_FULL_IMAGE:figures/full_fig_p047_3_2.png]

**Figure 3.3.** Figure 3.3: Key ablation studies: (a) evaluate the influence on personalization [PITH_FULL_IMAGE:figures/full_fig_p047_3_3.png]

**Figure 3.5.** Figure 3.5: Comparison between global stepsize (dashed lines) and individual stepsizes (solid lines). ensure a fair comparison with other baseline algorithms, we conducted an extensive search of the optimal hyperparameters for each algorithm. The performance assessment of the g…

**Figure 4.1.** Figure 4.1: Pipeline illustration of our proposed framework [PITH_FULL_IMAGE:figures/full_fig_p054_4_1.png]

**Figure 4.2.** Figure 4.2: Comparative Analysis of Layer Overlap Strategies: The left figure [PITH_FULL_IMAGE:figures/full_fig_p058_4_2.png]

**Figure 4.** Figure 4: presents a comparison of different [PITH_FULL_IMAGE:figures/full_fig_p058_4.png]

**Figure 4.3.** Figure 4.3: ResNet18 architecture [PITH_FULL_IMAGE:figures/full_fig_p059_4_3.png]

**Figure 4.4.** Figure 4.4: Comparative Analysis of Server to Client Global Pruning Strategies: [PITH_FULL_IMAGE:figures/full_fig_p060_4_4.png]

**Figure 4.5.** Figure 4.5: Comparison of various model aggregation strategies. p = 0.9. The underlying idea is that clients with more comprehensive network information should have greater weight in parameter contribution. A more flexible approach is attention averaging, where αi is learnable,…

**Figure 5.1.** Figure 5.1: The total communication cost (defined as [PITH_FULL_IMAGE:figures/full_fig_p064_5_1.png]

**Figure 5.2.** Figure 5.2: Analysis of total communication costs against local communication [PITH_FULL_IMAGE:figures/full_fig_p072_5_2.png]

**Figure 5.4.** Figure 5.4: Convergence analysis compared to popular [PITH_FULL_IMAGE:figures/full_fig_p073_5_4.png]

**Figure 5.5.** Figure 5.5: Server-hub-client hierarchical FL architecture [PITH_FULL_IMAGE:figures/full_fig_p074_5_5.png]

**Figure 5.6.** Figure 5.6: Communication cost for achieving 70% accuracy in hierarchical FL (c1 = 0.05, c2 = 1). 0 200 400 600 800 1000 Total Communication Cost 10 −1 10 0 Train Loss SPPM-NICE, °=1.0e+00 SPPM-NICE, °=1.0e+04 SPPM-NICE, °=1.0e+08 LocalGD, °=1.0e-01 LocalGD, °=5.0e-02 0 200 400…

**Figure 6.1.** Figure 6.1: Visualization of the dense weight matrix in LLaMA2-7b. [PITH_FULL_IMAGE:figures/full_fig_p086_6_1.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

254 extracted references · 25 canonical work pages

[1]

Deep learning with differential privacy

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308--318, 2016

2016
[2]

Sparse communication for distributed gradient descent

Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. arXiv preprint arXiv:1704.05021, 2017

arXiv 2017
[3]

Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction

Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. In Advances in Neural Information Processing Systems, 2022

2022
[4]

Optimal Gradient Compression for Distributed and Federated Learning

A. Albasyoni, M. Safaryan, L. Condat, and P. Richt\'arik . Optimal gradient compression for distributed and federated learning. preprint arXiv:2010.03246, 2020

work page Pith review arXiv 2010
[5]

Alistarh, D

D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. QSGD: Communication-efficient SGD via gradient quantization and encoding . In Proc. of 31st Conf. Neural Information Processing Systems (NIPS), pages 1709--1720, 2017

2017
[6]

Geo-indistinguishability: Differential privacy for location-based systems

Miguel E Andr \'e s, Nicol \'a s E Bordenabe, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. Geo-indistinguishability: Differential privacy for location-based systems. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, pages 901--914, 2013

2013
[7]

M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary. Federated learning with personalization layers. preprint arXiv:1912.00818, 2019

arXiv 1912
[8]

A tight convergence analysis for stochastic gradient descent with delayed updates

Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Algorithmic Learning Theory, pages 111--132. PMLR, 2020

2020

Show all 254 references

[9]

Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity

Hilal Asi and John C Duchi. Stochastic (approximate) proximal point methods: Convergence, optimality, and adaptivity . SIAM Journal on Optimization, 29 0 (3): 0 2257--2290, 2019

2019
[10]

Minibatch stochastic approximate proximal point methods

Hilal Asi, Karan Chadha, Gary Cheng, and John C Duchi. Minibatch stochastic approximate proximal point methods . In Advances in Neural Information Processing Systems, volume 33, pages 21958--21968. Curran Associates, Inc., 2020

2020
[11]

Attouch and J

H. Attouch and J. Bolte. On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. Math. Program., 116: 0 5--116, 2009

2009
[12]

J. Baek, W. Jeong, J. Jin, J. Yoon, and S. J. Hwang. Personalized subgraph federated learning. In Proc.\ of 40th Int. Conf. Machine Learning (ICML), PMLR 202 , pages 1396--1415, 2023

2023
[13]

L. P. Barnes, H. A. Inan, B. Isik, and A. \"Ozg\"ur . rTop-k : A statistical estimation approach to distributed SGD . IEEE J. Sel. Areas Inf. Theory, 1 0 (3): 0 897--907, November 2020

2020
[14]

Private empirical risk minimization: Efficient algorithms and tight error bounds

Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th annual symposium on foundations of computer science, pages 464--473. IEEE, 2014

2014
[15]

H. H. Bauschke and P. L. Combettes. Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, New York, 2nd edition, 2017

2017
[16]

Deep rewiring: Training very sparse deep networks

Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. In International Conference on Learning Representations, 2018

2018
[17]

Incremental proximal methods for large scale convex optimization

Dimitri P Bertsekas. Incremental proximal methods for large scale convex optimization . Mathematical Programming, 129 0 (2): 0 163--195, 2011

2011
[18]

Beznosikov, S

A. Beznosikov, S. Horv \'a th, P. Richt \'a rik, and M. Safaryan. On biased compression for distributed learning. preprint arXiv:2002.12410, 2020

2002 arXiv
[19]

On biased compression for distributed learning

Aleksandr Beznosikov, Samuel Horv \'a th, Peter Richt \'a rik, and Mher Safaryan. On biased compression for distributed learning. Journal of Machine Learning Research, 24 0 (276): 0 1--50, 2023

2023
[20]

Sebastian Bischoff, Stephan Günnemann, Martin Jaggi, and Sebastian U. Stich. On second-order optimization methods for federated learning . arXiv preprint arXiv:2303.10581, 2023

2023 arXiv
[21]

Towards federated learning at scale: Syste m design

Keith Bonawitz. Towards federated learning at scale: Syste m design. arXiv preprint arXiv:1902.01046, 2019

1902 arXiv
[22]

Language models are few-shot learners

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 0 1877--1901, 2020

1901
[23]

Quasi-Newton methods and their application to function minimisation

Charles G Broyden. Quasi-Newton methods and their application to function minimisation . Mathematics of Computation, 21 0 (99): 0 368--381, 1967

1967
[24]

D. Bui, K. Malik, J. Goetz, H. Liu, S. Moon, A. Kumar, and K. G. Shin. Federated user representation learning. preprint arXiv:1909.12535, 2019

1909 arXiv
[25]

Efficient implementation of stochastic proximal point algorithm for matrix and tensor completion

Aysegul Bumin and Kejun Huang. Efficient implementation of stochastic proximal point algorithm for matrix and tensor completion . In 29th European Signal Processing Conference (EUSIPCO), pages 1050--1054. IEEE, 2021

2021
[26]

Caldas, P

S. Caldas, P. Wua, T. Lia, Kone c n\' y J., B. McMahan, Virginia Smith, and Ameet Talwalkar. Leaf: a benchmark for federated settings. arXiv preprint arXiv:1812.01097, 2018

2018 arXiv
[27]

Accelerated, optimal and parallel: Some results on model-based stochastic optimization

Karan Chadha, Gary Cheng, and John Duchi. Accelerated, optimal and parallel: Some results on model-based stochastic optimization . In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 2811--2827. PMLR, 2022

2022
[28]

Chang and C.-J

C.-C. Chang and C.-J. Lin. Lib S V M : A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2 0 (3): 0 27, 2011

2011
[29]

Broadening the scope of differential privacy using metrics

Konstantinos Chatzikokolakis, Miguel E Andr \'e s, Nicol \'a s Emilio Bordenabe, and Catuscia Palamidessi. Broadening the scope of differential privacy using metrics. In Privacy Enhancing Technologies: 13th International Symposium, PETS 2013, Bloomington, IN, USA, July 10-12, ...

2013
[30]

D. Chen, L. Yao, D. Gao, B. Ding, and Y. Li. Efficient personalized federated learning via sparse model-adaptation. preprint arXiv:2305.02776, 2023

2023 arXiv
[31]

Visualgpt: Data-efficient adaptation of pretrained language models for image captioning

Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18030--18040, 2022

2022
[32]

Y. Chen, X. Qin, J. Wang, C. Yu, and W. Gao. Fedhealth: A federated transfer learning framework for wearable healthcare. IEEE Intelligent Systems, 35 0 (4): 0 83--93, 2020

2020
[33]

A comprehensive survey on model compression and acceleration

Tejalal Choudhary, Vipul Mishra, Anurag Goswami, and Jagannathan Sarangapani. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review, 53: 0 5113--5155, 2020

2020
[34]

Palm: Scaling language modeling with pathways

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022

2022 arXiv
[35]

Boolq: Exploring the surprising difficulty of natural yes/no questions

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

1905 arXiv
[36]

Think you have solved question answering? try arc, the ai2 reasoning challenge

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018

2018 arXiv
[37]

Emnist: Extending mnist to handwritten letters

Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921--2926. IEEE, 2017

2017
[38]

Condat and P

L. Condat and P. Richt\'arik . MURANA: A generic framework for stochastic variance-reduced optimization. In Proc. of the Mathematical and Scientific Machine Learning (MSML) conference, 2022

2022
[39]

Condat and P

L. Condat and P. Richt \'a rik. RandProx : P rimal-dual optimization algorithms with randomized proximal updates. In Proc.\ of Int. Conf. Learning Representations (ICLR) , 2023

2023
[40]

Condat, I

L. Condat, I. Agarsk \'y , and P. Richt \'a rik. Provably doubly accelerated federated learning: The first theoretically successful combination of local training and compressed communication. preprint arXiv:2210.13277, 2022 a

2022
[41]

Condat, D

L. Condat, D. Kitahara, A. Contreras, and A. Hirabayashi. Proximal splitting algorithms for convex optimization: A tour of recent advances, with new twists. SIAM Review, 2022 b . to appear

2022
[42]

Condat, G

L. Condat, G. Malinovsky, and P. Richt \'a rik. Distributed proximal splitting algorithms with rates and acceleration. Frontiers in Signal Processing, 1, January 2022 c

2022
[43]

Condat, I

L. Condat, I. Agarsk\'y , G. Malinovsky, and P. Richt \'a rik. TAMUNA : D oubly accelerated federated learning with local training, compression, and partial participation. preprint arXiv:2302.09832, 2023

2023 arXiv
[44]

Only tails matter: Average-case universality and robustness in the convex regime

Leonardo Cunha, Gauthier Gidel, Fabian Pedregosa, Damien Scieur, and Courtney Paquette. Only tails matter: Average-case universality and robustness in the convex regime. In International Conference on Machine Learning, pages 4474--4491. PMLR, 2022

2022
[45]

Large scale distributed deep networks

Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc'aurelio Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. Advances in neural information processing systems, 25, 2012

2012
[46]

Heterofl: Computation and communication efficient federated learning for heterogeneous clients

Enmao Diao, Jie Ding, and Vahid Tarokh. Heterofl: Computation and communication efficient federated learning for heterogeneous clients. In International Conference on Learning Representations, 2021

2021
[47]

Differentially private and communication efficient collaborative learning

Jiahao Ding, Guannan Liang, Jinbo Bi, and Miao Pan. Differentially private and communication efficient collaborative learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7219--7227, 2021

2021
[48]

C. T. Dinh, N. H. Tran, and T. D. Nguyen. Personalized federated learning with Moreau envelopes. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), volume 33, pages 21394--21405, 2020

2020
[49]

The llama 3 herd of models

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

2024 arXiv
[50]

Resist: Layer-wise decomposition of resnets for distributed training

Chen Dun, Cameron R Wolfe, Christopher M Jermaine, and Anastasios Kyrillidis. Resist: Layer-wise decomposition of resnets for distributed training. In Uncertainty in Artificial Intelligence, pages 610--620. PMLR, 2022

2022
[51]

Efficient and light-weight federated learning via asynchronous distributed dropout

Chen Dun, Mirian Hipolito, Chris Jermaine, Dimitrios Dimitriadis, and Anastasios Kyrillidis. Efficient and light-weight federated learning via asynchronous distributed dropout. In International Conference on Artificial Intelligence and Statistics, pages 6630--6660. PMLR, 2023

2023
[52]

Calibrating noise to sensitivity in private data analysis

Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pages 265--284. Springer, 2006

2006
[53]

The algorithmic foundations of differential privacy

Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science , 9 0 (3--4): 0 211--407, 2014

2014
[54]

Extreme compression of large language models via additive quantization

Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. In Forty-first International Conference on Machine Learning, 2024

2024
[55]

Cizsl++: Creativity inspired generative zero-shot learning

Mohamed Elhoseiny, Kai Yi, and Mohamed Elfeki. Cizsl++: Creativity inspired generative zero-shot learning. T-PAMI major revision, 2021

2021
[56]

Rigging the lottery: Making all tickets winners

Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference on machine learning, pages 2943--2952. PMLR, 2020

2020
[57]

Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach

Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. Advances in neural information processing systems, 33: 0 3557--3568, 2020

2020
[58]

Fatkhullin, I

I. Fatkhullin, I. Sokolov, E. Gorbunov, Z. Li, and P. Richt \'a rik. EF21 with bells & whistles: Practical algorithmic extensions of modern error feedback. preprint arXiv:2110.03294, 2021

2021 arXiv
[59]

Private stochastic convex optimization: optimal rates in linear time

Vitaly Feldman, Tomer Koren, and Kunal Talwar. Private stochastic convex optimization: optimal rates in linear time. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 439--449, 2020

2020
[60]

Fletcher

R. Fletcher. A new approach to variable metric algorithms . The Computer Journal, 13 0 (3): 0 317--322, 1970

1970
[61]

Gerald B. Folland. Real Analysis: Modern Techniques and Their Applications . 1984

1984
[62]

The lottery ticket hypothesis: Finding sparse, trainable neural networks

Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018

2018 arXiv
[63]

Sparsegpt: Massive language models can be accurately pruned in one-shot

Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323--10337. PMLR, 2023

2023
[64]

OPTQ : Accurate quantization for generative pre-trained transformers

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ : Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=tcbBPnfwxS

2023
[65]

vqSGD : V ector quantized stochastic gradient descent

Venkata Gandikota, Daniel Kane, Raj Kumar Maity, and Arya Mazumdar. vqSGD : V ector quantized stochastic gradient descent. preprint arXiv:1911.07971, 2019

1911 arXiv
[66]

A survey on heterogeneous federated learning

Dashan Gao, Xin Yao, and Qiang Yang. A survey on heterogeneous federated learning. arXiv preprint arXiv:2210.04505, 2022 a

2022 arXiv
[67]

A framework for few-shot language model evaluation

Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 10: 0 8--9, 2021

2021
[68]

Feddc: Federated learning with non-iid data via local drift decoupling and correction

Liang Gao, Huazhu Fu, Li Li, Yingwen Chen, Ming Xu, and Cheng-Zhong Xu. Feddc: Federated learning with non-iid data via local drift decoupling and correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10112--10121, June 2022 b

2022
[69]

Gasanov, A

E. Gasanov, A. Khaled, S. Horv\' a th, and P. Richt\' a rik. Flix: A simple and communication-efficient alternative to local methods in federated learning. In Proc.\ of 24th Int. Conf. Artificial Intelligence and Statistics (AISTATS) , 2022

2022
[70]

Stochastic first-and zeroth-order methods for nonconvex stochastic programming

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23 0 (4): 0 2341--2368, 2013

2013
[71]

An efficient framework for clustered federated learning

Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in Neural Information Processing Systems, 33: 0 19586--19597, 2020

2020
[72]

A family of variable-metric methods derived by variational means

Donald Goldfarb. A family of variable-metric methods derived by variational means . Mathematics of Computation, 24 0 (109): 0 23--26, 1970

1970
[73]

Gorbunov, F

E. Gorbunov, F. Hanzely, and P. Richt\' a rik. Local SGD : Unified theory and new efficient methods. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), 2020 a

2020
[74]

Gorbunov, F

E. Gorbunov, F. Hanzely, and P. Richt\' a rik . A unified theory of SGD: Variance reduction, sampling, quantization and coordinate descent. In Proc. of 23rd Int. Conf. Artificial Intelligence and Statistics (AISTATS), 2020 b

2020
[75]

Gorbunov, D

E. Gorbunov, D. Kovalev, D. Makarenko, and P. Richt\'arik . Linearly converging error compensated SGD . In Proc. of 34th Conf. Neural Information Processing Systems (NeurIPS), 2020 c

2020
[76]

Super-acceleration with cyclical step-sizes

Baptiste Goujaud, Damien Scieur, Aymeric Dieuleveut, Adrien B Taylor, and Fabian Pedregosa. Super-acceleration with cyclical step-sizes. In International Conference on Artificial Intelligence and Statistics, pages 3028--3065. PMLR, 2022

2022
[77]

R. M. Gower, N. Loizou, X. Qian, A. Sailanbayev, E. Shulgin, and P. Richt\' a rik. SGD : G eneral analysis and improved rates. In Proc.\ of 36th Int. Conf. Machine Learning (ICML), PMLR 97 , pages 5200--5209, 2019 a

2019
[78]

R. M. Gower, M. Schmidt, F. Bach, and P. Richt\'arik . Variance-reduced methods for machine learning. Proc. of the IEEE, 108 0 (11): 0 1968--1983, November 2020

1968
[79]

R. M. Gower, P. Richt\'arik , and F. Bach. Stochastic quasi-gradient methods: V ariance reduction via J acobian sketching. Math. Program., 188: 0 135--192, July 2021

2021
[80]

Sgd: General analysis and improved rates

Robert Mansel Gower, Nicolas Loizou, Xun Qian, Alibek Sailanbayev, Egor Shulgin, and Peter Richt \'a rik. Sgd: General analysis and improved rates. In International conference on machine learning, pages 5200--5209. PMLR, 2019 b

2019
[81]

Grudzie \'n , G

M. Grudzie \'n , G. Malinovsky, and P. Richt \'a rik. Can 5th Generation Local Training Methods Support Client Sampling? Yes! In Proc.\ of Int. Conf. Artificial Intelligence and Statistics (AISTATS) , April 2023

2023
[82]

Haddadpour and M

F. Haddadpour and M. Mahdavi. On the convergence of local descent methods in federated learning. preprint arXiv:1910.14425, 2019

1910 arXiv
[83]

Learning both weights and connections for efficient neural network

Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

2015
[84]

Hanzely and P

F. Hanzely and P. Richt \'a rik. Federated learning of a mixture of global and local models. preprint arXiv:2002.05516, 2020

2002 arXiv
[85]

One method to rule them all: Variance reduction for data, parameters and many new methods

Filip Hanzely and Peter Richt \'a rik. One method to rule them all: Variance reduction for data, parameters and many new methods. preprint arXiv:1905.11266, 2019

1905 arXiv
[86]

Personalized federated learning: A unified framework and universal optimization techniques

Filip Hanzely, Boxin Zhao, and Mladen Kolar. Personalized federated learning: A unified framework and universal optimization techniques. arXiv preprint arXiv:2102.09743, 2021

2021 arXiv
[87]

A damped newton method achieves global o(1/k^2) and local quadratic convergence rate

Slavom \' r Hanzely, Dmitry Kamzolov, Dmitry Pasechnyuk, Alexander Gasnikov, Peter Richt\' a rik, and Martin Tak\' a c . A damped newton method achieves global o(1/k^2) and local quadratic convergence rate. Advances in Neural Information Processing Systems, 35: 0 25320--25334, 2022

2022
[88]

Federated learning for mobile keyboard prediction

Andrew Hard, Kanishka Rao, Rajiv Mathews, Swaroop Ramaswamy, Fran c oise Beaufays, Sean Augenstein, Hubert Eichner, Chlo \'e Kiddon, and Daniel Ramage. Federated learning for mobile keyboard prediction . arXiv preprint arXiv:1811.03604, 2018

2018 arXiv
[89]

Fednas: Federated deep learning via neural architecture search

Chaoyang He, Erum Mushtaq, Jie Ding, and Salman Avestimehr. Fednas: Federated deep learning via neural architecture search. 2021

2021
[90]

Deep residual learning for image recognition

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770--778, 2016

2016
[91]

Methods of conjugate gradients for solving linear systems , volume 49

Magnus Rudolph Hestenes, Eduard Stiefel, et al. Methods of conjugate gradients for solving linear systems , volume 49. NBS Washington, DC, 1952

1952
[92]

Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks

Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22 0 (241): 0 1--124, 2021

2021
[93]

Horv\'ath , D

S. Horv\'ath , D. Kovalev, K. Mishchenko, S. Stich, and P. Richt\'arik . Stochastic distributed learning with gradient quantization and variance reduction. Optimization Methods and Software, 2022

2022
[94]

Natural compression for distributed deep learning

Samuel Horv\' a th, Chen-Yu Ho, Ludov\' i t Horv\' a th, Atal Narayan Sahu, Marco Canini, and Peter Richt\' a rik. Natural compression for distributed deep learning. preprint arXiv:1905.10988, 2019

1905 arXiv
[95]

Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout

Samuel Horv\' a th, Stefanos Laskaridis, Mario Almeida, Ilias Leontiadis, Stylianos Venieris, and Nicholas Lane. Fjord: Fair and accurate federated learning under heterogeneous targets with ordered dropout. Advances in Neural Information Processing Systems, 34: 0 12876--12889, 2021

2021
[96]

Fedtiny: Pruned federated learning towards specialized tiny models

Hong Huang, Lan Zhang, Chaoyue Sun, Ruogu Fang, Xiaoyong Yuan, and Dapeng Wu. Fedtiny: Pruned federated learning towards specialized tiny models. arXiv preprint arXiv:2212.01977, 2022

2022 arXiv
[97]

Privacy-preserving learning via deep net pruning

Yangsibo Huang, Yushan Su, Sachin Ravi, Zhao Song, Sanjeev Arora, and Kai Li. Privacy-preserving learning via deep net pruning. arXiv preprint arXiv:2003.01876, 2020

2003 arXiv
[98]

Quantized neural networks: Training neural networks with low precision weights and activations

Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. Journal of Machine Learning Research, 18 0 (187): 0 1--30, 2018

2018
[99]

Towards practical differentially private convex optimization

Roger Iyengar, Joseph P Near, Dawn Song, Om Thakkar, Abhradeep Thakurta, and Lun Wang. Towards practical differentially private convex optimization. In 2019 IEEE Symposium on Security and Privacy (SP), pages 299--316. IEEE, 2019

2019
[100]

Communication-efficient distributed dual coordinate ascent

Martin Jaggi, Virginia Smith, Martin Tak \'a c, Jonathan Terhorst, Sanjay Krishnan, Thomas Hofmann, and Michael I Jordan. Communication-efficient distributed dual coordinate ascent. Advances in neural information processing systems, 27, 2014

2014
[101]

Doubly adaptive scaled algorithm for machine learning using second-order information

Majid Jahani, Sergey Rusakov, Zheng Shi, Peter Richt \'a rik, Michael W Mahoney, and Martin Tak \'a c . Doubly adaptive scaled algorithm for machine learning using second-order information . arXiv preprint arXiv:2109.05198, 2021

2021 arXiv
[102]

Phi-2: The surprising power of small language models

Mojan Javaheripi, S \'e bastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio C \'e sar Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language models. Microsoft Research Blog, 2023

2023
[103]

Creative walk adversarial networks: Novel art generation with probabilistic random walk deviation from style norms

Divyansh Jha, Kai Yi, Ivan Skorokhodov, and Mohamed Elhoseiny. Creative walk adversarial networks: Novel art generation with probabilistic random walk deviation from style norms. In International Conference on Innovative Computing and Cloud Computing, 2022. URL https://api.sem...

2022
[104]

FedExP: Speeding up federated averaging via extrapolation

Divyansh Jhunjhunwala, Shiqiang Wang, and Gauri Joshi. FedExP: Speeding up federated averaging via extrapolation . arXiv preprint arXiv:2301.09604, 2023

2023 arXiv
[105]

Federated learning in smart city sensing: Challenges and opportunities

Ji Chu Jiang, Burak Kantarci, Sema Oktug, and Tolga Soyata. Federated learning in smart city sensing: Challenges and opportunities. Sensors, 20 0 (21): 0 6230, 2020

2020
[106]

Model pruning enables efficient federated learning on edge devices

Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022 a

2022
[107]

Model pruning enables efficient federated learning on edge devices

Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices. IEEE Transactions on Neural Networks and Learning Systems, 2022 b

2022
[108]

Advances and open problems in federated learning

Peter Kairouz, H Brendan McMahan, Brendan Avent, Aur \'e lien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning , 1...

2021
[109]

Kairouz et al

P. Kairouz et al. Advances and open problems in federated learning. Foundations and Trends in Machine Learning, 14 0 (1--2): 0 1--210, 2019

2019
[110]

Layer-wise and dimension-wise locally adaptive federated learning , 2022

Belhal Karimi, Ping Li, and Xiaoyun Li. Layer-wise and dimension-wise locally adaptive federated learning , 2022

2022
[111]

Linear convergence of gradient and proximal-gradient methods under the P olyak- ojasiewicz condition

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal-gradient methods under the P olyak- ojasiewicz condition. In Paolo Frasconi, Niels Landwehr, Giuseppe Manco, and Jilles Vreeken, editors, Machine Learning and Knowledge Discovery in Datab...

2016
[112]

Karimireddy, S

S. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. Suresh. SCAFFOLD : Stochastic controlled averaging for on-device federated learning. In Proc. of Int. Conf. Machine Learning (ICML), 2020 a

2020
[113]

Mime: Mimicking centralized stochastic algorithms in federated learning

Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale, Mehryar Mohri, Sashank J Reddi, Sebastian U Stich, and Ananda Theertha Suresh. Mime: Mimicking centralized stochastic algorithms in federated learning . arXiv preprint arXiv:2008.03606, 2020 b

2008 arXiv
[114]

Scaffold: Stochastic controlled averaging for federated learning

Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International conference on machine learning, pages 5132--5143. PMLR, 2020 c

2020
[115]

Khaled, K

A. Khaled, K. Mishchenko, and P. Richt\' a rik. First analysis of local GD on heterogeneous data. paper arXiv:1909.04715, presented at NeurIPS Workshop on Federated Learning for Data Privacy and Confidentiality, 2019

1909 arXiv
[116]

Khaled, K

A. Khaled, K. Mishchenko, and P. Richt\' a rik. Tighter theory for local SGD on identical and heterogeneous data. In Proc. of 23rd Int. Conf. Artificial Intelligence and Statistics (AISTATS), 2020 a

2020
[117]

Khaled, O

A. Khaled, O. Sebbouh, N. Loizou, R. M. Gower, and P. Richt\' a rik . Unified analysis of stochastic gradient methods for composite convex and smooth optimization. preprint arXiv:2006.11573, 2020 b

2006 arXiv
[118]

Faster federated optimization under second-order similarity

Ahmed Khaled and Chi Jin. Faster federated optimization under second-order similarity . In The Eleventh International Conference on Learning Representations, 2023

2023
[119]

Better theory for sgd in the nonconvex world

Ahmed Khaled and Peter Richt \'a rik. Better theory for sgd in the nonconvex world. arXiv preprint arXiv:2002.03329, 2020

2002 arXiv
[120]

Federated optimization: Distributed machine learning for on-device intelligence

Jakub Kone c n \`y , H Brendan McMahan, Daniel Ramage, and Peter Richt \'a rik. Federated optimization: Distributed machine learning for on-device intelligence. arXiv preprint arXiv:1610.02527, 2016

2016 arXiv
[121]

Brendan McMahan, Felix X

Jakub Kone c n\' y , H. Brendan McMahan, Felix X. Yu, Peter Richt\' a rik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. In NIPS Workshop on Private Multi-Party Machine Learning, 2016

2016
[122]

Learning multiple layers of features from tiny images

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

2009
[123]

Soft threshold weight reparameterization for learnable sparsity

Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, pages 5544--5555. PMLR, 2020

2020
[124]

Dynamic sparse training with structured sparsity

Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, and Yani Ioannou. Dynamic sparse training with structured sparsity. arXiv preprint arXiv:2305.02299, 2023

2023 arXiv
[125]

Optimal brain damage

Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989

1989
[126]

Snip: Single-shot network pruning based on connection sensitivity

Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340, 2018

2018 arXiv
[128]

Fedmd: Heterogenous federated learning via model distillation

Daliang Li and Junpu Wang. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019 b

1910 arXiv
[129]

Q. Li, B. He, and D. Song. Model-contrastive federated learning. In Proc. of IEEE/CVF Conf. Computer Vision and Pattern Recognition, pages 10713--10722, 2021 a

2021
[130]

Model-contrastive federated learning

Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10713--10722, 2021 b

2021
[131]

Federated learning: Challenges, methods, and future directions

Tian Li, Anit Kumar Sahu, Ameet Talwalkar, and Virginia Smith. Federated learning: Challenges, methods, and future directions. IEEE signal processing magazine, 37 0 (3): 0 50--60, 2020 a

2020
[132]

Federated optimization in heterogeneous networks

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks . 2020 b

2020
[133]

Federated optimization in heterogeneous networks

Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2: 0 429--450, 2020 c

2020
[134]

X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang. On the convergence of F ed A vg on non- IID data. In Proc. of Int. Conf. Learning Representations (ICLR), 2020 d

2020
[135]

E-sparse: Boosting the large language model inference through entropy-based n: M sparsity

Yun Li, Lin Niu, Xipeng Zhang, Kai Liu, Jianchen Zhu, and Zhanhui Kang. E-sparse: Boosting the large language model inference through entropy-based n: M sparsity. arXiv preprint arXiv:2310.15929, 2023

2023 arXiv
[136]

Z. Li, D. Kovalev, X. Qian, and P. Richt\'arik . Acceleration for compressed gradient descent in distributed and federated optimization. In Proc. of 37th Int. Conf. Machine Learning (ICML), 2020 e

2020
[137]

Simple and optimal stochastic gradient methods for nonsmooth nonconvex optimization

Zhize Li and Jian Li. Simple and optimal stochastic gradient methods for nonsmooth nonconvex optimization. The Journal of Machine Learning Research, 23 0 (1): 0 10891--10951, 2022

2022
[138]

Soteriafl: A unified framework for private federated learning with communication compression

Zhize Li, Haoyu Zhao, Boyue Li, and Yuejie Chi. Soteriafl: A unified framework for private federated learning with communication compression. Advances in Neural Information Processing Systems, 35: 0 4285--4300, 2022

2022
[139]

Adaptive channel sparsity for federated learning under system heterogeneity

Dongping Liao, Xitong Gao, Yiren Zhao, and Cheng-Zhong Xu. Adaptive channel sparsity for federated learning under system heterogeneity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20432--20441, 2023

2023
[140]

On the convergence of shallow neural network training with randomly masked neurons

Fangshuo Liao and Anastasios Kyrillidis. On the convergence of shallow neural network training with randomly masked neurons. Transactions on Machine Learning Research, 2022

2022
[141]

Differentially quantized gradient methods

Chung-Yi Lin, Victoria Kostina, and Babak Hassibi. Differentially quantized gradient methods. 68 0 (9): 0 6078--6097, September 2022

2022
[142]

Stochastic distributed optimization under average second-order similarity: Algorithms and analysis

Dachao Lin, Yuze Han, Haishan Ye, and Zhihua Zhang. Stochastic distributed optimization under average second-order similarity: Algorithms and analysis . Advances in Neural Information Processing Systems, 36, 2024

2024
[143]

Ensemble distillation for robust model fusion in federated learning

Tao Lin, Lingjing Kong, Sebastian U Stich, and Martin Jaggi. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33: 0 2351--2363, 2020

2020
[144]

Deep gradient compression: Reducing the communication bandwidth for distributed training

Yujun Lin, Song Han, Huizi Mao, Yu Wang, and William J Dally. Deep gradient compression: Reducing the communication bandwidth for distributed training. arXiv preprint arXiv:1712.01887, 2017

2017 arXiv
[145]

From distributed machine learning to federated learning: A survey

Ji Liu, Jizhou Huang, Yang Zhou, Xuhong Li, Shilei Ji, Haoyi Xiong, and Dejing Dou. From distributed machine learning to federated learning: A survey. Knowledge and Information Systems, 64 0 (4): 0 885--917, 2022

2022
[146]

Do we actually need dense over-parameterization? in-time over-parameterization in sparse training

Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense over-parameterization? in-time over-parameterization in sparse training. In International Conference on Machine Learning, pages 6989--7000. PMLR, 2021

2021
[147]

Private non-convex federated learning without a trusted server

Andrew Lowy, Ali Ghafelebashi, and Meisam Razaviyayn. Private non-convex federated learning without a trusted server. In International Conference on Artificial Intelligence and Statistics, pages 5749--5786. PMLR, 2023

2023
[148]

Adding vs

Chenxin Ma, Virginia Smith, Martin Jaggi, Michael Jordan, Peter Richt \'a rik, and Martin Tak \'a c. Adding vs. averaging in distributed primal-dual optimization. In International Conference on Machine Learning, pages 1973--1982. PMLR, 2015

1973
[149]

Apollo: An adaptive parameter-wise diagonal quasi-Newton method for nonconvex stochastic optimization

Xuezhe Ma. Apollo: An adaptive parameter-wise diagonal quasi-Newton method for nonconvex stochastic optimization . arXiv preprint arXiv:2009.13586, 2020

2009 arXiv
[150]

PV -tuning: Beyond straight-through estimation for extreme LLM compression

Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Pavlovich Burlachenko, Kai Yi, Dan Alistarh, and Peter Richt \'a rik. PV -tuning: Beyond straight-through estimation for extreme LLM compression. In The Thirty-eighth Annual Conference on Neural Informa...

2024
[151]

Malinovsky, D

G. Malinovsky, D. Kovalev, E. Gasanov, L. Condat, and P. Richt \'a rik. From local SGD to local fixed point methods for federated learning. In Proc.\ of 37th Int. Conf. Machine Learning (ICML) , 2020

2020
[152]

Malinovsky, K

G. Malinovsky, K. Yi, and P. Richt \'a rik. Variance reduced Proxskip : A lgorithm, theory and application to federated learning. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), 2022

2022
[153]

Server-side stepsizes and sampling without replacement provably help in federated optimization

Grigory Malinovsky, Konstantin Mishchenko, and Peter Richt \'a rik. Server-side stepsizes and sampling without replacement provably help in federated optimization . In Proceedings of the 4th International Workshop on Distributed Machine Learning, pages 85--104, 2023

2023
[154]

Maranjyan, M

A. Maranjyan, M. Safaryan, and P. Richt \'a rik. Gradskip: C ommunication-accelerated local gradient methods with better computational complexity. preprint arXiv:2210.16402, 2022

2022 arXiv
[155]

Regularisation d'inequations variationelles par approximations successives

Bernard Martinet. Regularisation d'inequations variationelles par approximations successives . Revue Francaise d'informatique et de Recherche operationelle, 4: 0 154--159, 1970

1970
[156]

RATQ : A universal fixed-length quantizer for stochastic optimization

Prathamesh Mayekar and Himanshu Tyagi. RATQ : A universal fixed-length quantizer for stochastic optimization. 67 0 (5): 0 3130--3154, 2021

2021
[157]

McMahan, E

B. McMahan, E. Moore, D. Ramage, and B. Ag\" u era y Arcas . Federated learning of deep networks using model averaging. preprint arXiv:1602.05629, 2016 a

2016 arXiv
[158]

Communication-efficient learning of deep networks from decentralized data

Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273--1282. PMLR, 2017 a

2017
[159]

Federated learning: Strategies for improving communication efficiency

H Brendan McMahan, FX Yu, P Richtarik, AT Suresh, D Bacon, et al. Federated learning: Strategies for improving communication efficiency. In Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS), Barcelona, Spain, pages 5--10, 2016 b

2016
[160]

Communication-efficient learning of deep networks from decentralized data

H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Ag\" u era y Arcas. Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), 2017 b

2017
[161]

Communication-efficient learning of deep networks from decentralized data

H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Ag\" u era y Arcas. Communication-efficient learning of deep networks from decentralized data . In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS), 2017 c

2017
[162]

Y. Mei, P. Guo, M. Zhou, and V. Patel. Resource-adaptive federated learning with all-in-one neural composition. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), 2022

2022
[163]

Prune at the clients, not the server: Accelerated sparse training in federated learning

Georg Meinhardt, Kai Yi, Laurent Condat, and Peter Richt \'a rik. Prune at the clients, not the server: Accelerated sparse training in federated learning. arXiv preprint arXiv:2405.20623, 2024

2024 arXiv
[164]

Local learning matters: Rethinking data heterogeneity in federated learning

Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, Zhengming Ding, and Chen Chen. Local learning matters: Rethinking data heterogeneity in federated learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8397--8406, June 2022

2022
[165]

Pointer sentinel mixture models

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016

2016 arXiv
[166]

Can a suit of armor conduct electricity? a new dataset for open book question answering

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789, 2018

2018 arXiv
[167]

99\ In Proc

Konstantin Mishchenko, Filip Hanzely, and Peter Richt \'a rik. 99\ In Proc. of 36th Conf. on Uncertainty in Artificial Intelligence (UAI), volume 124, pages 979--988, 2020

2020
[168]

Proximal and federated random reshuffling

Konstantin Mishchenko, Ahmed Khaled, and Peter Richtarik. Proximal and federated random reshuffling . In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 15718--15749. PMLR, 2022 a

2022
[169]

Prox S kip: Y es! L ocal gradient steps provably lead to communication acceleration! F inally! In 39th International Conference on Machine Learning (ICML 2022), 2022 b

Konstantin Mishchenko, Grigory Malinovsky, Sebastian Stich, and Peter Richt\' a rik. Prox S kip: Y es! L ocal gradient steps provably lead to communication acceleration! F inally! In 39th International Conference on Machine Learning (ICML 2022), 2022 b

2022
[170]

Distributed learning with compressed gradient differences

Konstantin Mishchenko, Eduard Gorbunov, Martin Tak \'a c , and Peter Richt \'a rik. Distributed learning with compressed gradient differences. Optimization Methods and Software, pages 1--16, 2024

2024
[171]

Mitra, R

A. Mitra, R. Jaafar, G. Pappas, and H. Hassani. Linear convergence in federated learning: T ackling client heterogeneity and sparse gradients. In Proc. of Conf. Neural Information Processing Systems (NeurIPS), 2021

2021
[172]

Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science

Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, 9 0 (1): 0 2383, 2018

2018
[173]

Proximité et dualité dans un espace hilbertien

Jean-Jacques Moreau. Proximité et dualité dans un espace hilbertien . Bulletin de la Société Mathématique de France, 93: 0 273--299, 1965

1965
[174]

Moritz, R

P. Moritz, R. Nishihara, I. Stoica, and M. I. Jordan. Spark N et: Training deep networks in S park. In Proc. of Int. Conf. Learning Representations (ICLR), 2016

2016
[175]

Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization

Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization. In International Conference on Machine Learning, pages 4646--4655. PMLR, 2019

2019
[176]

Introductory lectures on convex optimization: A basic course, volume 87

Yurii Nesterov. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media, 2003

2003
[177]

Parikh and S

N. Parikh and S. Boyd. Proximal algorithms. Foundations and Trends in Optimization, 3 0 (1): 0 127--239, 2014

2014
[178]

Nonasymptotic convergence of stochastic proximal point methods for constrained convex optimization

Andrei Patrascu and Ion Necoara. Nonasymptotic convergence of stochastic proximal point methods for constrained convex optimization . Journal of Machine Learning Research, 18 0 (198): 0 1--42, 2018

2018
[179]

Philippenko and A

C. Philippenko and A. Dieuleveut. Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees. arXiv:2006.14591, 2020

2006 arXiv
[180]

Povey, X

D. Povey, X. Zhang, and S. Khudanpur. Parallel training of DNN s with natural gradient and parameter averaging. preprint arXiv:1410.7455, 2014

2014 arXiv
[181]

X. Qian, A. Sailanbayev, K. Mishchenko, and P. Richt \'a rik. MISO is making a comeback with better proofs and rates. arXiv:1906.01474, June 2019

1906 arXiv
[182]

Exploring the limits of transfer learning with a unified text-to-text transformer

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21 0 (140): 0 1--67, 2020

2020
[183]

Federated learning for emoji prediction in a mobile keyboard

Swaroop Ramaswamy, Rajiv Mathews, Kanishka Rao, and Fran c oise Beaufays. Federated learning for emoji prediction in a mobile keyboard . arXiv preprint arXiv:1906.04329, 2019

1906 arXiv
[184]

Adaptive federated optimization

Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Kone c n \`y , Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization . arXiv preprint arXiv:2003.00295, 2020

2003 arXiv
[185]

Richt\'arik and M

P. Richt\'arik and M. Tak\'a c . Parallel coordinate descent methods for big data optimization. Math. Program., 156: 0 433--484, 2016

2016
[186]

Ef21: A new, simpler, theoretically better, and practically faster error feedback

Peter Richt \'a rik, Igor Sokolov, and Ilyas Fatkhullin. Ef21: A new, simpler, theoretically better, and practically faster error feedback. Advances in Neural Information Processing Systems, 34: 0 4384--4396, 2021 a

2021
[187]

EF21 : A new, simpler, theoretically better, and practically faster error feedback

Peter Richt \'a rik, Igor Sokolov, and Ilyas Fatkhullin. EF21 : A new, simpler, theoretically better, and practically faster error feedback. In Proc. of 35th Conf. Neural Information Processing Systems (NeurIPS), 2021 b

2021
[188]

J. H. Ro, A. T. Suresh, and K. Wu. FedJAX: Federated learning simulation with JAX . preprint arXiv:2108.02117, 2021

2021 arXiv
[189]

Stochastic proximal iteration: A non-asymptotic improvement upon stochastic gradient descent

Ernest Ryu and Stephen Boyd. Stochastic proximal iteration: A non-asymptotic improvement upon stochastic gradient descent . Technical report, Stanford University, 2016

2016
[190]

Smoothness matrices beat smoothness constants: Better communication compression techniques for distributed optimization

Mher Safaryan, Filip Hanzely, and Peter Richt \'a rik. Smoothness matrices beat smoothness constants: Better communication compression techniques for distributed optimization. Advances in Neural Information Processing Systems, 34: 0 25688--25702, 2021 a

2021
[191]

Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor

Mher Safaryan, Egor Shulgin, and Peter Richt \'a rik. Uncertainty principle for communication compression in distributed and federated learning and the search for an optimal compressor. Information and Inference: A Journal of the IMA, 2021 b

2021
[192]

Goldsmith

Rajarshi Saha, Mert Pilanci, and Andrea J. Goldsmith. Democratic source coding: An optimal fixed-length quantization scheme for distributed optimization under communication constraints. preprint arXiv:2103.07578, 2021

2021 arXiv
[193]

Winogrande: An adversarial winograd schema challenge at scale

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64 0 (9): 0 99--106, 2021

2021
[194]

1-bit stochastic gradient descent and application to data-parallel distributed training of speech DNNs

Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and application to data-parallel distributed training of speech DNNs . In Proc. of Annual Conf. of Int. Speech Communication Association (Interspeech), 2014

2014
[195]

Conditioning of quasi-Newton methods for function minimization

David F Shanno. Conditioning of quasi-Newton methods for function minimization . Mathematics of Computation, 24 0 (111): 0 647--656, 1970

1970
[196]

Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data

Micah J Sheller, Brandon Edwards, G Anthony Reina, Jason Martin, Sarthak Pati, Aikaterini Kotrotsou, Mikhail Milchenko, Weilin Xu, Daniel Marcus, Rivka R Colen, et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data....

2020
[197]

Efficient implementation of incremental proximal-point methods

Alex Shtoff. Efficient implementation of incremental proximal-point methods . arXiv preprint arXiv:2205.01457, 2022

2022 arXiv
[198]

Towards a better theoretical understanding of independent subnetwork training

Egor Shulgin and Peter Richt \'a rik. Towards a better theoretical understanding of independent subnetwork training. arXiv preprint arXiv:2306.16484, 2023

2023 arXiv
[199]

Federated multi-task learning

Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017

2017
[200]

Local sgd converges fast and communicates little

Sebastian U Stich. Local sgd converges fast and communicates little. arXiv preprint arXiv:1805.09767, 2018

2018 arXiv
[201]

On the role of server momentum in federated learning

Jianhui Sun, Xidong Wu, Heng Huang, and Aidong Zhang. On the role of server momentum in federated learning . In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15164--15172, 2024

2024
[202]

A simple and effective pruning approach for large language models

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2023 a

2023
[203]

Fedspeed: Larger local interval, less communication round, and higher generalization accuracy

Yan Sun, Li Shen, Tiansheng Huang, Liang Ding, and Dacheng Tao. Fedspeed: Larger local interval, less communication round, and higher generalization accuracy . arXiv preprint arXiv:2302.10429, 2023 b

2023 arXiv
[204]

Szlendak, A

R. Szlendak, A. Tyurin, and P. Richt \'a rik. Permutation compressors for provably faster distributed nonconvex optimization. In Proc. of Int. Conf. on Learning Representations (ICLR), 2022

2022
[205]

Permutation compressors for provably faster distributed nonconvex optimization

Rafa Szlendak, Alexander Tyurin, and Peter Richt \'a rik. Permutation compressors for provably faster distributed nonconvex optimization. arXiv preprint arXiv:2110.03300, 2021

2021 arXiv
[206]

Fedproto: Federated prototype learning across heterogeneous clients

Yue Tan, Guodong Long, Lu Liu, Tianyi Zhou, Qinghua Lu, Jing Jiang, and Chengqi Zhang. Fedproto: Federated prototype learning across heterogeneous clients. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8432--8440, 2022

2022
[208]

Llama: Open and efficient foundation language models

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 b

2023 arXiv
[209]

Llama 2: Open foundation and fine-tuned chat models

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 c

2023 arXiv
[210]

A survey on distributed machine learning

Joost Verbraeken, Matthijs Wolting, Jonathan Katzy, Jeroen Kloppenburg, Tim Verbelen, and Jan S Rellermeyer. A survey on distributed machine learning. Acm computing surveys (csur), 53 0 (2): 0 1--33, 2020

2020
[211]

Glue: A multi-task benchmark and analysis platform for natural language understanding

A Wang, A Singh, J Michael, F Hill, O Levy, and SR Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arxiv preprint arxiv: 180407461, 2018

2018
[212]

Theoretically better and numerically faster distributed optimization with smoothness-aware quantization techniques

Bokun Wang, Mher Safaryan, and Peter Richt \'a rik. Theoretically better and numerically faster distributed optimization with smoothness-aware quantization techniques. Advances in Neural Information Processing Systems, 35: 0 9841--9852, 2022

2022
[213]

Differentially private empirical risk minimization revisited: Faster and more general

Di Wang, Minwei Ye, and Jinhui Xu. Differentially private empirical risk minimization revisited: Faster and more general. Advances in Neural Information Processing Systems, 30, 2017

2017
[214]

A novel framework for the analysis and design of heterogeneous federated learning

Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. A novel framework for the analysis and design of heterogeneous federated learning . IEEE Transactions on Signal Processing, 69: 0 5234--5249, 2021 a

2021
[215]

Local adaptivity in federated learning: Convergence and consistency

Jianyu Wang, Zheng Xu, Zachary Garrett, Zachary Charles, Luyang Liu, and Gauri Joshi. Local adaptivity in federated learning: Convergence and consistency . arXiv preprint arXiv:2106.02305, 2021 b

2021 arXiv
[216]

Wang et al

J. Wang et al. A field guide to federated optimization. preprint arXiv:2107.06917, 2021

2021 arXiv
[217]

Transformers: State-of-the-art natural language processing

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. EMNLP 2020, page 38, 2020

2020
[218]

Gist: Distributed training for large-scale graph convolutional networks

Cameron R Wolfe, Jingkang Yang, Fangshuo Liao, Arindam Chowdhury, Chen Dun, Artun Bayer, Santiago Segarra, and Anastasios Kyrillidis. Gist: Distributed training for large-scale graph convolutional networks. Journal of Applied and Computational Topology, pages 1--53, 2023

2023
[219]

Y. Wu, S. Zhang, W. Yu, Y. Liu, Q. Gu, D. Zhou, H. Chen, and W. Cheng. Personalized federated learning under mixture of distributions. preprint arXiv:2305.01068, 2023

2023 arXiv
[220]

Smoothquant: Accurate and efficient post-training quantization for large language models

Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087--38099. PMLR, 2023

2023
[221]

Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms

Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017

2017 arXiv
[222]

Xu, C.-Y

H. Xu, C.-Y. Ho, A. M. Abdelmoniem, A. Dutta, E. H. Bergou, K. Karatsenidis, M. Canini, and P. Kalnis. Compressed communication for distributed deep learning: S urvey and quantitative evaluation. Technical report, KAUST, 2020

2020
[223]

FedCM: Federated learning with client-level momentum

Jing Xu, Sen Wang, Liwei Wang, and Andrew Chi-Chih Yao. FedCM: Federated learning with client-level momentum . arXiv preprint arXiv:2106.10874, 2021

2021 arXiv
[224]

H. Yang, H. He, W. Zhang, and X. Cao. Fedsteg: A federated transfer learning framework for secure image steganalysis. IEEE Trans. Network Science and Engineering, 8 0 (2): 0 1084--1094, 2020

2020
[225]

Applied federated learning: Improving Google keyboard query suggestions

Timothy Yang, Galen Andrew, Hubert Eichner, Haicheng Sun, Wei Li, Nicholas Kong, Daniel Ramage, and Fran c oise Beaufays. Applied federated learning: Improving Google keyboard query suggestions . arXiv preprint arXiv:1812.02903, 2018

2018 arXiv
[226]

Heterogeneous federated learning: State-of-the-art and research challenges

Mang Ye, Xiuwen Fang, Bo Du, Pong C Yuen, and Dacheng Tao. Heterogeneous federated learning: State-of-the-art and research challenges. ACM Computing Surveys, 56 0 (3): 0 1--44, 2023 a

2023
[227]

R. Ye, Z. Ni, F. Wu, S. Chen, and Y. Wang. Personalized federated learning with inferred collaboration graphs. In Proc.\ of 40th Int. Conf. Machine Learning (ICML), PMLR 202 , 2023 b

2023
[228]

Fedllm-bench: Realistic benchmarks for federated learning of large language models

Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, and Siheng Chen. Fedllm-bench: Realistic benchmarks for federated learning of large language models. arXiv preprint arXiv:2406.04845, 2024

2024 arXiv
[229]

Domain-aware continual zero-shot learning

Kai Yi, Paul Janson, Wenxuan Zhang, and Mohamed Elhoseiny. Domain-aware continual zero-shot learning. arXiv preprint arXiv:2112.12989, 2021 a

2021 arXiv
[230]

Disentangling semantic features of macromolecules in cryo-electron tomography

Kai Yi, Jianye Pang, Yungeng Zhang, Xiangrui Zeng, and Min Xu. Disentangling semantic features of macromolecules in cryo-electron tomography. arXiv preprint arXiv:2106.14192, 2021 b

2021 arXiv
[231]

Exploring hierarchical graph representation for large-scale zero-shot image classification

Kai Yi, Xiaoqian Shen, Yunhao Gou, and Mohamed Elhoseiny. Exploring hierarchical graph representation for large-scale zero-shot image classification. In European Conference on Computer Vision, pages 116--132. Springer, 2022

2022
[232]

Explicit personalization and local training: Double communication acceleration in federated learning

Kai Yi, Laurent Condat, and Peter Richt \'a rik. Explicit personalization and local training: Double communication acceleration in federated learning. arXiv preprint arXiv:2305.13170, 2023

2023 arXiv
[233]

Fedp3: Federated personalized and privacy-friendly network pruning under model heterogeneity

Kai Yi, Nidham Gazagnadou, Peter Richtarik, and Lingjuan Lyu. Fedp3: Federated personalized and privacy-friendly network pruning under model heterogeneity. ICLR, 2024

2024
[234]

Distributed learning of fully connected neural networks using independent subnet training

Binhang Yuan, Cameron R Wolfe, Chen Dun, Yuxin Tang, Anastasios Kyrillidis, and Chris Jermaine. Distributed learning of fully connected neural networks using independent subnet training. Proceedings of the VLDB Endowment, 15 0 (8): 0 1581--1590, 2022

2022
[235]

Sharper analysis for minibatch stochastic proximal point methods: Stability, smoothness, and deviation

Xiao-Tong Yuan and Ping Li. Sharper analysis for minibatch stochastic proximal point methods: Stability, smoothness, and deviation . Journal of Machine Learning Research, 24 0 (270): 0 1--52, 2023

2023
[236]

On convergence of FedProx : Local dissimilarity invariant bounds, non-smoothness and beyond

Xiaotong Yuan and Ping Li. On convergence of FedProx : Local dissimilarity invariant bounds, non-smoothness and beyond . Advances in Neural Information Processing Systems, 35: 0 10752--10765, 2022

2022
[237]

Visualizing and understanding convolutional networks

Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pages 818--833. Springer, 2014

2014
[238]

Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

1905 arXiv
[239]

Fedlab: A flexible federated learning framework

Dun Zeng, Siqi Liang, Xiangjing Hu, Hui Wang, and Zenglin Xu. Fedlab: A flexible federated learning framework. Journal of Machine Learning Research, 24 0 (100): 0 1--7, 2023

2023
[240]

Unsupervised domain alignment based open set structural recognition of macromolecules captured by cryo-electron tomography

Yuchen Zeng, Gregory Howe, Kai Yi, Xiangrui Zeng, Jing Zhang, Yi-Wei Chang, and Min Xu. Unsupervised domain alignment based open set structural recognition of macromolecules captured by cryo-electron tomography. In 2021 IEEE International Conference on Image Processing (ICIP),...

2021
[241]

Magr: Weight magnitude reduction for enhancing post-training quantization

Aozhong Zhang, Naigang Wang, Yanxia Deng, Xin Li, Zi Yang, and Penghang Yin. Magr: Weight magnitude reduction for enhancing post-training quantization. Advances in neural information processing systems, 2024 a

2024
[242]

Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model

Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George Dahl, Chris Shallue, and Roger B Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in neural information processing systems, 32, 2019

2019
[243]

Fedcr: Personalized federated learning based on across-client common representation with conditional mutual information regularization

Hao Zhang, Chenglin Li, Wenrui Dai, Junni Zou, and Hongkai Xiong. Fedcr: Personalized federated learning based on across-client common representation with conditional mutual information regularization. 2023 a

2023
[244]

Innovation compression for communication-efficient distributed optimization with linear convergence

Jiaqi Zhang, Keyou You, and Lihua Xie. Innovation compression for communication-efficient distributed optimization with linear convergence. preprint arXiv:2105.06697, 2021

2021 arXiv
[245]

Opt: Open pre-trained transformer language models

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022 a

2022 arXiv
[246]

Continual zero-shot learning through semantically guided generative random walks

Wenxuan Zhang, Paul Janson, Kai Yi, Ivan Skorokhodov, and Mohamed Elhoseiny. Continual zero-shot learning through semantically guided generative random walks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11574--11585, 2023 b

2023
[247]

Private and communication-efficient edge learning: a sparse differential gaussian-masking distributed sgd approach

Xin Zhang, Minghong Fang, Jia Liu, and Zhengyuan Zhu. Private and communication-efficient edge learning: a sparse differential gaussian-masking distributed sgd approach. In Proceedings of the Twenty-First International Symposium on Theory, Algorithmic Foundations, and Protocol...

2020
[248]

Plug-and-play: An efficient post-training pruning method for large language models

Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, 2024 b

2024
[249]

Learning best combination for efficient n: M sparsity

Yuxin Zhang, Mingbao Lin, Zhihang Lin, Yiting Luo, Ke Li, Fei Chao, Yongjian Wu, and Rongrong Ji. Learning best combination for efficient n: M sparsity. Advances in Neural Information Processing Systems, 35: 0 941--953, 2022 b

2022
[250]

Dynamic sparse no training: Training-free fine-tuning for sparse llms

Yuxin Zhang, Lirui Zhao, Mingbao Lin, Yunyun Sun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse llms. arXiv preprint arXiv:2310.08915, 2023 c

2023 arXiv
[251]

Local differential privacy-based federated learning for internet of things

Yang Zhao, Jun Zhao, Mengmeng Yang, Teng Wang, Ning Wang, Lingjuan Lyu, Dusit Niyato, and Kwok-Yan Lam. Local differential privacy-based federated learning for internet of things. IEEE Internet of Things Journal, 8 0 (11): 0 8836--8853, 2020

2020
[252]

Federated learning with non-iid data

Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582, 2018

2018 arXiv
[253]

Learning n: m fine-grained structured sparse neural networks from scratch

Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. arXiv preprint arXiv:2102.04010, 2021

2021 arXiv
[254]

Quadratic models for understanding neural network dynamics

Libin Zhu, Chaoyue Liu, Adityanarayanan Radhakrishnan, and Mikhail Belkin. Quadratic models for understanding neural network dynamics. arXiv preprint arXiv:2205.11787, 2022

2022 arXiv
[255]

Deep leakage from gradients

Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in neural information processing systems, 32, 2019

2019
[256]

A survey on model compression for large language models

Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. Transactions of the Association for Computational Linguistics, 12: 0 1556--1577, 2024

2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [1]

Deep learning with differential privacy

Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308--318, 2016

2016

[2] [2]

Sparse communication for distributed gradient descent

Alham Fikri Aji and Kenneth Heafield. Sparse communication for distributed gradient descent. arXiv preprint arXiv:1704.05021, 2017

arXiv 2017

[3] [3]

Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction

Samiul Alam, Luyang Liu, Ming Yan, and Mi Zhang. Fedrolex: Model-heterogeneous federated learning with rolling sub-model extraction. In Advances in Neural Information Processing Systems, 2022

2022

[4] [4]

Optimal Gradient Compression for Distributed and Federated Learning

A. Albasyoni, M. Safaryan, L. Condat, and P. Richt\'arik . Optimal gradient compression for distributed and federated learning. preprint arXiv:2010.03246, 2020

work page Pith review arXiv 2010

[5] [5]

Alistarh, D

D. Alistarh, D. Grubic, J. Li, R. Tomioka, and M. Vojnovic. QSGD: Communication-efficient SGD via gradient quantization and encoding . In Proc. of 31st Conf. Neural Information Processing Systems (NIPS), pages 1709--1720, 2017

2017

[6] [6]

Geo-indistinguishability: Differential privacy for location-based systems

Miguel E Andr \'e s, Nicol \'a s E Bordenabe, Konstantinos Chatzikokolakis, and Catuscia Palamidessi. Geo-indistinguishability: Differential privacy for location-based systems. In Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security, pages 901--914, 2013

2013

[7] [7]

M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary. Federated learning with personalization layers. preprint arXiv:1912.00818, 2019

arXiv 1912

[8] [8]

A tight convergence analysis for stochastic gradient descent with delayed updates

Yossi Arjevani, Ohad Shamir, and Nathan Srebro. A tight convergence analysis for stochastic gradient descent with delayed updates. In Algorithmic Learning Theory, pages 111--132. PMLR, 2020

2020