Pith. sign in

REVIEW 4 major objections 5 minor 42 references

BICompFL: Stochastic Federated Learning with Bi-Directional Compression

T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Two-way minimal-random-coding compression cuts federated communication cost by an order of magnitude while keeping accuracy.

desk verdict Solid experimental contribution on bidirectional MRC compression, but the theoretical proof chain has a monotonicity-direction error in Lemma 2 that currently invalidates the contraction and convergence claims. read the letter →

arxiv 2502.00206 v1 pith:NOBCJ7SN submitted 2025-01-31 cs.LG cs.DCcs.ITmath.ITstat.ML

Maximilian Egger , Rawad Bitar , Antonia Wachter-Zeh , Nir Weinberger , Deniz Gündüz This is my paper · ORCID
classification cs.LGcs.DCcs.ITmath.ITstat.ML
keywords federatedlearningbidirectionalcompressionstochasticminimalrandomcodingimportancesamplingconvergenceanalysisprobabilisticmasktrainingcommunicationefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper takes on the full communication bottleneck of federated learning: not just clients sending updates to the server, but also the server sending the refreshed model back to every client. It proposes BICompFL, which represents each model update as a distribution over binary mask values and transmits a sample of it through minimal random coding (MRC): both sides generate candidate samples from a shared prior, and the sender pays only for an index telling the receiver which candidate to use. Two variants handle global shared randomness versus per-client private randomness, since the extra downlink compression round in the private case adds noise that slows convergence. The paper reports that BICompFL cuts total communication cost by an order of magnitude relative to several compression baselines while keeping the same accuracy, and it proves a contraction property and a high-probability divergence bound that together support convergence analysis for MRC-based compression in both settings.

What carries the argument

The load-bearing object is minimal random coding (MRC), a stochastic compressor that turns a posterior distribution into a short index. Encoder and decoder first generate $n_{\mathrm{IS}}$ candidate samples from a shared prior using shared randomness; the encoder weights the candidates by likelihood ratios $Q(X)/P(X)$, samples an index from that categorical distribution, and sends the index using $\log_2(n_{\mathrm{IS}})$ bits, so the decoder can reproduce an approximate sample from $Q$. The cost is governed by the KL divergence between posterior and prior, so BICompFL's trick is to choose the previous global model estimate as the prior, making the transmitted amount shrink as training converges. The paper's new analytic engine is a Bernoulli-specific refinement of the MRC error bound that vanishes when posterior equals prior and depends explicitly on $n_{\mathrm{IS}}$; this feeds both the high-probability bound on the aggregated model's divergence (Theorem 1) and the contraction property (Lemma 1) that converts MRC into a biased contractive compressor usable in standard convergence proofs.

What would settle it

Simulate minimal random coding for a Bernoulli posterior $q$ and prior $p$ with $|q-p|$ large (say 0.4) and $n_{\mathrm{IS}}=256$, and measure $|\Pr(X=1)-q|$ across many replications; if the error consistently exceeds the Lemma 2 bound $\Delta'/n_{\mathrm{IS}}^2 + O((\Delta+\Delta^2)\sqrt{6p\log(2n_{\mathrm{IS}})/n_{\mathrm{IS}}})$, the refined MRC analysis underlying Theorem 1 is wrong.

Watch

Extended reading notes

Core claim

The central claim is that stochastic federated learning can be made communication-efficient in both directions without sacrificing accuracy, provided the compressor is driven by the right side information. In BICompFL, the server and each client share a prior distribution (initially the global model, later the previous model estimate); the client encodes a sample from its updated posterior by importance-sampling a fixed number of candidate samples from the prior and transmitting only the winning index, costing $\log_2(n_{\mathrm{IS}})$ bits per block. Two versions are analyzed: one with global shared randomness, where the server relays other clients' indices so everyone reconstructs the same global estimate, and one with only private randomness, where the server performs a separate downlink MRC round per client. The theoretical contribution is a refined analysis of MRC for Bernoulli distributions that bounds the per-client estimation error, leading to a high-probability upper bound on the KL divergence between the aggregated model and each client's prior, with an explicit $1/\sqrt{n_{\mathrm{UL}}}$ coupling between uplink and downlink cost. Combined with a contraction property showing the MRC-plus-quantization step shrinks the squared gradient norm by a factor $(1-\delta)$, the paper obtains convergence guarantees for conventional federated learning with stochastic compression as well.

Load-bearing premise

The cost and convergence bounds assume each client's local training moves the model distribution only a bounded distance from the shared prior, and that clients' priors stay close to one another; if those fail, the predicted communication savings do not apply.

Editorial extensions

If this is right

  • On MNIST, Fashion-MNIST, and CIFAR-10, BICompFL reaches roughly the same final accuracy as uncompressed FedAvg while transmitting about 0.06\text{--}0.34 bits per parameter instead of 64, and the paper reports at least a 5\text{--}32-fold saving over the best non-stochastic bidirectional baselines.
  • Because per-round cost is governed by the KL divergence between posterior and prior, communication automatically falls as training converges, and adaptive block allocation reduces it further with little accuracy loss.
  • The contraction property turns MRC plus a stochastic quantizer into a biased contractive compressor, so the same scheme can be dropped into conventional federated learning and inherits convergence guarantees, including with error feedback.
  • Under private (per-client) randomness the extra downlink compression round slows convergence, but splitting the downlink model across clients cuts downlink cost by the number of clients, which is attractive when broadcast is available.
  • Partial client participation is supported by the private-randomness variant, which global shared randomness does not allow.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the refined Bernoulli MRC error bound should transfer to any relative-entropy coding scheme with Bernoulli latents, such as compression of Bayesian neural networks outside federated learning.
  • The explicit $1/\sqrt{n_{\mathrm{UL}}}$ coupling in the divergence bound reads as a design rule: when the downlink is the bottleneck, spending a few more uplink samples per client buys a more accurate global estimate and cheaper downlink transmission.
  • A testable extension is per-block adaptive control of $n_{\mathrm{IS}}$: estimate local divergence per block rather than fixing $n_{\mathrm{IS}}=256$, which could cut bitrate further in heterogeneous regimes where divergence varies strongly across parameter blocks.
  • The restriction to Bernoulli posteriors leaves open whether analogous contraction and divergence bounds hold for categorical or Gaussian posteriors, which would broaden the method to regression and larger models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes BICompFL, a framework for stochastic federated learning with bi-directional compression based on minimal random coding (MRC). Two algorithms are given, one for globally shared randomness (BICompFL-GR) and one for private shared randomness (BICompFL-PR), together with an extension to conventional FL with stochastic quantization (BICompFL-GR-CFL). The experimental section reports large communication-cost reductions on MNIST, Fashion-MNIST, and CIFAR-10 compared with several non-stochastic baselines. The theoretical part claims (i) a contraction property for MRC composed with QSGD (Lemma 1), (ii) a refined MRC analysis for Bernoulli distributions (Lemma 2), (iii) a downlink KL-divergence bound (Theorem 1), and (iv) a convergence guarantee for an error-feedback version (Theorem 2).

Significance. If the theoretical results were correct, the paper would make a useful contribution to communication-efficient FL by extending MRC-based stochastic compression to the downlink and by providing a refined analysis of importance sampling for Bernoulli distributions. The experimental study is extensive, covering multiple datasets, heterogeneity regimes, ablations, and a careful accounting of communication bits, and the reported savings are substantial. The paper also provides a high level of reproducibility detail. However, the theoretical core is not currently established: the proof of Lemma 2 contains a sign error in a monotonicity argument, and Lemma 2 underpins Lemma 1, Theorem 1, and Theorem 2. In addition, the convergence theorem is only sketched and applies to a QSGD-based variant with s ≥ sqrt(2d), which is not the stochastic SignSGD method used in the experiments. These issues are load-bearing for the paper's central theoretical claims, though the experimental methodology and algorithmic framework remain valuable.

major comments (4)
  1. [Section 5, Lemma 1 and Theorem 2] The proof of Lemma 2 contains a sign error in the monotonicity argument. For the branch q < p, the random variable M takes the value q/p (which is smaller than (1-q)/(1-p)) with probability (L+1)/nIS. As L increases, E_Ber[M] decreases, so 1/E_Ber[M] is strictly increasing in L, not non-increasing as claimed. The subsequent inequality 1/E_Ber[M(L)] ≤ 1/E_Ber[M(nIS(p - η_δ))] therefore has the wrong direction, and the same issue appears in the q > p branch. Consequently, the displayed upper bound on |Pr(X_l = 1) - q| does not follow from the preceding steps. Since Lemma 2 is used to prove Lemma 1 and Theorem 1, the contraction property and the downlink KL-divergence bound are not established by the provided proofs. The bound may be repairable (e.g., by using the upper endpoint p + η_δ and an absolute value in the first-order term), but the current statement is not proved.
  2. [Appendix C, Theorem 2] The contraction property in Lemma 1 is stated for the QSGD quantizer with s ≥ sqrt(2d), which requires the number of quantization levels to grow exponentially with dimension; this is impractical for large d and is not the stochastic SignSGD compressor used in the experiments described in Section 4. Theorem 2, stated in Appendix C, is only a sketch and inherits the same condition. Thus the convergence guarantees do not apply to the experimentally evaluated BICompFL-GR-CFL method, and the paper should either provide convergence analysis for the actually implemented compressor or clearly separate the theoretical claims from the experimental validation.
  3. [Appendix B, proof of Theorem 1] In the proof of Theorem 1, the Hoeffding bound is applied to |\hat{q}_j - Pr(X_l = 1)| with a denominator involving nIS, writing sqrt(-ln(δ'/2)/(2 nIS)). However, the estimate \hat{q}_j is the average of K samples (corresponding to nUL in the theorem statement), so the sample count in the Hoeffding bound should be nUL, not nIS. The theorem statement correctly uses sqrt(ln(2/δ')/(2 nUL)), but the proof does not match the statement. This is a concrete gap in the derivation of the downlink communication-cost bound.
  4. [Appendix I, bitrate tables] The bitrate accounting for the broadcast case is not fully explained. For BICompFL-GR, the downlink in the point-to-point model consists of relaying the other clients' indices; the tables report a downlink bitrate of (n-1) times the uplink bitrate. In the broadcast column, the bitrate is reduced by roughly a factor of n, but it is unclear whether this assumes the server broadcasts the full set of indices once or uses a different downlink transmission scheme. The derivation of the bpp (BC) numbers should be made explicit so that the experimental comparisons are unambiguous.
minor comments (5)
  1. [Appendix B] In the displayed equation after defining M, the binomial coefficient is written as \binom{n - 1}{L} instead of \binom{nIS - 1}{L}; this is a typo that should be corrected.
  2. [Section 5] The assumption |q_j - p_j| ≤ ρ is stated as 'natural' and enforceable by projecting onto a KL ball, but the experimental local training in Algorithm 3 does not perform such a projection. The paper should clarify whether the theoretical results apply to the unmodified algorithm or to a projected variant.
  3. [Section 4] The abstract and Section 4 use the phrase 'order of magnitude' and 'order-wise improvements'; the reported reductions versus the best non-stochastic baselines are factors between 5 and 32 (e.g., Figure 2 and Tables 5-12), which is at most about 1.5 orders of magnitude. A more precise quantitative summary would avoid overstating the improvement.
  4. [Appendix I] In the non-i.i.d. CIFAR-10 results (Table 12), BICompFL-PR-Fixed-SplitDL achieves accuracy 0.47, much lower than BICompFL-GR-Fixed (0.665); the main text mentions this qualitatively but the large drop and its variance should be reported in the main-body discussion of partitioning.
  5. [Section 2] The notation nIS is used both for the number of importance samples per block and, in the proof of Theorem 1, for the number of samples used in the Hoeffding bound; this dual use is confusing and should be disambiguated (e.g., by using nUL consistently for the number of transmitted samples).

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the communication-cost analysis derives from external MRC theory and standard quantization bounds, not from fitted parameters or self-referential definitions.

full rationale

BICompFL's theoretical claims are not derived from its own fitted values or from a self-referential definition. The communication-cost model (nIS = Θ(exp(DKL(Q∥P)))) is taken from Chatterjee & Diaconis (2018) and Havasi et al. (2019); the paper's contribution is to bound the resulting downstream KL divergence for Bernoulli MRC in Theorem 1 via Lemma 2, which is a new inequality rather than a restatement of the bound it is used to prove. Lemma 1's contraction proof cites Weinberger & Yemini (2023, Lemma 15) for a logarithmic inequality, but that is an external published lemma with stated assumptions that do not include the target contraction, so it is independent support. The experimental communication-cost figures are direct measurements of transmitted bits under declared hyperparameters (nIS, B, nDL), not predictions obtained from fitted parameters; the accuracy-versus-bitrate plots are benchmark comparisons. The ρ and ζ assumptions in Theorem 1 are explicit premises about local training and prior disagreement, not definitions of the conclusion. A possible sign reversal in the proof of Lemma 2 would be a correctness defect, but it is not a circular reduction: the lemma is not equal by construction to the quantity it bounds. No circular step was found.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The central claim depends on the chosen hyperparameters (nIS, block size, nUL, nDL) that set the communication bitrate, and on the domain assumption that posteriors and priors are Bernoulli and remain close in KL divergence. The theoretical results rely on standard concentration inequalities and on bounded-divergence assumptions (ρ, ζ) that are stated in Theorem 1.

free parameters (3)
  • nIS (importance samples per block) = 256
    Set to 256 in all main experiments (Appendix J.5). The reported bitrate depends directly on log2(nIS) per block; larger nIS would increase cost, smaller nIS would increase compression error.
  • Block size d/B = 256 (default)
    Set to 256 in most experiments (Appendix J.4). Cost per parameter is log2(nIS) / (d/B), so this choice determines the bitrate. Adaptive schemes vary it.
  • nUL and nDL (samples per direction) = nUL=1, nDL=10
    Chosen as 'fair' defaults; nDL = n * nUL is used to reduce downlink variance (Section 3). These directly affect cost and accuracy.
assumptions (6)
  • standard math Chernoff, Jensen, Pinsker, and Hoeffding inequalities
    Used in proofs of Proposition 1, Lemma 2, and Theorem 1 (Appendix B).
  • domain assumption Model updates are Bernoulli distributions parameterized in [0,1]^d
    Section 3 defines the stochastic FL setting over Bernoulli mask parameters; the MRC analysis is specialized to Bernoulli in Theorem 1.
  • domain assumption Bounded local training progress: |q_j - p_j| ≤ ρ for all clients
    Assumed in Theorem 1 to bound the triangle inequality for |q̂_j - p_i|; not verified for the experimental training schedules.
  • domain assumption Bounded prior mismatch: |p_i - p_j| ≤ ζ for all clients
    Assumed in Theorem 1; holds exactly when all clients share the same prior (GR), but only approximately in PR.
  • domain assumption For Lemma 1, the quantization level satisfies s ≥ √(2d)
    Required to make δ ≤ 1 in the contraction bound; for large d this is impractical, and the experiments use stochastic SignSGD rather than QSGD.
  • domain assumption Lipschitz gradients and bounded stochastic gradient variance
    Standard assumptions for the convergence bound in Theorem 2 (Appendix C).

how reviews work

0 comments
Cite this review

Pith. "Pith review of BICompFL: Stochastic Federated Learning with Bi-Directional Compression." pith.science (2026). https://pith.science/paper/NOBCJ7SN

@misc{pith2026250200206,
  author       = {Pith},
  title        = {Pith review of: BICompFL: Stochastic Federated Learning with Bi-Directional Compression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NOBCJ7SN}},
  note         = {Machine review of arXiv:2502.00206}
}
read the original abstract

We address the prominent communication bottleneck in federated learning (FL). We specifically consider stochastic FL, in which models or compressed model updates are specified by distributions rather than deterministic parameters. Stochastic FL offers a principled approach to compression, and has been shown to reduce the communication load under perfect downlink transmission from the federator to the clients. However, in practice, both the uplink and downlink communications are constrained. We show that bi-directional compression for stochastic FL has inherent challenges, which we address by introducing BICompFL. Our BICompFL is experimentally shown to reduce the communication cost by an order of magnitude compared to multiple benchmarks, while maintaining state-of-the-art accuracies. Theoretically, we study the communication cost of BICompFL through a new analysis of an importance-sampling based technique, which exposes the interplay between uplink and downlink communication costs.

Figures

Figures reproduced from arXiv: 2502.00206 by the authors.

Figure 1
Figure 1. Test accuracy for BICOMPFL and baselines on Fashion MNIST 4CNN on i.i.d. data. since the previous global model estimate is likely to be similarly different from the posterior (in terms of the KL￾divergence) than the previous posterior estimate of the fed￾erator. Indeed, our numerical experiments have shown that the savings from choosing λ ̸= 1, i.e., priors other than ˆθi,t, are not significant. For simplicity, we t… view at source ↗
Figure 2
Figure 2. Maximum test accuracy as a function of the total communication cost measured as the bitrate per parameter. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. MNIST LeNet i.i.d. For LeNet5 on MNIST, it can be observed that all our proposed methods converge significantly faster to satisfying ac￾curacies with respect to the communication cost, while achieving higher maximum accuracies after 200 epochs than the non-stochastic baselines. Partitioning the model on the downlink can help to further reduce the communication cost with only a minor loss in performance, especially i… view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: MNIST LeNet non-i.i.d [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: MNIST 4CNN i.i.d [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: MNIST 4CNN non-i.i.d. In the non-i.i.d. case of 4CNN on MNIST, the adaptive average allocation strategy provides a significant reduction in the bitrate for BICOMPFL-GR, with similar loss in the accuracy as SplitDL for BICOMPFL-PR. In this setting, it is also apparent t…
Figure 7
Figure 7. Figure 7: Fashion MNIST 4CNN i.i.d [PITH_FULL_IMAGE:figures/full_fig_p026_7.png]
Figure 8
Figure 8. Figure 8: Fashion MNIST 4CNN non-i.i.d. The results for Fashion MNIST are similar compared to the MNIST case. However, it becomes clear that BICOMPFL-PR can significantly suffer from the unavailability of shared randomness in terms of the achieved accuracy when data is highly he…
Figure 9
Figure 9. Figure 9: CIFAR-10 6CNN i.i.d [PITH_FULL_IMAGE:figures/full_fig_p028_9.png]
Figure 10
Figure 10. Figure 10: CIFAR-10 6CNN non-i.i.d [PITH_FULL_IMAGE:figures/full_fig_p029_10.png]
Figure 11
Figure 11. Figure 11: Test Accuracy over Epochs 30 [PITH_FULL_IMAGE:figures/full_fig_p030_11.png]
Figure 12
Figure 12. Figure 12: BICOMPFL-GR and BICOMPFL-GR With Different Number of Clients 10 1 10 0 Bitrate p. P. 0.900 0.905 0.910 0.915 0.920 0.925 Maximum Test Accuracy BiCompFL-GR-C10 BiCompFL-GR-C50 BiCompFL-PR-C10 BiCompFL-PR-C50 [PITH_FULL_IMAGE:figures/full_fig_p031_12.png]
Figure 13
Figure 13. Figure 13: Bitrates for BICOMPFL-GR and BICOMPFL-GR With Different Number of Clients J.2 Optimization of the Prior As described in the main body of the paper, BICOMPFL-PR allows for optimizing the choice of the prior at the clients by optimizing the convexity parameter λ that mi…
Figure 14
Figure 14. Figure 14: BICOMPFL-PR With and Without Optimization over the Prior. Optimization over the Priors is denoted by OP. It can be found that, while optimizing the prior improves the accuracy over epochs and with respect to the communication cost compared to BICOMPFL-PR the improveme…
Figure 15
Figure 15. Figure 15: BICOMPFL-PR for Different Number of Downlink Samples and a Single Uplink Sample. J.4 Block Size We compare in [PITH_FULL_IMAGE:figures/full_fig_p033_15.png]
Figure 16
Figure 16. Figure 16: BICOMPFL-GR With Fixed Block Allocation for Varying Block Sizes (BS) d/B. J.5 Number of Importance Samples In [PITH_FULL_IMAGE:figures/full_fig_p033_16.png]
Figure 17
Figure 17. Figure 17: BICOMPFL-GR with Varying Number of Importance Samples nIS per Block. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 31 canonical work pages

  1. [1]

    Nested Dithered Quantization for Communication Reduction in Distributed Training

    Afshin Abdi and Faramarz Fekri. Nested dithered quantization for communication reduction in distributed training. arXiv preprint arXiv:1904.01197, 2019

    work page Pith review arXiv 1904

  2. [2]

    QSGD : Communication-efficient SGD via gradient quantization and encoding

    Dan Alistarh, Demjan Grubic, Jerry Li, Ryota Tomioka, and Milan Vojnovic. QSGD : Communication-efficient SGD via gradient quantization and encoding. In Advances in Neural Information Processing Systems, volume 30, 2017

    work page 2017

  3. [3]

    Kulkarni, and H

    Mohammad Mohammadi Amiri, Deniz Gunduz, Sanjeev R. Kulkarni, and H. Vincent Poor. Federated learning with quantized global model updates. arXiv preprint arXiv:2006.10672, 2020

    arXiv 2006

  4. [4]

    The sample size required in importance sampling

    Sourav Chatterjee and Persi Diaconis. The sample size required in importance sampling. The Annals of Applied Probability, 28 0 (2): 0 1099--1135, 2018

    work page 2018

  5. [5]

    Communication-Efficient Distributed Learning with Local Immediate Error Compensation

    Yifei Cheng, Li Shen, Linli Xu, Xun Qian, Shiwei Wu, Yiming Zhou, Tie Zhang, Dacheng Tao, and Enhong Chen. Communication-efficient distributed learning with local immediate error compensation. arXiv preprint arXiv:2402.11857, 2024

    work page Pith review arXiv 2024

  6. [6]

    Elements of information theory

    Thomas Cover and Joy A Thomas. Elements of information theory. Wiley-Interscience, 2006

    work page 2006

  7. [7]

    DoCoFL : Downlink compression for cross-device federated learning

    Ron Dorfman, Shay Vargaftik, Yaniv Ben-Itzhak, and Kfir Yehuda Levy. DoCoFL : Downlink compression for cross-device federated learning. In International Conference on Machine Learning, pp.\ 8356--8388, 2023

    work page 2023

  8. [8]

    The lottery ticket hypothesis: Finding sparse, trainable neural networks

    Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019

    2019

Show all 42 references
  1. [9]

    vqSGD : Vector quantized stochastic gradient descent

    Venkata Gandikota, Daniel Kane, Raj Kumar Maity, and Arya Mazumdar. vqSGD : Vector quantized stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics, volume 130, pp.\ 2197--2205, 2021

    2021

  2. [10]

    EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression

    Kaja Gruntkowska, Alexander Tyurin, and Peter Richt \'a rik. EF21-P and friends: Improved theoretical communication complexity for distributed optimization with bidirectional compression. In International Conference on Machine Learning, pp.\ 11761--11807, 2023

    2023

  3. [11]

    Improving the worst-case bidirectional communication complexity for nonconvex distributed optimization under function similarity

    Kaja Gruntkowska, Alexander Tyurin, and Peter Richt \'a rik. Improving the worst-case bidirectional communication complexity for nonconvex distributed optimization under function similarity. arXiv preprint arXiv:2402.06412, 2024

    2024 arXiv

  4. [12]

    Minimal random code learning: Getting bits back from compressed model parameters

    Marton Havasi, Robert Peharz, and José Miguel Hernández-Lobato. Minimal random code learning: Getting bits back from compressed model parameters. In International Conference on Learning Representations, 2019

    2019

  5. [13]

    Natural compression for distributed deep learning

    Samuel Horv \'o th, Chen-Yu Ho, Ludovit Horvath, Atal Narayan Sahu, Marco Canini, and Peter Richtarik. Natural compression for distributed deep learning. In Proceedings of Mathematical and Scientific Machine Learning, volume 190, pp.\ 129--141, 2022

    2022

  6. [14]

    Lower bounds and nearly optimal algorithms in distributed learning with communication compression

    Xinmeng Huang, Yiming Chen, Wotao Yin, and Kun Yuan. Lower bounds and nearly optimal algorithms in distributed learning with communication compression. Advances in Neural Information Processing Systems, 35: 0 18955--18969, 2022

    2022

  7. [15]

    Sparse random networks for communication-efficient federated learning

    Berivan Isik, Francesco Pase, Deniz Gunduz, Tsachy Weissman, and Zorzi Michele. Sparse random networks for communication-efficient federated learning. In International Conference on Learning Representations, 2023

    2023

  8. [16]

    Adaptive compression in federated learning via side information

    Berivan Isik, Francesco Pase, Deniz Gunduz, Sanmi Koyejo, Tsachy Weissman, and Michele Zorzi. Adaptive compression in federated learning via side information. In International Conference on Artificial Intelligence and Statistics, pp.\ 487--495, 2024

    2024

  9. [17]

    Error feedback fixes SignSGD and other gradient compression schemes

    Sai Praneeth Karimireddy, Quentin Rebjock, Sebastian Stich, and Martin Jaggi. Error feedback fixes SignSGD and other gradient compression schemes. In International Conference on Machine Learning, volume 97, pp.\ 3252--3261, 2019

    2019

  10. [18]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015

    2015

  11. [19]

    Lecun, L

    Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998

    1998

  12. [20]

    Communication-Efficient Learning of Deep Networks from Decentralized Data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data . In International Conference on Artificial Intelligence and Statistics, volume 54, pp.\ 1273--1282, 2017

    2017

  13. [21]

    Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees

    Constantin Philippenko and Aymeric Dieuleveut. Bidirectional compression in heterogeneous settings for distributed or federated learning with partial participation: tight convergence guarantees. arXiv preprint arXiv:2006.14591, 2020

    2006 arXiv

  14. [22]

    Preserved central model for faster bidirectional compression in distributed settings

    Constantin Philippenko and Aymeric Dieuleveut. Preserved central model for faster bidirectional compression in distributed settings. Advances in Neural Information Processing Systems, 34: 0 2387--2399, 2021

    2021

  15. [23]

    What's hidden in a randomly weighted neural network? In IEEE/CVF conference on computer vision and pattern recognition, pp.\ 11893--11902, 2020

    Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in a randomly weighted neural network? In IEEE/CVF conference on computer vision and pattern recognition, pp.\ 11893--11902, 2020

    2020

  16. [24]

    Ef21: A new, simpler, theoretically better, and practically faster error feedback

    Peter Richt \'a rik, Igor Sokolov, and Ilyas Fatkhullin. Ef21: A new, simpler, theoretically better, and practically faster error feedback. Advances in Neural Information Processing Systems, 34: 0 4384--4396, 2021

    2021

  17. [25]

    1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs

    Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech DNNs . In Interspeech, pp.\ 1058--1062, 2014

    2014

  18. [26]

    A distributed synchronous SGD algorithm with global top-k sparsification for low bandwidth networks

    Shaohuai Shi, Qiang Wang, Kaiyong Zhao, Zhenheng Tang, Yuxin Wang, Xiang Huang, and Xiaowen Chu. A distributed synchronous SGD algorithm with global top-k sparsification for low bandwidth networks. In International Conference on Distributed Computing Systems (ICDCS), pp.\ 2238...

    2019

  19. [27]

    Importance sampling: Applications in communications and detection

    Rajan Srinivasan. Importance sampling: Applications in communications and detection. Springer Science & Business Media, 2002

    2002

  20. [28]

    Sparsified SGD with memory

    Sebastian U Stich, Jean-Baptiste Cordonnier, and Martin Jaggi. Sparsified SGD with memory. Advances in Neural Information Processing Systems, 31, 2018

    2018

  21. [29]

    Doublesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression

    Hanlin Tang, Chen Yu, Xiangru Lian, Tong Zhang, and Ji Liu. Doublesqueeze: Parallel stochastic gradient descent with double-pass error-compensated compression. In International Conference on Machine Learning, pp.\ 6155--6165, 2019

    2019

  22. [30]

    DP - REC : Private & communication-efficient federated learning, 2022

    Aleksei Triastcyn, Matthias Reisser, and Christos Louizos. DP - REC : Private & communication-efficient federated learning, 2022

    2022

  23. [31]

    2Direction: theoretically faster distributed training with bidirectional communication compression

    Alexander Tyurin and Peter Richt \'a rik. 2Direction: theoretically faster distributed training with bidirectional communication compression. In Conference on Neural Information Processing Systems, 2023

    2023

  24. [32]

    QLSD: quantised langevin stochastic dynamics for bayesian federated learning

    Maxime Vono, Vincent Plassier, Alain Durmus, Aymeric Dieuleveut, and Eric Moulines. QLSD: quantised langevin stochastic dynamics for bayesian federated learning. In International Conference on Artificial Intelligence and Statistics, pp.\ 6459--6500, 2022

    2022

  25. [33]

    Gradient sparsification for communication-efficient distributed optimization

    Jianqiao Wangni, Jialei Wang, Ji Liu, and Tong Zhang. Gradient sparsification for communication-efficient distributed optimization. Advances in Neural Information Processing Systems, 31, 2018

    2018

  26. [34]

    Multi-armed bandits with self-information rewards

    Nir Weinberger and Michal Yemini. Multi-armed bandits with self-information rewards. IEEE Transactions on Information Theory, 2023

    2023

  27. [35]

    A survey on federated learning: challenges and applications

    Jie Wen, Zhixia Zhang, Yang Lan, Zhihua Cui, Jianghui Cai, and Wensheng Zhang. A survey on federated learning: challenges and applications. International Journal of Machine Learning and Cybernetics, 14 0 (2): 0 513--535, 2023

    2023

  28. [36]

    TernGrad : Ternary gradients to reduce communication in distributed deep learning

    Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. TernGrad : Ternary gradients to reduce communication in distributed deep learning. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  29. [37]

    Cser: Communication-efficient sgd with error reset

    Cong Xie, Shuai Zheng, Sanmi Koyejo, Indranil Gupta, Mu Li, and Haibin Lin. Cser: Communication-efficient sgd with error reset. Advances in Neural Information Processing Systems, 33: 0 12593--12603, 2020

    2020

  30. [38]

    A survey on federated learning

    Chen Zhang, Yu Xie, Hang Bai, Bin Yu, Weihong Li, and Yuan Gao. A survey on federated learning. Knowledge-Based Systems, 216: 0 106775, 2021

    2021

  31. [39]

    Communication-efficient distributed blockwise momentum SGD with error-feedback

    Shuai Zheng, Ziyue Huang, and James Kwok. Communication-efficient distributed blockwise momentum SGD with error-feedback. Advances in Neural Information Processing Systems, 32, 2019

    2019

  32. [40]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  33. [41]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  34. [42]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.