Pith. sign in

REVIEW 4 major objections 5 minor 54 references

ParaBlock claims that running communication and computation as two parallel threads—with a one-round-delayed correction—lets federated block-coordinate fine-tuning hide communication latency while preserving the standard convergence rate.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 20:22 UTC pith:QH2FUAQB

load-bearing objection ParaBlock's overlap idea is plausible and the experiments look useful, but Theorem 5.3's proof has a factor-of-8 misapplication that breaks the convergence claim. the 4 major comments →

arxiv 2511.19959 v2 pith:QH2FUAQB submitted 2025-11-25 cs.LG cs.DC

ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models

Yujia Wang , Yuanpu Cao , Jinghui Chen This is my paper
classification cs.LG cs.DC
keywords federated learningblock coordinate descentcommunication-computation overlaplarge language model fine-tuningconvergence ratecommunication efficiencystaleness correction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that the communication bottleneck in federated fine-tuning of large language models can be removed by running two parallel threads on each client: one that trains a block of the model and one that exchanges the previous round's update with the server. A one-round-delayed correction step makes the local model catch up to the global model, and the paper proves that this delayed design converges at the same O(1/√T) rate as standard federated block-coordinate descent. If correct, this means wall-clock time can be cut by hiding communication behind computation without sacrificing model quality. Experiments on instruction-following and mathematical reasoning with 3B and 8B models support the claim, showing roughly 30–40% runtime savings.

Core claim

The central discovery is a recursion identity: after the correction step, each client's block b_{t-1} equals the initial model plus η times the sum of all global updates to that block, identical to the expression for the global model. The one-round-old aggregate therefore fully repairs the inconsistency caused by parallel training, and the delayed global model is the same update sequence shifted by one round, not a degraded object. The paper proves that the average squared norm of the block gradient converges at O((F+σ²)/√(TKN) + (Nσ²+σ_g²)/T), the same order as standard federated block-coordinate descent.

What carries the argument

The two-thread client design together with the correction update [θ^i_{t+1}]_{b_{t-1}} ← [θ^i_t]_{b_{t-1}} + η(Δ_{t-1} − Δ^i_{t-1}). This step replaces a local-only update with the server-aggregated update one round late, and the recursion identities show that local and global models coincide on every already-synchronized block. That equality is what lets the convergence proof go through despite the staleness.

Load-bearing premise

The formal guarantee requires every client to participate and complete the send–aggregate–receive cycle every round, so when the communication thread runs slower than the computation thread the equality behind the correction breaks and the theorem no longer covers the very regime where the speedup is most needed.

What would settle it

Run ParaBlock in a 50-client setup with 50M/s bandwidth and effective batch size 2 on a 3B model: if per-round wall-clock equals the full communication duration plus computation (no overlap) and the runtime saving over the single-thread baseline disappears, the central efficiency claim fails in that regime. A convergence counterexample would be a run under the stated learning-rate bounds where the average squared block gradient fails to decrease as the bound predicts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Communication time can be almost entirely hidden whenever local computation takes at least as long as communication, so the wall-clock savings grow with model size.
  • The convergence rate matches standard federated block-coordinate descent, so the efficiency gain does not trade away the optimization guarantee.
  • A final extra aggregation step is needed to fold the last trained block into the global model; the paper presents this as a small fixed cost.
  • The method is orthogonal to compression and quantization, and combining it with top-k compression further reduces the communication that remains exposed.
  • Because the proof assumes full participation, the partial-participation results are empirical rather than covered by the convergence theorem.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • In the low-bandwidth, small-compute regime where communication exceeds computation, the overlap is incomplete and the runtime advantage shrinks; the paper's own time-efficiency figure shows an exposed-communication region at 50M/s with batch size 2.
  • A natural extension is a sampled-participant version of the proof: bounding the staleness when only a subset of clients send updates would let the formal guarantee cover the cross-silo setting the paper tests empirically.
  • The correction identity suggests the scheme can tolerate more than one round of staleness; the paper's two-round ablation shows only mild degradation, hinting that deeper overlap or asynchronous send/receive may be viable.
  • Because they target different resources, the parallelism here and bit-reduction methods such as top-k compression could be paired for further gains.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes ParaBlock, a federated block-coordinate learning method that runs a computation thread and a communication thread in parallel. Clients train the currently assigned block while concurrently exchanging the previous round's block update with the server, and then correct the local model using the received aggregate. The main theoretical claim (Theorem 5.3, Corollary 5.4) is that, under L-smoothness and bounded variance, ParaBlock achieves the same O((F+sigma^2)/sqrt(TKN) + (N sigma^2 + sigma_g^2)/T) non-convex convergence rate as standard federated block coordinate descent, despite the one-round staleness. Experiments on Llama 3-8B, Llama 3.2-3B, and Qwen-2.5-1.5B for instruction following and mathematical reasoning show competitive accuracy and reduced wall-clock time relative to FedBCD, FedCyBGD, and LoRA-based baselines.

Significance. The idea of overlapping communication and computation in federated BCD is practically motivated, and the empirical evaluation is broad: two datasets, multiple model sizes, bandwidth ablations, and partial-participation experiments. The correction mechanism described in Section 4 is coherent and the runtime savings are plausible. However, the paper's central theoretical contribution is not currently established. The proof of Theorem 5.3 contains a load-bearing algebraic error: substituting Lemma C.2 into Eq. (23) changes the coefficient of the gradient-norm term by a factor of 8, turning the intended negative coefficient into a positive one. Since the abstract and introduction explicitly rest the paper's novelty on the claim that ParaBlock 'achieves the same convergence rate as standard federated block coordinate descent methods,' the manuscript requires a corrected proof before the theoretical claim can be accepted. The experimental results are useful, but they do not compensate for the invalid proof of the main theorem.

major comments (4)
  1. [Appendix B.2, Eqs. (23)-(25)] The substitution of Lemma C.2 into Eq. (23) is incorrect. The term (4*eta^3*eta_l*L^2*K/N) * sum_t sum_i E||Delta^i_{t-1}||^2 in Eq. (23), when bounded using Lemma C.2, contributes 4*eta^3*eta_l*L^2*K * (1/(2*eta^2*L^2)) * sum_t E||nabla_{b_t} f(theta_t)||^2 = 2*eta*eta_l*K * sum_t E||nabla_{b_t} f(theta_t)||^2, not eta*eta_l*K/4 as written in Eq. (24). With the correct coefficient, the gradient-norm coefficient in Eq. (25) is -eta*eta_l*K/2 + 60*K^3*eta*eta_l^3*L^2 + 2*eta*eta_l*K, which is positive under the stated condition eta_l <= 1/(22*K*L). Therefore the descent inequality cannot be closed, and the displayed negative coefficient -eta*eta_l*K/8 is unjustified. Theorem 5.3 and Corollary 5.4 do not follow from the provided proof.
  2. [Theorem 5.3 vs. Eq. (27)] The theorem statement in Eq. (5) does not match the bound derived in Eq. (27). In Eq. (5), the last term is 64*eta^2*eta_l^2*L^2*K [sigma^2 + 10*eta_l^2*L^2*K (sigma^2 + K*sigma_g^2)], which gives a sigma_g^2 coefficient of 640*eta^2*K^3*eta_l^4*L^4. In Eq. (27), the corresponding sigma_g^2 term is 3840*eta^2*K^3*eta_l^4*L^4, a factor of 6 larger. This discrepancy needs to be resolved; if Eq. (27) is correct, the stated theorem overstates the bound by a constant factor.
  3. [Assumptions for Theorem 5.3 / Eq. (14)] The proof assumes full participation and synchronous aggregation. Eq. (14) sets theta^i_{t,0} = theta_{t-1} + eta*Delta^i_{t-1} and theta_t = theta_{t-1} + eta*Delta_{t-1}, which requires every client's previous-round update to be included in the aggregate. This is not satisfied in the partial-participation experiment of Table 7, where 20% of 50 clients are selected per round. The formal guarantee therefore does not cover a setting the paper explicitly claims to support. The authors should either extend the analysis to partial participation or clearly state in the main text that the theoretical result is restricted to full participation.
  4. [Appendix B.3 / Section 6.1] The experiments use AdamW as the local optimizer (Section 6.1), while Theorem 5.3 is stated and proved only for local SGD. Appendix B.3 gives an informal sketch for adaptive optimizers but does not provide a complete proof or a theorem statement. Thus the empirical setting is not covered by the formal convergence result. This is a scope limitation that should be acknowledged explicitly, especially since the abstract claims a general convergence guarantee.
minor comments (5)
  1. [Figure 2] The legend of Figure 2 lists 'FedCyBGD: communication' twice; the second occurrence appears to be a typo.
  2. [Table 7] The caption 'Ablation for the number of staleness rounds' does not match the table content, which reports cross-silo partial participation results. The caption should be corrected.
  3. [Lemma C.2 proof, Eq. (33)] In the definition of Delta^i_t, the proof writes '-eta' and then 'eta' instead of consistently using the local learning rate eta_l. This is a notational slip that should be fixed.
  4. [Section 6.2] The phrase 'outstanding less runtime' should be 'substantially less runtime'.
  5. [Remark 5.5] The comparison to FedBCD and FedBCGD is made by citing prior convergence rates rather than a side-by-side derivation. The claim 'same convergence rate' would be stronger if the comparison were made explicit with matching constant dependencies on K and N.

Circularity Check

0 steps flagged

No circular derivation; Theorem 5.3 follows from stated assumptions with no fitted constants or load-bearing self-citations.

full rationale

The paper's central claim, Theorem 5.3, is a conventional convergence bound derived in Appendix B from Assumptions 5.1-5.2 via smoothness (Eq. 9), local-step recursions (Eqs. 15-21), and supporting Lemmas C.1-C.2. None of these inputs contains the theorem's conclusion by construction: the bound's constants are algebraic functions of L, K, N, sigma^2, and sigma_g^2, not fitted values, and no quantity that is later called a 'prediction' is used to set a parameter. The correction mechanism in Section 4 is derived as an identity from the update rule (Eqs. 3-4), not used to predefine the rate. The comparison with FedBCD/FedBCGD is made by citing prior analyses rather than re-deriving them, which is a benchmark comparison, not a circular import. Self-citations appear only in a list of standard assumptions and related-work context; they do not carry the proof. Two manuscript passages do flag missing support: Appendix B.1 says 'We will discuss this in the revision' for blockwise variance, and B.3 gives only a sketch for AdamW; these are completeness/correctness limitations, not circularity. The skeptic's alleged factor-of-8 misapplication of Lemma C.2, if correct, would be a proof error rather than a self-referential reduction. Overall, the derivation is self-contained and not circular.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central theoretical result is parameter-free under standard assumptions and introduces no new entities. The empirical claims, however, rest on hand-tuned hyperparameters (LR grids, block size, heterogeneity levels, round counts) and on an unstated full-participation synchronous-aggregation assumption that the proof relies on.

free parameters (5)
  • local learning rate η_l = grid {3e-7, 1e-6, 3e-6, 1e-5, 3e-5} selected per baseline/task (Table 10)
    Best scores and runtimes depend on grid-searching η_l for each method separately; no variance is reported.
  • global learning rate η = 1 (fixed)
    Not fitted, but its value participates in the theoretical condition ηη_l ≤ 1/(4KL) and in the empirical comparison.
  • number of blocks / layers per block = 16 blocks for Llama 3-8B, 14 for Llama 3.2-3B; 2 layers chosen by ablation
    Block partitioning and the number of layers per block are hand-chosen and tuned via ablation (Table 2), and the efficiency claims are sensitive to this choice.
  • data heterogeneity parameters = Dirichlet(0.1) for Alpaca-GPT4, Dirichlet(0.6) for MathInstruct
    Non-i.i.d. partition parameters are selected by the authors; they affect all benchmark numbers.
  • number of global rounds = 32 (8B), 28 (3B) for BCD/LoRA methods; 3 for Full FT
    Full FT is given only 3 rounds, which makes its runtime comparison with BCD methods difficult to interpret.
axioms (4)
  • domain assumption L-smoothness of each local objective (Assumption 5.1)
    Used in the descent lemma and drift bounds throughout Section B.
  • domain assumption Unbiased stochastic gradients with bounded local and global variance (Assumption 5.2)
    Needed for Lemmas C.1-C.2 to control noise and heterogeneity.
  • ad hoc to paper Full participation and synchronous aggregation every round for the theoretical guarantee
    Algorithm 2 Lines 3-10 and Eq. (14) imply all N clients communicate and the server aggregates in one round; partial participation is only empirically tested (Table 7) without a matching theorem.
  • ad hoc to paper The local model at round start satisfies θ_i_{t,0} = θ_{t-1} + ηΔ_i_{t-1}
    This equality, used in Eq. (14), depends on the correction mechanism completing exactly one round and all clients having synchronized deltas.

pith-pipeline@v1.3.0-alltime-deepseek · 24711 in / 13469 out tokens · 136004 ms · 2026-08-03T20:22:22.869675+00:00 · methodology

0 comments
read the original abstract

Federated learning (FL) has been extensively studied as a privacy-preserving training paradigm. Recently, federated block coordinate descent scheme has become a popular option in training large-scale models, as it allows clients to train only a subset of the model locally instead of the entire model. However, in the era of large language models (LLMs), even a single block can contain a significant number of parameters, posing substantial communication latency, particularly for resource-constrained clients. To address this challenge in federated training/fine-tuning LLMs, we propose ParaBlock, a novel approach that establishes two parallel threads for communication and computation to enhance communication efficiency. We theoretically prove that the proposed ParaBlock achieves the same convergence rate as the standard federated block coordinate descent methods. Empirical evaluations on fine-tuning LLMs on general instruction following and mathematical reasoning confirm that ParaBlock not only maintains strong performance but also significantly improves communication efficiency.

Figures

Figures reproduced from arXiv: 2511.19959 by Jinghui Chen, Yuanpu Cao, Yujia Wang.

Figure 1
Figure 1. Figure 1: Comparison between the original federated [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Time efficiency: wall-clock runtime for various network communication bandwidths and [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

54 extracted references · 27 linked inside Pith

  1. [1]

    Llama 3 model card

    AI@Meta. Llama 3 model card. 2024

    2024

  2. [2]

    Slora: Federated parameter efficient fine-tuning of language models.arXiv preprint arXiv:2308.06522, 2023

    Sara Babakniya, Ahmed Roushdy Elkordy, Yahya H Ezzeldin, Qingfeng Liu, Kee-Bong Song, 13 Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models.arXiv preprint arXiv:2308.06522, 2023

    Pith/arXiv arXiv 2023

  3. [3]

    Federated fine-tuning of large language models under heterogeneous language tasks and client resources.arXiv e-prints, pages arXiv–2402, 2024

    Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous language tasks and client resources.arXiv e-prints, pages arXiv–2402, 2024

    2024

  4. [4]

    On the convergence of block coordinate descent type methods

    Amir Beck and Luba Tetruashvili. On the convergence of block coordinate descent type methods. SIAM journal on Optimization, 23(4):2037–2060, 2013

    2037

  5. [5]

    Open llm leaderboard, 2023

    Edward Beeching, Cl´ ementine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open llm leaderboard, 2023

    2023

  6. [6]

    Flower: A friendly federated learning research framework.arXiv preprint arXiv:2007.14390, 2020

    Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Hei Li Kwing, Titouan Parcollet, Pedro PB de Gusm˜ ao, and Nicholas D Lane. Flower: A friendly federated learning research framework.arXiv preprint arXiv:2007.14390, 2020

    Pith/arXiv arXiv 2007

  7. [7]

    Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020

    Pith/arXiv arXiv 2005

  8. [8]

    Heterogeneous low- rank approximation for federated fine-tuning of on-device foundation models.arXiv preprint arXiv:2401.06432, 2024

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous low- rank approximation for federated fine-tuning of on-device foundation models.arXiv preprint arXiv:2401.06432, 2024

    Pith/arXiv arXiv 2024

  9. [9]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  10. [10]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  11. [11]

    Fedmkt: Federated mutual knowledge transfer for large and small language models

    Tao Fan, Guoqiang Ma, Yan Kang, Hanlin Gu, Yuanfeng Song, Lixin Fan, Kai Chen, and Qiang Yang. Fedmkt: Federated mutual knowledge transfer for large and small language models. arXiv preprint arXiv:2406.02224, 2024

    Pith/arXiv arXiv 2024

  12. [12]

    Local SGD with periodic averaging: Tighter analysis and adaptive synchronization.Advances in Neural Information Processing Systems, 32, 2019

    Farzin Haddadpour, Mohammad Mahdi Kamani, Mehrdad Mahdavi, and Viveck Cadambe. Local SGD with periodic averaging: Tighter analysis and adaptive synchronization.Advances in Neural Information Processing Systems, 32, 2019

    2019

  13. [13]

    Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, 2021

    Pith/arXiv arXiv 2021

  14. [14]

    Sparse ran- dom networks for communication-efficient federated learning.arXiv preprint arXiv:2209.15328, 2022

    Berivan Isik, Francesco Pase, Deniz Gunduz, Tsachy Weissman, and Michele Zorzi. Sparse ran- dom networks for communication-efficient federated learning.arXiv preprint arXiv:2209.15328, 2022. 14

    Pith/arXiv arXiv 2022

  15. [15]

    Adaptive quantization of model updates for communication-efficient federated learning

    Divyansh Jhunjhunwala, Advait Gadhikar, Gauri Joshi, and Yonina C Eldar. Adaptive quantization of model updates for communication-efficient federated learning. InICASSP 2021- 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3110–3114. IEEE, 2021

    2021

  16. [16]

    Model pruning enables efficient federated learning on edge devices.IEEE Transactions on Neural Networks and Learning Systems, 34(12):10374–10386, 2022

    Yuang Jiang, Shiqiang Wang, Victor Valls, Bong Jun Ko, Wei-Han Lee, Kin K Leung, and Leandros Tassiulas. Model pruning enables efficient federated learning on edge devices.IEEE Transactions on Neural Networks and Learning Systems, 34(12):10374–10386, 2022

    2022

  17. [17]

    Stochastic-sign SGD for federated learning with theoretical guarantees.arXiv preprint arXiv:2002.10940, 2020

    Richeng Jin, Yufan Huang, Xiaofan He, Huaiyu Dai, and Tianfu Wu. Stochastic-sign SGD for federated learning with theoretical guarantees.arXiv preprint arXiv:2002.10940, 2020

    Pith/arXiv arXiv 2002

  18. [18]

    Scaffold: Stochastic controlled averaging for federated learning

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In International Conference on Machine Learning, pages 5132–5143. PMLR, 2020

    2020

  19. [19]

    Privacy preserving federated learning in medical imaging with uncertainty estimation

    Nikolas Koutsoubis, Yasin Yilmaz, Ravi P Ramachandran, Matthew Schabath, and Ghulam Rasool. Privacy preserving federated learning in medical imaging with uncertainty estimation. arXiv preprint arXiv:2406.12815, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking

    Ang Li, Jingwei Sun, Xiao Zeng, Mi Zhang, Hai Li, and Yiran Chen. Fedmask: Joint computation and communication-efficient personalized federated learning via heterogeneous masking. InProceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pages 42–55, 2021

    2021

  21. [21]

    Fedbat: Communication-efficient federated learning via learnable binarization.arXiv preprint arXiv:2408.03215, 2024

    Shiwei Li, Wenchao Xu, Haozhao Wang, Xing Tang, Yining Qi, Shijie Xu, Weihong Luo, Yuhua Li, Xiuqiang He, and Ruixuan Li. Fedbat: Communication-efficient federated learning via learnable binarization.arXiv preprint arXiv:2408.03215, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Federated optimization in heterogeneous networks.Proceedings of Machine Learning and Systems, 2:429–450, 2020

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks.Proceedings of Machine Learning and Systems, 2:429–450, 2020

    2020

  23. [23]

    Communication-efficient local decentralized SGD methods.arXiv preprint arXiv:1910.09126, 2019

    Xiang Li, Wenhao Yang, Shusen Wang, and Zhihua Zhang. Communication-efficient local decentralized SGD methods.arXiv preprint arXiv:1910.09126, 2019

    Pith/arXiv arXiv 1910

  24. [24]

    Fednlp: Bench- marking federated learning methods for natural language processing tasks.arXiv preprint arXiv:2104.08815, 2021

    Bill Yuchen Lin, Chaoyang He, Zihang Zeng, Hulin Wang, Yufen Huang, Christophe Dupuy, Rahul Gupta, Mahdi Soltanolkotabi, Xiang Ren, and Salman Avestimehr. Fednlp: Bench- marking federated learning methods for natural language processing tasks.arXiv preprint arXiv:2104.08815, 2021

    Pith/arXiv arXiv 2021

  25. [25]

    Fedbcgd: Communication-efficient accelerated block coordinate gradient descent for federated learning

    Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, and YunXiang Gong. Fedbcgd: Communication-efficient accelerated block coordinate gradient descent for federated learning. InProceedings of the 32nd ACM International Conference on Multimedia, pages 2955–2963, 2024

    2024

  26. [26]

    A communication efficient collaborative learning framework for distributed features.arXiv preprint arXiv:1912.11187, 2019

    Yang Liu, Yan Kang, Xinwei Zhang, Liping Li, Yong Cheng, Tianjian Chen, Mingyi Hong, and Qiang Yang. A communication efficient collaborative learning framework for distributed features.arXiv preprint arXiv:1912.11187, 2019. 15

    Pith/arXiv arXiv 1912

  27. [27]

    Badam: A memory efficient full parameter training method for large language models.arXiv preprint arXiv:2404.02827, 2024

    Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter training method for large language models.arXiv preprint arXiv:2404.02827, 2024

    Pith/arXiv arXiv 2024

  28. [28]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pages 1273–1282. PMLR, 2017

    2017

  29. [29]

    Efficiency of coordinate descent methods on huge-scale optimization problems

    Yu Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341–362, 2012

    2012

  30. [30]

    Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning.arXiv preprint arXiv:2403.17919, 2024

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning.arXiv preprint arXiv:2403.17919, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Instruction tuning with gpt-4.arXiv preprint arXiv:2304.03277, 2023

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4.arXiv preprint arXiv:2304.03277, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cn´ y, Sanjiv Kumar, and Hugh Brendan McMahan

    Sashank J. Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Koneˇ cn´ y, Sanjiv Kumar, and Hugh Brendan McMahan. Adaptive federated optimization. In International Conference on Learning Representations, 2021

    2021

  33. [33]

    Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization

    Amirhossein Reisizadeh, Aryan Mokhtari, Hamed Hassani, Ali Jadbabaie, and Ramtin Pedarsani. Fedpaq: A communication-efficient federated learning method with periodic averaging and quantization. InInternational Conference on Artificial Intelligence and Statistics, pages 2021–2031. PMLR, 2020

    2021

  34. [34]

    Conquering the communication constraints to enable large pre-trained models in federated learning.arXiv preprint arXiv:2210.01708, 2022

    Guangyu Sun, Umar Khalid, Matias Mendieta, Taojiannan Yang, Pu Wang, Minwoo Lee, and Chen Chen. Conquering the communication constraints to enable large pre-trained models in federated learning.arXiv preprint arXiv:2210.01708, 2022

    Pith/arXiv arXiv 2022

  35. [35]

    Improving lora in privacy-preserving federated learning.arXiv preprint arXiv:2403.12313, 2024

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning.arXiv preprint arXiv:2403.12313, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L´ eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram´ e, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Qwen2.5: A party of foundation models, September 2024

    Qwen Team. Qwen2.5: A party of foundation models, September 2024

    2024

  38. [38]

    Convergence of a block coordinate descent method for nondifferentiable minimiza- tion.Journal of optimization theory and applications, 109:475–494, 2001

    Paul Tseng. Convergence of a block coordinate descent method for nondifferentiable minimiza- tion.Journal of optimization theory and applications, 109:475–494, 2001

    2001

  39. [39]

    Federated learning with matched averaging

    Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khaz- aeni. Federated learning with matched averaging. InInternational Conference on Learning Representations, 2020. 16

    2020

  40. [40]

    Tackling the objective inconsistency problem in heterogeneous federated optimization.arXiv preprint arXiv:2007.07481, 2020

    Jianyu Wang, Qinghua Liu, Hao Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization.arXiv preprint arXiv:2007.07481, 2020

    Pith/arXiv arXiv 2007

  41. [41]

    Save it all: Enabling full parameter tuning for fed- erated large language models via cycle black gradient descent.arXiv preprint arXiv:2406.11187, 2024

    Lin Wang, Zhichao Wang, and Xiaoying Tang. Save it all: Enabling full parameter tuning for fed- erated large language models via cycle black gradient descent.arXiv preprint arXiv:2406.11187, 2024

    Pith/arXiv arXiv 2024

  42. [42]

    A lightweight method for tackling unknown participation probabilities in federated averaging.arXiv preprint arXiv:2306.03401, 2023

    Shiqiang Wang and Mingyue Ji. A lightweight method for tackling unknown participation probabilities in federated averaging.arXiv preprint arXiv:2306.03401, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Tackling the data heterogeneity in asynchronous federated learning with cached update calibration

    Yujia Wang, Yuanpu Cao, Jingcheng Wu, Ruoyu Chen, and Jinghui Chen. Tackling the data heterogeneity in asynchronous federated learning with cached update calibration. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  44. [44]

    Communication-efficient adaptive federated learning

    Yujia Wang, Lu Lin, and Jinghui Chen. Communication-efficient adaptive federated learning. InProceedings of the 39th International Conference on Machine Learning, pages 22802–22838. PMLR, 2022

    2022

  45. [45]

    Fadas: Towards federated adaptive asynchronous optimization.arXiv preprint arXiv:2407.18365, 2024

    Yujia Wang, Shiqiang Wang, Songtao Lu, and Jinghui Chen. Fadas: Towards federated adaptive asynchronous optimization.arXiv preprint arXiv:2407.18365, 2024

    Pith/arXiv arXiv 2024

  46. [46]

    Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations

    Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint arXiv:2409.05976, 2024

    Pith/arXiv arXiv 2024

  47. [47]

    Communication- efficient federated learning via knowledge distillation.Nature communications, 13(1):2032, 2022

    Chuhan Wu, Fangzhao Wu, Lingjuan Lyu, Yongfeng Huang, and Xing Xie. Communication- efficient federated learning via knowledge distillation.Nature communications, 13(1):2032, 2022

    2032

  48. [48]

    Cg-fedllm: How to compress gradients in federated fune-tuning for large language models

    Huiwen Wu, Xiaohan Li, Deyi Zhang, Xiaogang Xu, Jiafei Wu, Puning Zhao, and Zhe Liu. Cg-fedllm: How to compress gradients in federated fune-tuning for large language models. arXiv preprint arXiv:2405.13746, 2024

    arXiv 2024

  49. [49]

    Federated block coordinate descent scheme for learning global and personalized models

    Ruiyuan Wu, Anna Scaglione, Hoi-To Wai, Nurullah Karakoc, Kari Hreinsson, and Wing-Kin Ma. Federated block coordinate descent scheme for learning global and personalized models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10355–10362, 2021

    2021

  50. [50]

    Achieving linear speedup with partial worker partici- pation in non-IID federated learning

    Haibo Yang, Minghong Fang, and Jia Liu. Achieving linear speedup with partial worker partici- pation in non-IID federated learning. InInternational Conference on Learning Representations, 2021

    2021

  51. [51]

    Openfedllm: Training large language models on decentralized private data via federated learning

    Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6137–6147, 2024. 17

    2024

  52. [52]

    Mammoth: Building math generalist models through hybrid instruction tuning.arXiv preprint arXiv:2309.05653, 2023

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning.arXiv preprint arXiv:2309.05653, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Towards building the federatedgpt: Federated instruction tuning

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6915–6919. IEEE, 2024

    2024

  54. [54]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 18 A Additional Experiments A.1 Additional Results Additional experiments on multilingual settingsWe conducted mult...

    2023