Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves SP LoRA aggregation cannot reach the optimal convergence rate because SVD broadcast truncation corrupts the update, while PS aggregation can.

desk verdict Useful ABO framework for LoRA FL convergence, but the headline claim that SP cannot achieve the optimal rate rests on an infeasible unconstrained optimum; within the feasible rank-r class SP is actually the optimal broadcaster. read the letter →

arxiv 2508.01348 v2 pith:VFDDQZB5 submitted 2025-08-02 cs.LG cs.AI

Xin Chen , Shuaijun Chen , Omid Tavallaie , Nguyen Tran , Shuhuang Xiang , Albert Zomaya This is my paper
classification cs.LGcs.AI MSC 68T0790C26
keywords federatedlearninglow-rankadaptationLoRAaggregation-broadcastoperatorsum-productaggregationproduct-sumconvergencerateSVDbroadcasterror
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which way of aggregating LoRA adapters in federated fine-tuning can be trusted to converge, and answers with a unified theoretical framework. It defines an aggregation-broadcast operator for the pair of low-rank matrices $(B,A)$, proves that a weak convergence condition suffices for each client's local model to converge in the LoRA subspace at rate $O(1/\sqrt{T})$, and proves that a strong convergence condition is needed for the global model to converge at the same rate. The central result is that the Sum-Product rule---averaging the products $B_iA_i$, then broadcasting an SVD truncation of that average---satisfies only the weak condition: the truncation is a broadcast error that grows as the rank shrinks and blocks the optimal rate. The Product-Sum rule---separately averaging the $B_i$'s and the $A_i$'s---satisfies the strong condition and attains the optimal rate, with derived constants showing SP degrades with small rank while PS degrades more quickly with more local epochs between communication rounds.

What carries the argument

The central object is the Aggregation-Broadcast Operator (ABO), a pair of maps $(P,Q)$ that reads all clients' locally updated LoRA matrices $(A_i,B_i)$ and returns the matrices each client receives after the server's aggregation-and-broadcast phase. The mechanism that carries the argument is comparison with the average: the weak convergence condition bounds the distance between $P(\cdot)Q(\cdot)$ and each local $B_iA_i$, the strong condition bounds the distances of $P$ to the $B_i$'s and $Q$ to the $A_i$'s separately, and both bounds are minimized by the mean because the squared distance to a set of points is minimized at their average. This reduces the proof of convergence to checking how close a proposed aggregation rule comes to that average, and it isolates SVD truncation as the precise reason SP falls short.

What would settle it

A direct experiment would record, at every communication round, the actual broadcast error $\|\tilde U\Sigma\tilde V^\top - \frac{1}{m}\sum_i B_iA_i\|_F$ for SP at rank ratios 0.1, 0.5, and 1.0; the paper predicts this error grows monotonically as rank falls and that the growth is what slows convergence, so if the measured error is flat or the training curves do not separate in the predicted order, the broadcast-error explanation fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that every LoRA federated aggregation scheme can be viewed as a broadcast pair $(P,Q)$ applied to all locally updated matrices, and convergence of the global update $W_0+P(\cdot)Q(\cdot)$ is governed by how far that broadcast product sits from the local products $B_iA_i$. If the average squared distance is at most $R^2\eta^2$ (the weak convergence condition), Theorem 1 gives local convergence at $O(1/\sqrt{T})$; if the separate distances from $P$ to the $B_i$ and from $Q$ to the $A_i$ are each at most $P^2\eta^2$ and $Q^2\eta^2$ (the strong condition), Theorem 3 gives global convergence at the same rate. Minimizing these distances is a convex problem whose solution is the mean, so the optimality condition for both levels is the average. The proof then specializes: SP broadcasts $\tilde U\Sigma\tilde V^\top$ from the SVD of $\frac{1}{m}\sum_i B_iA_i$, which equals the mean only when the rank is full, so SP satisfies only the weak condition and suffers a broadcast error that grows as $r$ falls; PS broadcasts $\frac{1}{m}\sum_i B_i$ and $\frac{1}{m}\sum_i A_i$, which satisfies the strong condition and the optimality condition when all clients share a rank. The derived bounds scale as $R^2_{\mathrm{sp}}=O(E^2G^2(C_A^4+C_B^4))$ and $R^2_{\mathrm{ps}}=O(E^4G^4C_A^2C_B^2\eta^2)$, predicting the rank sensitivity of SP and the epoch sensitivity of PS observed in the experiments.

Load-bearing premise

The load-bearing premise is that the LoRA matrices stay bounded: there exist fixed $C_A$ and $C_B$ such that the Frobenius norm of every $A_i^{(t)}$ and $B_i^{(t)}$ stays below its constant for all clients and all steps, and if adapters drift upward during training, the bounds on $R^2$ and the derived $O(1/\sqrt{T})$ rates no longer follow.

Editorial extensions

If this is right

  • Any existing or future LoRA aggregation rule can be audited against the two conditions: satisfying the strong condition buys global convergence at the optimal rate, while satisfying only the weak condition buys local convergence with a possibly worse constant.
  • The Sum-Product rule, despite being the natural analogue of standard federated averaging, will underperform at small LoRA ranks; its convergence reaches the optimal rate only when the rank equals the full model rank, which negates LoRA's purpose.
  • The Product-Sum rule is robust to rank choice under a shared rank, but its convergence constant grows like $E^4$ in the number of local epochs, so frequent communication matters more for PS than for SP.
  • The bounds give a quantitative trade-off: users who want low communication should expect SP to need a higher rank, while PS needs fewer local epochs per round to keep the rate.
  • The framework unifies existing methods by showing that SP-type and PS-type methods are not two disconnected heuristics but two evaluations of the same optimality condition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same weak and strong conditions can be applied to any lossy broadcast---quantization, sparsification, or LoRA rank pruning---since all such schemes inflate $R$; the paper does not test this, but the framework yields a direct way to compare compression schemes by their convergence constant.
  • The theory assumes all clients share one LoRA rank, yet the strong condition is stated in terms of separate closeness of $P$ and $Q$; a heterogeneous-rank PS variant that pads or aligns ranks may still satisfy the condition, which is a testable extension the paper does not explore.
  • The derived $O(E^4)$ dependence for PS suggests a concrete scheduling recipe: halving the number of local epochs should more than square the improvement in the bound; an experiment measuring test accuracy versus $E$ at fixed total steps could confirm or reject this quantitative prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces an Aggregation-Broadcast Operator (ABO) abstraction for LoRA-based federated fine-tuning, proposes weak and strong convergence conditions under standard smoothness/boundedness assumptions, and proves O(1/sqrt(T)) gradient-norm convergence rates for local models (Theorem 1) and global models (Theorem 3). It then specializes the analysis to Sum-Product (SP) aggregation with SVD broadcast and Product-Sum (PS) aggregation, claiming that SP satisfies only the weak condition and cannot achieve the optimal rate due to broadcast error, while PS satisfies the strong condition and achieves the optimal rate. Experiments on MNIST with an MLP are presented as supporting evidence.

Significance. The ABO framework is a useful unifying language for comparing LoRA aggregation strategies, and the convergence theorems, if the assumptions hold, provide concrete sufficient conditions for local and global convergence. The paper is also honest in presenting Assumption 3 and full-client participation as limitations. However, the headline claim of an SP/PS asymmetry in achievable convergence rate is not established as stated: the 'optimal' benchmark used in Corollary 1 ignores the rank constraint on the broadcast product, so the conclusion that SP is suboptimal is either trivial or unsupported. The framework still has value as a general analysis tool, and the distinction between weak and strong conditions is meaningful, but the main comparative claim needs substantial revision.

major comments (4)
  1. [Section 4.2.1, Corollary 1 and Eq. (15)-(17)] The optimization problem in Eq. (15) and its solution in Appendix A.5 minimize f(X) = (1/m) sum_i ||X - X_i||_F^2 over all matrices X, but the broadcast operator is constrained to X = P Q with P in R^{d x r} and Q in R^{r x n}, i.e., rank at most r. When the average of the local products has rank greater than r, the unconstrained average is infeasible. By Eckart-Young, the minimizer of f over rank-r matrices is the rank-r truncated SVD of the average, which is exactly the SP broadcast in Eqs. (7)-(8). Therefore Eq. (16) cannot be satisfied by any rank-r broadcast operator, and the statement that SP 'cannot achieve the optimal convergence rate due to broadcast errors' (Section 4.2.1) is either trivial or unsupported. The SP/PS comparison must be reformulated with respect to the feasible rank-r class; otherwise the claimed asymmetry is not established.
  2. [Section 3, Algorithms 1 and 2 vs. Eqs. (3)-(4) and (7)-(10)] Algorithm 1 (lines 8-10) and Algorithm 2 (lines 8-9) use sums without the 1/m normalization: W(t) = W0 + sum_i B_i A_i, SVD of the sum, broadcast of U Sigma V^T, and similarly W(t) = W0 + (sum B_i)(sum A_i) with broadcast of sums. The equations and all theoretical analysis (Definitions 2-3, Corollaries 1-2, the proofs in Appendices A.5-A.7) use averages. This inconsistency changes the effective learning rate by a factor of m and undermines the claimed equivalence between the algorithms and the 'optimality conditions' in Eqs. (16), (27), and (28). The paper must either correct the pseudocode to use averages or rescale the theoretical bounds accordingly.
  3. [Sections 4.2.1 and 4.2.3, 'optimal convergence rate' claims] Both Theorem 1 and Theorem 3 establish O(1/sqrt(T)) rates, with the constant depending on R or on P,Q. The paper repeatedly refers to 'optimal convergence rate' (e.g., Corollary 1 and Corollary 2), but no lower bound is proved, so the term can only mean 'smallest constant R among a restricted family.' In particular, Corollary 1 claims that Eq. (16) achieves the optimal convergence rate, yet Eq. (16) is infeasible for rank-r SP broadcast. The claimed PS superiority in convergence rate is therefore not a statement about the exponent in T but about constants, and it must be justified by minimizing R over the actual feasible set of broadcast operators.
  4. [Section 4.2.3, Corollary 2 and Appendix A.7] The constants in Corollary 2 are inconsistent with the derivation. Corollary 2 states P^2 = 4 E^2 G^2 C_A^4 and Q^2 = 4 E^2 G^2 C_B^2, while Appendix A.7, Eqs. (80)-(83), derives P^2 = 4 E^2 C_A^2 G^2 and Q^2 = 4 E^2 C_B^2 G^2. Additionally, the expression for R^2 in Corollary 2 (64 E^4 G^4 C_A^2 C_B^2 eta^2 + 12 E^2 G^2 (Q^2 C_B^4 + P^2 C_A^4)) does not match Eq. (67) in Appendix A.6 (4 P^2 Q^2 eta^2 + 3 C_B^2 Q^2 + 3 C_A^2 P^2). These discrepancies must be resolved before the scaling claims R^2_sp = O(E^2) and R^2_ps = O(E^4) in Section 5.3 can be used.
minor comments (6)
  1. [Abstract and Introduction] There are several typos, e.g., 'categories' should be 'categorize', 'These theoretical analyze' should be 'These theoretical analyses', and 'Boradcast' in Algorithm 2 should be 'Broadcast'.
  2. [Theorem 1] The learning rate condition 'eta > xi > 0 for some xi > 0' is vague. The proof requires eta to be large enough so that the linear term 1/2 (R^2 + G^2) eta is dominated by M eta^2; the exact threshold should be stated explicitly.
  3. [Theorem 3] The theorem statement uses R^2 without defining it; the proof introduces R^2 = 4 P^2 Q^2 eta^2 + 3 C_B^2 Q^2 + 3 C_A^2 P^2. The theorem should define R^2 or replace it with the explicit expression.
  4. [Section 5.2] The figures and text are inconsistent about ranks: the text mentions delta = 0.1, 0.3, 0.5, 1.0, while Figure 1 labels 'Rank=10%, 30%, 50%' and the experimental setup in Section 5.1 defines delta values. Please align the notation.
  5. [Table 2] The caption says 'Side-by-side comparison of MNIST and FMNIST tables', but the experiments only use MNIST. The caption should be corrected.
  6. [Assumption 3] The assumption that LoRA matrices are uniformly bounded is strong and all bounds depend on C_A and C_B. The paper should discuss when this assumption is plausible during training (e.g., with weight decay or projection) and what happens if it is violated.

Circularity Check

1 steps flagged · score 6.0 of 10

SP's 'cannot achieve optimal convergence rate' conclusion is built into the optimality condition: Corollary 1 defines optimality as the unconstrained mean, while SP is the rank-constrained minimizer, so the broadcast 'error' is the feasibility gap by construction.

  1. self definitional [Section 4.2.1, Corollary 1 and Eqs. (15)-(17); Appendix A.5, Eqs. (60)-(62)]
    "We refer the Eq. (16) as the optimality condition under the weak convergence condition. It's obvious that the SP Aggregation Method (As shown in Eq. (3)) satisfy the Eq. (16) in Corollary. 1 during aggregation phase. However, issues arise during the broadcast phase. ... P(A(...))Q(A(...)) = Ũ[:, : r]Σ[: r, : r]Ṽ⊤[: r, :] ≠ 1/m Σ B_i A_i (17). These results indicate that the SP aggregate-broadcast strategy cannot achieve the optimal convergence rate due to the broadcast error, also referred to as broadcast loss."

    The 'optimal convergence rate' is characterized by Eq. (16), obtained in Appendix A.5 by minimizing f(X) = 1/m Σ_i ||X − X_i||²_F over all matrices X (Eqs. 60-62) and taking the unconstrained minimizer X = 1/m Σ_i X_i. But X = P(A,B)Q(A,B) is rank-constrained: rank(PQ) ≤ r. The constrained minimizer of the same objective (Eq. 15) is the rank-r truncated SVD of the mean, which is exactly the SP broadcast in Eqs. (7)-(8). Hence SP attains the smallest feasible R in Definition 2, and Eq. (17) is just the gap between the feasible rank-r set and the infeasible full-rank target.

full rationale

The convergence theorems (Theorem 1 and 3) and the sufficient conditions are derived from Assumptions 1-3 and Definitions 2-3 without fitting; no fitted parameter is renamed as a prediction. The self-citation to RBLA [3] is used only as an example of PS aggregation and for an experimental rank-scaling scheme, so it is not load-bearing for the convergence analysis. The one substantive circularity is the optimality claim behind the headline SP-versus-PS asymmetry: Corollary 1 defines 'optimal convergence rate' as equality to the unconstrained average (Eq. 16), and then SP's SVD broadcast is shown to violate Eq. (16) (Eq. 17). Because P and Q are rank-constrained, the genuine minimizer of Eq. (15) is the rank-r truncated SVD that SP already broadcasts; the broadcast 'error' is therefore an artifact of comparing against an infeasible optimum. This makes the central claim that SP cannot achieve the optimal convergence rate true by construction rather than derived. The rest of the derivation chain is self-contained, so the overall score is 6 rather than higher.

Assumptions & free parameters 0 free parameters · 6 assumptions · 1 invented entities

The central claims rest on standard smoothness and boundedness assumptions and on the fully-participating, synchronized-loop setting. No parameters are fitted to data; the constants in the bounds (G, C_A, C_B, D) are assumed to exist. The ABO is a new abstraction with no independent empirical handle.

assumptions (6)
  • standard math L-smoothness of loss functions (Assumption 1)
    Used throughout the proof of Theorems 1 and 3 via descent lemmas.
  • domain assumption Uniformly bounded stochastic gradients (Assumption 2)
    Bounded expected squared gradient norm, needed for Lemma 1 and the final rate.
  • domain assumption Uniformly bounded LoRA matrices (Assumption 3)
    Bounds on ||B|| and ||A|| appear in every lemma and corollary; weakest for real training where adapters may grow.
  • domain assumption Full participation and identical learning rate for all clients
    Stateed at the start of Section 4.2; avoids partial participation effects.
  • domain assumption All clients start each communication round with identical LoRA matrices
    Used in the proof of Corollary 1 to set A(nE)_j = A(nE)_i and B(nE)_j = B(nE)_i before local drift.
  • domain assumption All clients share the same LoRA rank for the PS optimality result
    Corollary 2 states 'if all client share the same LoRA rank'; the analysis does not cover heterogeneous ranks.
invented entities (1)
  • Aggregation-Broadcast Operator (ABO)
    purpose: To unify and analyze arbitrary LoRA aggregation and broadcast strategies in federated learning.
    A formal definition (Definition 1) that encapsulates aggregation rules; its usefulness is evaluated solely through the paper's theorems and experiments, with no external falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning." pith.science (2026). https://pith.science/paper/VFDDQZB5

@misc{pith2026250801348,
  author       = {Pith},
  title        = {Pith review of: Convergence Analysis of Aggregation-Broadcast in LoRA-enabled Distributed Fine-Tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VFDDQZB5}},
  note         = {Machine review of arXiv:2508.01348}
}
read the original abstract

Federated Learning (FL) enables collaborative model training across decentralized data sources while preserving data privacy. However, the growing size of Machine Learning (ML) models poses communication and computation challenges in FL. Low-Rank Adaptation (LoRA) has recently been introduced into FL as an efficient fine-tuning method, reducing communication overhead by updating only a small number of trainable parameters. Despite its effectiveness, how to aggregate LoRA-updated local models on the server remains a critical and understudied problem. In this paper, we provide a unified convergence analysis for LoRA-based FL. We first categories the current aggregation method into two major type: Sum-Product (SP) and Product-Sum (PS). Then we formally define the Aggregation-Broadcast Operator (ABO) and derive both weak and strong convergence condition under mild assumptions. Furthermore, we present both weak and strong convergence condition that guarantee convergence of the local model and the global model respectively. These theoretical analyze offer a principled understanding of various aggregation strategies. Notably, we prove that the SP and PS aggregation methods satisfy the weak and strong convergence condition respectively, but differ in their ability to achieve the optimal convergence rate. Extensive experiments on standard benchmarks validate our theoretical findings.

Figures

Figures reproduced from arXiv: 2508.01348 by the authors.

Figure 1
Figure 1. Comparison of PS and SP of homogeneous ranks on MNIST dataset. [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. Comparison of PS learning curve on homogeneous ranks. [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Comparison of SP learning curve on homogeneous ranks. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Subspace-Constrained Federated Learning with Low-Rank Adaptation

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    Subspace-Reg federated LoRA yields higher accuracy and near-perfect subspace overlap (~0.9999) on RoBERTa-large in 10-client non-IID HellaSwag experiments versus FedAvg and SVD baselines, but FedAvg wins on SmolLM-360M.

Reference graph

Works this paper leans on

40 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Federated fine-tuning of large language models under heterogeneous tasks and client resources

    Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    2024

  2. [2]

    Lora-fair: Federated lora fine-tuning with aggregation and initialization refinement

    Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu. Lora-fair: Federated lora fine-tuning with aggregation and initialization refinement. arXiv preprint arXiv:2411.14961 , 2024

    arXiv 2024

  3. [3]

    Rbla: Rank-based-lora- aggregation for fine-tuning heterogeneous models in flaas

    Shuaijun Chen, Omid Tavallaie, Niousha Nazemi, and Albert Y Zomaya. Rbla: Rank-based-lora- aggregation for fine-tuning heterogeneous models in flaas. In International Conference on Web Services, pages 47–62. Springer, 2024

    work page 2024

  4. [4]

    Robust feder- ated finetuning of foundation models via alternating minimization of lora

    Shuangyi Chen, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti. Robust feder- ated finetuning of foundation models via alternating minimization of lora. arXiv preprint arXiv:2409.02346, 2024

    arXiv 2024

  5. [5]

    Heterogeneous loRA for federated fine-tuning of on-device foundation models

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, Matt Barnes, and Gauri Joshi. Heterogeneous loRA for federated fine-tuning of on-device foundation models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023 , 2023

    work page 2023

  6. [6]

    Heterogeneous LoRA for federated fine-tuning of on-device foundation models

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12903–12913, Miami, Florida, USA, November 2024. Associat...

    work page 2024

  7. [7]

    Client selection in federated learning: Convergence analysis and power-of-choice selection strategies, 2021

    Yae Jee Cho, Jianyu Wang, and Gauri Joshi. Client selection in federated learning: Convergence analysis and power-of-choice selection strategies, 2021

    work page 2021

  8. [8]

    Shuiguang Deng, Hailiang Zhao, Weijia Fang, Jianwei Yin, Schahram Dustdar, and Albert Y. Zomaya. Edge intelligence: The confluence of edge computing and artificial intelligence. IEEE Internet of Things Journal , 7(8):7457–7469, 2020

    work page 2020

Show all 40 references
  1. [9]

    Qlora: Efficient finetun- ing of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetun- ing of quantized llms. Advances in Neural Information Processing Systems , 36, 2024

    2024

  2. [10]

    Selec- tive aggregation for low-rank adaptation in federated learning

    Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selec- tive aggregation for low-rank adaptation in federated learning. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  3. [11]

    Universal language model fine-tuning for text classification

    Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Iryna Gurevych and Yusuke Miyao, editors, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 328–339, Melbou...

    2018

  4. [12]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022

    2022

  5. [13]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022

    2022

  6. [14]

    Ad- vances and open problems in federated learning

    Peter Kairouz, H Brendan McMahan, Brendan Avent, Aur´ elien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, et al. Ad- vances and open problems in federated learning. Foundations and trends® in machine learning , ...

    2021

  7. [15]

    Scaffold: Stochastic controlled averaging for federated learning

    Sai Praneeth Karimireddy, Satyen Kale, Mehryar Mohri, Sashank Reddi, Sebastian Stich, and Ananda Theertha Suresh. Scaffold: Stochastic controlled averaging for federated learning. In ICML, 2020

    2020

  8. [16]

    Federated learning: Strategies for improving communication efficiency

    Jakub Koneˇ cn` y, H Brendan McMahan, Felix X Yu, Peter Richt´ arik, Ananda Theertha Suresh, and Dave Bacon. Federated learning: Strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492, 2016

    2016 arXiv

  9. [17]

    Towards robust and efficient federated low-rank adaptation with heterogeneous clients

    Jabin Koo, Minwoo Jang, and Jungseul Ok. Towards robust and efficient federated low-rank adaptation with heterogeneous clients. arXiv preprint arXiv:2410.22815 , 2024

    2024 arXiv

  10. [18]

    Model-contrastive federated learning

    Qinbin Li, Bingsheng He, and Dawn Song. Model-contrastive federated learning. In CVPR, 2021

    2021

  11. [19]

    Federated optimization in heterogeneous networks

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems, 2:429–450, 2020

    2020

  12. [20]

    Fair resource allocation in federated learning

    Tian Li, Maziar Sanjabi, Ahmad Beirami, and Virginia Smith. Fair resource allocation in federated learning. arXiv preprint arXiv:1905.10497 , 2019

    1905 arXiv

  13. [21]

    On the convergence of fedavg on non-iid data

    Xiang Li, Kaixuan Huang, Wenhao Yang, Shusen Wang, and Zhihua Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations , 2020

    2020

  14. [22]

    A kernel-based view of language model fine-tuning

    Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning , pages 23610–23641. PMLR, 2023

    2023

  15. [23]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelli- gence and statistics , pages 1273–1282. PMLR, 2017

    2017

  16. [24]

    Le, and Kok-Seng Wong

    Ngoc-Hieu Nguyen, Tuan-Anh Nguyen, Tuan Minh Nguyen, Vu Tien Hoang, Dung D. Le, and Kok-Seng Wong. Towards efficient communication and secure federated recommendation system via low-rank training. In The Web Conference 2024 , 2024

    2024

  17. [25]

    Fedinc: One-shot federated tuning for collaborative incident recognition

    Huangsiyuan Qin and Ying Li. Fedinc: One-shot federated tuning for collaborative incident recognition. In International Conference on Artificial Neural Networks , pages 174–185. Springer, 2024

    2024

  18. [26]

    Fedex-loRA: Exact aggregation for federated and efficient fine-tuning of foundation models, 2024

    Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. Fedex-loRA: Exact aggregation for federated and efficient fine-tuning of foundation models, 2024

    2024

  19. [27]

    Federated multi-task learning

    Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet Talwalkar. Federated multi-task learning. In NeurIPS, 2017

    2017

  20. [28]

    Improving loRA in privacy-preserving feder- ated learning

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preserving feder- ated learning. In The Twelfth International Conference on Learning Representations , 2024

    2024

  21. [29]

    Tackling the objective inconsistency problem in heterogeneous federated optimization

    Jihun Wang, Qing Liu, Haoran Liang, Gauri Joshi, and H Vincent Poor. Tackling the objective inconsistency problem in heterogeneous federated optimization. In NeurIPS, 2020

    2020

  22. [30]

    FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations

    Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024

    2024

  23. [31]

    FedloRA: When personalized federated learning meets low-rank adaptation, 2024

    Xinghao Wu, Xuefeng Liu, Jianwei Niu, Haolin Wang, Shaojie Tang, and Guogang Zhu. FedloRA: When personalized federated learning meets low-rank adaptation, 2024

    2024

  24. [32]

    QA-loRA: Quantization-aware low-rank adaptation of large language models

    Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, XIAOPENG ZHANG, and Qi Tian. QA-loRA: Quantization-aware low-rank adaptation of large language models. In The Twelfth International Conference on Learning Representations , 2024. 15

    2024

  25. [33]

    Federated low-rank adaptation for foundation models: A survey

    Yiyuan Yang, Guodong Long, Qinghua Lu, Liming Zhu, Jing Jiang, and Chengqi Zhang. Federated low-rank adaptation for foundation models: A survey. arXiv preprint arXiv:2505.13502 , 2025

    2025

  26. [34]

    Fedlora: Model-heterogeneous personalized federated learning with lora tuning

    Liping Yi, Han Yu, Gang Wang, and Xiaoguang Liu. Fedlora: Model-heterogeneous personalized federated learning with lora tuning. arXiv preprint arXiv:2310.13283 , 2023

    2023 arXiv

  27. [35]

    Parallel restarted sgd with faster convergence and less communication: demystifying why model averaging works for deep learning

    Hao Yu, Sen Yang, and Shenghuo Zhu. Parallel restarted sgd with faster convergence and less communication: demystifying why model averaging works for deep learning. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applic...

    2019

  28. [36]

    Towards building the federatedgpt: Federated instruction tuning

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p...

    2024

  29. [37]

    Towards building the federatedgpt: Federated instruction tuning

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pa...

    2024

  30. [38]

    Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations , 2023

    2023

  31. [39]

    AutoLoRA: Automatically tuning matrix ranks in low-rank adaptation based on meta learning

    Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Pengtao Xie. AutoLoRA: Automatically tuning matrix ranks in low-rank adaptation based on meta learning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapte...

    2024

  32. [40]

    Federated learning with non-iid data

    Yue Zhao, Meng Li, Liangzhen Lai, Naveen Suda, Damon Civin, and Vikas Chandra. Federated learning with non-iid data. arXiv preprint arXiv:1806.00582 , 2018. 16 A Appendix This subsection presents proofs for the theorems and corollaries in the main text, including convergence a...

    2018 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.