Pith. sign in

REVIEW 3 major objections 4 minor 91 references

SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Federated LoRA's rank clash and bilinear mismatch both dissolve when clients send fixed-size linear sketches of their updates instead of factor matrices, with an $\mathcal{O}(1/T)$ convergence rate up to a spectral-tail error floor.

desk verdict Novel sketching trick for federated LoRA that cleanly solves rank heterogeneity and the bilinear mismatch, but the advertised O(1/T) convergence rate is not supported by the proof (it's O(1/sqrt(T)) plus a floor). read the letter →

arxiv 2608.10144 v1 pith:HOEDRMZP submitted 2026-08-10 cs.LG cs.DC

Yue Xia , Tayyebeh Jahani-Nezhad , Mayank Bakshi , Rawad Bitar This is my paper · ORCID
classification cs.LGcs.DC
keywords federatedlearninglow-rankadaptationLoRAmatrixsketchingrankheterogeneitybilinearmismatchparameter-efficientfine-tuningconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Federated fine-tuning of large models with LoRA runs into two problems that neither setting has alone: clients may pick different adapter ranks, making their factor matrices impossible to average directly, and even with matching ranks, averaging the factors first and multiplying afterwards gives a different matrix from averaging the full updates — the bilinear mismatch. SeFoRA sidesteps both by having each client send two fixed-size linear sketches of its local update $\Delta W_i = B_i A_i$ rather than the factors themselves. Because sketching is linear, the federator's average of the sketches is exactly the sketch of the averaged update, so rank heterogeneity stops mattering to the aggregation and the mismatch is gone. For the rank-homogeneous version SeFoRA-Ho the paper proves convergence to a neighborhood of a first-order stationary point at rate $\mathcal{O}(1/T)$, with the neighborhood size bounded by the spectral tail energy of the aggregated client updates. Experiments fine-tuning RoBERTa-Large on four GLUE tasks show SeFoRA matching or beating state-of-the-art rank-heterogeneous baselines at lower server-side computation.

What carries the argument

The load-bearing object is the two-sided random sketching operator of [48]. For a target matrix $W$ and independent standard Gaussian matrices $\Omega\in\mathbb{R}^{s\times r}$ and $\Psi\in\mathbb{R}^{k\times d}$ with $k>r+1$, the pair of sketches $Y=W\Omega$ and $Z=\Psi W$ determines a rank-$r$ approximation $\hat{W}=\hat{B}\hat{A}$ whose expected squared error is at most $(1 + \frac{r}{k-r-1})(1 + \frac{\varrho}{r-\varrho-1})\tau^2_{\varrho+1}(W)$, where $\tau^2_{\varrho+1}(W)$ is the spectral tail energy of $W$ beyond rank $\varrho+1$ (Lemma 2). The property that does the work is linearity: averaging clients' sketch pairs gives the sketch of the averaged update, so the federator aggregates in a small subspace of size $dr + ks$ instead of the full model space. The inverse map (Algorithm 1) is a short QR-based factorization that runs $Y = \hat{B}R$, factors $(\Psi\hat{B}) = UT$, and returns $\hat{A} = T^{-1}U^\top Z$.

What would settle it

Measure the average spectral tail energy $\tau^2_{\varrho+1}(\sum_i p_i B_i A_i)$ of the aggregated client updates during a real federated fine-tuning run: if it is large relative to the early-round gradient norms, Corollary 1 predicts a stagnation plateau in the gradient norm that should be visible as training proceeds. Conversely, run the rank-heterogeneous SeFoRA on a synthetic task where clients' updates are constructed to occupy disjoint rank-$r$ subspaces; the paper gives no convergence bound for this setting, so an experiment showing the gradient norm stalling or diverging there would delimit the method's reach.

Watch

Extended reading notes

Core claim

The central claim is that moving federated LoRA aggregation into a sketched subspace removes both the bilinear mismatch and rank heterogeneity at once. Each client sketches its local update $\Delta W_i$ into $Y_i = \Delta W_i\Omega \in \mathbb{R}^{d\times r}$ and $Z_i = \Psi\Delta W_i \in \mathbb{R}^{k\times s}$ using shared Gaussian matrices $\Omega$ and $\Psi$ with $k > r+1$; the federator averages the sketches, and the sketch-inversion procedure of [48] recovers a rank-$r$ approximation of the aggregated update $\sum_i p_i\Delta W_i$. Linearity of the sketch operator makes the average of the sketches exactly the sketch of the average, eliminating the factor-level mismatch that plagues FedAvg-style LoRA, and the fixed sketch dimensions make client ranks irrelevant to the aggregation. The paper further proves (Theorem 1, Corollary 1, Section 4) that the rank-homogeneous variant SeFoRA-Ho converges to a neighborhood of a first-order stationary point of the original LoRA objective at rate $\mathcal{O}(1/T + c)$, where the floor $c$ is bounded by the spectral tail energy of the aggregated client updates (Assumption 4); for the rank-heterogeneous version the claim is supported by experiments rather than a convergence theorem.

Load-bearing premise

The guarantee and the practical accuracy both rest on the aggregated client update matrix being well approximated by a rank-$r$ matrix: if a meaningful share of its energy sits beyond rank $r$, the sketching error floor $c$ dominates the $\mathcal{O}(1/T)$ term and the final model can sit far from a stationary point, and no bound is proven for the rank-heterogeneous algorithm where that assumption is most strained.

Editorial extensions

If this is right

  • SeFoRA-Ho carries a non-convex convergence rate on the original LoRA objective ($\mathcal{O}(1/T)$ up to the sketching floor) while neither aggregating in the full model space nor freezing one factor — a combination the cited baselines do not achieve.
  • Server-side cost per layer drops from full-space reconstruction (such as $O(d^2s)$ SVD or $O(Ndrs)$ update reconstruction) to $O((N+r)r(d+s))$, making large-model federation cheaper at the federator.
  • Clients can change their local ranks during training, growing or shrinking them, because the sketch dimensions stay fixed; adaptive rank allocation therefore combines directly with aggregation.
  • Because aggregation is linear, secure aggregation and lower-noise differential privacy for federated LoRA become reachable, which the paper names as future work.
  • The error floor $c$ is a real accuracy ceiling set by how well the aggregated update is captured by a rank-$r$ matrix, so tasks with high-rank shared updates pay a visible price.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit: the federator can monitor the reconstruction error $\|\hat{W} - \sum_i p_i\Delta W_i\|_F$ each round and grow the sketch width $k$ when it rises, trading communication for a smaller floor $c$ — a practical rule that follows directly from Corollary 1 but is not stated.
  • The convergence proof covers only the rank-homogeneous variant; whether the rank-truncation and rank-adaptation steps in the heterogeneous algorithm keep the tail energy small is untested, so the natural next theorem is a tail-energy bound for SeFoRA with adaptive client ranks.
  • The experiments show client ranks shrinking as training progresses, which suggests the global rank $r$ itself could decay over time; whether the guarantee degrades gracefully under such rank decay is an open question the paper does not address.
  • The similar accuracy of FlexLoRA and SeFoRA in several settings hints that the deciding factor between full-space and sketched aggregation is the tail energy of the aggregate rather than the aggregation space itself — a hypothesis testable by measuring tail energy directly on the same runs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SeFoRA, a federated low-rank adaptation method in which clients transmit linear sketches of their local LoRA updates and the federator aggregates these sketches directly, thereby avoiding the bilinear mismatch and full-space aggregation while supporting heterogeneous client ranks. A rank-homogeneous variant, SeFoRA-Ho, is introduced and analyzed. The authors prove convergence of SeFoRA-Ho to a neighborhood of a first-order stationary point under smoothness, bounded-moment, and bounded-factor assumptions, and they bound the error floor through an assumed bound on the spectral tail energy of the aggregated client updates. Experiments on RoBERTa-Large over four GLUE tasks compare SeFoRA-Ho with FedIT and FFA-LoRA, and SeFoRA with FlexLoRA, FLoRA, and FSLoRA, reporting competitive accuracy and reduced server-side computation.

Significance. If the claims are correct, this is a meaningful step for federated LoRA: the sketching-based aggregation is elegant, enables linear aggregation, and avoids the bilinear mismatch without communicating full-model updates. The proof is detailed and builds on an external, well-established sketching bound rather than on a circular argument. The method is also practically plausible, with experiments covering rank-homogeneous and rank-heterogeneous settings. The main theoretical claim, however, is overstated: the stated convergence rate O(1/T) is not what the proof delivers, and this is a load-bearing issue that affects the abstract, the contributions list, and Table 1. The experimental comparison also omits HetLoRA, one of the most directly relevant baselines, so the empirical claim of outperforming the state of the art needs qualification.

major comments (3)
  1. [Section 4, Corollary 1; Abstract; Contribution 2 (Section 3); Table 1] The advertised convergence rate O(1/T) is not supported by the proof. In Corollary 1, the learning rate is chosen as eta = gamma_T/E with gamma_T = sqrt((Delta_0/T + chi*epsilon + 2*mu*epsilon^2)/(chi*C3 + 2*C4)). Substituting this into the bound gives an average squared gradient norm bounded by 4*gamma_T*(chi*C3 + 2*C4) plus O(gamma_T^2 + gamma_T^3). Even when the sketching error is exactly zero (epsilon = 0), gamma_T = Theta(1/sqrt(T)), so the bound is Theta(1/sqrt(T)), not Theta(1/T). For epsilon > 0, the bound tends to a positive constant of order sqrt(epsilon) as T grows, so the floor is O(sqrt(epsilon)), not an additive c in an O(1/T + c) statement. This is not a stylistic mismatch: the rate is a headline theoretical contribution. The abstract, Contribution 2, and Table 1 should be revised to state convergence to an O(sqrt(epsilon))-neighborhood at rate O(1/sqrt(T)), or the proof must be modified to yield the claimed rate.
  2. [Section 5, Rank-heterogeneous results (Fig. 1, Table 3)] The experimental section omits HetLoRA, which the paper itself identifies as one of the most relevant rank-heterogeneous federated LoRA baselines and which appears in the comparison Table 1. Without this baseline, the claim that SeFoRA 'outperforms the state-of-the-art' is not fully substantiated for the rank-heterogeneous setting. The reported results also show SeFoRA-Ho performing comparably to FedIT, not clearly better; the text should be adjusted to reflect this accurately.
  3. [Assumption 4, Section 4] The convergence guarantee in Corollary 1 is conditioned on the aggregated client update matrix having small spectral tail energy beyond rank r. This assumption is central: if the tail energy is not small, the sketching floor dominates the optimization error. The paper does not empirically measure this tail energy or verify Assumption 4 in the experiments, despite using k = r + 2 in several settings. The authors should either provide measurements of the spectral tail energy of the aggregated updates or explicitly state that the practical regime of validity of the convergence guarantee is unverified.
minor comments (4)
  1. [Section 6, paragraph 'Combining PEFT with FL'] The sentence 'Combining LoRA with FL raises several challenges faces challenges that do not exist' contains a duplicated phrase and should be rewritten.
  2. [References] Reference [5] contains the typo 'Antrhopic' and reference [48] has a spacing issue in 'V olkan'; these should be corrected.
  3. [Figure 1 and Section 5] The caption of Figure 1 reports accuracy curves but does not state the random seed count or error bars; adding standard-deviation shading or a note that the same three-seed protocol as Table 2 is used would improve clarity.
  4. [Algorithm 2, Step 2] The pseudocode says the federator 'send the shared seed' while the text says 'communicates them to the clients via a shared random seed'; the wording should be unified for consistency.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the convergence derivation relies on an external sketching bound and explicit tail-energy assumptions, not on its own conclusions.

full rationale

The paper's derivation chain is self-contained and non-circular. SeFoRA's aggregation is linear in the sketches by construction (Y = W\Omega, Z = \Psi W), and the federator sums client sketches before unsketching; this is a design property, not an assumed conclusion. The convergence analysis of SeFoRA-Ho (Theorem 1) is built on standard Assumptions 1-3 (smoothness, bounded stochastic gradients, bounded LoRA factors) plus Lemma 2, which is an external low-rank sketching guarantee from Tropp et al. (2017). The reconstruction error E_sketch is defined as the difference between the unsketched aggregate and the true aggregate, and its expectation is controlled through Lemma 2; it is not set equal to the theorem's conclusion. Corollary 1 then bounds the average sketching floor by invoking Assumption 4, which explicitly bounds the spectral tail energy of the aggregated client updates. This is a transparent, data-dependent assumption rather than a fitted parameter renamed as a prediction: tau^2 enters the definition of epsilon^2 and the final bound exactly as stated. There are no load-bearing self-citations: none of the cited results come from the present authors, and no uniqueness theorem from prior work is used to force the algorithm. The claimed O(1/T+c) rate is supported by the constant-learning-rate form of Theorem 1, where the first term is 2Delta_0/(eta E T); whether a T-dependent learning-rate schedule in Corollary 1 yields a O(1/sqrt(T)) term is a question of rate optimality and is a correctness risk, not a circularity. No step of the derivation reduces to its own input by definition.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. The free parameters are the algorithm's hyperparameters (rank, sketch dimension, truncation threshold). The main load-bearing assumptions are the standard smoothness and bounded-gradient conditions plus the data-dependent tail-energy assumption that controls the sketching floor; the latter is the most fragile.

free parameters (3)
  • Maximum LoRA rank r = 16 in experiments
    Chosen by the federator as the rank cap for the sketch reconstruction; a design choice, not derived from the theory.
  • Sketch dimension k = 18 (r+2) in experiments
    Selected as r+2; the paper shows empirically that this is sufficient, but no theory specifies the optimal k.
  • Rank-update energy threshold tau = 90% in experiments
    Used in Algorithm 4 to decide when to truncate local ranks; affects empirical behavior but is not analyzed theoretically.
assumptions (5)
  • domain assumption Per-sample Lipschitz smoothness of the loss (Assumption 1)
    Standard smoothness condition used to enable the descent lemma in the convergence proof; introduced in Section 4.
  • domain assumption Unbiased stochastic gradients with bounded second and fourth moments (Assumption 2)
    Standard bounded-variance assumption used throughout Lemmas 3-6 and the proof of Theorem 1.
  • domain assumption Bounded LoRA factors (Assumption 3)
    Assumes spectral norms of all local and global adapters are bounded by constants; used pervasively in the drift and gradient bounds. The paper notes the global B from QR has norm 1, but local B_i after gradient steps must also be bounded.
  • ad hoc to paper Bounded average spectral tail energy of the aggregated update (Assumption 4)
    Introduced in Section 4 specifically to convert the sketching error floor into a constant; without this assumption, the convergence neighborhood is uncontrolled and the 'O(1/T)' claim is vacuous.
  • standard math Low-rank approximation error bound for Algorithm 1 (Lemma 2, from Tropp et al. 2017)
    External mathematical guarantee for the sketch-inversion procedure; imported as a known result and used to bound the reconstruction error.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks." pith.science (2026). https://pith.science/paper/HOEDRMZP

@misc{pith2026260810144,
  author       = {Pith},
  title        = {Pith review of: SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HOEDRMZP}},
  note         = {Machine review of arXiv:2608.10144}
}
abstract

We consider federated parameter efficient fine-tuning of large neural networks with low-rank adaptation (LoRA,~Hu et al.\ 2022). Combining LoRA with federated PEFT introduces challenges absent from either setting alone: clients may use different LoRA ranks, making their factor matrices dimension-incompatible, and factor-wise averaging suffers from a bilinear mismatch. We propose SeFoRA, a sketch-aggregated federated LoRA algorithm in which each client transmits a linear sketch of its local updates, enabling direct aggregation at the federator. As a result, SeFoRA alleviates the bilinear mismatch, and allows for aggregation in a small subspace of the full model. We introduce a rank-homogeneous version called SeFoRA-Ho which allows for direct adapter aggregation in this setting. We prove convergence to a neighborhood of the first-order stationary point at rate $\cO(1/T)$ for the rank-homogeneous setting. Numerical experiments on fine-tuning RoBERTa-Large on GLUE datasets show how our algorithms outperform the state-of-the-art.

Figures

Figures reproduced from arXiv: 2608.10144 by the authors.

Figure 1
Figure 1. We report the accuracy in the rank-heterogeneous setting. We use [PITH_FULL_IMAGE:figures/full_fig_p014_1.png] view at source ↗
Figure 2
Figure 2. We report the evolution of the clients’ average adaptive rank for [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

91 extracted references · 51 canonical work pages

  1. [1]

    GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171–4186, 2019

    2019

  3. [3]

    PaLM 2 technical report.arXiv preprint arXiv:2305.10403, 2023

    Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM 2 technical report.arXiv preprint arXiv:2305.10403, 2023

    arXiv 2023

  4. [4]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    arXiv 2023

  5. [5]

    The claude 3 model family: Opus, sonnet, haiku

    Antrhopic. The claude 3 model family: Opus, sonnet, haiku. online. URLhttps://www-cdn.anthropic. com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf. accessed on 03.Aug.2026

    2026

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    arXiv 2010

  7. [7]

    Fine-tuning a llm using reinforcement learning from human feedback for a therapy chatbot application, 2023

    Desirée Bill and Theodor Eriksson. Fine-tuning a llm using reinforcement learning from human feedback for a therapy chatbot application, 2023

    2023

  8. [8]

    Towards next-generation intelligent assistants leveraging llm techniques

    Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. Towards next-generation intelligent assistants leveraging llm techniques. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5792–5793, 2023

    2023

Show all 91 references
  1. [9]

    Bing chat: The future of search engines?Proceedings of the Association for Information Science and Technology, 60(1):1007–1009, 2023

    Dominique Kelly, Yimin Chen, Sarah E Cornwell, Nicole S Delellis, Alex Mayhew, Sodiq Onaolapo, and Victoria L Rubin. Bing chat: The future of search engines?Proceedings of the Association for Information Science and Technology, 60(1):1007–1009, 2023

    2023

  2. [10]

    Large language models encode clinical knowledge

    Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, 620(7972):172–180, 2023

    2023

  3. [11]

    Parameter-Efficient Transfer Learning for NLP

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efficient Transfer Learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, pages 2790–27...

    2019

  4. [12]

    Towards a Unified View of Parameter-Efficient Transfer Learning

    Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a Unified View of Parameter-Efficient Transfer Learning. InInternational Conference on Learning Representations, October 2021. 17

    2021

  5. [13]

    Prefix-Tuning: Optimizing Continuous Prompts for Generation

    Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors,Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Jo...

    2021 doi

  6. [14]

    The Power of Scale for Parameter-Efficient Prompt Tuning

    Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processin...

    2021 doi

  7. [15]

    BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models, September 2022

    Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models, September 2022

    2022

  8. [16]

    Parameter-Efficient Fine-Tuning without Introducing New Latency

    Baohao Liao, Yan Meng, and Christof Monz. Parameter-Efficient Fine-Tuning without Introducing New Latency. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors,Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2023 doi

  9. [17]

    LoRA: Low-rank adaptation of large language models.Iclr, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. LoRA: Low-rank adaptation of large language models.Iclr, 1(2):3, 2022

    2022

  10. [18]

    DoRA: Weight-Decomposed Low-Rank Adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-Decomposed Low-Rank Adaptation. InProceedings of the 41st International Conference on Machine Learning, pages 32100–32121. PMLR, July 2024

    2024

  11. [19]

    HydraLoRA: An Asymmetric LoRA architecture for efficient fine-tuning

    Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Chengzhong Xu. HydraLoRA: An Asymmetric LoRA architecture for efficient fine-tuning. InProceedings of the 38th International Conference on Neural Information Processing Systems, volume 37 ofNIPS ’24, pages 9565–9584, Red Hook, N...

  12. [20]

    AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning

    Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512, 2023

    2023 arXiv

  13. [21]

    DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation

    Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. DyLoRA: Parameter-efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Lingu...

    2023

  14. [22]

    Parameter-Efficient Fine-Tuning for Large Models: A Comprehen- sive Survey.Transactions on Machine Learning Research, October 2024

    Zeyu Han, Chao Gao, and Jinyang Liu. Parameter-Efficient Fine-Tuning for Large Models: A Comprehen- sive Survey.Transactions on Machine Learning Research, October 2024

    2024

  15. [23]

    Communication-efficient learning of deep networks from decentralized data

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-efficient learning of deep networks from decentralized data. InArtificial intelligence and statistics, pages 1273–1282. Pmlr, 2017

    2017

  16. [24]

    Advances and open problems in federated learning.Foundations and trends in machine learning, 14(1-2):1–210, 2021

    Peter Kairouz and H Brendan McMahan. Advances and open problems in federated learning.Foundations and trends in machine learning, 14(1-2):1–210, 2021

    2021

  17. [25]

    Federated large language models: Feasibility, robustness, security and future directions

    Wenhao Jiang, Yuchuan Luo, Guilin Deng, Silong Chen, Xu Yang, Shihong Wu, Xinwen Gao, Lin Liu, and Shaojing Fu. Federated large language models: Feasibility, robustness, security and future directions. arXiv preprint arXiv:2505.08830, 2025

    2025 arXiv

  18. [26]

    A survey on federated learning systems: Vision, hype and reality for data privacy and protection.IEEE Transactions on Knowledge and Data Engineering, 35(4):3347–3366, 2021

    Qinbin Li, Zeyi Wen, Zhaomin Wu, Sixu Hu, Naibo Wang, Yuan Li, Xu Liu, and Bingsheng He. A survey on federated learning systems: Vision, hype and reality for data privacy and protection.IEEE Transactions on Knowledge and Data Engineering, 35(4):3347–3366, 2021

    2021

  19. [27]

    Efficient federated learning for modern NLP

    Dongqi Cai, Yaozong Wu, Shangguang Wang, Felix Xiaozhu Lin, and Mengwei Xu. Efficient federated learning for modern NLP. InProceedings of the 29th Annual International Conference on Mobile Computing and Networking (MobiCom), pages 1–16, 2023

    2023

  20. [28]

    Communication-efficient and tensorized federated fine-tuning of large language models.arXiv preprint arXiv:2410.13097, 2024

    Sajjad Ghiasvand, Yifan Yang, Zhiyu Xue, Mahnoosh Alizadeh, Zheng Zhang, and Ramtin Pedarsani. Communication-efficient and tensorized federated fine-tuning of large language models.arXiv preprint arXiv:2410.13097, 2024. 18

    2024 arXiv

  21. [29]

    Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning

    Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Fedprompt: Communication-efficient and privacy-preserving prompt tuning in federated learning. InICASSP, pages 1–5, 2023

    2023

  22. [30]

    Text-driven prompt generation for vision-language models in federated learning.arXiv preprint arXiv:2310.06123, 2023

    Chen Qiu, Xingyu Li, Chaithanya Kumar Mummadi, Madan Ravi Ganesh, Zhenzhen Li, Lu Peng, and Wan-Yi Lin. Text-driven prompt generation for vision-language models in federated learning.arXiv preprint arXiv:2310.06123, 2023

    2023 arXiv

  23. [31]

    Pablo Muñoz, and Ali Jannesari

    Sixing Yu, J. Pablo Muñoz, and Ali Jannesari. Bridging the gap between foundation models and heteroge- neous federated learning.arXiv preprint arXiv:2310.00247, 2023

    2023 arXiv

  24. [32]

    Rethinking LoRA for data heterogeneous federated learning: Subspace and state alignment.arXiv preprint arXiv:2602.01746, 2026

    Hongyi Peng, Han Yu, Xiaoxiao Li, and Qiang Yang. Rethinking LoRA for data heterogeneous federated learning: Subspace and state alignment.arXiv preprint arXiv:2602.01746, 2026

    2026

  25. [33]

    LoRA+: Efficient low rank adaptation of large models

    Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In International Conference on Machine Learning (ICML), 2024

    2024

  26. [34]

    DoRA: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. InInternational Conference on Machine Learning (ICML), 2024

    2024

  27. [35]

    LoRA-drop: Efficient LoRA parameter pruning based on output evaluation

    Hongyun Zhou, Xiangyu Lu, Wang Xu, Conghui Zhu, Tiejun Zhao, and Muyun Yang. LoRA-drop: Efficient LoRA parameter pruning based on output evaluation. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2025

    2025

  28. [36]

    LLaMA-LoRA neural prompt engineering: A deep tuning framework for automatically generating chinese text logical reasoning thinking chains.Data Intelligence, 2024

    Songlin Chen, Weicheng Wang, Xiaoliang Chen, Peng Lu, Zaiyan Yang, and Yajun Du. LLaMA-LoRA neural prompt engineering: A deep tuning framework for automatically generating chinese text logical reasoning thinking chains.Data Intelligence, 2024

    2024

  29. [37]

    LoRA training provably converges to a low-rank global minimum or it fails loudly (but it probably won’t fail)

    Junsu Kim, Jaeyeon Kim, and Ernest K Ryu. LoRA training provably converges to a low-rank global minimum or it fails loudly (but it probably won’t fail). InInternational Conference on Machine Learning, volume 2025, 2025

    2025

  30. [38]

    Towards building the federated GPT: Federated instruction tuning

    Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federated GPT: Federated instruction tuning. InIEEE international conference on acoustics, speech and signal processing (ICASSP), pages 6915–6919. IEEE, 2024

    2024

  31. [39]

    Flexlora: Entropy-guided flexible low-rank adaptation.arXiv preprint arXiv:2601.22905, 2026

    Muqing Liu, Chongjie Si, and Yuheng Jia. Flexlora: Entropy-guided flexible low-rank adaptation.arXiv preprint arXiv:2601.22905, 2026

    2026

  32. [40]

    Rethinking loRA for privacy-preserving federated learning in large models

    Jin Liu, Yinbin Miao, Ning Xi, and Junkang Liu. Rethinking loRA for privacy-preserving federated learning in large models. InThe Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview.net/forum?id=BPzSV4uw0x

    2026

  33. [41]

    Heterogeneous LoRA for federated fine-tuning of on-device foundation models

    Yae Jee Cho, Luyang Liu, Zheng Xu, Aldi Fahrezi, and Gauri Joshi. Heterogeneous LoRA for federated fine-tuning of on-device foundation models. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 12903–12913, 2024

    2024

  34. [42]

    Robust federated finetuning of foundation models via alternating minimization of lora.arXiv preprint arXiv:2409.02346, 2024

    Shuangyi Chen, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti. Robust federated finetuning of foundation models via alternating minimization of lora.arXiv preprint arXiv:2409.02346, 2024

    2024 arXiv

  35. [43]

    Selective aggregation for low-rank adaptation in federated learning

    Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation in federated learning. In13th International Conference on Learning Representations Iclr 2025, 2025

    2025

  36. [44]

    Improving loRA in privacy-preserving federated learning

    Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in privacy-preserving federated learning. InThe Twelfth International Conference on Learning Representations, 2024. URL https: //openreview.net/forum?id=NLPzL6HWNl

    2024

  37. [45]

    FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations.Advances in Neural Information Processing Systems, 37:22513–22533, 2024

    Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. FLoRA: Federated fine-tuning large language models with heterogeneous low-rank adaptations.Advances in Neural Information Processing Systems, 37:22513–22533, 2024

    2024

  38. [46]

    Federated fine-tuning of large language models under heterogeneous tasks and client resources.Advances in Neural Information Processing Systems, 37:14457–14483, 2024

    Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous tasks and client resources.Advances in Neural Information Processing Systems, 37:14457–14483, 2024

    2024

  39. [47]

    Federated sketching LoRA: On-device collaborative fine-tuning of large language models

    Wenzhi Fang, Dong-Jun Han, Liangqi Yuan, Seyyedali Hosseinalipour, and Christopher G Brinton. Federated sketching LoRA: On-device collaborative fine-tuning of large language models. InInternational Conference on Machine Learning (ICML), 2026. 19

    2026

  40. [48]

    Practical sketching algorithms for low-rank matrix approximation.SIAM Journal on Matrix Analysis and Applications, 38(4):1454–1485, 2017

    Joel A Tropp, Alp Yurtsever, Madeleine Udell, and V olkan Cevher. Practical sketching algorithms for low-rank matrix approximation.SIAM Journal on Matrix Analysis and Applications, 38(4):1454–1485, 2017

    2017

  41. [49]

    Towards federated low-rank adaptation of language models with rank hetero- geneity

    Yuji Byun and Jaeho Lee. Towards federated low-rank adaptation of language models with rank hetero- geneity. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Sho...

    2025

  42. [50]

    RB-LoRA: Rank-balanced aggregation for low-rank adaptation with federated fine-tuning

    Sihyeon Ha, Yongjeong Oh, and Yo-Seb Jeon. RB-LoRA: Rank-balanced aggregation for low-rank adaptation with federated fine-tuning. InFindings of the Association for Computational Linguistics: EACL 2026, pages 1737–1746, 2026

    2026

  43. [51]

    Fedsrd: Sparsify-reconstruct- decompose for communication-efficient federated large language models fine-tuning

    Guochen Yan, Luyuan Xie, Qingni Shen, Yuejian Fang, and Zhonghai Wu. Fedsrd: Sparsify-reconstruct- decompose for communication-efficient federated large language models fine-tuning. InProceedings of the ACM Web Conference 2026, pages 5087–5098, 2026

    2026

  44. [52]

    Adaptive rank allocation for federated parameter-efficient fine-tuning of language models.IEEE Transactions on Computers, 2026

    Fei Wu, Jia Hu, Geyong Min, and Shiqiang Wang. Adaptive rank allocation for federated parameter-efficient fine-tuning of language models.IEEE Transactions on Computers, 2026

    2026

  45. [53]

    Sparse low-rank adaptation of pre-trained language models

    Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. InProceedings of the 2023 conference on empirical methods in natural language processing, pages 4133–4145, 2023

    2023

  46. [54]

    Robust federated finetuning of LLMs via alternating optimization of LoRA.Advances in Neural Information Processing Systems, 38:120038–120090, 2026

    Shuangyi Chen, Yuanxin Guo, Yue Ju, Hardik Dalal, Zhongwen Zhu, and Ashish Khisti. Robust federated finetuning of LLMs via alternating optimization of LoRA.Advances in Neural Information Processing Systems, 38:120038–120090, 2026

    2026

  47. [55]

    Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  48. [56]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. InProceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP...

    2018

  49. [57]

    Federated learning for open banking

    Guodong Long, Yue Tan, Jing Jiang, and Chengqi Zhang. Federated learning for open banking. In Federated Learning: Privacy and Incentive, pages 240–254. Springer, 2020

    2020

  50. [58]

    Pushpita Chatterjee, Debashis Das, and Danda B. Rawat. Federated learning empowered recommendation model for financial consumer services.IEEE Transactions on Consumer Electronics, 2023

    2023

  51. [59]

    Specificity-preserving federated learning for mr image reconstruction.IEEE Transactions on Medical Imaging, 42(7):2010–2021, 2022

    Chun-Mei Feng, Yunlu Yan, Shanshan Wang, Yong Xu, Ling Shao, and Huazhu Fu. Specificity-preserving federated learning for mr image reconstruction.IEEE Transactions on Medical Imaging, 42(7):2010–2021, 2022

    2010

  52. [60]

    Roth, Wenqi Li, Dong Yang, Can Zhao, Vishwesh Nath, Daguang Xu, Qi Dou, and Ziyue Xu

    Meirui Jiang, Holger R. Roth, Wenqi Li, Dong Yang, Can Zhao, Vishwesh Nath, Daguang Xu, Qi Dou, and Ziyue Xu. Fair federated medical image segmentation via client contribution estimation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR...

    2023

  53. [61]

    Learning federated visual prompt in null space for mri reconstruction

    Chun-Mei Feng, Bangjun Li, Xinxing Xu, Yong Liu, Huazhu Fu, and Wangmeng Zuo. Learning federated visual prompt in null space for mri reconstruction. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8064–8073, 2023

    2023

  54. [62]

    Cross-modal vertical federated learning for mri reconstruction.IEEE Journal of Biomedical and Health Informatics, 2024

    Yunlu Yan, Hong Wang, Yawen Huang, Nanjun He, Lei Zhu, Yong Xu, Yuexiang Li, and Yefeng Zheng. Cross-modal vertical federated learning for mri reconstruction.IEEE Journal of Biomedical and Health Informatics, 2024

    2024

  55. [63]

    Federated learning driven large language models for swarm intelligence: A survey.arXiv preprint arXiv:2406.09831, 2024

    Youyang Qu. Federated learning driven large language models for swarm intelligence: A survey.arXiv preprint arXiv:2406.09831, 2024

    2024 arXiv

  56. [64]

    FATE-LLM: An industrial grade federated learning framework for large language models.arXiv preprint arXiv:2310.10049, 2023

    Tao Fan, Yan Kang, Guoqiang Ma, Weijing Chen, Wenbin Wei, Lixin Fan, and Qiang Yang. FATE-LLM: An industrial grade federated learning framework for large language models.arXiv preprint arXiv:2310.10049, 2023. 20

    2023 arXiv

  57. [65]

    FederatedScope-LLM: A comprehensive package for fine-tuning large language models in federated learning

    Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. FederatedScope-LLM: A comprehensive package for fine-tuning large language models in federated learning. InProceedings of the 30th ACM SIGKDD Con...

    2024

  58. [66]

    OpenFedLLM: Training large language models on decentralized private data via federated learning

    Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. OpenFedLLM: Training large language models on decentralized private data via federated learning. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and D...

    2024

  59. [67]

    FedLLM- Bench: Realistic benchmarks for federated learning of large language models.Advances in Neural Information Processing Systems, 37:111106–111130, 2024

    Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Yaxin Du, Yang Liu, Yanfeng Wang, and Siheng Chen. FedLLM- Bench: Realistic benchmarks for federated learning of large language models.Advances in Neural Information Processing Systems, 37:111106–111130, 2024

    2024

  60. [68]

    Fed- BCGD: Communication-efficient accelerated block coordinate gradient descent for federated learning

    Junkang Liu, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Yuangang Li, and YunXiang Gong. Fed- BCGD: Communication-efficient accelerated block coordinate gradient descent for federated learning. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 2955–2963, 2024

    2024

  61. [69]

    Improving generalization in federated learning with highly heterogeneous data via momentum-based stochastic controlled weight averaging

    Junkang Liu, Yuanyuan Liu, Fanhua Shang, Hongying Liu, Jin Liu, and Wei Feng. Improving generalization in federated learning with highly heterogeneous data via momentum-based stochastic controlled weight averaging. InInternational Conference on Machine Learning (ICML), 2024

    2024

  62. [70]

    FedAdamW: A communication-efficient optimizer with convergence and generalization guarantees for federated large models.arXiv preprint arXiv:2510.27486, 2025

    Junkang Liu, Fanhua Shang, Kewen Zhu, Hongying Liu, Yuanyuan Liu, and Jin Liu. FedAdamW: A communication-efficient optimizer with convergence and generalization guarantees for federated large models.arXiv preprint arXiv:2510.27486, 2025

    arXiv

  63. [71]

    Consistency of local and global flatness for federated learning

    Junkang Liu, Fanhua Shang, Yuxuan Tian, Hongying Liu, and Yuanyuan Liu. Consistency of local and global flatness for federated learning. InProceedings of the 33rd ACM International Conference on Multimedia, pages 3875–3883, 2025

    2025

  64. [72]

    FedMuon: Accelerating federated learning with matrix orthogonalization.arXiv preprint arXiv:2510.27403, 2025

    Junkang Liu, Fanhua Shang, Junchao Zhou, Hongying Liu, Yuanyuan Liu, and Jin Liu. FedMuon: Accelerating federated learning with matrix orthogonalization.arXiv preprint arXiv:2510.27403, 2025

    2025

  65. [73]

    DP-FedPGN: Finding global flat minima for differentially private federated learning via penalizing gradient norm.arXiv preprint arXiv:2510.27504, 2025

    Junkang Liu, Yuxuan Tian, Fanhua Shang, Yuanyuan Liu, Hongying Liu, Junchao Zhou, and Daorui Ding. DP-FedPGN: Finding global flat minima for differentially private federated learning via penalizing gradient norm.arXiv preprint arXiv:2510.27504, 2025

  66. [74]

    Federated residual low-rank adaptation of large language models

    Yunlu Yan, Chun-Mei Feng, Wangmeng Zuo, Rick Siow Mong Goh, Yong Liu, and Lei Zhu. Federated residual low-rank adaptation of large language models. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  67. [75]

    Differentially private federated low rank adaptation beyond fixed-matrix

    Ming Wen, Jiaqi Zhu, Yuedong Xu, Yipeng Zhou, and Dingding Han. Differentially private federated low rank adaptation beyond fixed-matrix. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URLhttps://openreview.net/forum?id=TecJ926Vgn

    2025

  68. [76]

    Towards robust parameter-efficient fine-tuning for federated learning.Advances in Neural Information Processing Systems, 38:141777–141800, 2026

    Xiuwen Fang and Mang Ye. Towards robust parameter-efficient fine-tuning for federated learning.Advances in Neural Information Processing Systems, 38:141777–141800, 2026

    2026

  69. [77]

    Personalized federated fine- tuning for LLMs via data-driven heterogeneous model architectures

    Yicheng Zhang, Zhen Qin, Zhaomin Wu, Jian Hou, and Shuiguang Deng. Personalized federated fine- tuning for LLMs via data-driven heterogeneous model architectures. InProceedings of the ACM Web Conference 2026, pages 5099–5110, 2026

    2026

  70. [78]

    FDLoRA: Personalized federated learning of large language model via dual lora tuning.arXiv preprint arXiv:2406.07925, 2026

    Yao Lu, Jiaxing QI, Zhongzhi Luan, Shaohan Huang, Carol Fung, Hailong Yang, and Depei Qian. FDLoRA: Personalized federated learning of large language model via dual lora tuning.arXiv preprint arXiv:2406.07925, 2026. URLhttps://arxiv.org/abs/2406.07925

    2026 arXiv

  71. [79]

    FedEx-LoRA: Exact aggregation for federated and efficient fine-tuning of foundation models.arXiv preprint arXiv:2410.09432, 2024

    Raghav Singhal, Kaustubh Ponkshe, and Praneeth Vepakomma. FedEx-LoRA: Exact aggregation for federated and efficient fine-tuning of foundation models.arXiv preprint arXiv:2410.09432, 2024

    2024 arXiv

  72. [80]

    Fedsvd: Adaptive orthogonalization for private federated learning with lora.Advances in Neural Information Processing Systems, 38:119733–119757, 2026

    Seanie Lee, Sangwoo Park, Dong Bok Lee, Dominik Wagner, Haebin Seong, Tobias Bocklet, Juho Lee, and Sung Ju Hwang. Fedsvd: Adaptive orthogonalization for private federated learning with lora.Advances in Neural Information Processing Systems, 38:119733–119757, 2026

    2026

  73. [81]

    Towards robust and efficient federated low-rank adaptation with heterogeneous clients

    Jabin Koo, Minwoo Jang, and Jungseul Ok. Towards robust and efficient federated low-rank adaptation with heterogeneous clients. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 416–429, 2025

    2025

  74. [82]

    FedRot-LoRA: Mitigating rotational misalignment in federated lora.arXiv preprint arXiv:2602.23638, 2026

    Haoran Zhang, Dongjun Kim, Seohyeon Cha, and Haris Vikalo. FedRot-LoRA: Mitigating rotational misalignment in federated lora.arXiv preprint arXiv:2602.23638, 2026. 21

    2026 arXiv

  75. [83]

    LoRA-FAIR: Federated LoRA fine-tuning with aggregation and initialization refinement.arXiv preprint arXiv:2411.14961, 2024

    Jieming Bian, Lei Wang, Letian Zhang, and Jie Xu. LoRA-FAIR: Federated LoRA fine-tuning with aggregation and initialization refinement.arXiv preprint arXiv:2411.14961, 2024

    2024

  76. [84]

    SLoRA: Federated parameter efficient fine-tuning of language models

    Sara Babakniya, Ahmed Elkordy, Yahya Ezzeldin, Qingfeng Liu, Kee-Bong Song, MOSTAFA EL-Khamy, and Salman Avestimehr. SLoRA: Federated parameter efficient fine-tuning of language models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunct...

    2023

  77. [85]

    FedMomentum: Preserving lora training momentum in federated fine-tuning.arXiv preprint arXiv:2603.08014, 2026

    Peishen Yan, Yang Hua, Hao Wang, Jiaru Zhang, Xiaoyu Wu, Tao Song, and Haibing Guan. FedMomentum: Preserving lora training momentum in federated fine-tuning.arXiv preprint arXiv:2603.08014, 2026

    2026

  78. [86]

    AFLoRA: Adaptive federated fine-tuning of large language models with resource-aware low-rank adaption.arXiv preprint arXiv:2505.24773, 2025

    Yajie Zhou, Xiaoyi Pang, and Zhibo Wang. AFLoRA: Adaptive federated fine-tuning of large language models with resource-aware low-rank adaption.arXiv preprint arXiv:2505.24773, 2025

    2025 arXiv

  79. [87]

    Preventing rank collapse in federated low-rank adaptation with client heterogeneity.arXiv preprint arXiv:2602.13486, 2026

    Fei Wu, Jia Hu, Geyong Min, and Shiqiang Wang. Preventing rank collapse in federated low-rank adaptation with client heterogeneity.arXiv preprint arXiv:2602.13486, 2026

    2026 arXiv

  80. [88]

    Fedhl: Federated learning for heterogeneous low-rank adaptation via unbiased aggregation.arXiv preprint arXiv:2505.18494, 2025

    Zihao Peng, Jiandian Zeng, Boyuan Li, Guo Li, Shengbo Chen, and Tian Wang. Fedhl: Federated learning for heterogeneous low-rank adaptation via unbiased aggregation.arXiv preprint arXiv:2505.18494, 2025

    2025 arXiv

  81. [89]

    Florg: Federated fine-tuning with low-rank gram matrices and procrustes alignment

    Chuiyang Meng, Ming Tang, and Vincent Wong. Florg: Federated fine-tuning with low-rank gram matrices and procrustes alignment. InInternational Conference on Learning Representations, volume 2026, pages 82804–82824, 2026

    2026

  82. [90]

    LX l=1 ∇∆WLfull i (∆W(t,e1) i,l ;ξ (e1) i )(A(t,e1) i,l )⊤ ·(B (t,e2) i,l )⊤∇∆WLfull i (∆W(t,e2) i,l ;ξ (e2) i ) 2 F # (b) ≤ 1 E2 X e1,e2 Et

    Neranjan Senarath, Rohit Muralitharan, and Sadia Asif. Subspace-constrained federated learning with low-rank adaptation.arXiv preprint arXiv:2606.22724, 2026. A Algorithmic representation ofSeFoRA-Ho SeFoRA-Hois summarized in Algorithm 5. B Proof of Theorem 1 and Corollary 1 I...

    2026 arXiv

  83. [2025]

    Curran Associates Inc

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.