Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Decentralized Low-Rank Fine-Tuning of Large Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Dec-LoRA replaces the parameter server with peer-to-peer averaging of LoRA factors and proves $O(T^{-1/2})$ convergence, with experiments matching centralized LoRA accuracy.

desk verdict Decentralized LoRA with gossip is a sensible method and the experiments are the strongest part, but the central O(1/sqrt(T)) guarantee rests on a boundedness assumption that the algorithm does not enforce. read the letter →

arxiv 2501.15361 v5 pith:PI4S6HSV submitted 2025-01-26 cs.LG

Sajjad Ghiasvand , Mahnoosh Alizadeh , Ramtin Pedarsani This is my paper · ORCID
classification cs.LG
keywords decentralizedfederatedlearninglow-rankadaptation(LoRA)largelanguagemodelsparameter-efficientfine-tuningnon-convexoptimizationgossipaveragingconvergenceanalysisquantization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Dec-LoRA is a fully decentralized version of LoRA: a network of clients fine-tunes the low-rank factors $A$ and $B$ locally on private data, exchanges only those small matrices with neighbors, and averages them with a doubly stochastic mixing matrix $Q$ instead of sending anything to a server. The paper's central claim is twofold. Empirically, Dec-LoRA matches the accuracy of centralized LoRA on RoBERTa and LLaMA-2 across GLUE and SuperGLUE-style tasks, including with 4-bit quantized base models and non-i.i.d. data, and it outperforms decentralized Adapter, BitFit, and IA3 baselines on average. Theoretically, under $L$-smoothness, bounded stochastic gradients, and a uniform bound on the LoRA factors, the algorithm converges to a stationary point of the global loss in the averaged variables at rate $O(T^{-1/2})$ in the number of communication rounds $T$. If true, this establishes that serverless peer-to-peer fine-tuning is a practical and provably convergent alternative to centralized federated LoRA.

What carries the argument

The engine is a consensus-averaged LoRA step. Each client runs $K$ local SGD steps on $A_i$ and $B_i$ with learning rate $\eta=1/(K\sqrt{T})$, then sets $A_i^{(t+1)}=\sum_j q_{ij} A_j^{(t)+K}$ and similarly for $B_i$, where $Q$ is symmetric doubly stochastic with second-largest-magnitude eigenvalue $\beta<1$. The proof separates two effects: geometric contraction of the deviation from the network mean (Lemma 4.7) and smoothness of the loss as a function of each factor when the other is fixed (Lemma 4.10, with constants $LC_B^2$ and $LC_A^2$). Summing the descent inequalities for $A$ and $B$ cancels the cross terms and yields the bound in Theorem 4.8; the tractable stationarity metric is the squared norm of $\nabla_{(A,B)} f(\bar B \bar A)$, which is what one can actually control when $A$ and $B$ are aggregated separately.

What would settle it

On a smooth nonconvex problem with bounded data, run Dec-LoRA for growing $T$ with $\eta=1/(K\sqrt{T})$ and record $\frac{1}{T}\sum_t (\|\nabla_A f(\bar B^{(t)}\bar A^{(t)})\|_F^2 + \|\nabla_B f(\bar B^{(t)}\bar A^{(t)})\|_F^2)$; if the metric does not approach zero at the predicted $O(T^{-1/2})$ rate while the iterates stay bounded, the theorem is contradicted. Alternatively, if $\max_{i,t,k}(\|A_i^{(t)+k}\|_F + \|B_i^{(t)+k}\|_F)$ diverges as $T$ grows, then Assumption 4.3 is violated and the theorem's conditions do not hold for the actual trajectory.

Watch

Extended reading notes

Core claim

The discovery is that LoRA's separate $A$ and $B$ updates can be averaged peer-to-peer without destroying convergence, provided stationarity is measured at the averaged point $(\bar B^{(t)}, \bar A^{(t)})$ rather than at the average of the products $B_i A_i$. The paper proves (Theorem 4.8) that when each client performs $K$ local SGD steps and neighbors mix via $Q$, the averaged squared gradient norms, $\frac{1}{T}\sum_{t} (\mathbb{E}\|\nabla_A f(\bar B^{(t)}\bar A^{(t)})\|_F^2 + \mathbb{E}\|\nabla_B f(\bar B^{(t)}\bar A^{(t)})\|_F^2)$, decay as $O(T^{-1/2})$ with step size $\eta = 1/(K\sqrt{T})$. The experiments support the same story: on RoBERTa-base, classification accuracy is within a point or two of centralized LoRA, and on LLaMA-2-7B the gap is small on most benchmarks; 4-bit quantization barely changes accuracy; non-i.i.d. splits cause only a modest drop.

Load-bearing premise

The proof assumes every client's LoRA matrices stay bounded in norm by fixed constants $C_A, C_B$ for all rounds and local steps, and the algorithm never enforces this; if the factors drift as $T$ grows, the stated convergence-rate proof no longer applies.

Editorial extensions

If this is right

  • A language model can be fine-tuned on private data spread over many devices with no parameter server and no single point of failure; per-client communication stays at $O((d_1+d_2)r)$ LoRA parameters per round, independent of the number of clients.
  • The convergence rate $O(T^{-1/2})$ means that, up to constants, decentralized fine-tuning needs the same order of communication rounds as standard nonconvex stochastic optimization, so removing the server does not change the asymptotic convergence order.
  • The bound degrades as the mixing matrix's second eigenvalue approaches 1, so the theory predicts—and the paper's figures confirm—that sparser topologies converge slower than well-connected ones.
  • The paper reports that 4-bit quantized base models preserve almost the same Dec-LoRA accuracy as full precision, while non-i.i.d. data causes only a small drop when client counts are small.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The convergence guarantee is for stationarity at the averaged factors $(\bar B,\bar A)$, not at the average product $\frac{1}{n}\sum_i B_i A_i$; those two points need not coincide, so the theory leaves the product-average gap unquantified.
  • Assumption 4.3's uniform bounds on $\|A_i\|_F$ and $\|B_i\|_F$ are not enforced by the update rules, so a natural testable extension is to add norm clipping or projection after each aggregation and see whether the same $O(T^{-1/2})$ rate holds with explicit constants.
  • Because the deviation bounds in Lemma 4.7 grow like $(1-\beta^2)^{-2}$ and a ring's $\beta$ approaches 1 as $1-O(1/n^2)$, the theory implies that very large ring networks will need many more rounds; the paper's Table 5 shows exactly this qualitative pattern.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Dec-LoRA, a decentralized gossip-style algorithm for fine-tuning large language models with low-rank adapters. Clients perform K local SGD steps on their LoRA factors A and B, exchange the updated factors with neighbors, and aggregate them with a doubly stochastic mixing matrix. The authors introduce a factorized stationarity metric evaluated at the averages \bar{B} \bar{A}, prove an O(1/T^{1/2}) convergence rate to a stationary point under smoothness, bounded stochastic gradients, uniform boundedness of the LoRA iterates, and a spectral gap condition on the mixing matrix, and report experiments on RoBERTa-base and LLaMA-2-7B/13B and OPT-2.7B comparing Dec-LoRA with centralized LoRA and other PEFT baselines under different topologies, ranks, quantization, and data heterogeneity.

Significance. If the convergence claim and the empirical comparisons hold, Dec-LoRA would be a useful contribution: it is, to my knowledge, the first decentralized LoRA fine-tuning method with a non-convex convergence guarantee, and the paper includes a substantial experimental study across two model families, several GLUE/SuperGLUE tasks, ring and Erdős-Rényi topologies, quantization, and non-i.i.d. data. The explicit factorized stationarity metric is a reasonable response to the known difficulty of analyzing averages of products BA, and the appendix contains a detailed proof with clearly stated lemmas. The authors also acknowledge limitations of the method under data heterogeneity in Section F, which is helpful. The main weakness is that the central convergence theorem rests on an unverified uniform-boundedness assumption that the algorithm does not enforce; this must be fixed before the theoretical contribution can be accepted as stated.

major comments (4)
  1. [§4, Assumption 4.3 and Remark 4.4; Lemma E.6; Theorem 4.8] Assumption 4.3 is load-bearing but is not guaranteed by the algorithm. It is used exactly where the proof needs control of the LoRA chain-rule gradients: Lemma E.6 obtains LC_B^2- and LC_A^2-smoothness, Lemma E.8 bounds stacked gradients by C_B^2G^2 and C_A^2G^2, and the final bound in Theorem 4.8 contains terms L(C_A^4+C_B^4)G^2/T^{1/2}. Remark 4.4 argues that the assumption holds if all elements remain finite, but for each fixed T this only yields constants C_A(T), C_B(T) that may depend on T. The theorem requires T-independent constants; if the iterates grow with T, the term L(C_A^4+C_B^4)G^2/T^{1/2} need not vanish at the claimed rate. The proof provides no projection, regularization, or stepsize-dependent invariant that would enforce uniform boundedness. Thus the O(1/T^{1/2}) guarantee is conditional on an unproven trajectory property, not on Assumptions 4.1, 4.2, and 4.5 alone.
  2. [§5.2–5.3, Tables 1–3 and Figures 2–3] The empirical comparisons are reported as single-run best validation accuracies with no error bars, standard deviations, or number of seeds. The central empirical claim that Dec-LoRA achieves performance comparable to centralized LoRA is therefore not quantitatively supported; for example, Table 3 shows DROP accuracy differences of roughly 3–4 points (e.g., 48.32 vs. 44.35 at rank 2) with no indication of run-to-run variability. Given that the convergence theorem is also conditional, the experiments should include repeated runs or confidence intervals before the abstract's 'performance comparable to centralized LoRA' claim can be evaluated.
  3. [Appendix D.5 vs. Theorem 4.8] The validation of the local-update effect is internally inconsistent with the stated theorem. Appendix D.5 says 'the final convergence bound is proportional to 1/T^{1/3}', but Theorem 4.8 states an O(1/T^{1/2}) bound after setting eta = 1/(K T^{1/2}). The qualitative conclusion in Fig. 2(c) and (f) may still be correct, but the passage as written does not follow from the theory and needs correction.
  4. [Eq. (7) and Theorem 4.8] The stationarity statement is with respect to the factorized metric at \bar{B}^{(t)}\bar{A}^{(t)}, not with respect to the original objective in W. This is disclosed and is a reasonable choice given separate aggregation of A and B, but the abstract and introduction should state more prominently that 'converges to a stationary point' means a stationary point of the factorized metric, not necessarily of f(W0+BA) over the low-rank manifold. As written, the phrasing in the abstract and Section 1 could mislead readers into expecting a stronger guarantee.
minor comments (5)
  1. [Lemma E.7, Eq. (24)] The second displayed inequality in Lemma E.7 is labeled with E||∇A fi(BA) - ∇A fi(B'A')||^2 again, but from the proof and the following text it should be E||∇B fi(BA) - ∇B fi(B'A')||^2; please correct the statement.
  2. [Remark 4.4] The dimension notation in Remark 4.4 is inconsistent with the setup: it says A_i is r × k and B_i is d × r, while the paper defines A ∈ R^{r×d2} and B ∈ R^{d1×r}. Please harmonize the notation.
  3. [Section 5.2.2 / Appendix D.5] The statement in Section 5.2.2 that 'We validate these empirical results with our theoretical findings in Appendix D.5' is not fully accurate because the appendix discussion uses a different rate (1/T^{1/3}) than the theorem and only addresses monotonicity in β, not the precise finite-T behavior of the bound.
  4. [Appendix B.4] There is a duplicated citation marker in the text: '... converges to zero as T → ∞[30, 31]:[30, 31]:' appears twice; please remove the duplicate.
  5. [Table 4] The table entry for FedSA-LoRA says 'Week metric'; this should be 'Weak metric'.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: Dec-LoRA's convergence proof is self-contained; the non-standard stationarity metric and the conditional boundedness assumption are strength/correctness issues, not circularity.

full rationale

The central result, Theorem 4.8, is a conditional bound on the explicitly defined metric (7), and the proof derives it from Assumptions 4.1, 4.2, 4.3, and 4.5 via Lemmas E.6-E.11. The metric is defined independently of the algorithm's output and is not fitted to any data; no parameter is tuned on a subset and then 'predicted'. The self-citations [17,84] appear only in related-work and limitations contexts and are not used to justify the convergence claim. The main mathematical concern - Assumption 4.3's uniform boundedness is not enforced by the update rule, and Remark 4.4's finiteness argument does not supply a T-independent constant - is a correctness gap in the argument, not a circular reduction: the theorem does not assume its own conclusion, and the bound is not equal to an input by construction. The new stationarity metric is weaker than the standard full-gradient metric (6), which limits the strength of the claim, but it is not a renamed or fitted input. For these reasons the paper has no significant circularity, though the convergence guarantee is conditional on an unverified boundedness property.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The theoretical result rests on the four assumptions in Section 4 plus the newly introduced convergence metric. The free parameters listed are experimental hyperparameters chosen by hand. No new physical or mathematical entities are postulated.

free parameters (5)
  • learning rate eta = 1e-3 (BERT), 1e-4 (LLaMA); theory sets eta=1/(K sqrt(T))
    Chosen per model and not fitted to data, but the empirical results depend on it; theory specifies a value that depends on T and K.
  • LoRA rank r = 16 for main BERT results; 2,4,8 in ablations
    Hyperparameter affecting parameter count and accuracy; not derived from data.
  • local updates K = 1,5 for BERT; 3 for LLaMA
    Controls communication efficiency and affects convergence; chosen by the experimenter.
  • communication rounds T = 20,10,100 for BERT; 10 for LLaMA
    Determines the amount of training; results are reported after these round counts.
  • edge probability p_c (ER topology) = 0.2,0.6
    Changes network connectivity and hence convergence; chosen to show the effect.
assumptions (5)
  • domain assumption Each local objective f_i is L-smooth (Assumption 4.1).
    Standard smoothness used to bound gradient changes; invoked in Lemma E.6 and Theorem 4.8.
  • domain assumption Stochastic gradients are unbiased and uniformly bounded by G (Assumption 4.2).
    Standard bounded-variance assumption needed for the convergence proof.
  • ad hoc to paper A and B iterates are uniformly bounded by C_A and C_B for all clients, rounds, and local steps (Assumption 4.3).
    Necessary for smoothness with respect to A and B and for the rate; not proven by the algorithm and only hand-waved in Remark 4.4.
  • domain assumption The mixing matrix Q is symmetric, doubly stochastic, and has second-largest absolute eigenvalue beta < 1 (Assumption 4.5).
    Standard for consensus in decentralized optimization; used in Lemma E.9.
  • ad hoc to paper The factorized stationarity metric at \bar{B}\bar{A} is the accepted notion of convergence for the algorithm.
    The paper introduces this metric because the standard product-averaging metric is intractable; it is weaker than the standard metric and not part of the prior literature.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decentralized Low-Rank Fine-Tuning of Large Language Models." pith.science (2026). https://pith.science/paper/PI4S6HSV

@misc{pith2026250115361,
  author       = {Pith},
  title        = {Pith review of: Decentralized Low-Rank Fine-Tuning of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PI4S6HSV}},
  note         = {Machine review of arXiv:2501.15361}
}
read the original abstract

While parameter-efficient fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) offer computationally efficient adaptations of Large Language Models (LLMs), their practical deployment often assumes centralized data and training environments. However, real-world scenarios frequently involve distributed, privacy-sensitive datasets that require decentralized solutions. Federated learning (FL) addresses data privacy by coordinating model updates across clients, but it is typically based on centralized aggregation through a parameter server, which can introduce bottlenecks and communication constraints. Decentralized learning, in contrast, eliminates this dependency by enabling direct collaboration between clients, improving scalability and efficiency in distributed environments. Despite its advantages, decentralized LLM fine-tuning remains underexplored. In this work, we propose Dec-LoRA, a decentralized fine-tuning algorithm for LLMs based on LoRA. Through extensive experiments on BERT and LLaMA-2 models, we demonstrate that Dec-LoRA achieves performance comparable to centralized LoRA under various conditions, including data heterogeneity and quantization constraints. Additionally, we provide a rigorous theoretical guarantee proving the convergence of our algorithm to a stationary point for non-convex and smooth loss functions. These findings highlight the potential of Dec-LoRA for scalable LLM fine-tuning in decentralized environments.

Figures

Figures reproduced from arXiv: 2501.15361 by the authors.

Figure 1
Figure 1. (a): Illustration of the Dec-LoRA algorithm. The process includes three stages: (1) local training of low-rank matrices A and B on each client for K iterations using their private data, (2) communication of updated parameters between neighboring clients in the network, and (3) aggregation of received updates by each client using the mixing matrix Q to compute the next round’s parameters. (b) Convergence speed of dec… view at source ↗
Figure 2
Figure 2. (a) and (d): Effect of the number of clients on accuracy for the Ring topology with [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. A comparative analysis of centralized LoRA and [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DeCAF: Decentralized Consensus-And-Factorization for Low-Rank Adaptation of Foundation Models

    cs.LG 2025-05 reject novelty 6.0 of 10

    A truncated-SVD consensus step for decentralized LoRA is claimed to reach O(1/sqrt T) convergence, matching decentralized SGD, with supporting CLIP and LLAMA2-7B experiments.

Reference graph

Works this paper leans on

85 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar,et al., “Llama: Open and efficient foundation language models,”arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  3. [3]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,”arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

  4. [4]

    On the opportunities and risks of foundation models,

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill,et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  5. [5]

    Attention is all you need,

    A. Vaswani, “Attention is all you need,”Advances in Neural Information Processing Systems, 2017

    2017

  6. [6]

    Universal language model fine-tuning for text classification,

    J. Howard and S. Ruder, “Universal language model fine-tuning for text classification,”arXiv preprint arXiv:1801.06146, 2018

    arXiv 2018

  7. [7]

    Parameter-efficient transfer learning for nlp,

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly, “Parameter-efficient transfer learning for nlp,” inInternational conference on machine learning, pp. 2790–2799, PMLR, 2019

    2019

  8. [8]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, 2021

    2021

Show all 85 references
  1. [9]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,”arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  2. [10]

    Parameter-efficient fine-tuning of large-scale pre-trained language models,

    N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen,et al., “Parameter-efficient fine-tuning of large-scale pre-trained language models,”Nature Machine Intelligence, vol. 5, no. 3, pp. 220–235, 2023. 12

    2023

  3. [11]

    Federated multilingual models for medical transcript analysis,

    A. Manoel, M. d. C. H. Garcia, T. Baumel, S. Su, J. Chen, R. Sim, D. Miller, D. Karmon, and D. Dimitriadis, “Federated multilingual models for medical transcript analysis,” inConference on Health, Inference, and Learning, pp. 147–162, PMLR, 2023

    2023

  4. [12]

    Federated learning of medical concepts embedding using behrt,

    O. B. Shoham and N. Rappoport, “Federated learning of medical concepts embedding using behrt,” arXiv preprint arXiv:2305.13052, 2023

    2023 arXiv

  5. [13]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” inArtificial intelligence and statistics, pp. 1273–1282, PMLR, 2017

    2017

  6. [14]

    Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models,

    Z. Zhang, Y. Yang, Y. Dai, Q. Wang, Y. Yu, L. Qu, and Z. Xu, “Fedpetuning: When federated learning meets the parameter-efficient tuning methods of pre-trained language models,” inAnnual Meeting of the Association of Computational Linguistics 2023, pp. 9963–9977, Association fo...

    2023

  7. [15]

    Fate-llm: A industrial grade federated learning framework for large language models,

    T. Fan, Y. Kang, G. Ma, W. Chen, W. Wei, L. Fan, and Q. Yang, “Fate-llm: A industrial grade federated learning framework for large language models,”arXiv preprint arXiv:2310.10049, 2023

    2023 arXiv

  8. [16]

    Fedprompt: Communication-efficient and privacy- preserving prompt tuning in federated learning,

    H. Zhao, W. Du, F. Li, P. Li, and G. Liu, “Fedprompt: Communication-efficient and privacy- preserving prompt tuning in federated learning,” inICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1–5, IEEE, 2023

    2023

  9. [17]

    Communication- efficient and tensorized federated fine-tuning of large language models,

    S. Ghiasvand, Y. Yang, Z. Xue, M. Alizadeh, Z. Zhang, and R. Pedarsani, “Communication- efficient and tensorized federated fine-tuning of large language models,” arXiv preprint arXiv:2410.13097, 2024

    2024 arXiv

  10. [18]

    SLoRA: federated parameter efficient fine-tuning of language models,

    S. Babakniya, A. R. Elkordy, Y. H. Ezzeldin, Q. Liu, K.-B. Song, M. El-Khamy, and S. Aves- timehr, “SLoRA: federated parameter efficient fine-tuning of language models,”arXiv preprint arXiv:2308.06522, 2023

    2023 arXiv

  11. [19]

    FeDeRA: efficient fine-tuning of language models in federated learning leveraging weight decomposition,

    Y. Yan, S. Tang, Z. Shi, and Q. Yang, “FeDeRA: efficient fine-tuning of language models in federated learning leveraging weight decomposition,”arXiv preprint arXiv:2404.18848, 2024

    2024 arXiv

  12. [20]

    Heterogeneous lora for federated fine-tuning of on-device foundation models,

    Y. J. Cho, L. Liu, Z. Xu, A. Fahrezi, M. Barnes, and G. Joshi, “Heterogeneous lora for federated fine-tuning of on-device foundation models,” inInternational Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, 2023

    2023

  13. [21]

    Federated fine-tuning of large language models under heterogeneous language tasks and client resources,

    J. Bai, D. Chen, B. Qian, L. Yao, and Y. Li, “Federated fine-tuning of large language models under heterogeneous language tasks and client resources,”arXiv preprint arXiv:2402.11505, 2024

    2024 arXiv

  14. [22]

    Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations,

    Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li, “Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations,”arXiv preprint arXiv:2409.05976, 2024

    2024 arXiv

  15. [23]

    Federated lora with sparse communication,

    K. Kuo, A. Raje, K. Rajesh, and V. Smith, “Federated lora with sparse communication,”arXiv preprint arXiv:2406.05233, 2024

    2024 arXiv

  16. [24]

    Improving LoRA in privacy-preserving federated learning,

    Y. Sun, Z. Li, Y. Li, and B. Ding, “Improving LoRA in privacy-preserving federated learning,” arXiv preprint arXiv:2403.12313, 2024. 13

    2024 arXiv

  17. [25]

    Robust federated finetuning of foundation models via alternating minimization of LoRA,

    S. Chen, Y. Ju, H. Dalal, Z. Zhu, and A. J. Khisti, “Robust federated finetuning of foundation models via alternating minimization of LoRA,” inWorkshop on Efficient Systems for Foundation Models II@ ICML2024

  18. [26]

    Decentralized federated learning: A survey and perspective,

    L. Yuan, Z. Wang, L. Sun, S. Y. Philip, and C. G. Brinton, “Decentralized federated learning: A survey and perspective,”IEEE Internet of Things Journal, 2024

    2024

  19. [27]

    Large language model based multi-agents: A survey of progress and challenges,

    T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: A survey of progress and challenges,”arXiv preprint arXiv:2402.01680, 2024

    2024 arXiv

  20. [28]

    Scalable multi-robot collaboration with large language models: Centralized or decentralized systems?,

    Y. Chen, J. Arkin, Y. Zhang, N. Roy, and C. Fan, “Scalable multi-robot collaboration with large language models: Centralized or decentralized systems?,” in2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 4311–4317, IEEE, 2024

    2024

  21. [29]

    Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,

    W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou, “Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning,” inProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mi...

    2024

  22. [30]

    Personalized federated learning: A meta-learning approach,

    A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning: A meta-learning approach,” arXiv preprint arXiv:2002.07948, 2020

    2002 arXiv

  23. [31]

    Tackling the objective inconsistency problem in heterogeneous federated optimization,

    J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,”Advances in neural information processing systems, vol. 33, pp. 7611–7623, 2020

    2020

  24. [32]

    Accelerating gossip sgd with periodic global averaging,

    Y. Chen, K. Yuan, Y. Zhang, P. Pan, Y. Xu, and W. Yin, “Accelerating gossip sgd with periodic global averaging,” inInternational Conference on Machine Learning, pp. 1791–1802, PMLR, 2021

    2021

  25. [33]

    Roberta: A robustly optimized bert pretraining approach,

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,”arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  26. [34]

    Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,

    E. B. Zaken, S. Ravfogel, and Y. Goldberg, “Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models,”arXiv preprint arXiv:2106.10199, 2021

    2021

  27. [35]

    Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,

    H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel, “Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 1950–1965, 2022

    1950

  28. [36]

    Peft: State-of-the-art parameter-efficient fine-tuning methods

    S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan, “Peft: State-of-the-art parameter-efficient fine-tuning methods.”https://github.com/huggingface/peft, 2022

    2022

  29. [37]

    Glue: A multi- task benchmark and analysis platform for natural language understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “Glue: A multi- task benchmark and analysis platform for natural language understanding,”arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  30. [38]

    Qlora: Efficient finetuning of quantized llms,

    T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, “Qlora: Efficient finetuning of quantized llms,”Advances in Neural Information Processing Systems, vol. 36, 2024. 14

    2024

  31. [39]

    The non-iid data quagmire of decentralized machine learning,

    K. Hsieh, A. Phanishayee, O. Mutlu, and P. Gibbons, “The non-iid data quagmire of decentralized machine learning,” inInternational Conference on Machine Learning, pp. 4387–4398, PMLR, 2020

    2020

  32. [40]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,”Proceedings of Machine learning and systems, vol. 2, pp. 429–450, 2020

    2020

  33. [41]

    Superglue: Astickierbenchmarkforgeneral-purposelanguageunderstandingsystems,

    A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “Superglue: Astickierbenchmarkforgeneral-purposelanguageunderstandingsystems,” Advances in neural information processing systems, vol. 32, 2019

    2019

  34. [42]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “Squad: 100,000+ questions for machine comprehension of text,”arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  35. [43]

    Drop: A read- ing comprehension benchmark requiring discrete reasoning over paragraphs,

    D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner, “Drop: A read- ing comprehension benchmark requiring discrete reasoning over paragraphs,”arXiv preprint arXiv:1903.00161, 2019

    1903 arXiv

  36. [44]

    Opt: Open pre-trained transformer language models,

    S.Zhang, S.Roller, N.Goyal, M.Artetxe, M.Chen, S.Chen, C.Dewan, M.Diab, X.Li, X.V.Lin, et al., “Opt: Open pre-trained transformer language models,”arXiv preprint arXiv:2205.01068, 2022

    2022 arXiv

  37. [45]

    Parameter-efficient fine-tuning for large models: A comprehensive survey,

    Z. Han, C. Gao, J. Liu, S. Q. Zhang,et al., “Parameter-efficient fine-tuning for large models: A comprehensive survey,”arXiv preprint arXiv:2403.14608, 2024

    2024 arXiv

  38. [46]

    Towards a unified view of parameter-efficient transfer learning,

    J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, “Towards a unified view of parameter-efficient transfer learning,”arXiv preprint arXiv:2110.04366, 2021

    2021 arXiv

  39. [47]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,”arXiv preprint arXiv:2101.00190, 2021

    2021 arXiv

  40. [48]

    Parameter-efficient fine-tuning without introducing new latency,

    B. Liao, Y. Meng, and C. Monz, “Parameter-efficient fine-tuning without introducing new latency,” arXiv preprint arXiv:2305.16742, 2023

    2023 arXiv

  41. [49]

    Dora: Weight-decomposed low-rank adaptation,

    S.-Y. Liu, C.-Y. Wang, H. Yin, P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen, “Dora: Weight-decomposed low-rank adaptation,”arXiv preprint arXiv:2402.09353, 2024

    2024 arXiv

  42. [50]

    J. N. Tsitsiklis, Problems in decentralized decision making and computation. PhD thesis, Massachusetts Institute of Technology, 1984

    1984

  43. [51]

    Gossip-based computation of aggregate information,

    D. Kempe, A. Dobra, and J. Gehrke, “Gossip-based computation of aggregate information,” in 44th Annual IEEE Symposium on Foundations of Computer Science, 2003. Proceedings., pp. 482–491, IEEE, 2003

    2003

  44. [52]

    Randomized gossip algorithms,

    S. Boyd, A. Ghosh, B. Prabhakar, and D. Shah, “Randomized gossip algorithms,” IEEE transactions on information theory, vol. 52, no. 6, pp. 2508–2530, 2006

    2006

  45. [53]

    Distributed subgradient methods for multi-agent optimization,

    A. Nedic and A. Ozdaglar, “Distributed subgradient methods for multi-agent optimization,” IEEE Transactions on Automatic Control, vol. 54, no. 1, pp. 48–61, 2009

    2009

  46. [54]

    A randomized incremental subgradient method for distributed optimization in networked systems,

    B. Johansson, M. Rabi, and M. Johansson, “A randomized incremental subgradient method for distributed optimization in networked systems,”SIAM Journal on Optimization, vol. 20, no. 3, pp. 1157–1170, 2010. 15

    2010

  47. [55]

    Federated learning for connected and automated vehicles: A survey of existing approaches and challenges,

    V. P. Chellapandi, L. Yuan, C. G. Brinton, S. H. Żak, and Z. Wang, “Federated learning for connected and automated vehicles: A survey of existing approaches and challenges,”IEEE Transactions on Intelligent Vehicles, 2023

    2023

  48. [56]

    Swarm learning for decentralized and confidential clinical machine learning,

    S. Warnat-Herresthal, H. Schultze, K. L. Shastry, S. Manamohan, S. Mukherjee, V. Garg, R. Sarveswara, K. Händler, P. Pickkers, N. A. Aziz,et al., “Swarm learning for decentralized and confidential clinical machine learning,”Nature, vol. 594, no. 7862, pp. 265–270, 2021

    2021

  49. [57]

    Decentralized federated learning for industrial iot with deep echo state networks,

    W. Qiu, W. Ai, H. Chen, Q. Feng, and G. Tang, “Decentralized federated learning for industrial iot with deep echo state networks,”IEEE Transactions on Industrial Informatics, vol. 19, no. 4, pp. 5849–5857, 2022

    2022

  50. [58]

    Iot with blockchain: A new infrastructure proposal,

    H. Hexmoor and E. Maghsoudlou, “Iot with blockchain: A new infrastructure proposal,” Proceedings of 39th International Confer, vol. 98, pp. 15–24, 2024

    2024

  51. [59]

    Network anomaly detection for iot using hyperdimensional computing on nsl-kdd,

    G. Ghajari, A. Ghimire, E. Ghajari, and F. Amsaad, “Network anomaly detection for iot using hyperdimensional computing on nsl-kdd,”arXiv preprint arXiv:2503.03031, 2025

    2025

  52. [60]

    Spreadgnn: Decentralized multi-task federated learning for graph neural networks on molecular data,

    C. He, E. Ceyani, K. Balasubramanian, M. Annavaram, and S. Avestimehr, “Spreadgnn: Decentralized multi-task federated learning for graph neural networks on molecular data,” in Proceedings of the AAAI conference on artificial intelligence, vol. 36, pp. 6865–6873, 2022

    2022

  53. [61]

    Lora training in the ntk regime has no spurious local minima,

    U. Jang, J. D. Lee, and E. K. Ryu, “Lora training in the ntk regime has no spurious local minima,” in Proceedings of the 41st International Conference on Machine Learning, pp. 21306–21328, 2024

    2024

  54. [62]

    Neural tangent kernel: Convergence and generalization in neural networks,

    A. Jacot, F. Gabriel, and C. Hongler, “Neural tangent kernel: Convergence and generalization in neural networks,”Advances in neural information processing systems, vol. 31, 2018

    2018

  55. [63]

    Asymmetry in low-rank adapters of foundation models,

    J. Zhu, K. Greenewald, K. Nadjahi, H. S. de Ocáriz Borde, R. B. Gabrielsson, L. Choshen, M. Ghassemi, M. Yurochkin, and J. Solomon, “Asymmetry in low-rank adapters of foundation models,” in Forty-first International Conference on Machine Learning

  56. [64]

    Gradient dynamics for low-rank fine-tuning beyond kernels,

    A. K. Dayi and S. Chen, “Gradient dynamics for low-rank fine-tuning beyond kernels,”arXiv preprint arXiv:2411.15385, 2024

    2024 arXiv

  57. [65]

    Implicit balancing and regularization: Generalization and convergence guarantees for overparameterized asymmetric matrix sensing,

    M. Soltanolkotabi, D. Stöger, and C. Xie, “Implicit balancing and regularization: Generalization and convergence guarantees for overparameterized asymmetric matrix sensing,” inThe Thirty Sixth Annual Conference on Learning Theory, pp. 5140–5142, PMLR, 2023

    2023

  58. [66]

    On the crucial role of initialization for matrix factorization,

    B. Li, L. Zhang, A. Mokhtari, and N. He, “On the crucial role of initialization for matrix factorization,” in OPT 2024: Optimization for Machine Learning

    2024

  59. [67]

    One-step full gradient suffices for low-rank fine-tuning, provably and efficiently,

    Y. Zhang, F. Liu, and Y. Chen, “One-step full gradient suffices for low-rank fine-tuning, provably and efficiently,”arXiv preprint arXiv:2502.01235, 2025

    2025 arXiv

  60. [68]

    Randomized asymmetric chain of lora: The first meaningful theoretical framework for low-rank adaptation,

    G. Malinovsky, U. Michieli, H. A. A. K. Hammoud, T. Ceritli, H. Elesedy, M. Ozay, and P. Richtárik, “Randomized asymmetric chain of lora: The first meaningful theoretical framework for low-rank adaptation,”arXiv preprint arXiv:2410.08305, 2024

    arXiv

  61. [69]

    Selective aggregation for low-rank adaptation in federated learning,

    P. Guo, S. Zeng, Y. Wang, H. Fan, F. Wang, and L. Qu, “Selective aggregation for low-rank adaptation in federated learning,”CoRR, 2024. 16

    2024

  62. [70]

    Federated residual low-rank adaptation of large language models,

    Y. Yan, C.-M. Feng, W. Zuo, R. S. M. Goh, Y. Liu, and L. Zhu, “Federated residual low-rank adaptation of large language models,” inThe Thirteenth International Conference on Learning Representations

  63. [71]

    FedEx-LoRA: Exact aggregation for federated and efficient fine-tuning of large language models,

    R. Singhal, K. Ponkshe, and P. Vepakomma, “FedEx-LoRA: Exact aggregation for federated and efficient fine-tuning of large language models,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)(W. Che, J. Nabende, E. ...

    2025

  64. [72]

    Towards robust and efficient federated low-rank adaptation with heterogeneous clients,

    J. Koo, M. Jang, and J. Ok, “Towards robust and efficient federated low-rank adaptation with heterogeneous clients,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)(W. Che, J. Nabende, E. Shutova, and M. T. Pile...

    2025

  65. [73]

    Fed-SB: A silver bullet for extreme communication efficiency and performance in (private) federated loRA fine-tuning,

    R. Singhal, K. Ponkshe, R. Vartak, L. R. Varshney, and P. Vepakomma, “Fed-SB: A silver bullet for extreme communication efficiency and performance in (private) federated loRA fine-tuning,” in ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, 2025

    2025

  66. [74]

    A unified theory of decentralized sgd with changing topology and local updates,

    A. Koloskova, N. Loizou, S. Boreiri, M. Jaggi, and S. Stich, “A unified theory of decentralized sgd with changing topology and local updates,” inInternational conference on machine learning, pp. 5381–5393, PMLR, 2020

    2020

  67. [75]

    Fast linear iterations for distributed averaging,

    L. Xiao and S. Boyd, “Fast linear iterations for distributed averaging,”Systems & Control Letters, vol. 53, no. 1, pp. 65–78, 2004

    2004

  68. [76]

    The largest eigenvalue of sparse random graphs,

    M. Krivelevich and B. Sudakov, “The largest eigenvalue of sparse random graphs,”Combinatorics, Probability and Computing, vol. 12, no. 1, pp. 61–72, 2003

    2003

  69. [77]

    Recursive deep models for semantic compositionality over a sentiment treebank,

    R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” inProceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642, 2013

    2013

  70. [78]

    The pascal recognising textual entailment challenge,

    I. Dagan, O. Glickman, and B. Magnini, “The pascal recognising textual entailment challenge,” in Machine learning challenges workshop, pp. 177–190, Springer, 2005

    2005

  71. [79]

    A broad-coverage challenge corpus for sentence understanding through inference,

    A. Williams, N. Nangia, and S. R. Bowman, “A broad-coverage challenge corpus for sentence understanding through inference,”arXiv preprint arXiv:1704.05426, 2017

    2017 arXiv

  72. [80]

    Know what you don’t know: Unanswerable questions for squad,

    P. Rajpurkar, R. Jia, and P. Liang, “Know what you don’t know: Unanswerable questions for squad,” arXiv preprint arXiv:1806.03822, 2018

    2018 arXiv

  73. [81]

    Fine-tuning language models with just forward passes,

    S. Malladi, T. Gao, E. Nichani, A. Damian, J. D. Lee, D. Chen, and S. Arora, “Fine-tuning language models with just forward passes,”arXiv preprint arXiv:2305.17333, 2023

    2023 arXiv

  74. [82]

    On the convergence of decentralized gradient descent,

    K. Yuan, Q. Ling, and W. Yin, “On the convergence of decentralized gradient descent,”SIAM Journal on Optimization, vol. 26, no. 3, pp. 1835–1854, 2016

    2016

  75. [83]

    Decentralized gradient tracking with local steps,

    Y. Liu, T. Lin, A. Koloskova, and S. U. Stich, “Decentralized gradient tracking with local steps,” Optimization Methods and Software, pp. 1–28, 2024

    2024

  76. [84]

    Robust decentralized learning with local updates and gradient tracking,

    S. Ghiasvand, A. Reisizadeh, M. Alizadeh, and R. Pedarsani, “Robust decentralized learning with local updates and gradient tracking,”IEEE Transactions on Networking, 2025. 17 Algorithm 1 Dec-LoRA 1: Input: Step size η, number of communication rounds T , number of local updates...

    2025

  77. [85]

    β ≈ 1 − Θ( 1√n ), when pc ≫ log n n . D.5 Impact of Number of Clients, Edge Probabilities, and Number of Local Updates In this section, we validate the empirical results presented in Section 5.2.2 using our theoretical findings. First, note that a smaller value ofβ ∈ (0, 1) in...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.