Pith. sign in

REVIEW 18 cited by

SLoRA: Federated Parameter Efficient Fine-Tuning of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2308.06522 v1 pith:OIPV75YX submitted 2023-08-12 cs.LG cs.AI

Sara Babakniya , Ahmed Roushdy Elkordy , Yahya H. Ezzeldin , Qingfeng Liu , Kee-Bong Song , Mostafa El-Khamy , Salman Avestimehr This is my paper · ORCID
classification cs.LGcs.AI
keywords fine-tuningdataefficientfederatedmodelssloraacrossedge
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Transfer learning via fine-tuning pre-trained transformer models has gained significant success in delivering state-of-the-art results across various NLP tasks. In the absence of centralized data, Federated Learning (FL) can benefit from distributed and private data of the FL edge clients for fine-tuning. However, due to the limited communication, computation, and storage capabilities of edge devices and the huge sizes of popular transformer models, efficient fine-tuning is crucial to make federated training feasible. This work explores the opportunities and challenges associated with applying parameter efficient fine-tuning (PEFT) methods in different FL settings for language tasks. Specifically, our investigation reveals that as the data across users becomes more diverse, the gap between fully fine-tuning the model and employing PEFT methods widens. To bridge this performance gap, we propose a method called SLoRA, which overcomes the key limitations of LoRA in high heterogeneous data scenarios through a novel data-driven initialization technique. Our experimental results demonstrate that SLoRA achieves performance comparable to full fine-tuning, with significant sparse updates with approximately $\sim 1\%$ density while reducing training time by up to $90\%$.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Personalized Federated Learning via Variance-Aware Nonparametric Empirical Bayes

    stat.ML 2026-08 conditional novelty 7.0 of 10

    VANEB generalizes nonparametric empirical Bayes to parameter-dependent noise and uses it to personalize federated models by shrinking local estimates toward a learned population prior.

  2. FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A federated LoRA fine-tuning method that builds a client-similarity tree and adapts aggregation depth layer-by-layer outperforms flat or global aggregation baselines on NLU and NLG tasks.

  3. From LLMs to Edge: Parameter-Efficient Fine-Tuning on Edge Devices

    cs.LG 2025-07 conditional novelty 6.0 of 10

    On depthwise-separable CNNs, PEFT memory savings drop to about half of LLM levels, though LoRA and DoRA still cut update FLOPs by up to 95%.

  4. Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix

    cs.CR 2025-07 reject novelty 6.0 of 10

    FedASK uses double sketching and server-side SVD to aggregate federated LoRA updates under differential privacy, allowing both A and B matrices to be updated instead of freezing one.

  5. ConsNoTrainLoRA: Data-driven Weight Initialization of Low-rank Adapters using Constraints

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LoRA adapters can be initialized with a closed-form estimate derived from constraint sets linking source and target activations, improving fine-tuning speed and accuracy.

  6. FedRPCA: Enhancing Federated LoRA Aggregation Using Robust PCA

    cs.LG 2025-06 conditional novelty 6.0 of 10

    FedRPCA decomposes federated LoRA client updates with Robust PCA into common and client-specific components, averaging the common part and scaled-averaging the sparse part, which improves accuracy and convergence over...

  7. Assortment of Attention Heads: Accelerating Federated PEFT with Head Pruning and Strategic Client Selection

    cs.CL 2025-05 conditional novelty 6.0 of 10

    A federated fine-tuning method prunes 90% of attention heads, weights updates by attention importance, and selects clients by loss gap, cutting communication 1.8x and training compute 3.9x with under 2% accuracy drop.

  8. Decentralized Low-Rank Fine-Tuning of Large Language Models

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Decentralized LoRA fine-tuning with gossip averaging converges to a stationary point at O(1/sqrt(T)) and matches centralized LoRA accuracy on several NLP tasks.

  9. HeteroTune: Efficient Federated Learning for Large Heterogeneous Models

    cs.LG 2024-11 reject novelty 6.0 of 10

    HeteroTune is a federated fine-tuning method that combines local and shared adapters with gradient whitening to aggregate updates from heterogeneous model sizes.

  10. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0 of 10

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

  11. ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models

    cs.LG 2025-11 conditional novelty 5.0 of 10

    ParaBlock hides communication latency in federated block-coordinate LLM fine-tuning by running last round's upload/download in parallel with current computation, preserving the O(1/√T) convergence rate.

  12. A Comprehensive Data-centric Overview of Federated Graph Learning

    cs.LG 2025-07 conditional novelty 5.0 of 10

    A data-centric taxonomy for Federated Graph Learning that classifies 79 studies by data characteristics and data utilization, plus a discussion of integration with pre-trained large models.

  13. Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning with Heterogeneous LoRA Allocation

    cs.LG 2025-06 conditional novelty 5.0 of 10

    Fed-HeLLo allocates different LoRA layers to clients of different resource levels using importance scores and geometric patterns, improving federated fine-tuning accuracy over random allocation baselines.

  14. AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption

    cs.LG 2025-05 conditional novelty 4.0 of 10

    AFLoRA prunes low-rank adapter dimensions per client, trains only the client-specific part locally and the shared part on the server, and aggregates heterogeneous updates by zero-padding and rank-aware weighting.

  15. Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression

    cs.LG 2025-05 conditional novelty 4.0 of 10

    ACBench tests compressed LLMs on agentic tasks and finds 4-bit quantization keeps tool use and workflow generation strong while hurting real-world application performance.

  16. Federated Adapter on Foundation Models: An Out-Of-Distribution Approach

    cs.LG 2025-05 reject novelty 4.0 of 10

    FedOA regularizes personalized adapters toward the global model in feature space to improve OOD generalization in federated foundation models, but the proof is incomplete and the empirical gains are modest.

  17. A Survey on Parameter-Efficient Fine-Tuning for Foundation Models in Federated Learning

    cs.LG 2025-04 conditional novelty 3.0 of 10

    A review that sorts recent federated-learning PEFT approaches into additive, selective, and reparameterized (LoRA-style) families and maps them onto NLP and vision applications.

  18. Federated Continual Learning for Edge-AI: A Comprehensive Survey

    cs.LG 2024-11 conditional novelty 3.0 of 10

    A survey that organizes federated continual learning methods into class, domain, and task continual learning categories for edge-AI applications.

Pith tools