Pith. sign in

REVIEW 10 cited by

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2409.05976 v1 pith:O5AR2KQP submitted 2024-09-09 cs.LG cs.DC

FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations

Ziyao Wang , Zheyu Shen , Yexiao He , Guoheng Sun , Hongyi Wang , Lingjuan Lyu , Ang Li This is my paper
classification cs.LG cs.DC
keywords fine-tuningfederatedheterogeneousloraaggregationllmsadaptersclients
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The rapid development of Large Language Models (LLMs) has been pivotal in advancing AI, with pre-trained LLMs being adaptable to diverse downstream tasks through fine-tuning. Federated learning (FL) further enhances fine-tuning in a privacy-aware manner by utilizing clients' local data through in-situ computation, eliminating the need for data movement. However, fine-tuning LLMs, given their massive scale of parameters, poses challenges for clients with constrained and heterogeneous resources in FL. Previous methods employed low-rank adaptation (LoRA) for efficient federated fine-tuning but utilized traditional FL aggregation strategies on LoRA adapters. These approaches led to mathematically inaccurate aggregation noise, reducing fine-tuning effectiveness and failing to address heterogeneous LoRAs. In this work, we first highlight the mathematical incorrectness of LoRA aggregation in existing federated fine-tuning methods. We introduce a new approach called FLORA that enables federated fine-tuning on heterogeneous LoRA adapters across clients through a novel stacking-based aggregation method. Our approach is noise-free and seamlessly supports heterogeneous LoRA adapters. Extensive experiments demonstrate FLORA' s superior performance in both homogeneous and heterogeneous settings, surpassing state-of-the-art methods. We envision this work as a milestone for efficient, privacy-preserving, and accurate federated fine-tuning of LLMs. Our code is available at https://github.com/ATP-1010/FederatedLLM.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Beyond End-to-End: Dynamic Chain Optimization for Private LLM Adaptation on the Edge

    cs.DC 2026-04 unverdicted novelty 6.0

    ChainFed achieves memory-efficient private LLM fine-tuning on edge devices through sequential layer-by-layer adapter training with dynamic co-tuning, perceptive optimization, and adaptive starting point selection, imp...

  2. FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

    cs.LG 2026-02 conditional novelty 6.0

    A federated LoRA fine-tuning method that builds a client-similarity tree and adapts aggregation depth layer-by-layer outperforms flat or global aggregation baselines on NLU and NLG tasks.

  3. Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models

    cs.LG 2026-02 unverdicted novelty 6.0

    LA-LoRA decouples LoRA matrix updates in DPFL settings to improve robustness to privacy noise, delivering up to 16.83% higher accuracy than prior LoRA variants on Swin-B under strict epsilon=1.

  4. Task-agnostic Low-rank Residual Adaptation for Efficient Federated Continual Fine-Tuning

    cs.LG 2025-05 unverdicted novelty 6.0

    Fed-TaLoRA uses task-agnostic low-rank residual adaptation with post-aggregation calibration to enable efficient federated continual fine-tuning across sequential tasks under non-IID conditions.

  5. LAARA: Layer-Aware Adaptive Rank Allocation for Parameter-Efficient Fine-Tuning

    cs.LG 2026-07 conditional novelty 5.0

    LAARA allocates LoRA ranks per layer from diagonal Fisher (gradient-based) estimates, reporting improved accuracy with fewer trainable parameters on GLUE and MathInstruct.

  6. Amortizing Federated Adaptation: Hypernetwork Driven LoRA for Personalized Foundation Models

    cs.AI 2026-06 unverdicted novelty 5.0

    HyperLoRA amortizes federated LoRA adaptation via hypernetwork-generated initializations and product-space aggregation to fix structural bias and initialization lag.

  7. ParaBlock: Communication-Computation Parallel Block Coordinate Federated Learning for Large Language Models

    cs.LG 2025-11 conditional novelty 5.0

    ParaBlock hides communication latency in federated block-coordinate LLM fine-tuning by running last round's upload/download in parallel with current computation, preserving the O(1/√T) convergence rate.

  8. FedShield-LLM: A Secure and Scalable Federated Fine-Tuned Large Language Model

    cs.CR 2025-06 unverdicted novelty 5.0

    FedShield-LLM integrates pruning and FHE on LoRA parameters to support secure, scalable federated fine-tuning of LLMs such as Llama-2.

  9. FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints

    cs.LG 2025-09 unverdicted novelty 4.0

    FediLoRA is a lightweight federated LoRA aggregation method that jointly mitigates missing modalities and heterogeneous ranks in collaborative fine-tuning of foundation models.

  10. DP-FedLoRA: Privacy-Enhanced Federated Fine-Tuning for On-Device Large Language Models

    cs.CR 2025-09 reject novelty 3.0

    DP-FedLoRA clips and adds Gaussian noise to per-client LoRA matrices in federated LLM fine-tuning, claiming unbiased updates and bounded variance, but the privacy calibration and experiments have significant gaps.