REVIEW 10 cited by
FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning
read the original abstract
Large language models (LLMs) have emerged as important components across various fields, yet their training requires substantial computation resources and abundant labeled data. It poses a challenge to robustly training LLMs for individual users (clients). To tackle this challenge, the intuitive idea is to introduce federated learning (FL), which can collaboratively train models on distributed private data. However, existing methods suffer from the challenges of data heterogeneity, system heterogeneity, and model size, resulting in suboptimal performance and high costs. In this work, we proposed a variant of personalized federated learning (PFL) framework, namely FDLoRA, which allows the client to be a single device or a cluster and adopts low-rank adaptation (LoRA) tuning. FDLoRA sets dual LoRA modules on each client to capture personalized and global knowledge, respectively, and only the global LoRA module uploads parameters to the central server to aggregate cross-client knowledge. Finally, an adaptive fusion approach is employed to combine the parameters of the dual LoRAs. This enables FDLoRA to make effective use of private data distributed across different clients, thereby improving performance on the client without incurring high communication and computing costs. We conducted extensive experiments in two practice scenarios. The results demonstrate that FDLoRA outperforms six baselines in terms of performance, stability, robustness, computation cost, and communication cost.
Forward citations
Cited by 10 Pith papers
-
Test-Time Personalization: A Diagnostic Framework and Probabilistic Fix for Scaling Failures
Test-time scaling for personalized LLMs follows a logarithmic utility curve under oracle selection but standard reward models suffer user-level collapse and query-level hacking; a probabilistic reward model with learn...
-
Toward Efficient Membership Inference Attacks against Federated Large Language Models: A Projection Residual Approach
ProjRes achieves near-100% accuracy in membership inference on FedLLMs by measuring projection residuals of hidden embeddings on gradient subspaces, outperforming prior methods by up to 75.75% even under differential privacy.
-
Rethinking LoRA for Privacy-Preserving Federated Learning in Large Models
LA-LoRA decouples LoRA matrix updates in DPFL settings to improve robustness to privacy noise, delivering up to 16.83% higher accuracy than prior LoRA variants on Swin-B under strict epsilon=1.
-
SecureGate: Learning When to Reveal PII Safely via Token-Gated Dual-Adapters for Federated LLMs
SecureGate reduces PII leakage up to 31.66X in federated LLM fine-tuning via token-gated dual LoRA adapters while preserving utility and achieving perfect routing reliability.
-
Safe-FedLLM: Delving into the Safety of Federated Large Language Models
Safe-FedLLM shows that a lightweight logistic-regression probe on LoRA B-matrix deltas can detect malicious federated clients, and step/client/shadow defense levels restore most of the safety lost to data poisoning.
-
Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning
Dysco reduces cross-client interference in federated LoRA fine-tuning by fixing each client's update subspace to directions insensitive to other clients' activations, yielding tighter bounds and better benchmark accuracy.
-
DECA: Decentralizing Block-Wise Adam for Efficient LLM Full-Parameter Fine-Tuning on Non-IID Data
DECA partitions LLM parameters into blocks for sequential block-wise Adam optimization in decentralized non-IID settings to support efficient full-parameter fine-tuning.
-
FedSpy-LLM: Towards Scalable and Generalizable Data Reconstruction Attacks from Gradients on LLMs
FedSpy-LLM uses gradient decomposition and iterative alignment to reconstruct larger batches and longer sequences of training data from LLM gradients in federated settings, including with PEFT methods.
-
Safe-FedLLM: Delving into the Safety of Federated Large Language Models
Safe-FedLLM detects malicious client LoRA updates in federated LLM training via step-, client-, and shadow-level probes with lightweight classifiers, improving robustness while preserving benign performance.
-
FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints
FediLoRA is a lightweight federated LoRA aggregation method that jointly mitigates missing modalities and heterogeneous ranks in collaborative fine-tuning of foundation models.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.