REVIEW 7 cited by
FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
FedPara: Low-Rank Hadamard Product for Communication-Efficient Federated Learning
read the original abstract
In this work, we propose a communication-efficient parameterization, FedPara, for federated learning (FL) to overcome the burdens on frequent model uploads and downloads. Our method re-parameterizes weight parameters of layers using low-rank weights followed by the Hadamard product. Compared to the conventional low-rank parameterization, our FedPara method is not restricted to low-rank constraints, and thereby it has a far larger capacity. This property enables to achieve comparable performance while requiring 3 to 10 times lower communication costs than the model with the original layers, which is not achievable by the traditional low-rank methods. The efficiency of our method can be further improved by combining with other efficient FL optimizers. In addition, we extend our method to a personalized FL application, pFedPara, which separates parameters into global and local ones. We show that pFedPara outperforms competing personalized FL methods with more than three times fewer parameters.
Forward citations
Cited by 7 Pith papers
-
Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs
UQ4CT integrates functional-level uncertainty calibration into mixture-of-experts LoRA fine-tuning via a dedicated loss, cutting expected calibration error by over 25% on multiple-choice and generative QA tasks.
-
Manifold-based Algorithms for the Hadamard Decomposition
The paper introduces manifold-based algorithms and initializations for Hadamard decomposition, reformulating it as a low-rank factorization on manifolds and demonstrating efficiency on synthetic and real data.
-
Q-LocalAdam: Memory-Efficient Client-Side Adaptive Optimization for Edge Federated Learning
Q-LocalAdam reduces optimizer memory by 3.37x via tailored 8-bit quantization for Adam states while maintaining or improving accuracy under data heterogeneity in edge federated learning.
-
Beyond Feature Fusion: Contextual Bayesian PEFT for Multimodal Uncertainty Estimation
CoCo-LoRA uses audio context to modulate uncertainty in Bayesian low-rank adapters for multimodal text tasks, offering a lightweight alternative to feature fusion that matches or exceeds baselines.
-
Exploiting Correlations in Federated Learning: Opportunities and Practical Limitations
A correlation-based taxonomy unifies existing FL compression methods, experiments show correlation strengths vary by task and architecture, and adaptive mode-switching designs are proposed to exploit this.
-
Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
LoRA-Over injects auxiliary parameters into low-rank adapters during training and decomposes them back into standard LoRA at inference, with static or dynamic scheduling to allocate extra capacity where needed, yieldi...
-
CoLLM: Continuous Adaptation for SLO-Aware LLM Serving on Shared GPU Clusters
CoLLM unifies FL PEFT and inference on shared edge replicas via intra-replica model sharing and two-timescale inter-replica coordination, achieving up to 3x higher goodput than prior LLM systems.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.