REVIEW 8 cited by
Distilled One-Shot Federated Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Distilled One-Shot Federated Learning
read the original abstract
Current federated learning algorithms take tens of communication rounds transmitting unwieldy model weights under ideal circumstances and hundreds when data is poorly distributed. Inspired by recent work on dataset distillation and distributed one-shot learning, we propose Distilled One-Shot Federated Learning (DOSFL) to significantly reduce the communication cost while achieving comparable performance. In just one round, each client distills their private dataset, sends the synthetic data (e.g. images or sentences) to the server, and collectively trains a global model. The distilled data look like noise and are only useful to the specific model weights, i.e., become useless after the model updates. With this weight-less and gradient-less design, the total communication cost of DOSFL is up to three orders of magnitude less than FedAvg while preserving between 93% to 99% performance of a centralized counterpart. Afterwards, clients could switch to traditional methods such as FedAvg to finetune the last few percent to fit personalized local models with local datasets. Through comprehensive experiments, we show the accuracy and communication performance of DOSFL on both vision and language tasks with different models including CNN, LSTM, Transformer, etc. We demonstrate that an eavesdropping attacker cannot properly train a good model using the leaked distilled data, without knowing the initial model weights. DOSFL serves as an inexpensive method to quickly converge on a performant pre-trained model with less than 0.1% communication cost of traditional methods.
Forward citations
Cited by 8 Pith papers
-
Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks
Gradient-based dataset distillation of two-layer ReLU nets on multi-index models encodes the r-dimensional principal subspace into synthetic data of memory complexity Θ̃(r²d+L) that recovers high generalization.
-
CRIP: Channel Level Representation Injection for Personalized One-Shot Federated Learning
CRIP improves one-shot federated learning by matching and injecting only the most compatible feature channels from other clients' models, without extra training or public data.
-
Provable Sparse Inversion and Token Relabel Enhanced One-shot Federated Learning with ViTs
FedMITR uses sparse model inversion and token relabeling to improve one-shot federated learning with ViTs under non-IID conditions, delivering a tighter generalization bound via algorithmic stability analysis and bett...
-
Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integration
FedHD performs federated distillation for whole slide images by generating one synthetic feature set per real slide via Gaussian-mixture alignment and adding them via curriculum integration, outperforming prior federa...
-
Federated Distillation for Whole Slide Image via Gaussian-Mixture Feature Alignment and Curriculum Integration
FedHD is a federated learning framework for whole slide images that distills one-to-one synthetic features aligned via Gaussian mixtures and progressively integrates cross-site features through curriculum learning to ...
-
FedBiCross: Personalized One-Shot Federated Learning on Medical Images
FedBiCross clusters clients by model similarity, uses bi-level cross-cluster optimization for adaptive knowledge transfer, and applies personalized distillation to outperform baselines in non-IID data-free one-shot fe...
-
FedBiCross: Personalized One-Shot Federated Learning on Medical Images
FedBiCross adds client clustering, adaptive cross-cluster weighting, and local fine-tuning to data-free one-shot federated learning, reporting large accuracy jumps on four medical image datasets.
-
Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
A survey organizing knowledge distillation techniques for addressing privacy, heterogeneity, communication, and personalization challenges in federated learning.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.