TallyTrain is a hard-label distillation protocol for federated learning that uses argmax transmission and optional sparse merges to match soft-label performance at up to 1000x lower communication cost.
hub
Fedmd: Heterogenous federated learning via model distillation
22 Pith papers cite this work, alongside 480 external citations. Polarity classification is still indexing.
22
Pith papers citing it
480
external citations · Pith
abstract
Federated learning enables the creation of a powerful centralized model without compromising data privacy of multiple participants. While successful, it does not incorporate the case where each participant independently designs its own model. Due to intellectual property concerns and heterogeneous nature of tasks and data, this is a widespread requirement in applications of federated learning to areas such as health care and AI as a service. In this work, we use transfer learning and knowledge distillation to develop a universal framework that enables federated learning when each agent owns not only their private data, but also uniquely designed models. We test our framework on the MNIST/FEMNIST dataset and the CIFAR10/CIFAR100 dataset and observe fast improvement across all participating models. With 10 distinct participants, the final test accuracy of each model on average receives a 20% gain on top of what's possible without collaboration and is only a few percent lower than the performance each model would have obtained if all private datasets were pooled and made directly available for all participants.
hub tools
citation-role summary
background 4
citation-polarity summary
roles
background 4polarities
background 4representative citing papers
Logit-based federated learning leaks private model information to a semi-honest server via shared logits even with unrelated public data, enabling an adaptive stealing attack with theoretical bounds and a logit-perturbation defense.
FedQHD achieves closed-form federated Q-learning via hyperdimensional encoders with linear readouts, formalizes the federation gap under heterogeneous encoders, and reports competitive performance on continuous-state benchmarks with reduced computation.
FedSAF shifts prototype alignment in heterogeneous federated learning from coordinate matching to inter-class structural relations and reports up to 3.52% gains over prior methods.
LNTrust has nodes learn compact trust functions from validation evidence that both guide training distillation and define deployment ensembles, yielding higher accuracy with less communication than prior output-only baselines.
Proposes a covariance-aware tuning-free shrinkage framework and sequential algorithm for multi-source estimation that attains oracle risk asymptotically and improves on single-step methods.
Proposes a sparsity-plus-fusion-penalty federated estimator for tail index regression with non-asymptotic rates, oracle group recovery, ADMM solver, and debiased inference that improves on target-only methods.
Q-LocalAdam reduces optimizer memory by 3.37x via tailored 8-bit quantization for Adam states while maintaining or improving accuracy under data heterogeneity in edge federated learning.
Low-resolution data improves high-resolution model performance when high-resolution samples are limited, via KL-divergence bounds and experiments on vision transformers and CNNs.
FedF-ADMM uses function-space ADMM updates projected via knowledge distillation plus a PI-like stabilization term to deliver faster, more stable convergence and higher accuracy than prior decentralized FL methods under severe non-IID conditions.
HARMONY mitigates representation skew in heterogeneous hybrid split federated learning via meta-learning to simulate diverse extractors and server-side contrastive learning to align features, delivering up to 43% accuracy gains.
Harmonization works better than personalization for appearance-based domain shifts in federated medical imaging while personalization is superior for structural shifts, with both performing similarly when shifts are small.
Collate jointly trains heterogeneous models under per-device latency constraints via dynamic zeroizing-recovering and proto-corrected aggregation, gaining ~2–3% accuracy over prior heterogeneous FL.
COSMOS personalizes federated learning across heterogeneous client models via server-side clustering and distillation using only pseudo-labels, with claimed exponential personalization risk contraction.
DFL-AA removes link-quality distortion in expectation from gossip aggregation in asynchronous decentralized federated learning by combining online IPW for selection bias correction with AoI weighting for staleness mitigation.
HASA computes client heterogeneity scores from local data and assigns wider subnets to less heterogeneous clients, raising mean client test accuracy from 13.82% to 14.32% and improving worst-client accuracy versus uniform and partial-training baselines under matched compute budgets on a seven-client
BiFedKD improves ECG classification accuracy by 3.52% and Macro-F1 by 9.93% on MIT-BIH while cutting communication overhead 40% and computation cost 71.7% versus baseline federated methods.
FedProxy replaces weak adapters with a proxy SLM for federated LLM fine-tuning, outperforming prior methods and approaching centralized performance via compression, heterogeneity-aware aggregation, and training-free fusion.
FedBB addresses inter-case, inter-class, and inter-client imbalances in federated learning via Positive Negative Balanced loss and Client Balanced Reweighting, outperforming baselines on X-ray and natural image datasets while using limited statistics for privacy.
FedMTFI clusters heterogeneous clients, trains cluster prototypes, and applies multi-teacher distillation with SHAP to improve accuracy over standard FL in non-IID settings.
FedKD-hybrid is a hybrid federated knowledge distillation framework for multi-model lithography hotspot detection that outperforms prior methods on ICCAD-2012 and real-world FAB datasets.
A survey organizing knowledge distillation techniques for addressing privacy, heterogeneity, communication, and personalization challenges in federated learning.
citing papers explorer
-
TallyTrain: Communication-Efficient Federated Distillation
TallyTrain is a hard-label distillation protocol for federated learning that uses argmax transmission and optional sparse merges to match soft-label performance at up to 1000x lower communication cost.
-
Quantifying and Defending against the Privacy Risk in Logit-based Federated Learning
Logit-based federated learning leaks private model information to a semi-honest server via shared logits even with unrelated public data, enabling an adaptive stealing attack with theoretical bounds and a logit-perturbation defense.
-
FedQHD: Closed-Form Function-Space Federated Reinforcement Learning
FedQHD achieves closed-form federated Q-learning via hyperdimensional encoders with linear readouts, formalizes the federation gap under heterogeneous encoders, and reports competitive performance on continuous-state benchmarks with reduced computation.
-
From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning
FedSAF shifts prototype alignment in heterogeneous federated learning from coordinate matching to inter-class structural relations and reports up to 3.52% gains over prior methods.
-
Learned Neighbor Trust for Collaborative Deployment in Model-Agnostic Decentralized Learning
LNTrust has nodes learn compact trust functions from validation evidence that both guide training distillation and define deployment ensembles, yielding higher accuracy with less communication than prior output-only baselines.
-
Tuning-Free Efficient Estimation for Multi-Source Data via Covariance-Aware Shrinkage
Proposes a covariance-aware tuning-free shrinkage framework and sequential algorithm for multi-source estimation that attains oracle risk asymptotically and improves on single-step methods.
-
Efficient Federated Estimation and Inference for High-Dimensional Tail Index Regression
Proposes a sparsity-plus-fusion-penalty federated estimator for tail index regression with non-asymptotic rates, oracle group recovery, ADMM solver, and debiased inference that improves on target-only methods.
-
Q-LocalAdam: Memory-Efficient Client-Side Adaptive Optimization for Edge Federated Learning
Q-LocalAdam reduces optimizer memory by 3.37x via tailored 8-bit quantization for Adam states while maintaining or improving accuracy under data heterogeneity in edge federated learning.
-
On What We Can Learn from Low-Resolution Data
Low-resolution data improves high-resolution model performance when high-resolution samples are limited, via KL-divergence bounds and experiments on vision transformers and CNNs.
-
Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspective
FedF-ADMM uses function-space ADMM updates projected via knowledge distillation plus a PI-like stabilization term to deliver faster, more stable convergence and higher accuracy than prior decentralized FL methods under severe non-IID conditions.
-
HARMONY: Bridging the Personalization-Generalization Gap by Mitigating Representation Skew in Heterogeneous Split Federated Learning
HARMONY mitigates representation skew in heterogeneous hybrid split federated learning via meta-learning to simulate diverse extractors and server-side contrastive learning to align features, delivering up to 43% accuracy gains.
-
When To Adapt? Adapting the Model or Data in Federated Medical Imaging
Harmonization works better than personalization for appearance-based domain shifts in federated medical imaging while personalization is superior for structural shifts, with both performing similarly when shifts are small.
-
Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems
Collate jointly trains heterogeneous models under per-device latency constraints via dynamic zeroizing-recovering and proto-corrected aggregation, gaining ~2–3% accuracy over prior heterogeneous FL.
-
COSMOS: Model-Agnostic Personalized Federated Learning with Clustered Server Models and Pseudo-Label-Only Communication
COSMOS personalizes federated learning across heterogeneous client models via server-side clustering and distillation using only pseudo-labels, with claimed exponential personalization risk contraction.
-
Asynchronous Decentralized Federated Learning over Lossy Wireless Links via Reception- and Age-Aware Aggregation
DFL-AA removes link-quality distortion in expectation from gossip aggregation in asynchronous decentralized federated learning by combining online IPW for selection bias correction with AoI weighting for staleness mitigation.
-
HASA: Subnet Allocation for Compute-Constrained Model-Heterogeneous Federated Learning
HASA computes client heterogeneity scores from local data and assigns wider subnets to less heterogeneous clients, raising mean client test accuracy from 13.82% to 14.32% and improving worst-client accuracy versus uniform and partial-training baselines under matched compute budgets on a seven-client
-
BiFedKD: Bidirectional Federated Knowledge Distillation Framework for Non-IID and Long-Tailed ECG Monitoring
BiFedKD improves ECG classification accuracy by 3.52% and Macro-F1 by 9.93% on MIT-BIH while cutting communication overhead 40% and computation cost 71.7% versus baseline federated methods.
-
FedProxy: Federated Fine-Tuning of LLMs via Proxy SLMs and Heterogeneity-Aware Fusion
FedProxy replaces weak adapters with a proxy SLM for federated LLM fine-tuning, outperforming prior methods and approaching centralized performance via compression, heterogeneity-aware aggregation, and training-free fusion.
-
Multi-Level Analyzation of Imbalance to Resolve Non-IID-Ness in Federated Learning
FedBB addresses inter-case, inter-class, and inter-client imbalances in federated learning via Positive Negative Balanced loss and Client Balanced Reweighting, outperforming baselines on X-ray and natural image datasets while using limited statistics for privacy.
-
FedMTFI: Feature Importance Based Optimized Multi Teacher Knowledge Distillation in Heterogeneous Federated Learning Environment
FedMTFI clusters heterogeneous clients, trains cluster prototypes, and applies multi-teacher distillation with SHAP to improve accuracy over standard FL in non-IID settings.
-
Federated Knowledge Distillation for Multi-Model Architectures Lithography Hotspot Detection
FedKD-hybrid is a hybrid federated knowledge distillation framework for multi-model lithography hotspot detection that outperforms prior methods on ICCAD-2012 and real-world FAB datasets.
-
Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions
A survey organizing knowledge distillation techniques for addressing privacy, heterogeneity, communication, and personalization challenges in federated learning.