FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning

Jiayu Liu; Jing Yang; Nianbin Wang; Xiaohui Tao; Yong Wang

REVIEW 5 major objections 5 minor 45 references

FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FedDW claims that a single consistency regularizer, aligning each client's row-softmaxed classifier Gram matrix with the global soft-label matrix, outperforms 10 state-of-the-art federated learning methods by an average of about 3% in…

desk verdict FedDW has a genuinely new regularizer and a broad empirical study, but the theory is broken and the central consistency assumption is under-supported. read the letter →

arxiv 2412.04521 v1 pith:OKJ6G567 submitted 2024-12-05 cs.LG cs.CE

Jiayu Liu , Yong Wang , Nianbin Wang , Jing Yang , Xiaohui Tao This is my paper · ORCID

classification cs.LGcs.CE

keywords federatedlearningnon-IIDdatasoftlabelsknowledgedistillationclassifierweightregularizationGrammatrixheterogeneousconsistencyoptimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FedDW claims that federated learning with non-IID data can be repaired by a single regularization term: pull each client's classifier-head weight Gram matrix toward a globally aggregated matrix of soft labels. The paper argues that in IID training these two matrices carry the same inter-class similarity information, so the global soft-label matrix is a legitimate target for local classifier parameters. Across MNIST, CIFAR-10/100, and IMDB, the method reports top or near-top accuracy among 10 baselines, with the largest gains in highly heterogeneous settings. A convergence proof gives an O(1/T) rate, and the extra backpropagation cost is limited to the last layer.

What carries the argument

The Class Relation (CR) matrix $\omega\omega^\top$ is the last-layer weight matrix multiplied by its own transpose, whose $(i,j)$ entry records the similarity between class $i$ and class $j$ through the inner product of their classifier weight vectors. The paper pairs this with the global soft-label (SL) matrix $\Omega$, a weighted average of per-class softmax outputs across clients, and regularizes via Eq. (11): $L_{\text{reg}} = \frac{1}{|C|^2}\|\Omega - \sigma(\omega\omega^\top)\|_F^2$. Because the gradient of this term depends only on the classification layer, backpropagation cost does not grow with model size.

What would settle it

Train a FedAvg model on an IID partition of CIFAR-10, compute the global soft-label matrix $\Omega$ and the row-softmaxed CR matrix $\sigma(\omega\omega^\top)$ at the end of training, and measure $\|\Omega - \sigma(\omega\omega^\top)\|_F$. If this distance is not close to zero, or does not shrink with more IID data or more classes, the consistency principle at the core of FedDW is violated and the regularization target is biased.

Watch

Extended reading notes

Core claim

Under IID data the per-class average soft-label matrix and the row-softmaxed product of the last-layer weight matrix with its transpose coincide. FedDW turns this equality into a training objective: each client minimizes the Frobenius distance between the server-aggregated soft-label matrix and its local $\sigma(\omega\omega^\top)$, thereby steering the classifier head back toward IID-like parameter geometry even when local data are skewed. The paper reports that this simple regularizer outperforms 10 existing FL methods, raising accuracy on average by about 3% in pathological non-IID settings, and that the added computation is negligible because the regularization gradient touches only the classifier parameters.

Load-bearing premise

FedDW presupposes that under evenly spread (IID) data, the matrix of average per-class predictions equals the classifier's weight-similarity matrix, which makes the global prediction matrix a valid target for local classifier weights.

Editorial extensions

If this is right

Any federated classifier with a final linear layer can add FedDW by uploading per-class average softmax vectors, so adoption requires no change to the backbone architecture.
The extra communication is a $|C|\times|C|$ matrix per client per round, which stays small relative to model parameters even for large networks.
Because the regularization gradient involves only the classification layer, the per-epoch client runtime stays near FedAvg; the paper measures a 0.06 ms/epoch difference on ResNet-18.
The DW regularizer can be stacked on other FL methods, improving accuracy when combined with FedAvg, FedDistill, MOON, and FedProx in the reported experiments.
The convergence bound of $O(1/T)$ matches standard federated rates under convex loss, so the added term does not worsen the asymptotic optimization guarantee.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A reader could test the core IID consistency directly: train any classifier on an IID split and compare the average soft-label matrix with the row-softmaxed weight Gram matrix; if the Frobenius distance is not small, the regularization target is approximate and the reported gains may be due to implicit averaging rather than exact consistency.
The method's global target is the average of per-class soft labels, so classes that are rare across all clients get less reliable entries; reweighting the SL matrix by class frequency might make FedDW more robust on long-tailed federated data.
Uploading per-class average softmax vectors reveals class proportions on each client; combining FedDW with differential privacy or a public dataset, as the paper suggests, would be needed to close that privacy gap.
For large models, the flat back-propagation cost is the most distinctive property; whether that advantage survives when the classifier head itself is huge is an open extension.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The short version: FedDW introduces a genuinely new regularizer for non-IID federated learning and backs it with an unusually broad set of experiments. The theoretical sections, however, are not reliable as written, and the paper's core assumption—that under IID data the row-softmaxed classifier Gram matrix equals the global soft-label matrix—is supported only by visual evidence.

What's new: the method aligns each client's CR matrix (sigma(omega omega^T)) with the globally averaged soft-label matrix from all clients. That specific mechanism is not in the cited soft-label or classifier-regularization papers. It's a clean, simple idea with negligible additional compute, since the regularizer only touches the last layer. The experimental work is a real strength: four datasets spanning CV and NLP, multiple backbones (CNN, ResNet, LSTM), two heterogeneity regimes, client-count and round-scalability tests, and a compatibility table showing the regularizer boosts several existing FL methods. Five runs with standard deviations are reported, which is more than many FL papers bother with. The roughly 3-point gains over strong baselines in pathological settings are plausible from the tables.

Soft spots. The theory is in poor shape. Theorem 3's proof is explicitly omitted, and the displayed convergence bound looks wrong: Eq. (21) gives an upper bound on T, which would imply convergence only for small T, the reverse of what you'd need. More seriously, Section IV-B defines A as sigma(omega omega^T) and then analyzes Lreg(A) = ||Omega - A^T A||_F^2, but the actual regularizer in Eq. (11) is ||Omega - A||_F^2. The derivatives and non-convexity discussion are about a different objective. That's a load-bearing error in the theory.

The key consistency hypothesis also has thin evidence. The paper shows a t-SNE plot and heatmaps under IID, but never measures the distance between SL and CR quantitatively, or compares it to a chance baseline. And because the regularizer forces CR toward SL, the agreement shown in the non-IID case is partly by construction. A referee should ask for an independent test: train a model on IID data, freeze it, compute both matrices, report Frobenius distance relative to a random permutation.

Minor issues: mu is tuned on the same test sets, which can inflate results; the 'DLE data' terminology is a new name for known ideas but that's harmless; the privacy discussion correctly notes that uploading soft labels may leak information but doesn't analyze it.

Bottom line: the empirical contribution is worth taking seriously, but the theory needs to be corrected or removed, and the consistency claim needs real evidence. I'd send this to peer review with major-revision expectations, not desk-reject it.

Referee Report

5 major / 5 minor

Summary. The paper proposes FedDW, a federated learning method that regularizes each client's last-layer classifier weights by matching the row-softmaxed Gram matrix σ(ωω^T) to a globally aggregated soft-label matrix Ω. The central claim is that under IID data these two matrices coincide and encode inter-class relationships, so the global soft-label matrix can serve as a target to mitigate performance loss under non-IID data. The method requires uploading only a |C|×|C| matrix per client and adds a regularization term that depends only on the classifier layer. The paper reports experiments on MNIST, CIFAR-10/100, and IMDB against ten baselines, scalability tests, efficiency measurements, compatibility experiments, and a theoretical convergence analysis.

Significance. If the key consistency assumption were rigorously established, FedDW would be a simple and communication-efficient regularizer for non-IID federated classification. The experimental work is broad: four datasets, ten baselines, five-run means with standard deviations, and scalability experiments in client count, training rounds, and network architectures, with code publicly released. The empirical results are internally consistent and the method is competitive or state-of-the-art in most settings. However, the theoretical sections currently do not support the paper's claims: the proof of Theorem 3 is omitted, the derivative analysis in Section IV-B is performed on a different objective, and the central IID equality is only demonstrated visually. These issues prevent acceptance in the current form.

major comments (5)

[Section III-B, Eq. (11)] The central assumption that under IID data the global soft-label matrix Ω_global equals σ(ωω^⊤) is not quantitatively established. The supporting evidence (Figures 4, 5, and 8) is visual: weight-modulus trends, a t-SNE plot, and heatmaps, with no distance, correlation, or error measure between Ω and σ(ωω^⊤). Because Eq. (11) directly penalizes the difference between these two matrices under non-IID conditions, an approximate or dataset-specific equality would bias the regularization target, and the reported gains could stem from generic classifier smoothing rather than the claimed consistency. Please provide a quantitative IID test, such as the average Frobenius distance or cosine similarity between the two matrices across datasets, compared against a baseline such as a randomly permuted matrix.
[Section III-C1, Eq. (4)] In the pathological non-IID setting (β=0.1), a client may have no data from a particular class, making |D_i^n|=0 and Eq. (4) undefined for that row. The paper does not specify how to construct the local average soft-label matrix for missing classes, even though Figure 4 explicitly shows clients with zero data in some classes. The implementation must define a convention for these rows (for example, omitting them from aggregation, using a uniform distribution, or assigning a default row), and the aggregation in Eq. (7) should be stated in terms of that convention. Please clarify how missing classes are handled in the experiments and in Algorithm 1.
[Section IV-B, Eqs. (23)-(25)] The derivative analysis is performed on a surrogate objective rather than the actual regularizer. In Eq. (23), σ(ωω^⊤) is replaced by a generic row-stochastic matrix A, and the regularizer is rewritten as L_reg(A)=||Ω−A^⊤A||_F^2, which is not the same function as Eq. (11), whose argument is σ(ωω^⊤) and whose variable is ω. Even for the surrogate, Eq. (24) is incorrect: the gradient of ||Ω−A^⊤A||_F^2 with respect to A is −4A(Ω−A^⊤A) (for symmetric Ω), not −2(Ω−A^⊤A)A. Consequently, Eq. (25) does not give the Hessian of the true objective, and the claims about computational cost and convexity of the actual L_reg are not supported by this section.
[Section IV-A, Theorem 3, Eqs. (20)-(22)] Theorem 3 is stated without proof, with the text noting that the proof is omitted due to space limitations. The statement also appears to have technical problems: Eq. (21) gives an upper bound on T for a given ε, whereas a convergence guarantee of the form (1/T)Σ E||∇L||² ≤ ε requires T to be sufficiently large, not bounded above; Eq. (22) can involve a negative denominator or an imaginary square root for allowed parameters; and Eq. (20) appears to contain a typographical error ('Eη²' instead of the Rη²σ² term that appears in Eq. (19)). Moreover, Assumption 3 assumes convexity of L_cla, but the theorem claims a non-convex convergence rate, and the paper does not explain how the convex assumption is reconciled with that claim. A complete, correct proof is needed before the convergence analysis can support the paper's claims.
[Abstract and Table I] The abstract claims that FedDW 'outperforms 10 state-of-the-art FL methods, improving accuracy by an average of 3% in highly heterogeneous settings.' The numbers in Table I do not clearly support this figure: averaged over the four pathological-heterogeneity tasks, FedDW improves over the best baseline by about 1.1 percentage points (and it is 0.12 points below FedBABU on MNIST), while averaged over all eight practical and pathological tasks the improvement over the best baseline is below 1 point. Please state explicitly which baselines and metric (absolute percentage points, relative improvement, comparison with FedAvg or the best baseline) produce the 3% average, or adjust the claim accordingly.

minor comments (5)

[Eq. (23)] The summation index is written as '∑_{j=1}^n A_ij = 1'; the upper limit should be |C|, and similar typographical inconsistencies with 'n' instead of '|C|' appear in the surrounding text. Please correct these.
[Table I caption] The caption says 'The font deepening represents the best result'; this should read 'boldface indicates the best result'. Additionally, the term 'counterexample' in the Figure 4 caption is not defined; please clarify what constitutes a counterexample in the 100 experiments.
[Section V-A] The sentence 'The other two datasets have not changed much' is vague; please report the µ values selected for MNIST and IMDB and describe how the selection was made.
[Related Work] The text cites 'FedDyn [14]', but reference [14] is the FedProto paper; FedDyn appears not to be cited elsewhere. Please correct the citation and ensure all references are used consistently.
[Section III-B, Figure 5] The statement 'before visualization, we need to perform Vector Unitization' is unclear; please specify what normalization was applied to the weight vectors and features before the t-SNE embedding.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity: the accuracy claim is an external benchmark; the only circular-adjacent element is the post-training SL-CR similarity in Figure 8, which is enforced by the FedDW loss rather than independently discovered.

self definitional [Section V-F (heatmap analysis, Figure 8; cf. Eq. (11))]
"In non-IID environments, FedDW’s SL and CR matrices both resemble those under IID conditions, and the two matrices are similar to each other as well. This demonstrates that our method can mitigate the issue of data heterogeneity."

The similarity between SL and CR under FedDW is not an emergent finding: Eq. (11) directly minimizes ||Omega_global - sigma(omega omega^T)||_F^2 at every local update, so the CR matrix is explicitly trained toward the SL matrix. Presenting this post-training agreement as evidence for the consistency principle or for the method's mechanism is circular in that the agreement is the optimization target, not an independent confirmation. However, this step is supporting visualization only; the main accuracy result (Table I) is measured on held-out test data against external baselines, so the central performance claim does not reduce to this construction.

full rationale

The central claim—FedDW outperforms 10 FL methods by about 3% in heterogeneous settings—is an empirical comparison on MNIST, CIFAR-10/100, and IMDB test sets, so it is not derived from the regularizer's own definition. The SL-CR consistency premise in Section III-B is an empirical hypothesis supported by weight-modulus trends, a t-SNE plot, and FedAvg-IID heatmaps; even if that evidence is weak, it is not circular because the IID SL and CR matrices are not coupled by a loss. The convergence analysis cites FedProto [14] and FedGH [11], but these are external works, not self-citations, and the cited proofs are not the source of the accuracy gains. The only circular-adjacent step is Section V-F's use of FedDW's own post-training SL-CR similarity as a demonstration that the method works; since Eq. (11) enforces exactly this similarity, that illustration is partly by construction. Because the held-out accuracy numbers are independent and no load-bearing argument reduces to a self-citation or to the regularizer's own target, the overall circularity is minor.

Assumptions & free parameters 1 free parameters · 6 assumptions · 1 invented entities

FedDW rests on an asserted numerical equality between soft-label matrices and classifier-head Gram matrices, on unverified statistical assumptions about local losses (including convexity for the convergence theorem), and on per-dataset tuning of the regularization weight mu. The DLE data definition also asserts a privacy guarantee that the paper itself later qualifies. None of these are machine-checked or reproduced with released artifacts.

free parameters (1)

mu (regularization weight) = 0.1 for CIFAR-10, 10 for CIFAR-100; search grid {0.01, 0.1, 1, 10, 100}
Tuned per dataset on the same benchmark used for reporting, so the reported gains partly depend on favorable per-dataset selection (Section V-A, Figure 6).

assumptions (6)

domain assumption Assumptions 1 and 2: local losses are L1-smooth, gradients are unbiased with bounded variance.
Section IV-A Eqs. (14)-(16); used for Theorems 2 and 3; not verified for the tested neural networks.
domain assumption Assumption 3: the local classification loss is convex.
Section IV-A Eq. (17); the paper calls this a strong condition; real neural network losses are not convex, and the theorem applies only to a simplified setting.
ad hoc to paper Under IID data, the global SL matrix and the CR matrix sigma(omega omega^T) are numerically equal.
Section III-B and Eq. (11); this is the load-bearing premise that makes global soft labels a valid target for local classifier weights. It is validated only qualitatively through t-SNE and heatmaps.
domain assumption Under IID data, classifier weight moduli are uniform enough that the inner product can replace cosine similarity.
Section III-B; the paper's own experiments show weight moduli correlate negatively with class proportions, so the uniformity claim is asserted rather than derived.
ad hoc to paper The globally aggregated SL matrix retains IID-like class-relationship information under non-IID aggregation.
Section V-F and Eq. (7); needed for the regularizer to be a valid global target; only supported by visual heatmap inspection.
ad hoc to paper DLE data cannot be decoded to reconstruct data or derive statistical results.
Definition 1 in Section I; the paper later states in Section III-C that uploading the average SL matrix may leak privacy, so the premise is internally qualified.

invented entities (1)

DLE data (Deep Learning Encrypted data)
purpose: Umbrella term for soft labels, feature vectors, and model parameters shared during federated learning.
Defined in Definition 1 as data that cannot be decoded, but the paper later acknowledges that soft-label upload may leak privacy. It is a labeling device rather than an independently evidenced entity.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning." pith.science (2026). https://pith.science/paper/OKJ6G567

@misc{pith2026241204521,
  author       = {Pith},
  title        = {Pith review of: FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKJ6G567}},
  note         = {Machine review of arXiv:2412.04521}
}

read the original abstract

Federated Learning (FL) is an innovative distributed machine learning paradigm that enables neural network training across devices without centralizing data. While this addresses issues of information sharing and data privacy, challenges arise from data heterogeneity across clients and increasing network scale, leading to impacts on model performance and training efficiency. Previous research shows that in IID environments, the parameter structure of the model is expected to adhere to certain specific consistency principles. Thus, identifying and regularizing these consistencies can mitigate issues from heterogeneous data. We found that both soft labels derived from knowledge distillation and the classifier head parameter matrix, when multiplied by their own transpose, capture the intrinsic relationships between data classes. These shared relationships suggest inherent consistency. Therefore, the work in this paper identifies the consistency between the two and leverages it to regulate training, underpinning our proposed FedDW framework. Experimental results show FedDW outperforms 10 state-of-the-art FL methods, improving accuracy by an average of 3% in highly heterogeneous settings. Additionally, we provide a theoretical proof that FedDW offers higher efficiency, with the additional computational load from backpropagation being negligible. The code is available at https://github.com/liuvvvvv1/FedDW.

Figures

Figures reproduced from arXiv: 2412.04521 by the authors.

**Figure 2.** The client uses DLE data for regular optimization, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The entire training process of FedDW. With the increasing size of models in today’s training landscape, especially with the advent of large models [35], [36], [37], the additional computational cost of regularizing data heterogeneity is something that absolutely cannot be ignored. In this regard, FedDW offers a significant advantage. III. METHODOLOGY A. Problem Statement We assume that there are N clients, denoted … view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: We test each dataset 100 times. Due to space constraints, only three different data (D1, D2, D3) are randomly selected [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: We use ”X” to represent the weight vector of each [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Accuracy for different µ values on CIFAR10 and CIFAR100 To simulate heterogeneous data distribution, we use the popular Dirichlet distribution Dir(β) to form non-IID data distribution, where the parameter β controls the degree of data heterogeneity. The smaller the β, …

**Figure 7.** Figure 7: Test accuracy under different communication rounds in both Pathological Heterogeneous and Practical Heterogeneous [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

**Figure 8.** Figure 8: Heatmap Visualization of SL matrix(Left) and CR matrix(Right) under three different conditions. Non-IID refers to the [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 27 canonical work pages

[1]

Challenges and future directions of secure federated learning: A survey,

K. Zhang, X. Song, C. Zhang, and S. Yu, “Challenges and future directions of secure federated learning: A survey,” Front. Comput. Sci., vol. 16, pp. 1–8, 2022

work page 2022
[2]

Advances and open problems in federated learning,

P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. , “Advances and open problems in federated learning,” Found. Trends Mach. Learn., vol. 14, no. 1–2, pp. 1–210, 2021

work page 2021
[4]

Communication-efficient learning of deep networks from decentralized data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Aguera y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proc. Artif. Intell. Statist. , 2017, pp. 1273–1282

work page 2017
[5]

Not all minorities are equal: Empty-class-aware distillation for heterogeneous federated learning,

K. Guo, Y . Ding, J. Liang, R. He, Z. Wang, and T. Tan, “Not all minorities are equal: Empty-class-aware distillation for heterogeneous federated learning,” arXiv preprint arXiv:2401.02329 , 2024

arXiv 2024
[6]

Federated optimization in heterogeneous networks,

T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,”Proc. Mach. Learn. Syst., vol. 2, pp. 429–450, 2020

work page 2020
[7]

Heterogeneous feder- ated learning: State-of-the-art and research challenges,

M. Ye, X. Fang, B. Du, P. C. Yuen, and D. Tao, “Heterogeneous feder- ated learning: State-of-the-art and research challenges,” ACM Comput. Surveys, vol. 56, no. 3, pp. 1–44, 2023

work page 2023
[8]

Federated learning with non-IID data,

Y . Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V . Chandra, “Federated learning with non-IID data,” arXiv preprint arXiv:1806.00582 , 2018

arXiv 2018
[9]

Clip-guided federated learning on heterogeneous and long-tailed data,

J. Shi, S. Zheng, X. Yin, Y . Lu, Y . Xie, and Y . Qu, “Clip-guided federated learning on heterogeneous and long-tailed data,”arXiv preprint arXiv:2312.08648, 2023

arXiv 2023

Show all 45 references

[10]

Decoupling representation and classifier for long-tailed recognition,

B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y . Kalantidis, “Decoupling representation and classifier for long-tailed recognition,” arXiv preprint arXiv:1910.09217 , 2019

1910 arXiv
[11]

FedGH: Heterogeneous federated learning with generalized global header,

L. Yi, G. Wang, X. Liu, Z. Shi, and H. Yu, “FedGH: Heterogeneous federated learning with generalized global header,” in Proc. 31st ACM Int. Conf. Multimedia , 2023, pp. 8686–8696

2023
[12]

FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning,

J. Zhang, Y . Liu, Y . Hua, and J. Cao, “FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 15, pp. 1676...

2024
[13]

Fedx: Unsupervised federated learning with cross knowledge distillation,

S. Han, S. Park, F. Wu, S. Kim, C. Wu, X. Xie, and M. Cha, “Fedx: Unsupervised federated learning with cross knowledge distillation,” in Proc. Eur. Conf. Comput. Vis. (ECCV) , 2022, pp. 691–707

2022
[14]

Fedproto: Federated prototype learning across heterogeneous clients,

Y . Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “Fedproto: Federated prototype learning across heterogeneous clients,” in Proc. AAAI Conf. Artif. Intell. , vol. 36, no. 8, 2022, pp. 8432–8440

2022
[15]

PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees,

C. Xie, D.-A. Huang, W. Chu, D. Xu, C. Xiao, B. Li, and A. Anand- kumar, “PerAda: Parameter-Efficient Federated Learning Personalization with Generalization Guarantees,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2024, pp. 23838–23848

2024
[16]

MH-pFLID: Model heterogeneous personalized federated learning via injection and distillation for medical data analysis,

L. Xie, M. Lin, T. Luan, C. Li, Y . Fang, Q. Shen, and Z. Wu, “MH-pFLID: Model heterogeneous personalized federated learning via injection and distillation for medical data analysis,” arXiv preprint arXiv:2405.06822, 2024

2024 arXiv
[17]

Communication-efficient on-device machine learning: Federated dis- tillation and augmentation under non-IID private data,

E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Communication-efficient on-device machine learning: Federated dis- tillation and augmentation under non-IID private data,” arXiv preprint arXiv:1811.11479, 2018

2018 arXiv
[18]

Distilling the knowledge in a neural network,

G. Hinton, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

2015 arXiv
[19]

Federated learning based on dynamic regularization,

A. E. Durmus, Z. Yue, M. Ramon, M. Mattina, P. Whatmough, and V . Saligrama, “Federated learning based on dynamic regularization,” in Proc. Int. Conf. Learn. Representations (ICLR) , 2021

2021
[20]

Model-contrastive federated learning,

Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, pp. 10713– 10722

2021
[21]

Deep leakage from gradients,

L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” Adv. Neural Inf. Process. Syst. , vol. 32, 2019

2019
[22]

Momentum contrast for unsupervised visual representation learning,

K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2020, pp. 9729–9738

2020
[23]

Debiased contrastive learning for sequential recommendation,

Y . Yang, C. Huang, L. Xia, C. Huang, D. Luo, and K. Lin, “Debiased contrastive learning for sequential recommendation,” in Proc. ACM Web Conf. 2023, 2023, pp. 1063–1073

2023
[24]

CPCL: Cross-modal prototypical contrastive learning for weakly supervised text-based person re-identification,

Y . Zheng, X. Zhao, C. Lan, X. Zhang, B. Huang, J. Yang, and D. Yu, “CPCL: Cross-modal prototypical contrastive learning for weakly supervised text-based person re-identification,” arXiv preprint arXiv:2401.10011, 2024

2024 arXiv
[25]

A simple framework for contrastive learning of visual representations,

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in Proc. Int. Conf. Mach. Learn. (ICML) , 2020, pp. 1597–1607

2020
[26]

An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning,

J. Zhang, Y . Liu, Y . Hua, and J. Cao, “An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR) , 2024, pp. 12109–12119

2024
[27]

Fair federated learning under domain skew with local consistency and domain diversity,

Y . Chen, W. Huang, and M. Ye, “Fair federated learning under domain skew with local consistency and domain diversity,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2024, pp. 12077–12086

2024
[28]

Dafkd: Domain-aware federated knowledge distillation,

H. Wang, Y . Li, W. Xu, R. Li, Y . Zhan, and Z. Zeng, “Dafkd: Domain-aware federated knowledge distillation,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2023, pp. 20412–20421

2023
[29]

Fedaux: Leveraging unlabeled auxiliary data in federated learning,

F. Sattler, T. Korjakow, R. Rischke, and W. Samek, “Fedaux: Leveraging unlabeled auxiliary data in federated learning,” IEEE Trans. Neural Netw. Learn. Syst. , vol. 34, no. 9, pp. 5531–5543, 2021

2021
[30]

Exploiting shared representations for personalized federated learning,

L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai, “Exploiting shared representations for personalized federated learning,” in Proc. Int. Conf. Mach. Learn. (ICML) , 2021, pp. 2089–2099

2021
[31]

Fedala: Adaptive local aggregation for personalized federated learning,

J. Zhang, Y . Hua, H. Wang, T. Song, Z. Xue, R. Ma, and H. Guan, “Fedala: Adaptive local aggregation for personalized federated learning,” in Proc. AAAI Conf. Artif. Intell., vol. 37, no. 9, 2023, pp. 11237–11244

2023
[32]

Federated learning with personalization layers,

M. G. Arivazhagan, V . Aggarwal, A. K. Singh, and S. Choud- hary, “Federated learning with personalization layers,” arXiv preprint arXiv:1912.00818, 2019

1912 arXiv
[33]

Federated recommendation with additive personalization,

Z. Li, G. Long, and T. Zhou, “Federated recommendation with additive personalization,” in Proc. 12th Int. Conf. Learn. Representations (ICLR) , 2024. [Online]. Available: https://openreview.net/forum?id=xkXdE81mOK

2024
[34]

Towards personalized federated learning,

A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” IEEE Trans. Neural Netw. Learn. Syst. , vol. 34, no. 12, pp. 9587–9603, 2022

2022
[35]

Large language models: A survey,

S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “Large language models: A survey,” arXiv preprint arXiv:2402.06196, 2024

2024 arXiv
[36]

A survey on large language models for recommendation,

L. Wu, Z. Zheng, Z. Qiu, H. Wang, H. Gu, T. Shen, C. Qin, C. Zhu, H. Zhu, Q. Liu, et al. , “A survey on large language models for recommendation,” World Wide Web, vol. 27, no. 5, p. 60, 2024

2024
[37]

A survey on evaluation of large language models,

Y . Chang, X. Wang, J. Wang, Y . Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y . Wang, et al. , “A survey on evaluation of large language models,” ACM Trans. Intell. Syst. Technol. , vol. 15, no. 3, pp. 1–45, 2024

2024
[38]

Deep learning with differential privacy,

M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang, “Deep learning with differential privacy,” inProc. ACM SIGSAC Conf. Comput. Commun. Secur. , 2016, pp. 308–318

2016
[39]

FedUV: Uniformity and variance for heterogeneous federated learning,

H. M. Son, M.-H. Kim, T.-M. Chung, C. Huang, and X. Liu, “FedUV: Uniformity and variance for heterogeneous federated learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. , 2024, pp. 5863–5872

2024
[40]

Fedbabu: Towards enhanced representation for federated image classification,

J. Oh, S. Kim, and S.-Y . Yun, “Fedbabu: Towards enhanced representation for federated image classification,” arXiv preprint arXiv:2106.06042, 2021

2021 arXiv
[41]

Personalized federated learning: A meta-learning approach,

A. Fallah, A. Mokhtari, and A. Ozdaglar, “Personalized federated learning: A meta-learning approach,” arXiv preprint arXiv:2002.07948 , 2020

2002 arXiv
[42]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778

2016
[43]

Pytorch: An imperative style, high-performance deep learning library,

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,” Adv. Neural Inf. Process. Syst., vol. 32, 2019

2019
[44]

Adam: A Method for Stochastic Optimization,

D. P. Kingma and J. Ba, “Adam: A Method for Stochastic Optimization,” International Conference on Learning Representa- tions (ICLR) , San Diego, CA, USA, 2015. [Online]. Available: https://arxiv.org/abs/1412.6980

2015 arXiv
[45]

Shufflenet: An extremely efficient convolutional neural network for mobile devices,

X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely efficient convolutional neural network for mobile devices,” inProc. IEEE Conf. Comput. Vis. Pattern Recognit. , 2018, pp. 6848–6856

2018
[46]

Mobilenets: Efficient convolutional neural networks for mobile vision applications,

A. G. Howard, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861 , 2017

2017 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

Challenges and future directions of secure federated learning: A survey,

K. Zhang, X. Song, C. Zhang, and S. Yu, “Challenges and future directions of secure federated learning: A survey,” Front. Comput. Sci., vol. 16, pp. 1–8, 2022

work page 2022

[2] [2]

Advances and open problems in federated learning,

P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al. , “Advances and open problems in federated learning,” Found. Trends Mach. Learn., vol. 14, no. 1–2, pp. 1–210, 2021

work page 2021

[3] [4]

Communication-efficient learning of deep networks from decentralized data,

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Aguera y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proc. Artif. Intell. Statist. , 2017, pp. 1273–1282

work page 2017

[4] [5]

Not all minorities are equal: Empty-class-aware distillation for heterogeneous federated learning,

K. Guo, Y . Ding, J. Liang, R. He, Z. Wang, and T. Tan, “Not all minorities are equal: Empty-class-aware distillation for heterogeneous federated learning,” arXiv preprint arXiv:2401.02329 , 2024

arXiv 2024

[5] [6]

Federated optimization in heterogeneous networks,

T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,”Proc. Mach. Learn. Syst., vol. 2, pp. 429–450, 2020

work page 2020

[6] [7]

Heterogeneous feder- ated learning: State-of-the-art and research challenges,

M. Ye, X. Fang, B. Du, P. C. Yuen, and D. Tao, “Heterogeneous feder- ated learning: State-of-the-art and research challenges,” ACM Comput. Surveys, vol. 56, no. 3, pp. 1–44, 2023

work page 2023

[7] [8]

Federated learning with non-IID data,

Y . Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V . Chandra, “Federated learning with non-IID data,” arXiv preprint arXiv:1806.00582 , 2018

arXiv 2018

[8] [9]

Clip-guided federated learning on heterogeneous and long-tailed data,

J. Shi, S. Zheng, X. Yin, Y . Lu, Y . Xie, and Y . Qu, “Clip-guided federated learning on heterogeneous and long-tailed data,”arXiv preprint arXiv:2312.08648, 2023

arXiv 2023