Pith. sign in

REVIEW 6 major objections 4 minor 44 references

Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data

T0 review · 6 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Client training losses can replace data-size weights in federated distillation.

desk verdict New multi-domain non-IID benchmark for federated distillation, but the central claim that adaptive weighting helps is unproven because no ablation isolates it from the switch to L2 distillation. read the letter →

arxiv 2508.20557 v1 pith:X4VBRL5R submitted 2025-08-28 cs.CL cs.AI

Jiahao Xiao , Jiangming Liu This is my paper · ORCID
classification cs.CLcs.AI
keywords federateddistillationnon-IIDmulti-domainnaturallanguageprocessingadaptiveensembletraininglossweightingbenchmarkheterogeneousmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that non-IID data in NLP federated learning should be understood as diversity in language domain as well as diversity in labels, and that earlier benchmarks missed the input-side heterogeneity. It introduces a benchmark built from five product-review domains and proposes AdaFD, a federated distillation method that weights each client's public-data predictions by a function of that client's private training loss. Two loss-based weighting schemes, RNWC and ENWC, plus a large-language-model-based variant, are compared with baselines that weight clients equally or by data size. In sentiment classification experiments, the loss-weighted variants achieve the best global F1 score among decentralized baselines in both same-architecture and different-architecture settings.

What carries the argument

The central object is the adaptive ensemble weighting of client logits: weights $w^k = (1/l^k_{\min})/\|v\|_1$ for RNWC and $w^k = e^{-\beta l^k_{\min}}/\|v\|_1$ for ENWC, where $l^k_{\min}$ is client $k$'s minimum training loss over local epochs, $\beta$ is a sharpness hyperparameter, and $v$ is the vector of unnormalized scores. These weights replace the equal or data-size weights used in earlier federated distillation methods, and they produce the aggregated prediction $\hat{z} = \sum_k w^k z^k$ that distills the central model through the squared-error objective $\|\hat{z} - \tilde{z}\|_2^2$. The weighting is what adapts: a client whose training loss is low is taken to fit the global distribution better and therefore receives more influence in the ensemble.

What would settle it

Run AdaFD on the released benchmark while recording, for each round, each client's training-loss rank and the change in global F1 when that client's weight is increased; if clients with lower training loss are not positively correlated with improvements in global F1, or if a deliberately overfitted client with near-zero loss on its own domain receives the largest weight and drags global F1 down, the central claim is falsified.

Watch

Extended reading notes

Core claim

The central claim is that adaptive ensemble weights derived from local training losses make federated distillation more robust to multi-domain non-IID text data than equal weighting or data-size weighting. Specifically, the paper proposes two weighting formulas: a reciprocal of each client's minimum training loss (RNWC) and an exponentially decaying function of that loss (ENWC), both normalized to sum to one and used to aggregate client logits on a shared public dataset. It further claims that distilling the central model with a squared-error loss, an approximation of KL divergence at high temperature, lets the global model track the global distribution even when clients hold different domains and different model architectures. The reported evidence is that AdaFD with ENWC achieves the best global F1 among decentralized baselines in homogeneous and heterogeneous settings, on both original and downsampled data, in both client-specific and global test sets.

Load-bearing premise

The load-bearing premise is that a client's minimum local training loss (or its exponential transform) reliably indicates how much that client's predictions should count in the global ensemble; if low loss can coincide with predictions that hurt the global distribution, the adaptive weighting provides no benefit.

Editorial extensions

If this is right

  • If the loss proxy holds, AdaFD's weighting should transfer to other text classification benchmarks without re-engineering the aggregation step.
  • The two loss-based schemes reach the reported global F1 without LLM API calls, so they are cheaper than the LLM-based variant while matching or beating it.
  • The best global F1 occurs at five communication rounds and declines with more rounds, so early stopping is a practical component of the method.
  • The benchmark's protocol—Dirichlet-skewed labels plus domain-specific clients—gives the community a common setting for comparing federated distillation under input-and-output non-IID.
  • The L2 distillation objective approximates KL divergence as temperature grows, removing the need to tune a temperature parameter.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same weighting could be tested on label-diverse-only non-IID data to see whether the loss proxy helps when domain misalignment is not the main difficulty.
  • A natural extension is to compare rank-based weighting, which uses only the ordering of losses, with value-based weighting, to separate the information in loss magnitudes from the information in their order.
  • The paper's five domains are all product reviews; a harder test would use structurally different text types, where a low training loss might reflect domain narrowness rather than alignment with the global distribution.
  • One could also examine whether the server's access to per-client losses leaks information about local data difficulty, a privacy question the paper does not address.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. The paper proposes AdaFD, an adaptive federated distillation framework for multi-domain non-IID textual data. It introduces a benchmark in which five Amazon review domains are assigned to five clients, creating both label and language-domain heterogeneity. AdaFD aggregates client logits on public data using weights derived from client training losses via three strategies (RNWC, ENWC, LLMWC) and distills the central model with an L2 loss that the authors derive as a large-temperature limit of KL divergence. Experiments report F1 scores on client-specific and global test sets in homogeneous and heterogeneous settings, at original and downsampled scales, claiming improvements over FedAvg, DS-FL, MHAT, and FedKD. The code is released.

Significance. The multi-domain non-IID benchmark is a useful contribution: most prior federated NLP work models non-IID only through label skew, and the proposed domain-based partitioning is more realistic for text. The adaptive weighting idea, using training loss as a proxy for a client's contribution to the global distribution, is interesting and plausibly useful. If the reported improvements are robust, the method is a practical addition to federated distillation. The code release and the systematic comparison across two settings, two data scales, and three weighting variants are also strengths. However, the empirical evidence as presented does not yet isolate the adaptive-weighting contribution, and several statistical and reproducibility issues need to be resolved before the central claim can be accepted.

major comments (6)
  1. [§V-A, §V-B, Tables I-II] The reported gains cannot be attributed to the adaptive weighting scheme because AdaFD differs from every baseline in two respects simultaneously: the weighting rule and the distillation objective. FedAvg, DS-FL, MHAT, and FedKD use KL/entropy-based distillation (Eqs. 3-4), while AdaFD uses the L2 loss of Eq. (10). No ablation holds the L2 objective fixed and replaces the ENWC/RNWC weights with equal weights or data-size weights. Without such an ablation, the 0.5-2.0 F1 improvements in Tables I and II may be caused by the distillation objective alone. Please add an ablation that isolates the adaptive weighting component.
  2. [§V-B, §VI-B] Eq. (10) is justified as the approximation of KL divergence when the temperature τ tends to infinity, but Section VI-B states that the temperature τ is set to 1 in the experiments. This is internally inconsistent: either the temperature is not actually used in the L2 loss, in which case the derivation should be corrected, or the experiments use τ=1, in which case the claimed connection to the τ→∞ limit is not what was implemented. Please clarify the role of τ and reconcile Eq. (10) with the reported setting.
  3. [Tables I and II captions] The table captions claim that AdaFD achieves statistically significant improvements over baselines with p<0.001, but the paper reports a single F1 number per condition with no variance, no number of replicates, and no description of the significance test. This makes the p-value unverifiable and the robustness claim unsupported. Please report means and standard deviations across at least several independent runs and describe the exact statistical test used, including any multiple-comparison correction.
  4. [§VII-C, §VII-D, §VII-E] The core assumption that a client's minimum training loss (Eq. 6) or exponentially scaled training loss (Eq. 7) reflects how well that client's predictions align with the global data distribution is asserted in Section VII-E but not supported by quantitative evidence. Section VII-E only shows loss curves and concludes that training losses are 'reasonable and effective' without reporting a correlation between loss and contribution quality. In addition, Section VII-D selects β=5 because it yields the highest F1 on the test set, which is a form of test-set fitting. Please provide an empirical correlation or a direct ablation supporting the loss-to-weight mapping, and select β on a validation split or report sensitivity without test-set optimization.
  5. [Algorithm 1, §V-A] The LLMWC branch in Algorithm 1 is incomplete: the pseudocode says 'generate the weights by LLM' but never assigns the generated weights to a variable and never shows how the ensemble logits bz are computed in that branch, since Eq. (8) appears only in the else branch. This makes the LLMWC results in Tables I and II non-reproducible. Please specify the full LLMWC procedure, including the weight variable, normalization, and how the weighted sum is formed.
  6. [§VI-D.2] The construction of the global test set is under-specified. The text says it is 'constructed from all domains' and 'sampled with α=0.8' and that its size is consistent with the client-specific test sets, but it does not state whether this set is disjoint from the client-specific test sets, how it relates to the 8:1:1 split of private data, or whether the same set was used for model selection such as choosing β. Please describe the exact sampling procedure and ensure that no test data are used for hyperparameter selection.
minor comments (4)
  1. [§V-B] Equation (9) uses σ without defining it; please state that σ denotes the softmax function.
  2. [Abstract, Index Terms] There are minor language issues: 'can not capture' should be 'cannot capture', and the Index Terms entry 'natural language process' should be 'natural language processing'.
  3. [Figures 4-6] The axis labels and legends in Figures 4-6 are difficult to read in the current rendering; please provide higher-resolution versions so that the reported trends are legible.
  4. [Section VI-B] The sentence 'the adaptive ensemble hyperparameter β and temperature τ is set to 5 and 1' has a subject-verb agreement issue and should be rephrased as 'β and τ are set to 5 and 1, respectively.'

Circularity Check

1 steps flagged · score 4.0 of 10

Headline F1 is partly a test-set-selected maximum: β is chosen by the F1 it then reports.

  1. fitted input called prediction [Section VI.B (Training and Inference Details) and Section VII.D (Effect of Hyperparameter)]
    "Meanwhile, the number of clients is fixed at 5, corresponding to the number of domain, and the adaptive ensemble hyperparameter β and temperature τ is set to 5 and 1, respectively. ... To investigate the impact of β on the performance of the central model, we set β to 1, 5, 10, 15, and 20, and conduct experiments under heterogeneous settings and with original data scale. ... Withβ = 5, the central model achieves its highest F1 score of 84.6"

    β is a free hyperparameter of the ENWC weighting rule (Eq. 7). The paper selects β=5 by sweeping β∈{1,5,10,15,20} and observing that β=5 'achieves its highest F1 score of 84.6' on the central model, then uses β=5 in the main experiments. The value 84.6 is the very heterogeneous-original-scale global F1 reported for AdaFD w/ ENWC in Table I. So the headline F1 is the maximum of a search over β evaluated on the same F1 metric that is later offered as evidence of superiority; it is a selected statistic, not an independent prediction. This is the fitted-input-called-prediction pattern: the reported performance is partly constructed by the choice of β, and the comparison with baselines inherits that selection.

full rationale

The paper's derivation chain is largely self-contained: AdaFD's weighting equations (Eqs. 6-8) are defined from local training losses, the distillation objective (Eq. 10) is stated, and the empirical claim is evaluated against external baselines (FedAvg, DS-FL, MHAT, FedKD) on public Amazon review data. I found no equation-level step where a derived quantity is identical to its own input by construction. The one concrete circular step is the hyperparameter β: Section VII.D reports that β was swept over {1,5,10,15,20} and that β=5 'achieves its highest F1 score of 84.6', and Section VI.B fixes β=5 for the main experiments; 84.6 is exactly the headline heterogeneous global F1 in Table I. That particular reported number is therefore a maximum over a test-metric sweep, i.e., a fitted input renamed as a result, rather than an independent estimate of AdaFD's performance. This is partial, not total, circularity: the adaptive weighting scheme itself is not fitted to its target, and there is genuine independent content in comparing against external baselines. The self-citation to FedID [9], which shares co-author Jiangming Liu, is related-work context and is not load-bearing; no uniqueness theorem or prior-work-derived ansatz is invoked to force the method. The unsupported p<0.001 caption claims, the τ=1 versus τ→∞ derivation of Eq. 10, and the missing ablation isolating the adaptive weights from the L2 distillation objective are serious correctness and attribution risks, but they are experimental/reporting deficiencies rather than circular derivation steps, so they do not raise the circularity score further.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The method leans on standard pretrained model backbones and a heuristic weighting scheme; it introduces no new physical entities. The main free parameter is beta, tuned on the evaluation set. The LLMWC variant relies on an external LLM whose weighting quality is not independently verified.

free parameters (3)
  • beta (ENWC smoothing) = 5
    Chosen in Section VII-D as the value giving highest global F1 (84.6); performance drops for beta=20. Tuned on the evaluation set.
  • Dirichlet alpha for label distribution = 1
    Chosen in Section VI-A as the benchmark configuration for label skew; not swept, but affects the non-IID scenario.
  • temperature tau = 1
    Reported in Section VI-B as set to 1, but the adaptive distillation uses L2 loss (Eq. 10) said to correspond to tau approaching infinity; the role of tau=1 is unclear.
assumptions (4)
  • domain assumption Pretrained language model representations transfer to downstream sentiment classification
    The method relies on BERT/RoBERTa/XLNet backbones initialized with pretrained weights; the paper does not study low-resource or non-pretrained settings.
  • standard math KL divergence with infinite temperature is approximated by L2 loss
    Invoked in Eq. (10) without derivation or citation; it is a standard result in distillation literature.
  • domain assumption Unlabeled public data sampled 20% per domain approximates the global distribution
    The public dataset D0 is created by random 20% sampling from each of the 5 domains; the ensemble is distilled on this proxy.
  • ad hoc to paper GPT-4 can output useful aggregation weights from local statistics
    LLMWC uses a prompt with training loss, F1, logits, entropy, and trusts the LLM's numeric output; no validation of the LLM's weighting quality is provided besides final accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data." pith.science (2026). https://pith.science/paper/X4VBRL5R

@misc{pith2026250820557,
  author       = {Pith},
  title        = {Pith review of: Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/X4VBRL5R}},
  note         = {Machine review of arXiv:2508.20557}
}
read the original abstract

The widespread success of pre-trained language models has established a new training paradigm, where a global PLM is fine-tuned using task-specific data from local clients. The local data are highly different from each other and can not capture the global distribution of the whole data in real world. To address the challenges of non-IID data in real environments, privacy-preserving federated distillation has been proposed and highly investigated. However, previous experimental non-IID scenarios are primarily identified with the label (output) diversity, without considering the diversity of language domains (input) that is crucial in natural language processing. In this paper, we introduce a comprehensive set of multi-domain non-IID scenarios and propose a unified benchmarking framework that includes diverse data. The benchmark can be used to evaluate the federated learning framework in a real environment. To this end, we propose an Adaptive Federated Distillation (AdaFD) framework designed to address multi-domain non-IID challenges in both homogeneous and heterogeneous settings. Experimental results demonstrate that our models capture the diversity of local clients and achieve better performance compared to the existing works. The code for this paper is available at: https://github.com/jiahaoxiao1228/AdaFD.

Figures

Figures reproduced from arXiv: 2508.20557 by the authors.

Figure 1
Figure 1. Diversity in language distributions across clients. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework of AdaFD. where hyperparameter β affects the smoothness of the weight distribution and is used to adjust the weight allocation in the adaptive ensemble strategy of the model. The weights calculated by the above two strategies are used to aggregate the logits predicted by the clients, resulting in adaptive ensemble logits: zb = X K k=1 w k z k . (8) • Large Language Models Weight Calculation (LLMWC) Due… view at source ↗
Figure 3
Figure 3. The losses and weights of varying data sizes on the central model in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The weights vary with changes in the hyperparameter [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: The training losses (left) and F1 scores (right) of Local and AdaFD [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 30 canonical work pages

  1. [1]

    The singapore personal data protection act and an assess- ment of future trends in data privacy reform,

    W. B. Chik, “The singapore personal data protection act and an assess- ment of future trends in data privacy reform,” Computer Law & Security Review, vol. 29, no. 5, pp. 554–575, 2013

    work page 2013

  2. [2]

    General data protection regulation,

    F. D. Protection, “General data protection regulation,” Intersoft Consult- ing, Accessed in October , vol. 24, no. 1, 2018

    work page 2018

  3. [3]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics . PMLR, 2017, pp. 1273– 1282

    2017

  4. [4]

    Fedbert: When federated learning meets pre-training,

    Y . Tian, Y . Wan, L. Lyu, D. Yao, H. Jin, and L. Sun, “Fedbert: When federated learning meets pre-training,” ACM Transactions on Intelligent Systems and Technology (TIST) , vol. 13, no. 4, pp. 1–26, 2022

    2022

  5. [5]

    Collaborating Heterogeneous Natural Language Processing Tasks via Federated Learning

    C. Dong, Y . Xie, B. Ding, Y . Shen, and Y . Li, “Collaborating heteroge- neous natural language processing tasks via federated learning,” arXiv preprint arXiv:2212.05789, 2022

    work page Pith review arXiv 2022

  6. [6]

    Federated model decomposition with private vocabulary for text classification,

    Z. Zhang, X. Hu, L. Qu, Q. Wang, and Z. Xu, “Federated model decomposition with private vocabulary for text classification,” in Em- pirical Methods in Natural Language Processing 2022 . Association for Computational Linguistics (ACL), 2022, pp. 6413–6425

    work page 2022

  7. [7]

    Fednlp: Benchmarking federated learning methods for natural language processing tasks,

    B. Y . Lin, C. He, Z. Ze, H. Wang, Y . Hua, C. Dupuy, R. Gupta, M. Soltanolkotabi, X. Ren, and S. Avestimehr, “Fednlp: Benchmarking federated learning methods for natural language processing tasks,” in Findings of the Association for Computational Linguistics: NAACL 2022, 2022, pp. 157–175

    work page 2022

  8. [8]

    Flightner: A federated learn- ing approach to lightweight named-entity recognition,

    M. Abadeer, W. Shi, and J.-P. Corriveau, “Flightner: A federated learn- ing approach to lightweight named-entity recognition,” in 2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom) . IEEE, 2022, pp. 687–694

    work page 2022

Show all 44 references
  1. [9]

    FedID: Federated interactive distillation for large-scale pretraining language models,

    X. Ma, J. Liu, J. Wang, and X. Zhang, “FedID: Federated interactive distillation for large-scale pretraining language models,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing . Singapore: Association for Computational Linguistics, Dec....

    2023

  2. [10]

    Communication-efficient on-device machine learning: Federated distil- lation and augmentation under non-iid private data,

    E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Communication-efficient on-device machine learning: Federated distil- lation and augmentation under non-iid private data,” in Proceedings of the 32nd Conference on Neural Information Processing Systems , 2018

    2018

  3. [11]

    Fedmd: Heterogenous federated learning via model distillation,

    D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” in Workshop on Federated Learning for Data Privacy and Confidentiality, NeurIPS 2019 , 2019

    2019

  4. [12]

    Cronus: Ro- bust and heterogeneous collaborative learning with black-box knowledge transfer,

    H. Chang, V . Shejwalkar, R. Shokri, and A. Houmansadr, “Cronus: Ro- bust and heterogeneous collaborative learning with black-box knowledge transfer,” arXiv preprint arXiv:1912.11279 , 2019

    1912 arXiv

  5. [13]

    Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data,

    S. Itahara, T. Nishio, Y . Koda, M. Morikura, and K. Ya- mamoto, “Distillation-based semi-supervised federated learning for communication-efficient collaborative training with non-iid private data,” IEEE Transactions on Mobile Computing , vol. 22, no. 1, pp. 191–205, 2021

    2021

  6. [14]

    Mhat: An efficient model-heterogenous aggregation training scheme for federated learning,

    L. Hu, H. Yan, L. Li, Z. Pan, X. Liu, and Z. Zhang, “Mhat: An efficient model-heterogenous aggregation training scheme for federated learning,” Information Sciences, vol. 560, pp. 493–503, 2021

    2021

  7. [15]

    Preserving privacy in federated learning with ensem- ble cross-domain knowledge distillation,

    X. Gong, A. Sharma, S. Karanam, Z. Wu, T. Chen, D. Doermann, and A. Innanje, “Preserving privacy in federated learning with ensem- ble cross-domain knowledge distillation,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 11, 2022, pp. 11 891– 11 899

    2022

  8. [16]

    Who speaks what language to whom and when?

    J. A. Fishman, “Who speaks what language to whom and when?” in The bilingualism reader . Routledge, 2020, pp. 55–70

    2020

  9. [17]

    Fedner: Privacy- preserving medical named entity recognition with federated learning,

    S. Ge, F. Wu, C. Wu, T. Qi, Y . Huang, and X. Xie, “Fedner: Privacy- preserving medical named entity recognition with federated learning,” arXiv preprint arXiv:2003.09288 , 2020

    2003 arXiv

  10. [18]

    Feded: Federated learning via ensemble distillation for medical relation extraction,

    D. Sui, Y . Chen, J. Zhao, Y . Jia, Y . Xie, and W. Sun, “Feded: Federated learning via ensemble distillation for medical relation extraction,” in Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP) , 2020, pp. 2118–2128

    2020

  11. [19]

    Faster adaptive federated learn- ing,

    X. Wu, F. Huang, Z. Hu, and H. Huang, “Faster adaptive federated learn- ing,” in Proceedings of the AAAI conference on artificial intelligence , vol. 37, no. 9, 2023, pp. 10 379–10 387

    2023

  12. [20]

    Rethinking federated learning with domain shift: A prototype view,

    W. Huang, M. Ye, Z. Shi, H. Li, and B. Du, “Rethinking federated learning with domain shift: A prototype view,” in 2023 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2023, pp. 16 312–16 322

    2023

  13. [21]

    Vertical federated learning: Concepts, advances, and challenges,

    Y . Liu, Y . Kang, T. Zou, Y . Pu, Y . He, X. Ye, Y . Ouyang, Y .-Q. Zhang, and Q. Yang, “Vertical federated learning: Concepts, advances, and challenges,” IEEE Transactions on Knowledge and Data Engineering , pp. 3615–3634, 2024

    2024

  14. [22]

    A robust privacy-preserving federated learning model against model poisoning attacks,

    A. Yazdinejad, A. Dehghantanha, H. Karimipour, G. Srivastava, and R. M. Parizi, “A robust privacy-preserving federated learning model against model poisoning attacks,” IEEE Transactions on Information Forensics and Security, 2024

    2024

  15. [23]

    Federated learning meets natural language processing: A survey,

    M. Liu, S. Ho, M. Wang, L. Gao, Y . Jin, and H. Zhang, “Federated learning meets natural language processing: A survey,” arXiv preprint arXiv:2107.12603, 2021

    2021 arXiv

  16. [24]

    Extracting training data from large language models,

    N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., “Extracting training data from large language models,” in 30th USENIX Security Symposium (USENIX Security 21) , 2021, pp. 2633–2650

    2021

  17. [25]

    Ensemble distillation for robust model fusion in federated learning,

    T. Lin, L. Kong, S. U. Stich, and M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” Advances in neural information processing systems , vol. 33, pp. 2351–2363, 2020

    2020

  18. [26]

    Fedproto: Federated prototype learning across heterogeneous clients,

    Y . Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang, “Fedproto: Federated prototype learning across heterogeneous clients,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 8, 2022, pp. 8432–8440

    2022

  19. [27]

    To distill or not to distill: towards fast, accurate and communication efficient federated distillation learning,

    Y . Zhang, W. Zhang, L. Pu, T. Lin, and J. Yan, “To distill or not to distill: towards fast, accurate and communication efficient federated distillation learning,” IEEE Internet of Things Journal , 2023

    2023

  20. [28]

    Feddkd: Federated learning with decentral- ized knowledge distillation,

    X. Li, B. Chen, and W. Lu, “Feddkd: Federated learning with decentral- ized knowledge distillation,” Applied Intelligence , vol. 53, no. 15, pp. 18 547–18 563, 2023

    2023

  21. [29]

    Dafkd: Domain-aware federated knowledge distillation,

    H. Wang, Y . Li, W. Xu, R. Li, Y . Zhan, and Z. Zeng, “Dafkd: Domain-aware federated knowledge distillation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 20 412–20 421

    2023

  22. [30]

    Fedfed: Feature distillation against data heterogeneity in federated learning,

    Z. Yang, Y . Zhang, Y . Zheng, X. Tian, H. Peng, T. Liu, and B. Han, “Fedfed: Feature distillation against data heterogeneity in federated learning,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  23. [31]

    Federated distillation and blockchain empowered secure knowledge sharing for internet of medical things,

    X. Zhou, W. Huang, W. Liang, Z. Yan, J. Ma, Y . Pan, I. Kevin, and K. Wang, “Federated distillation and blockchain empowered secure knowledge sharing for internet of medical things,” Information Sciences, vol. 662, p. 120217, 2024

    2024

  24. [32]

    Fedal: Black-box federated knowledge distillation enabled by adversarial learning,

    P. Han, X. Shi, and J. Huang, “Fedal: Black-box federated knowledge distillation enabled by adversarial learning,” IEEE Journal on Selected Areas in Communications , 2024

    2024

  25. [33]

    Advances and open problems in federated learning,

    P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings et al. , “Advances and open problems in federated learning,” Foundations and trends® in machine learning , vol. 14, no. 1–2, pp. 1–210, 2021

    2021

  26. [34]

    A survey on federated learning systems: Vision, hype and reality for data privacy and protection,

    Q. Li, Z. Wen, Z. Wu, S. Hu, N. Wang, Y . Li, X. Liu, and B. He, “A survey on federated learning systems: Vision, hype and reality for data privacy and protection,” IEEE Transactions on Knowledge and Data Engineering, vol. 35, no. 4, pp. 3347–3366, 2021

    2021

  27. [35]

    Model-contrastive federated learning,

    Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 10 713–10 722

    2021

  28. [36]

    Addressing class imbalance in federated learning,

    L. Wang, S. Xu, X. Wang, and Q. Zhu, “Addressing class imbalance in federated learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, 2021, pp. 10 165–10 173

    2021

  29. [37]

    Efficient federated domain translation,

    Z. Zhou, S. S. Azam, C. G. Brinton, and D. I. Inouye, “Efficient federated domain translation,” in The Eleventh International Conference on Learning Representations , 2023

    2023

  30. [38]

    Federated learning for speech recognition: Revisiting current trends towards large-scale asr,

    S. S. Azam, M. Pelikan, V . Feldman, K. Talwar, J. Silovsky, and T. Likhomanenko, “Federated learning for speech recognition: Revisiting current trends towards large-scale asr,” in International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with...

    2023

  31. [39]

    Personalized federated learning on non-iid data via group-based meta-learning,

    L. Yang, J. Huang, W. Lin, and J. Cao, “Personalized federated learning on non-iid data via group-based meta-learning,” ACM Transactions on Knowledge Discovery from Data , vol. 17, no. 4, pp. 1–20, 2023

    2023

  32. [40]

    Data independent warmup scheme for non-iid federated learning,

    M. Arafeh, H. Ould-Slimane, H. Otrok, A. Mourad, C. Talhi, and E. Damiani, “Data independent warmup scheme for non-iid federated learning,” Information Sciences, vol. 623, pp. 342–360, 2023

    2023

  33. [41]

    Federated learning with non-iid data: A survey,

    Z. Lu, H. Pan, Y . Dai, X. Si, and Y . Zhang, “Federated learning with non-iid data: A survey,” IEEE Internet of Things Journal , 2024

    2024

  34. [42]

    Feature matching data synthesis for non-iid federated learning,

    Z. Li, Y . Sun, J. Shao, Y . Mao, J. H. Wang, and J. Zhang, “Feature matching data synthesis for non-iid federated learning,” IEEE Transac- tions on Mobile Computing , 2024

    2024

  35. [43]

    Distributions of the kullback–leibler divergence with applications,

    D. I. Belov and R. D. Armstrong, “Distributions of the kullback–leibler divergence with applications,” British Journal of Mathematical and Statistical Psychology, vol. 64, no. 2, pp. 291–309, 2011

    2011

  36. [44]

    Measuring the effects of non- identical data distribution for federated visual classification,

    T.-M. H. Hsu, H. Qi, and M. Brown, “Measuring the effects of non- identical data distribution for federated visual classification,” arXiv preprint arXiv:1909.06335, 2019. APPENDIX A PROMPT TEMPLATE You are an expert in federated learning and federated distillation. Analyze and...

    1909 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.