Pith. sign in

REVIEW 3 major objections 5 minor 51 references

Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FedSSD selectively weights the global model's distilled knowledge by class and sample credibility, improving accuracy and convergence speed in non-IID federated learning.

desk verdict A useful new FL method with solid, mostly positive experiments, but the convergence theorem is algebraically inconsistent and should be fixed or dropped. read the letter →

arxiv 2504.14694 v1 pith:AIUKLZ3O submitted 2025-04-20 cs.LG cs.AI

Yuting He , Yiqiang Chen , XiaoDong Yang , Hanchao Yu , Yi-Hua Huang , Yang Gu This is my paper · ORCID
classification cs.LGcs.AI
keywords federatedlearningnon-IIDdataknowledgedistillationself-distillationclientdriftcatastrophicforgettingheterogeneityconvergenceanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that catastrophic forgetting of global knowledge during local training is a main cause of slow convergence and poor accuracy in federated learning on non-IID data. It proposes FedSSD, a method that makes each client distill the global model's output logits into its local model, but weights that distillation by how credible the global model is for each class and for each sample. Class credibility is measured by recall and confusion on a small server-held auxiliary set; sample credibility is the global model's confidence in the true label. The paper proves convergence under a descent condition and reports that FedSSD beats FedAvg, FedProx, MOON, FedNTD and others in accuracy and speed on CIFAR10, CIFAR100 and TinyImageNet. The central idea is that adaptive credibility weighting is what makes self-distillation work under heterogeneity, while a constant-weight distillation is too blunt.

What carries the argument

The central object is the selective distillation weight vector $M(x)\in\mathbb{R}^K$ (Eq. 4), a per-class, per-sample credibility mask applied element-wise to both teacher and student logits before an MSE. Class credibility is $M_{\text{class}}[k_1] = A_{k_1,k_1}(1-\max_{k\neq k_1} A_{k,k_1})$, computed from the global model's confusion matrix $A_t$ on a small server-side auxiliary set, so it is high when the global model both recalls class $k_1$ and rarely confuses other classes with it. Sample credibility is $M_{\text{sample}}(x) = 1 - (1-p_g(x)[k_2])^{0.5}$, which grows with the global model's predicted probability of the true label $k_2$. The product, clipped and scaled by $M_{\text{max}}$, controls how strongly each logit channel is distilled into the local model.

What would settle it

Run FedSSD on CIFAR10 with the paper's default settings and, at each round, measure for each client the quantity $\|\nabla L_{CE,i}(w_{t+1}^i) + M_{max}L_f(w_{t+1}^i - w_t)\|$ and compare it with $\eta \|\nabla L_i(w_t)\|$ for $\eta < 1$. If a substantial fraction of client updates violate the inequality, the theorem's descent guarantee does not cover the actual optimization trajectory.

Watch

Extended reading notes

Core claim

FedSSD's central discovery is that the global model in federated learning, while better overall than local models, is not uniformly trustworthy as a distillation teacher: its logits vary in reliability across classes and across individual samples, especially in early rounds and on skewed data. The method therefore adds a selective self-distillation loss $L_{SSD} = E(\|M\odot z_g - M\odot z\|_2^2)$ to the local cross-entropy objective, where the channel-wise weight $M(x)[k_1]$ combines a class-level credibility score (recall of class $k_1$ minus the maximum rate at which other classes are mistaken for $k_1$) and a sample-level score (the global model's predicted probability of the true label, mapped to $[0,1]$). Because the weights suppress unreliable logits, local models preserve global knowledge without being misled, and the aggregate model converges in fewer rounds. The paper also proves a non-convex convergence result, adapted from FedProx's theorem, under bounded-dissimilarity and Lipschitz-smoothness assumptions plus a technical condition on the local optimizer.

Load-bearing premise

The convergence theorem relies on Assumption 2, which asserts that after each round of local SGD, the optimized local parameter satisfies a particular norm bound involving the gradient and the distance from the global model; the paper does not verify that ordinary local training actually lands inside that bound.

Editorial extensions

If this is right

  • On non-IID image benchmarks, FedSSD should require fewer communication rounds to reach FedAvg's final accuracy: 33 rounds vs 100 on CIFAR10, 48 vs 100 on CIFAR100, and 10 vs 30 on TinyImageNet in the paper's runs.
  • FedSSD's gains widen as heterogeneity increases, e.g., on CIFAR10 with #K=2 it reports 53.37% versus the best baseline's 50.75%, whereas on near-IID it is within roughly two points, so the method is most useful where client drift is worst.
  • Using MSE logit matching with selective weights should dominate both plain MSE and KL distillation at the same coefficient, as the paper's ablation in Table 3 shows SSD highest across $\alpha$ values.
  • Beyond FedAvg's model transmission, the only extra communication is one $K\times K$ credibility matrix per round, keeping FedSSD communication-cheap relative to methods that transmit Fisher diagonals or control variates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The credibility-weighting scheme is teacher-agnostic: the same class-level and sample-level masks could be applied to any distillation setting (e.g., partially trained teachers in data-free or ensemble distillation) where teacher reliability varies along those two axes.
  • A natural testable extension is to replace the sample-level confidence $p_g(x)[k_2]$ with the teacher's predictive entropy or margin, which would also penalize confidently wrong samples; the paper's square-root mapping only distinguishes confident-correct from unconfident cases.
  • Because the class credibility matrix is estimated on a small auxiliary set, the method inherits sensitivity to that set's composition; one could instead estimate per-class credibility from the uploaded local models or from the server's own predictions on generated data, removing the public-data assumption.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes FedSSD, a federated learning algorithm that adds an adaptive self-distillation regularizer to each client's local training. The regularizer matches the local model's logits to the frozen global model's logits, with the matching weighted by a class-level credibility matrix computed on a server-side auxiliary dataset and by a sample-level confidence score for each local example. The authors motivate the method by showing that local models forget global knowledge and that the global model's reliability varies across classes and samples. The manuscript presents Algorithm 1, a convergence theorem (Theorem 1) based on Assumptions 1 and 2, and experiments on CIFAR-10, CIFAR-100, and TinyImageNet comparing FedSSD with FedAvg, FedProx, FedCurv, MOON, FedNTD, and the authors' prior FedCAD. The central claims are that FedSSD achieves better generalization and robustness in fewer communication rounds than other state-of-the-art FL methods and that its convergence is theoretically guaranteed.

Significance. The FedSSD idea is practically attractive: it requires no extra uplink communication beyond the model update, since the server sends only the model and a K-by-K credibility matrix, and the reported results show consistent improvements on CIFAR-10 and TinyImageNet across several heterogeneity settings. The ablation study and sensitivity analyses are useful and support the claim that the selective weighting mechanism contributes beyond fixed KL or MSE distillation. The paper is also honest in disclosing that it extends the authors' earlier FedCAD paper. However, the theoretical convergence guarantee is not established, because the key assumption used in Theorem 1 is not derived from the actual FedSSD local objective and the proof is delegated to an external theorem in a way that does not match the algorithm. In addition, the CIFAR-100 result in Table 1 contradicts the abstract's broad 'better generalization' claim. The empirical core is promising, but the theoretical claim and the headline wording need substantive revision before the paper can be accepted.

major comments (3)
  1. [Section 4.3, Lemma 1 and Assumption 2] The bridge from the FedSSD objective to the convergence assumption is not established. The actual local loss in Eq. (6) is L_i = L_CE,i + L_SSD,i, where L_SSD,i (Eq. (3)) is a sum of per-sample masked MSE logit losses whose mask M(x) depends on the sample and on the round-dependent credibility matrix. Lemma 1 replaces this with the surrogate L_tilde(w,w_t) = L_CE(w) + M_max^2 L_f^2 ||w - w_t||^2. Three problems arise. First, the coefficient L_f^2 in Lemma 1 is inconsistent with Assumption 1.3 as printed, which yields L_f after squaring the displayed bound, so the lemma's algebra is incorrect. Second, even if the bound were correct, the gradient of L_tilde is grad L_CE(w) + 2 M_max^2 L_f^2 (w - w_t), not grad L_CE(w) + M_max L_f (w - w_t) as Assumption 2 states. Third, Lemma 1 is a function-value bound for an exact minimizer of L_tilde, whereas Algorithm 1 runs E epochs of SGD on L_i, and no argument shows that the resulting w_{t+1}^i satisfies either the lemma's condition or Assumption 2. Assumption 2 is therefore an unverified postulate, and Theorem 1's descent guarantee does not follow from the stated premises.
  2. [Section 4.3, Theorem 1] No proof of Theorem 1 appears in the manuscript; the sentence 'The theorem derives from the Theorem 4 in [8]' is not a proof. FedProx's Theorem 4 relies on an inexactness condition for a proximal objective with a fixed coefficient, whereas FedSSD's regularizer is sample-dependent and is not a single proximal term, so the reduction to FedProx is not immediate. The authors should either provide a self-contained proof that verifies the required assumptions for Algorithm 1 or clearly state that the theoretical analysis is conditional on an unverified Assumption 2; in the latter case the advertised 'convergence guarantee' in the abstract should be removed.
  3. [Table 1, CIFAR-100 row] FedSSD's final global test accuracy on CIFAR-100 (66.19 ± 0.28) is lower than MOON (67.32 ± 0.74), yet the abstract claims that FedSSD 'achieves better generalization ... compared with other state-of-the-art FL methods' and Section 5.2 states a 'similar conclusion' for deeper networks. The paper's caveat that average local accuracy and convergence speed are higher does not rescue the global-accuracy claim. Please qualify the claim to the datasets and settings where it holds, or provide a statistical comparison showing that the 1.13-point deficit is not significant; as written, the central claim is contradicted by the paper's own table.
minor comments (5)
  1. [Section 4.3, Assumption 1.1] The quantifier in Assumption 1.1 is garbled; it should read 'For each client i in [N] and any parameter w in R^d, E_i[||grad L_i(w)||^2] <= ||grad L(w)||^2 B^2.'
  2. [Section 4.3, Assumption 1.3 and Lemma 1] The inequality in Assumption 1.3 as printed uses sqrt(2 L_f), but Lemma 1's derivation requires a factor of L_f^2 after squaring; the assumption and the lemma should be made dimensionally consistent.
  3. [Section 4.2, text near Eq. (4)] 'Global locit' is a typo for 'global logit'.
  4. [Table 1] The T entries for FedCurv, FedNTD, and FedCAD on CIFAR-100 appear to be missing from the table.
  5. [Section 5.1 and Section 5.2] The method relies on a server-side auxiliary dataset D_V with 64 samples per class, but the paper does not discuss the privacy and availability assumptions for this dataset or the sensitivity of the results to its composition; a discussion would help readers judge the method's practical scope.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: FedSSD's empirical claims are benchmarked externally, and the only self-citations (to FedCAD) are contextual rather than load-bearing.

full rationale

Walking the derivation chain, FedSSD's local objective (Eq. 6) is L_i = L_CE,i + L_SSD,i, with L_SSD defined in Eq. 3 as a weighted MSE between global and local logits. The weights M in Eq. 4 are computed from the global model's confusion matrix on an auxiliary server dataset and from the global model's predicted probability for the true class; they are not fitted to the reported test accuracies, and the reported convergence times and accuracies are measured against external baselines. The proof of Theorem 1 explicitly states that it derives from FedProx's Theorem 4, an external result; the stated Assumptions 1 and 2 are conditions, not the paper's own prior claims. The self-citation to [11] (FedCAD) appears only in the extension description and as a baseline, so it carries no load-bearing argument. The most serious weakness is in Section 4.3: Assumption 2 is asserted to be well-posed based on Lemma 1, but the paper does not show that an SGD update in Algorithm 1 satisfies the inequality, and the linear term M_max L_f (w_{t+1}^i - w_t) is not the gradient of the quadratic surrogate in Lemma 1 (which would be 2 M_max^2 L_f^2 (w_{t+1}^i - w_t)). This is a correctness gap, not a circular reduction: the convergence conclusion is not identical to the premise, and no fitted constant is renamed as a prediction. Since no claim reduces to its inputs by construction, the circularity score is low (2), reflecting only the minor and non-load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The algorithm introduces one tuned hyperparameter Mmax and several hand-chosen constants. The convergence proof rests on strong assumptions (Lipschitz, bounded dissimilarity, and a custom Assumption 2) that are not verified empirically. No new physical or mathematical entities are postulated.

free parameters (3)
  • Mmax (upper bound of selective distillation loss) = 0.01 (CIFAR10), 0.001 (CIFAR100), 0.001 (TinyImageNet)
    Tuned per dataset; Section 5.1 states the best upper bound Mmax for CIFAR10, CIFAR100, TinyImageNet are 0.01, 0.001 and 0.001; Table 3 shows sensitivity to this coefficient.
  • Msample exponent = 0.5
    Hand-chosen exponent in Msample(x) = 1 - (1 - pg(x)[k2])^0.5 in Eq. (4); no sensitivity analysis is provided.
  • Bias offset in class-wise weight = -0.1
    Hand-chosen constant inside the positive part of Eq. (4); no justification or ablation is given.
assumptions (5)
  • domain assumption Assumption 1.1: bounded dissimilarity between local gradients and global gradient (||∇Li(w)||^2 ≤ ||∇L(w)||^2 B^2)
    Standard but strong in heterogeneous FL; the bound B may be large under extreme non-IID splits.
  • domain assumption Assumption 1.2: each local objective is L-Lipschitz smooth and its Hessian is lower bounded by Lm I
    Standard smoothness condition; not verified for deep networks used in experiments.
  • domain assumption Assumption 1.3: the logits function f(w,x) is Lipschitz continuous with constant Lf
    Assumed to hold when inputs and parameters are bounded; used in Lemma 1.
  • ad hoc to paper Assumption 2: the locally optimized parameter satisfies ||∇L_CE,i(w_{t+1}^i) + M_{max} L_f (w_{t+1}^i - w_t)|| ≤ η ||∇L_i(w_t)||
    This is a custom condition designed to make Theorem 1 hold; it is not derived from the SGD update and is not empirically validated.
  • domain assumption The server-side auxiliary dataset D_V is representative of the global data distribution
    The credibility matrix A^t = P(w_t, D_V) is computed on D_V; if D_V is not representative, the class-level weights are unreliable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data." pith.science (2026). https://pith.science/paper/AIUKLZ3O

@misc{pith2026250414694,
  author       = {Pith},
  title        = {Pith review of: Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AIUKLZ3O}},
  note         = {Machine review of arXiv:2504.14694}
}
read the original abstract

Federated learning (FL) enables multiple clients to collaboratively train a global model while keeping local data decentralized. Data heterogeneity (non-IID) across clients has imposed significant challenges to FL, which makes local models re-optimize towards their own local optima and forget the global knowledge, resulting in performance degradation and convergence slowdown. Many existing works have attempted to address the non-IID issue by adding an extra global-model-based regularizing item to the local training but without an adaption scheme, which is not efficient enough to achieve high performance with deep learning models. In this paper, we propose a Selective Self-Distillation method for Federated learning (FedSSD), which imposes adaptive constraints on the local updates by self-distilling the global model's knowledge and selectively weighting it by evaluating the credibility at both the class and sample level. The convergence guarantee of FedSSD is theoretically analyzed and extensive experiments are conducted on three public benchmark datasets, which demonstrates that FedSSD achieves better generalization and robustness in fewer communication rounds, compared with other state-of-the-art FL methods.

Figures

Figures reproduced from arXiv: 2504.14694 by the authors.

Figure 1
Figure 1. The framework of FedAvg. difficult [3], [4], but also leads to performance degrada￾tion and convergence slowdown [5], [6]. Specifically, when each client trains a local model on biased data, it will be re-optimized towards the local optima and deviate from the global objective. This causes a drift in the client local updates, which is called ”Client-drift” [7]. Moreover, the server averages the divergent local model… view at source ↗
Figure 2
Figure 2. The catastrophic forgetting issue on non-IID CI [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Examples of confusion matrices for training on local [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: An overview of FedSSD in the heterogeneous setting. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Visualization of statistical heterogeneity among clients, where the x-axis indicates client IDs, the y-axis indicates [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: The learning curves on the benchmark datasets with default settings. The global test accuracy is the performance [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: The learning curves on CIFAR10 and CIFAR100 with different client sample ratios. [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Visualized the final global model’s performance with [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 29 canonical work pages

  1. [8]

    Federated optimization in heterogeneous networks,

    T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V . Smith, “Federated optimization in heterogeneous networks,” in Proceedings of Machine Learning and Systems (MLSys) , vol. 2, 2020, pp. 429–450

    2020

  2. [1]

    The eu general data protec- tion regulation (gdpr),

    P . Voigt and A. Von dem Bussche, “The eu general data protec- tion regulation (gdpr),” A Practical Guide, 1st Ed., Cham: Springer International Publishing, vol. 10, no. 3152676, pp. 10–5555, 2017

    2017

  3. [2]

    Communication-efficient learning of deep networks from decentralized data,

    B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Ar- cas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics . PMLR, 2017, pp. 1273–1282

    2017

  4. [3]

    Tighter theory for local sgd on identical and heterogeneous data,

    A. Khaled, K. Mishchenko, and P . Richt ´arik, “Tighter theory for local sgd on identical and heterogeneous data,” in International Conference on Artificial Intelligence and Statistics . PMLR, 2020, pp. 4519–4529. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 11

    work page 2020

  5. [4]

    On the conver- gence of fedavg on non-iid data,

    X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the conver- gence of fedavg on non-iid data,” in 8th International Conference on Learning Representations (ICLR), 2020

    work page 2020

  6. [5]

    Feder- ated learning with non-iid data,

    Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V . Chandra, “Feder- ated learning with non-iid data,” arXiv preprint arXiv:1806.00582 , 2018

    arXiv 2018

  7. [6]

    Federated learning: Challenges, methods, and future directions,

    T. Li, A. K. Sahu, A. Talwalkar, and V . Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Process- ing Magazine, vol. 37, no. 3, pp. 50–60, 2020

    work page 2020

  8. [7]

    Scaffold: Stochastic controlled averaging for federated learning,

    S. P . Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in International Conference on Machine Learning (ICML) . PMLR, 2020, pp. 5132–5143

    work page 2020

Show all 51 references
  1. [9]

    Model-contrastive federated learning,

    Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), 2021, pp. 10 713–10 722

    2021

  2. [10]

    Overcoming forgetting in federated learning on non-iid data,

    N. Shoham, T. Avidor, A. Keren, N. Israel, D. Benditkis, L. Mor- Yosef, and I. Zeitak, “Overcoming forgetting in federated learning on non-iid data,” arXiv preprint arXiv:1910.07796, 2019

    1910 arXiv

  3. [11]

    Class-wise adaptive self distillation for heterogeneous federated learning,

    Y. He, Y. Chen, X. Yang, Y. Zhang, and B. Zeng, “Class-wise adaptive self distillation for heterogeneous federated learning,” in International Workshop on Trustable, Verifiable and Auditable Federated Learning in Conjunction with AAAI 2022 (FL-AAAI-22) , 2022

    2022

  4. [12]

    Tackling the objective inconsistency problem in heterogeneous federated optimization,

    J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V . Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” Advances in neural information processing systems , vol. 33, pp. 7611–7623, 2020

    2020

  5. [13]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska et al. , “Overcoming catastrophic forgetting in neural networks,” Proceedings of the national academy of sciences , vol. 114, no. 13, pp. 3521–3...

    2017

  6. [14]

    Bayesian nonparametric federated learning of neural networks,

    M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y. Khazaeni, “Bayesian nonparametric federated learning of neural networks,” in International Conference on Machine Learning . PMLR, 2019, pp. 7252–7261

    2019

  7. [15]

    Federated learning with matched averaging,

    H. Wang, M. Yurochkin, Y. Sun, D. S. Papailiopoulos, and Y. Khaz- aeni, “Federated learning with matched averaging,” in 8th Inter- national Conference on Learning Representations (ICLR), 2020

    2020

  8. [16]

    Inverse distance aggregation for federated learning with non-iid data,

    Y. Yeganeh, A. Farshad, N. Navab, and S. Albarqouni, “Inverse distance aggregation for federated learning with non-iid data,” in Domain Adaptation and Representation Transfer, and Distributed and Collaborative Learning. Springer, 2020, pp. 150–159

    2020

  9. [17]

    Adaptive personalized federated learning,

    Y. Deng, M. M. Kamani, and M. Mahdavi, “Adaptive personalized federated learning,” arXiv preprint arXiv:2003.13461, 2020

    2003 arXiv

  10. [18]

    Fedhealth 2: Weighted fed- erated transfer learning via batch normalization for personalized healthcare,

    Y. Chen, W. Lu, J. Wang, and X. Qin, “Fedhealth 2: Weighted fed- erated transfer learning via batch normalization for personalized healthcare,” arXiv preprint arXiv:2106.01009, 2021

    2021 arXiv

  11. [19]

    Personalized cross-silo federated learning on non-iid data,

    Y. Huang, L. Chu, Z. Zhou, L. Wang, J. Liu, J. Pei, and Y. Zhang, “Personalized cross-silo federated learning on non-iid data,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 35, no. 9, 2021, pp. 7865–7873

    2021

  12. [20]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  13. [21]

    Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,

    W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou, “Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers,” Advances in Neural Information Process- ing Systems, vol. 33, pp. 5776–5788, 2020

    2020

  14. [22]

    Patient knowledge dis- tillation for bert model compression,

    S. Sun, Y. Cheng, Z. Gan, and J. Liu, “Patient knowledge dis- tillation for bert model compression,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing . Associatio...

    2019

  15. [23]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation,

    L. Zhang, J. Song, A. Gao, J. Chen, C. Bao, and K. Ma, “Be your own teacher: Improve the performance of convolutional neural networks via self distillation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), 2019, pp. 3713– 3722

    2019

  16. [24]

    Regularizing class-wise predictions via self-knowledge distillation,

    S. Yun, J. Park, K. Lee, and J. Shin, “Regularizing class-wise predictions via self-knowledge distillation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2020, pp. 13 876–13 885

    2020

  17. [25]

    Fitnets: Hints for thin deep nets,

    A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, “Fitnets: Hints for thin deep nets,” in 3rd International Conference on Learning Representations (ICLR), 2015

    2015

  18. [26]

    A comprehensive overhaul of feature distillation,

    B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi, “A comprehensive overhaul of feature distillation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR) , 2019, pp. 1921–1930

    2019

  19. [27]

    Paraphrasing complex network: Network compression via factor transfer,

    J. Kim, S. Park, and N. Kwak, “Paraphrasing complex network: Network compression via factor transfer,” Advances in neural infor- mation processing systems, vol. 31, 2018

    2018

  20. [28]

    Understanding and improving knowledge distillation,

    J. Tang, R. Shivanna, Z. Zhao, D. Lin, A. Singh, E. H. Chi, and S. Jain, “Understanding and improving knowledge distillation,” arXiv preprint arXiv:2002.03532, 2020

    2002 arXiv

  21. [29]

    Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective,

    H. Zhou, L. Song, J. Chen, Y. Zhou, G. Wang, J. Yuan, and Q. Zhang, “Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective,” in 9th International Conference on Learning Representations (ICLR), 2021

    2021

  22. [30]

    Knowledge distillation as semiparametric inference,

    T. Dao, G. M. Kamath, V . Syrgkanis, and L. Mackey, “Knowledge distillation as semiparametric inference,” in 9th International Con- ference on Learning Representations (ICLR), 2021

    2021

  23. [31]

    Teacher’s pet: understanding and mitigating biases in distilla- tion,

    M. Lukasik, S. Bhojanapalli, A. K. Menon, and S. Kumar, “Teacher’s pet: understanding and mitigating biases in distilla- tion,” arXiv preprint arXiv:2106.10494, 2021

    2021 arXiv

  24. [32]

    Selective knowledge dis- tillation for neural machine translation,

    F. Wang, J. Yan, F. Meng, and J. Zhou, “Selective knowledge dis- tillation for neural machine translation,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 20...

    2021

  25. [33]

    Online distilling from checkpoints for neural machine translation,

    H.-R. Wei, S. Huang, R. Wang, X. Dai, and J. Chen, “Online distilling from checkpoints for neural machine translation,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019, pp. ...

    2019

  26. [34]

    Multilingual neural machine translation with knowledge distillation,

    X. Tan, Y. Ren, D. He, T. Qin, Z. Zhao, and T. Liu, “Multilingual neural machine translation with knowledge distillation,” in 7th International Conference on Learning Representations (ICLR), 2019

    2019

  27. [35]

    Comparing kullback- leibler divergence and mean squared error loss in knowledge distillation,

    T. Kim, J. Oh, N. Kim, S. Cho, and S.-Y. Yun, “Comparing kullback- leibler divergence and mean squared error loss in knowledge distillation,” arXiv preprint arXiv:2105.08919, 2021

    2021 arXiv

  28. [36]

    Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data,

    E. Jeong, S. Oh, H. Kim, J. Park, M. Bennis, and S.-L. Kim, “Communication-efficient on-device machine learning: Federated distillation and augmentation under non-iid private data,” arXiv preprint arXiv:1811.11479, 2018

    2018 arXiv

  29. [37]

    Federated knowledge distillation,

    H. Seo, J. Park, S. Oh, M. Bennis, and S.-L. Kim, “Federated knowledge distillation,” arXiv preprint arXiv:2011.02367, 2020

    2011 arXiv

  30. [38]

    Fedmd: Heterogenous federated learning via model distillation,

    D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” arXiv preprint arXiv:1910.03581, 2019

    1910 arXiv

  31. [39]

    Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer,

    H. Chang, V . Shejwalkar, R. Shokri, and A. Houmansadr, “Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer,” arXiv preprint arXiv:1912.11279, 2019

    1912 arXiv

  32. [40]

    Ensemble distillation for robust model fusion in federated learning,

    T. Lin, L. Kong, S. U. Stich, and M. Jaggi, “Ensemble distillation for robust model fusion in federated learning,” in Advances in Neural Information Processing Systems, 2020

    2020

  33. [41]

    Preservation of the global knowledge by not-true self knowledge distillation in federated learning,

    G. Lee, Y. Shin, M. Jeong, and S.-Y. Yun, “Preservation of the global knowledge by not-true self knowledge distillation in federated learning,” arXiv preprint arXiv:2106.03097, 2021

    2021 arXiv

  34. [42]

    Local-global knowledge distillation in heterogeneous federated learning with non-iid data,

    D. Yao, W. Pan, Y. Dai, Y. Wan, X. Ding, H. Jin, Z. Xu, and L. Sun, “Local-global knowledge distillation in heterogeneous federated learning with non-iid data,” arXiv preprint arXiv:2107.00051, 2021

    2021 arXiv

  35. [43]

    Acceleration of federated learning with alleviated forgetting in local training,

    C. Xu, Z. Hong, M. Huang, and T. Jiang, “Acceleration of federated learning with alleviated forgetting in local training,” arXiv preprint arXiv:2203.02645, 2022

    2022 arXiv

  36. [44]

    The stability-plasticity dilemma: Investigating the continuum from catastrophic forget- ting to age-limited learning effects,

    M. Mermillod, A. Bugaiska, and P . Bonin, “The stability-plasticity dilemma: Investigating the continuum from catastrophic forget- ting to age-limited learning effects,” Frontiers in psychology, vol. 4, p. 504, 2013

    2013

  37. [45]

    Harmofl: Harmonizing local and global drifts in federated learning on heterogeneous medical images,

    M. Jiang, Z. Wang, and Q. Dou, “Harmofl: Harmonizing local and global drifts in federated learning on heterogeneous medical images,” arXiv preprint arXiv:2112.10775, 2021

    2021 arXiv

  38. [46]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al. , “Learning multiple layers of features from tiny images,” Tech Report, 2009

    2009

  39. [47]

    Tiny imagenet visual recognition challenge,

    Y. Le and X. Yang, “Tiny imagenet visual recognition challenge,” CS 231N, vol. 7, no. 7, p. 3, 2015

    2015

  40. [48]

    Federated learning on non-iid data silos: An experimental study,

    Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,”arXiv preprint arXiv:2102.02079, 2021. JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015 12

    2021 arXiv

  41. [49]

    Addressing class imbalance in federated learning,

    L. Wang, S. Xu, X. Wang, and Q. Zhu, “Addressing class imbalance in federated learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, 2021, pp. 10 165–10 173

    2021

  42. [50]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2016, pp. 770–778

    2016

  43. [51]

    Pytorch: An im- perative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga et al., “Pytorch: An im- perative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, pp. 8026–8037, 2019

    2019

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.