Pith. sign in

REVIEW 5 major objections 8 minor 54 references

DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer

T0 review · 5 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DeepKD splits distillation gradients into three noise-weighted streams and reports consistent accuracy gains across CIFAR-100, ImageNet, and MS-COCO.

desk verdict A broadly applicable distillation trainer with consistent empirical gains, but the claimed GSNR theory is asserted rather than derived and the gradient formulas in Sec. 3.2 don't match the appendix. read the letter →

arxiv 2505.15133 v1 pith:4VCGMWYD submitted 2025-05-21 cs.CV cs.AIcs.LG

Haiduo Huang , Jiangcheng Song , Yadong Zhang , Pengju Ren This is my paper
classification cs.CVcs.AIcs.LG MSC 68T0768T45
keywords knowledgedistillationgradientsignal-to-noiseratiomomentumdecouplingdynamictop-kmaskdarkcurriculumlearninglogitmodelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that knowledge distillation fails when task-oriented, target-class, and non-target-class gradient flows are optimized with shared momentum, because the three flows have different reliabilities and interfere with one another. DeepKD therefore decouples the student update into three momentum buffers, with the momentum coefficient of each buffer scaled by the gradient signal-to-noise ratio (GSNR) of that component. It adds a dynamic top-k mask that progressively includes more non-target classes during training, filtering out low-confidence logits that act as noise. If correct, the framework converts a generic distillation loss into a curriculum over knowledge sources and improves any logit-based distillation method it is attached to.

What carries the argument

The load-bearing object is the gradient signal-to-noise ratio, defined as $\mathrm{GSNR} = \|\mathbb{E}[g]\|_2^2 / \mathrm{Var}[g]$, estimated from a short temporal window of gradients. It supplies the paper's criterion for splitting the student update in Eq. (7) into three momentum streams, and it is paired with the dynamic top-k mask of Eqs. (8)--(9), which selects the $K_i$ largest non-target teacher logits each iteration. The momentum allocation is what separates the knowledge flows; the mask is what removes low-confidence dark knowledge during early training.

What would settle it

Train the same student-teacher pairs on CIFAR-100 with momentum coefficients deliberately inverted relative to the measured GSNR ordering, giving small momentum to NCG and TOG and large momentum to TCG while keeping the top-k mask unchanged; if accuracy stays at the DeepKD level, the GSNR-to-momentum rule is not what drives the gains.

Watch

Extended reading notes

Core claim

The central claim is that a distillation trainer should treat task-learning gradients, target-class distillation gradients, and non-target-class distillation gradients as separate optimization streams rather than as one summed loss. The paper derives closed-form gradient expressions for the three components, estimates each component's GSNR over a sliding window, and assigns momentum $\mu+\Delta$ to the task-oriented and non-target-class streams and $\mu-\Delta$ to the target-class stream, on the observation that their GSNRs are ordered that way. Combined with a dynamic top-k mask (DTM) that grows from 5% of classes to all classes over training, the resulting DeepKD framework reports gains over baseline KD, DKD, MLKD, and CRLD on CIFAR-100, ImageNet-1K, and MS-COCO object detection, with the largest single gain in the paper being +4.15% top-1 on ResNet50-to-MobileNet-V1.

Load-bearing premise

Everything rests on the hypothesis that a gradient component's optimal momentum grows with its signal-to-noise ratio, an ordering the paper observes empirically but never derives.

Editorial extensions

If this is right

  • Attaching DeepKD to a logit-based distillation method should yield the reported gains without changing the teacher or the student architecture, since the framework only rewires the gradient update and masks logits.
  • The dynamic top-k schedule implies that early training should trust only a few semantically close non-target classes and later expand, so the student is protected from noise in the easy phase.
  • Because only first-order momentum is modified, the decoupling should carry over to Adam-style optimizers, as shown for DeiT-Tiny on ImageNet.
  • The consistent gains on MS-COCO indicate the same decoupling applies to dense prediction tasks, not just image classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the GSNR-ordering rule is causal, it suggests a cheap diagnostic: compute per-component GSNR early in training and expect the method to help exactly when NCG and TOG rank above TCG; when that ordering is absent, gains should vanish.
  • The same momentum-allocation rule could be applied to feature-distillation losses, treating each alignment loss as another gradient stream, which the authors mention as future work but do not test.
  • The curriculum over K could be made adaptive per sample or per class rather than global, which would be a direct test of whether semantic closeness, not just confidence rank, drives the benefit.
  • A comparison of dynamic top-k against the best static top-k at every phase would separate the contribution of the curriculum schedule from the contribution of masking itself, a distinction the paper does not resolve.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. The paper proposes DeepKD, a knowledge distillation training framework with two main components: (1) a dual-level gradient decoupling that splits the distillation loss into task-oriented (TOG), target-class (TCG), and non-target-class (NCG) gradients and assigns them different momentum coefficients based on an observed positive relationship between gradient signal-to-noise ratio (GSNR) and optimal momentum; and (2) a dynamic top-k mask (DTM) that progressively filters low-confidence non-target logits from teacher and student outputs according to a curriculum schedule. The authors report extensive experiments on CIFAR-100, ImageNet-1K, and MS-COCO, showing consistent accuracy improvements when DeepKD is combined with several logit-based distillation baselines (KD, DKD, MLKD, CRLD).

Significance. If the empirical results hold, the paper demonstrates a broadly applicable and simple recipe for improving logit-based knowledge distillation: decoupled momentum buffers for the three gradient components plus a curriculum-based top-k filter. The experimental coverage is strong, spanning homogeneous and heterogeneous architectures, ImageNet, and object detection, and the code is promised publicly. However, the paper's advertised theoretical contribution—that momentum coefficients should be positively related to GSNR—is not derived, and the main-text gradient formulas conflict with the appendix derivation. The DTM contribution is interesting but its ablation contains unexplained outliers. The potential practical value is real, but the theoretical framing and the consistency of the technical presentation need substantial work.

major comments (5)
  1. [Section 3.2, Eqs. (4)-(7)] The central claim that optimal momentum coefficients should be positively related to GSNR is asserted rather than derived. The text first states 'We hypothesize that gradient components with higher SNR should be prioritized with heuristic weighting' and then, based on the observed GSNR ordering in Figure 2(a), assigns μ+Δ to TOG and NCG and μ−Δ to TCG in Eq. (7). No argument is made connecting the GSNR definition in Eq. (4) to the momentum update in Eq. (6), and the cited references ([20], [22], [21]) are not used to justify the direction of the relation. This is load-bearing because the entire 'GSNR-driven momentum allocation' is presented as the paper's theoretical contribution. Please either provide a derivation or explicitly reframe the contribution as an empirical heuristic and soften the abstract and Section 2 accordingly.
  2. [Section 3.2, Eq. (5) vs Appendix A.2, Eqs. (22) and (31)] The gradient formulas in the main text do not match the appendix. Eq. (5) gives TCG_j = −pS_j(pS_j − pT_j) for j≠t and NCG_j = pS_j − pT_j for j≠t, whereas the appendix derives TCG_j = −\hat{pS}_j(pS_t − pT_t) (Eq. 22) and NCG_j = \hat{pS}_j − \hat{pT}_j (Eq. 31). Since the GSNR ordering in Figure 2(a) motivates the momentum assignment in Eq. (7), the discrepancy matters: if the GSNR curves were computed with the appendix formulas, the main text must be corrected, and the authors should confirm that the momentum assignment remains valid under the corrected formulas. If the curves were computed with the main-text formulas, the appendix should be reconciled.
  3. [Table 5, right panel] The dynamic top-k ablation contains anomalous and unexplained results that undermine the robustness claim. For example, (k=55, Phase1=40, Phase2=170) gives top-1 = 69.98, far below the KD baseline of 73.33, while (k=60, Phase1=40, Phase2=170) gives 77.20; similarly, (k=60, Phase1=60, Phase2=170) gives 70.19, while (k=60, Phase1=60, Phase2=160) gives 77.29. These discontinuities suggest either a typo, a row/column misalignment, or a genuine sensitivity of the curriculum schedule that requires discussion. The authors should correct the table and either demonstrate robustness or qualify the claim in Section 5.
  4. [Section 5, Table 5 (left panel) and Section 4.1] The momentum difference Δ is tuned per method and dataset: Δ=0.075 for KD+DeepKD on CIFAR-100, Δ=0.05 for DKD/MLKD/CRLD+DeepKD on CIFAR-100, and Δ=0.05 for all ImageNet variants. The paper states that DeepKD introduces only one hyperparameter, but this hyperparameter is not shared across configurations. The sensitivity evidence is limited to one teacher-student pair (Table 5). Please provide additional sensitivity results for the main configurations or give a principled selection rule for Δ; otherwise the practical contribution of the decoupling mechanism is weakened.
  5. [Table 3 vs Appendix Table 9] There are small numerical inconsistencies between the main-text ImageNet table and the appendix version. For instance, DKD+Ours (w. top-k) for RegNetY-16GF/DeiT-Tiny reports top-5 = 93.36 in Table 3 but 93.35 in Table 9, and MLKD+Ours (w. top-k) reports 93.73 vs 93.75. These differences suggest the tables were generated from different runs or contain typos. All numbers should be reconciled and a single set of results used consistently.
minor comments (8)
  1. [Eq. (7) and Algorithm 1] The notation in Eq. (7) is ambiguous because v appears on both sides of the assignment; please use explicit subscripts such as v_new and v_old, as in Algorithm 1, to avoid confusion.
  2. [Section 3.2, last paragraph] The statement that DeepKD 'is equally applicable to the Adam optimizer by modifying only its first-order momentum' is not supported by any experiment or detailed derivation; please either provide evidence or mark it as a conjecture.
  3. [Figure 1(a) and Figure 2] The GSNR and BSNR subplots use different y-axis scales across methods, making the visual claim of 'higher GSNR' for DeepKD hard to assess; please use a consistent scale or annotate the ranges.
  4. [References] Several references are duplicated (e.g., [31] vs [55], [32] vs [56], [33] vs [51], [53] vs [57], [58] vs [11]); the bibliography should be cleaned and duplicates removed.
  5. [Section 4.2] The claim 'Performance remains stable (variance ≤0.5%) under hyperparameter variations' is not supported by any reported variance or error bar; please quantify or point to the relevant table.
  6. [Section 3.3, Eq. (8)] The rank() notation is not defined clearly: the text says 'rank of logits in ascending order,' but the mask should select the largest K logits. Please clarify the ordering convention so that Eq. (8) is unambiguous.
  7. [Section 2] The claim of establishing 'the first theoretical connection between GSNR and momentum allocation in KD' is an overstatement given the absence of a derivation; please qualify this statement if the theoretical claim is downgraded.
  8. [Appendix A.3] The subsection 'Additional Results' is empty; please either populate it with the stated results or remove the heading.

Circularity Check

1 steps flagged · score 6.0 of 10

GSNR-momentum 'validation' restates the design rule; Tables 1-4 remain independent external evidence.

  1. fitted input called prediction [Section 3.2, Eq. (7), Figures 2(a)-2(d)]
    "Through analysis of the GSNR in Figure 2(a), we observe that NCG and TOG maintain higher GSNR compared to TCG. This key observation motivates our adaptive momentum allocation strategy: vTOG = T OG+ (µ + ∆)vTOG; vTCG = T CG+ (µ − ∆)vTCG; vNCG = N CG+ (µ + ∆)vNCG ... As shown in Figure 2(b) & (d), our DeepKD with different momentum coefficients achieves significantly improved GSNR in both gradient buffers and raw gradients, further validating the necessity of our deep momentum decoupling approach for gradient components."

    The allocation in Eq. 7 is chosen directly from the observed GSNR ordering in Fig. 2(a): TOG and NCG have higher GSNR and receive µ+∆, while TCG has lower GSNR and receives µ−∆. The paper then cites the resulting GSNR traces in Fig. 2(b)/(d) and the sentence 'Empirical validation further corroborates that momentum coefficients for gradient components positively correlate with their respective SNRs' as evidence for the proposition. But this is the same correlation that was used to set the coefficients in the first place, so the 'validation' is not an independent test of the hypothesis; it restates the design rule. No derivation links GSNR to optimal momentum; Eq. 7 simply encodes the observed ordering. Thus the advertised theoretical finding ('optimal momentum coefficients ...

full rationale

The single genuine circular step is in Section 3.2: the momentum coefficients are read off the GSNR ordering of vanilla KD in Fig. 2(a), and then the improved GSNR of the resulting method (Figs. 2(b), 2(d)) is presented as validating the proposition that momentum should be positively related to GSNR. That is a fitted input called prediction: the positive correlation is enforced by the construction of Eq. 7 before it is 'empirically corroborated.' The cited stochastic-optimization sources [20,22] are not connected to the sign of ∆, so the claimed 'theoretical analysis' is actually an untested heuristic. This circularity is partial: the accuracy claims in Tables 1-4 are external test-set results against fixed baselines and are not forced by the momentum rule, so the paper's empirical contribution retains independent content. An additional non-circular correctness concern is that Eq. 5 states TCG_j = -pS_j(pS_j-pT_j) and NCG_j = pS_j-pT_j, whereas Appendix A.2 derives TCG_j = -hat(pS)_j(pS_t-pT_t) and NCG_j = hat(pS)_j-hat(pT)_j; because Fig. 2(a) is the sole motivation for the momentum assignment, this inconsistency makes the observed GSNR ordering non-reproducible from the paper's own equations. Hyperparameter choices (∆, k-value, Phase1/Phase2) are tuned on validation data and reported as ablation findings, which is standard practice rather than a circular prediction. Overall score 6 reflects one load-bearing 'prediction' that reduces to its input, while the headline empirical gains remain independently measured.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities. It relies on one unproven core heuristic (GSNR-to-momentum proportionality), one unproven denoising assumption (low-confidence logits are noise), and several hand-set scheduling hyperparameters (Δ, k_opt, phase boundaries, sampling interval, initial k).

free parameters (5)
  • Δ (momentum difference) = 0.075 for KD+DeepKD on CIFAR-100; 0.05 for DKD/MLKD/CRLD+DeepKD on CIFAR-100 and for ImageNet variants
    Chosen per method and dataset following DOT; controls the momentum split between TCG (μ-Δ) and TOG/NCG (μ+Δ), Section 5.
  • Static top-k value (k_opt) = 55 for CIFAR-100 in the ablation; determined per experiment via ablation or 20% of training data
    The dynamic top-k schedule interpolates to k_opt during the transition phase, so its value directly affects which classes are masked, Section 3.3 and Section 5.
  • Phase boundaries (Phase1, Phase2) = 60 and 170 epochs for 240-epoch CIFAR-100 runs; ratios 0.3 and 0.7 of total epochs in Algorithm 2
    The curriculum schedule's easy/transition/hard phases are fixed fractions chosen empirically, Section 3.3 and Algorithm 2.
  • GSNR sampling interval = 200 iterations
    'Empirically, we find that gradient sampling at intervals of 200 iterations yields better performance', Section 3.2. This window is used to estimate the GSNR that drives the momentum allocation.
  • Initial k ratio = 5% of total classes
    The mask starts at 5% of classes and expands; this starting point is set by hand, Section 3.3 and Algorithm 2.
assumptions (5)
  • standard math Softmax derivative identities and the KL decomposition of the KD loss are valid.
    Used in Appendix A.2 (Equations 13-31) to derive the TOG, TCG, and NCG gradient expressions; these are standard calculus results.
  • domain assumption Gradient noise is zero-mean.
    Appendix A.2, Eq. (33): 'Suppose the noise is zero-mean: E[n_t]=0'. This justifies treating the sample mean as the signal.
  • ad hoc to paper Momentum coefficients should be positively related to GSNR.
    Section 3.2: 'We hypothesize that gradient components with higher SNR should be prioritized with heuristic weighting.' This is the paper's core heuristic and has no derivation.
  • ad hoc to paper Low-confidence non-target logits are noise and should be filtered.
    Section 3.3: 'Low-confidence logits may introduce optimization noise that outweighs their informational value.' This motivates the dynamic top-k mask and is not proven.
  • ad hoc to paper The curriculum phase fractions (0.3 and 0.7 of total epochs) are appropriate.
    Algorithm 2 uses 0.3 and 0.7 of total epochs as phase boundaries; these are chosen empirically without analysis.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer." pith.science (2026). https://pith.science/paper/4VCGMWYD

@misc{pith2026250515133,
  author       = {Pith},
  title        = {Pith review of: DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4VCGMWYD}},
  note         = {Machine review of arXiv:2505.15133}
}
read the original abstract

Recent advances in knowledge distillation have emphasized the importance of decoupling different knowledge components. While existing methods utilize momentum mechanisms to separate task-oriented and distillation gradients, they overlook the inherent conflict between target-class and non-target-class knowledge flows. Furthermore, low-confidence dark knowledge in non-target classes introduces noisy signals that hinder effective knowledge transfer. To address these limitations, we propose DeepKD, a novel training framework that integrates dual-level decoupling with adaptive denoising. First, through theoretical analysis of gradient signal-to-noise ratio (GSNR) characteristics in task-oriented and non-task-oriented knowledge distillation, we design independent momentum updaters for each component to prevent mutual interference. We observe that the optimal momentum coefficients for task-oriented gradient (TOG), target-class gradient (TCG), and non-target-class gradient (NCG) should be positively related to their GSNR. Second, we introduce a dynamic top-k mask (DTM) mechanism that gradually increases K from a small initial value to incorporate more non-target classes as training progresses, following curriculum learning principles. The DTM jointly filters low-confidence logits from both teacher and student models, effectively purifying dark knowledge during early training. Extensive experiments on CIFAR-100, ImageNet, and MS-COCO demonstrate DeepKD's effectiveness. Our code is available at https://github.com/haiduo/DeepKD.

Figures

Figures reproduced from arXiv: 2505.15133 by the authors.

Figure 1
Figure 1. Analysis of optimization dynamics and knowledge transfer of ResNet32 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of gradient and buffer SNR between vanilla KD and DeepKD: (a) KD GSNR [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Analysis of top-k masking strategy. (a) Distribution of teacher model’s confidence on [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Detailed architecture of our DeepKD framework. Input images flow through teacher and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Difference of student and teacher logits. DeepKD leads to a significantly smaller difference [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: The t-SNE [54] feature visualization of ResNet32 [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 37 canonical work pages

  1. [20]

    Medhi, Stochastic processes

    J. Medhi, Stochastic processes. New Age International, 1994

    work page 1994

  2. [22]

    Towards understanding how momentum improves generalization in deep learning,

    S. Jelassi and Y . Li, “Towards understanding how momentum improves generalization in deep learning,” in International Conference on Machine Learning. PMLR, 2022, pp. 9965–10 040

    work page 2022

  3. [21]

    Understanding why neural networks generalize well through gsnr of parameters,

    J. Liu, G. Jiang, Y . Bai, T. Chen, and H. Wang, “Understanding why neural networks generalize well through gsnr of parameters,” arXiv preprint arXiv:2001.07384, 2020

    arXiv 2001

  4. [1]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015

    arXiv 2015

  5. [2]

    Pkd: General distillation framework for object detectors via pearson correlation coefficient,

    W. Cao, Y . Zhang, J. Gao, A. Cheng, K. Cheng, and J. Cheng, “Pkd: General distillation framework for object detectors via pearson correlation coefficient,” Advances in Neural Information Processing Systems, vol. 35, pp. 15 394–15 406, 2022

    work page 2022

  6. [3]

    Cross-image relational knowledge distillation for semantic segmentation,

    C. Yang, H. Zhou, Z. An, X. Jiang, Y . Xu, and Q. Zhang, “Cross-image relational knowledge distillation for semantic segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12 319–12 328

    work page 2022

  7. [4]

    Relational diffusion distillation for efficient image generation,

    W. Feng, C. Yang, Z. An, L. Huang, B. Diao, F. Wang, and Y . Xu, “Relational diffusion distillation for efficient image generation,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 205–213

    work page 2024

  8. [5]

    Knowledge distillation: A survey,

    J. Gou, B. Yu, S. J. Maybank, and D. Tao, “Knowledge distillation: A survey,”International Journal of Computer Vision, vol. 129, no. 6, pp. 1789–1819, 2021

    2021

Show all 54 references
  1. [6]

    Knowledge distillation from single-task teachers to multi-task student for end-to-end au- tonomous driving,

    P. Agand, “Knowledge distillation from single-task teachers to multi-task student for end-to-end au- tonomous driving,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 21, 2024, pp. 23 375–23 376

    2024

  2. [7]

    Do we really need a complex agent system? distill embodied agent into a single model,

    Z. Zhao, K. Ma, W. Chai, X. Wang, K. Chen, D. Guo, Y . Zhang, H. Wang, and G. Wang, “Do we really need a complex agent system? distill embodied agent into a single model,” arXiv preprint arXiv:2404.04619, 2024

    2024 arXiv

  3. [8]

    Magdi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models,

    J. C.-Y . Chen, S. Saha, E. Stengel-Eskin, and M. Bansal, “Magdi: Structured distillation of multi-agent interaction graphs improves reasoning in smaller language models,” arXiv preprint arXiv:2402.01620, 2024

    2024 arXiv

  4. [9]

    Adaptive multi-teacher knowledge distillation with meta-learning,

    H. Zhang, D. Chen, and C. Wang, “Adaptive multi-teacher knowledge distillation with meta-learning,” in 2023 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 2023, pp. 1943–1948

    2023

  5. [10]

    Multi-teacher knowledge distillation with reinforcement learning for visual recognition,

    C. Yang, X. Yu, H. Yang, Z. An, C. Yu, L. Huang, and Y . Xu, “Multi-teacher knowledge distillation with reinforcement learning for visual recognition,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 9, 2025, pp. 9148–9156

    2025

  6. [11]

    A comprehensive overhaul of feature distillation,

    B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y . Choi, “A comprehensive overhaul of feature distillation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1921–1930

    2019

  7. [12]

    What makes a

    H. Wang, S. Lohit, M. N. Jones, and Y . Fu, “What makes a" good" data augmentation in knowledge distillation-a statistical perspective,” Advances in Neural Information Processing Systems , vol. 35, pp. 13 456–13 469, 2022

    2022

  8. [13]

    Cross-view consistency regularisation for knowledge distillation,

    W. Zhang, D. Liu, W. Cai, and C. Ma, “Cross-view consistency regularisation for knowledge distillation,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 2011–2020

    2024

  9. [14]

    Why logit distillation works: A novel knowledge distillation technique by deriving target augmentation and logits distortion,

    M. I. Hossain, S. Akhter, N. I. Mahbub, C. S. Hong, and E.-N. Huh, “Why logit distillation works: A novel knowledge distillation technique by deriving target augmentation and logits distortion,”Information Processing & Management, vol. 62, no. 3, p. 104056, 2025

    2025

  10. [15]

    Single teacher, multiple perspectives: Teacher knowledge augmentation for enhanced knowledge distillation,

    M. I. Hossain, S. Akhter, C. S. Hong, and E.-N. Huh, “Single teacher, multiple perspectives: Teacher knowledge augmentation for enhanced knowledge distillation,” inThe Thirteenth International Conference on Learning Representations, 2025

    2025

  11. [16]

    Revisiting knowledge distillation via label smoothing regularization,

    L. Yuan, F. E. Tay, G. Li, T. Wang, and J. Feng, “Revisiting knowledge distillation via label smoothing regularization,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 3903–3911

    2020

  12. [17]

    Debiased distillation for consistency regularization,

    L. Wang, L. Xu, X. Yang, Z. Huang, and J. Cheng, “Debiased distillation for consistency regularization,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 8, 2025, pp. 7799–7807

    2025

  13. [18]

    Decoupled knowledge distillation,

    B. Zhao, Q. Cui, R. Song, Y . Qiu, and J. Liang, “Decoupled knowledge distillation,” inProceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2022, pp. 11 953–11 962

    2022

  14. [19]

    Dot: A distillation-oriented trainer,

    B. Zhao, Q. Cui, R. Song, and J. Liang, “Dot: A distillation-oriented trainer,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 6189–6198. 10

    2023

  15. [23]

    Visualizing the loss landscape of neural nets,

    H. Li, Z. Xu, G. Taylor, C. Studer, and T. Goldstein, “Visualizing the loss landscape of neural nets,” Advances in neural information processing systems, vol. 31, 2018

    2018

  16. [24]

    Tighter variational bounds are not necessarily better,

    T. Rainforth, A. Kosiorek, T. A. Le, C. Maddison, M. Igl, F. Wood, and Y . W. Teh, “Tighter variational bounds are not necessarily better,” in International Conference on Machine Learning. PMLR, 2018, pp. 4277–4285

    2018

  17. [25]

    On large-batch training for deep learning: Generalization gap and sharp minima,

    N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang, “On large-batch training for deep learning: Generalization gap and sharp minima,” arXiv preprint arXiv:1609.04836, 2016

    2016 arXiv

  18. [26]

    Averaging weights leads to wider optima and better generalization,

    P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  19. [27]

    Curriculum learning,

    Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” inProceedings of the 26th annual international conference on machine learning, 2009, pp. 41–48

    2009

  20. [28]

    Curriculum temperature for knowledge distillation,

    Z. Li, X. Li, L. Yang, B. Zhao, R. Song, L. Luo, J. Li, and J. Yang, “Curriculum temperature for knowledge distillation,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 2, 2023, pp. 1504–1512

    2023

  21. [29]

    Improving knowledge distillation via head and tail categories,

    L. Xu, J. Ren, Z. Huang, W. Zheng, and Y . Chen, “Improving knowledge distillation via head and tail categories,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 5, pp. 3465–3480, 2023

    2023

  22. [30]

    Fitnets: Hints for thin deep nets,

    A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y . Bengio, “Fitnets: Hints for thin deep nets,” arXiv preprint arXiv:1412.6550, 2014

    2014 arXiv

  23. [34]

    From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels,

    Z. Yang, A. Zeng, Z. Li, T. Zhang, C. Yuan, and Y . Li, “From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 17 185–17 194

    2023

  24. [35]

    Reducing the teacher-student gap via spherical knowledge disitllation,

    J. Guo, M. Chen, Y . Hu, C. Zhu, X. He, and D. Cai, “Reducing the teacher-student gap via spherical knowledge disitllation,” arXiv preprint arXiv:2010.07485, 2020

    2010 arXiv

  25. [36]

    Mdr: Multi-stage decoupled relational knowledge distillation with adaptive stage selection,

    J. Wang, L. Lu, M. Chi, and J. Chen, “Mdr: Multi-stage decoupled relational knowledge distillation with adaptive stage selection,” in Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 2175–2183

    2024

  26. [37]

    Ntce-kd: Non-target-class-enhanced knowledge distillation,

    C. Li, X. Teng, Y . Ding, and L. Lan, “Ntce-kd: Non-target-class-enhanced knowledge distillation,” Sensors, vol. 24, no. 11, 2024. [Online]. Available: https://www.mdpi.com/1424-8220/24/11/3617

    2024

  27. [38]

    Teach less, learn more: On the undistillable classes in knowledge distillation,

    Y . Zhu, N. Liu, Z. Xu, X. Liu, W. Meng, L. Wang, Z. Ou, and J. Tang, “Teach less, learn more: On the undistillable classes in knowledge distillation,” Advances in Neural Information Processing Systems, vol. 35, pp. 32 011–32 024, 2022

    2022

  28. [39]

    Knowledge distillation with refined logits,

    W. Sun, D. Chen, S. Lyu, G. Chen, C. Chen, and C. Wang, “Knowledge distillation with refined logits,” arXiv preprint arXiv:2408.07703, 2024

    2024 arXiv

  29. [40]

    Domain generalization guided by gradient signal to noise ratio of parameters,

    M. Michalkiewicz, M. Faraki, X. Yu, M. Chandraker, and M. Baktashmotlagh, “Domain generalization guided by gradient signal to noise ratio of parameters,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 6177–6188. 11

    2023

  30. [41]

    On the importance of initialization and momentum in deep learning,

    I. Sutskever, J. Martens, G. Dahl, and G. Hinton, “On the importance of initialization and momentum in deep learning,” in International conference on machine learning. PMLR, 2013, pp. 1139–1147

    2013

  31. [42]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  32. [43]

    Training data-efficient image transformers & distillation through attention,

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou, “Training data-efficient image transformers & distillation through attention,” in International conference on machine learning. PMLR, 2021, pp. 10 347–10 357

    2021

  33. [44]

    Logit standardization in knowledge distillation,

    S. Sun, W. Ren, J. Li, R. Wang, and X. Cao, “Logit standardization in knowledge distillation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 15 731– 15 740

    2024

  34. [45]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009

    2009

  35. [46]

    ImageNet large scale visual recognition challenge,

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., “ImageNet large scale visual recognition challenge,” IJCV, 2015

    2015

  36. [47]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in ECCV, 2014

    2014

  37. [48]

    Knowledge distillation with the reused teacher classifier,

    D. Chen, J.-P. Mei, H. Zhang, C. Wang, Y . Feng, and C. Chen, “Knowledge distillation with the reused teacher classifier,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 11 933–11 942

    2022

  38. [49]

    Class attention transfer based knowledge distillation,

    Z. Guo, H. Yan, H. Li, and X. Lin, “Class attention transfer based knowledge distillation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 11 868–11 877

    2023

  39. [50]

    Multi-level logit distillation,

    Y . Jin, J. Wang, and D. Lin, “Multi-level logit distillation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 24 276–24 285

    2023

  40. [51]

    Distilling knowledge via knowledge review,

    P. Chen, S. Liu, H. Zhao, and J. Jia, “Distilling knowledge via knowledge review,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021, pp. 5008–5017

    2021

  41. [52]

    Faster r-cnn: Towards real-time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in NeurIPS, 2015

    2015

  42. [54]

    Visualizing data using t-sne

    L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” JMLR, 2008

    2008

  43. [55]

    Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,

    S. Zagoruyko and N. Komodakis, “Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer,”arXiv preprint arXiv:1612.03928, 2016

    2016 arXiv

  44. [56]

    Relational knowledge distillation,

    W. Park, D. Kim, Y . Lu, and M. Cho, “Relational knowledge distillation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3967–3976

    2019

  45. [57]

    Contrastive representation distillation,

    Y . Tian, D. Krishnan, and P. Isola, “Contrastive representation distillation,” arXiv preprint arXiv:1910.10699, 2019

    1910 arXiv

  46. [58]

    A comprehensive overhaul of feature distillation,

    B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y . Choi, “A comprehensive overhaul of feature distillation,” in Proceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 1921–1930. A Technical Appendices and Supplementary Material A.1 Distillation fi...

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.