TopKD: Top-scaled Knowledge Distillation

Jinjia Zhou; Qi Wang

REVIEW 5 major objections 4 minor 53 references

TopKD: Top-scaled Knowledge Distillation

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that the most valuable signal for knowledge distillation lives in the teacher's Top-K logits, and that scaling these logits and supervising with a contrastive plus decoupled-cosine loss lets logit-based distillation match

desk verdict Plausible top-K distillation idea, but the core TSM algorithm is missing from the appendix—unverifiable as submitted. read the letter →

arxiv 2508.04539 v1 pith:22Y5ZNEU submitted 2025-08-06 cs.CV

Qi Wang , Jinjia Zhou This is my paper · ORCID

classification cs.CV

keywords knowledgedistillationlogit-basedTop-Klogitscontrastivelearningcosinesimilaritylossvisiontransformersimageclassificationmodelcompression

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to show that the most useful knowledge a teacher passes to a student sits in the teacher's Top-K output logits, not in the full softened probability vector that classic distillation matches. The authors propose TopKD, which rescales those top logits—giving an extra boost to the ground-truth class when the teacher's own Top-1 guess is wrong—and trains the student with a contrastive loss plus a decoupled cosine loss that treats positive Top-K, negative Top-K, and non-Top-K logits separately. They report that this logit-only recipe outperforms both logit-based and feature-based distillation methods on CIFAR-100 and ImageNet, transfers better to STL-10 and Tiny-ImageNet, works for Vision Transformer students, and can be dropped into existing distillation methods as a plug-in. If true, it would mean that lightweight logit distillation no longer needs to lag behind feature-matching approaches.

What carries the argument

The central objects are the Top-K Scaling Module (TSM) and the Top-K Decoupled Loss (TDL). TSM rescales the teacher logits as $z'_i = z_i \cdot w_i + \Delta$ on the Top-K indices plus the ground-truth class, where $w_i$ is a rank-dependent weight and $\Delta$ a bias tied to the mean Top-K/non-Top-K gap. TDL splits the student-teacher cosine similarity into positive Top-K, negative Top-K, and non-Top-K components with weights $\alpha$ and $\beta$. Together they amplify the semantically informative part of the teacher's logits and align the student's logit geometry directionally rather than by exact value matching.

What would settle it

Run TopKD once the appendix's Algorithm ?? is supplied: if Table 1's accuracies reproduce, the claim is supported. A sharper test is to replace the rank-dependent $w_i$ with a single constant scale on all Top-K logits and a fixed boost on the ground-truth class; if the simplified scaling matches full TopKD within noise, then rank-dependence is not the operative mechanism.

Watch

Extended reading notes

Core claim

The central claim is that Top-K knowledge—the few highest-confidence and most-negative entries in a teacher's logit vector—carries an inter-class semantic structure that KL-Div-based distillation erases. TopKD operationalizes this in two pieces. The Top-K Scaling Module (TSM) multiplies the teacher's Top-K logits by a rank-dependent weight and adds a bias $\Delta$ proportional to the gap between Top-K and non-Top-K logits; when the teacher's Top-1 prediction is wrong, it also amplifies the ground-truth logit to correct the teacher's bias. The Top-K Decoupled Loss (TDL) replaces the overall cosine similarity with three decoupled terms—positive Top-K, negative Top-K, and non-Top-K—so that high

Load-bearing premise

The load-bearing premise is that the paper's specific rank-dependent scaling of the teacher's Top-K logits—including a larger boost to the ground-truth class when the teacher's Top-1 is wrong—is the right way to expose Top-K knowledge, and the exact computation of that scaling is deferred to an Algorithm ?? that is not present in this submission, so the central mechanism cannot be independently checked from the text.

Editorial extensions

If this is right

Logit-based distillation can match or beat feature-based distillation on standard classification benchmarks, removing the need for architectural alignment layers and feature-matching overhead.
The TSM and TDL components can be inserted into existing distillation pipelines as plug-ins, raising accuracy without retraining the teacher or changing student backbones.
Students trained with TopKD transfer better to new datasets such as STL-10 and Tiny-ImageNet, indicating that the learned representations are more reusable.
TopKD also improves Vision Transformer students, suggesting that the Top-K signal generalizes across convolutional and attention-based architectures.
With stronger teachers, TopKD students continue to improve where several comparison methods stagnate or degrade, helping to mitigate the 'bigger teacher is not always better' problem.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct extension not tested in the paper: on very large label spaces, such as ImageNet-21k or long-tail datasets, the optimal $K$ may need to vary per sample or per teacher confidence rather than staying fixed.
The sharp accuracy drop at $K=50$ in the paper's own ablation suggests Top-K knowledge has a sweet spot; an adaptive or confidence-weighted $K$ could be a natural follow-up.
Because TSM and TDL only modify the loss, the same idea could transfer to other domains where top-of-the-distribution knowledge dominates, such as language-model token distributions or retrieval logits.
The exact computation of $w_i$ and $\Delta$ is deferred to an Algorithm ?? that does not appear in this submission, so the quantitative claims are contingent on the authors' implementation being reproduced.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear [Colleague],

The punchline: this is a workable idea shipwrecked by a missing appendix. TopKD claims that emphasizing the teacher's top-K logits with a decoupled cosine loss and contrastive alignment beats state-of-the-art logit- and feature-based KD across CIFAR-100, ImageNet, STL-10, and Tiny-ImageNet. If the numbers are real, that would be a useful, deployment-friendly result. But the central mechanism—the TSM scaling in Eq. (2), specifically how w_i and Delta are computed—is deferred to "Algorithm ?? in the appendix," and the appendix is absent. Without that, the method is unverifiable from the submitted text. You cannot implement it, you cannot separate the proposed mechanism from the contrastive core, and you cannot tell whether the gains come from the top-K scaling or from hyperparameter tuning on the benchmark.

Credit where it's due. The intuition is sound: teacher misclassifications often land on semantically related classes, so explicit emphasis on top-K logits is a reasonable extension of DKD's positive/negative decomposition. The loss structure is clearly written at the equation level. The experiments are extensive in scope—many teacher/student pairs, four datasets, and plug-and-play gains on existing methods. The ablations for K and α/β are thoughtful and show a sensible pattern, with performance collapsing at K=50, which supports the premise that top-K knowledge is the informative part.

Soft spots, in proportion. The missing algorithm is the load-bearing one; it is a conditional issue, not a fatal contradiction. The paper also defers detailed results and transformer/detection tables to the same missing appendix, so the boldest claims (ViT and detection gains) are unsubstantiated in the text. There are no error bars or significance tests, and the hyperparameters (α=3, β=1, K=10) are tuned on CIFAR-100, which is also a main benchmark—a legitimate concern for a "consistent SOTA" claim. Finally, the contrastive loss in Eq. (1) closely resembles the authors' own prior MCLD (ref [44]), which appears in the bibliography but is never cited in the body. That is a worrying citation gap and it weakens the novelty claim. None of these are fatal by themselves, but together they mean the paper is not ready as submitted.

Who's this for? Anyone working on logit-based distillation. It deserves a serious referee—the idea is plausible and the preliminary numbers are good enough to warrant a careful look at the missing appendix and code. But it needs major revision before I'd trust the claims. Recommendation: send to peer review, but the reviewers should demand the actual algorithm and code before accepting.

Referee Report

5 major / 4 minor

Summary. The paper proposes TopKD, a logit-based knowledge-distillation method. TopKD combines (i) a Top-K Scaling Module (TSM) that rescales the teacher's top-K logits, with a ground-truth boost when the teacher's top-1 prediction is wrong, and (ii) a Top-K Decoupled Loss (TDL) that applies separate cosine-similarity terms to positive top-K, negative top-K, and non-top-K logit dimensions, added to a contrastive batch-alignment loss. The authors claim consistent state-of-the-art results on CIFAR-100, ImageNet, STL-10, and Tiny-ImageNet, plug-and-play improvements when TSM/TDL are added to existing KD methods, and further gains for Vision Transformers and object detection. The paper contains the main loss equations, several tables of classification results, ablations over K, α, and β, and qualitative visualizations. However, the central TSM computation is deferred to an appendix algorithm that is absent from the submitted text, and other detailed experiments are also referenced to a missing appendix.

Significance. If the mechanism were fully specified and the reported results reproducible, the paper would make a useful contribution: it offers a lightweight, architecture-agnostic logit-distillation recipe that could complement feature-based methods, and the modular TSM/TDL idea is simple enough to be adopted by practitioners. The contrastive formulation and the emphasis on top-K structure are reasonable and potentially interesting directions. The paper is also honest in reporting ablations over the main hyperparameters. However, the significance is currently capped by three issues: (1) the core TSM scaling factors w_i and Δ are not defined anywhere in the manuscript, so the proposed method cannot be implemented or independently evaluated; (2) the headline claim of "consistently surpasses state-of-the-art" is contradicted by several cells in the paper's own tables; and (3) large parts of the experimental evidence, including ViT and detection results, are deferred to a nonexistent appendix. These are fixable in revision, which is why I am not recommending rejection, but they are load-bearing for the central claims.

major comments (5)

[§3.2, Eq. (2)] The definition of the Top-K Scaling Module is incomplete. Equation (2) rescales teacher logits using a rank-dependent factor w_i and a bias Δ, but neither quantity is specified. The text says the procedure is "provided in Algorithm ?? in the appendix," and no such algorithm or appendix appears in the manuscript. Since z'_t enters TDL (Eq. 4) and the final loss (Eq. 5), every reported result depends on this unspecified computation. This is the paper's central novelty and must be fully specified, including defaults, exact formulas, and any edge cases (e.g., when y_g is already in I_top). Without this, the method is not reproducible and cannot be distinguished from simpler logit-scaling heuristics.
[§4.1, Tables 1–3 and Abstract] The abstract and text claim TopKD "consistently surpasses state-of-the-art distillation methods," but the paper's own tables contain counterexamples. In Table 2, the WRN-40-2 → WRN-40-1 cell reports TopKD at 75.09, while FCFD achieves 75.53. In Table 3, setting (a) (ResNet-34/ResNet-18) reports TopKD Top-1 at 71.52, below DKD+LS (71.88), WTTM (72.19), and WKD-L (72.49). These cells undermine the "consistently surpasses" claim. The text should be revised to accurately state where TopKD is best and where it is competitive but not superior, or the experiments should be re-run and reported with corrected claims.
[§4.2, Tables 5–6 and §4.3] The hyperparameters K, α, and β are selected by ablations on CIFAR-100 and then adopted as defaults for all experiments, including ImageNet, STL-10, Tiny-ImageNet, and the ViT/detection experiments reported only in the missing appendix. This is a selection-on-the-benchmark concern for the main claims. The paper should provide, at minimum, the sensitivity of the method on at least one additional dataset or architecture, and should disclose clearly that the defaults were tuned on CIFAR-100. The current presentation overstates the generality of the chosen values.
[§4.3 and 'Detailed results ... appendix (Table??)'] Several load-bearing experimental claims are not verifiable because the supporting material is absent. Section 4.1 refers to "Detailed results are in the appendix (Table??)"; Section 4.3 states that ViT and object-detection results are "see appendix for details"; and the training details section says "More implementation details are provided in the appendix." No appendix is present in the submitted text. The ViT experiments and detection experiments are announced but not reported, so the claims of "substantial effectiveness" and "broad applicability" are unsupported. The authors must include the appendix or remove these claims.
[References / §3.3] Reference [44] (Wang and Zhou, arXiv:2411.10693, "Multi-perspective Contrastive Logit Distillation") is listed in the bibliography but never cited in the body. The contrastive loss in Eq. (1) appears conceptually close to that prior work, and the reader is left to guess how TopKD differs. The authors should cite and contrast their method with [44] explicitly, and state what is new in TopKD relative to that paper. This is essential for assessing novelty and for the reader to understand the incremental contribution.

minor comments (4)

[§3.1, Eq. (1)] The notation y ∈ R^{B×1} with indices 0 to B−1 is unusual for a cross-entropy label; it would be clearer to write y as a vector of class indices or one-hot labels. Also, the temperature τ is introduced but its default value is not stated here or in the training details.
[§4.1, Table 3] The table formatting is hard to read: the colon-aligned header "SettingMetricTch. Stu." and the merged Feature/Logits columns make it difficult to tell which baseline belongs to which setting. Please reformat and add clear column separators.
[§4.1, Baselines] The sentence "We report LSKD [37], select DKD+LS [52, 37] as the representative setting" is grammatically unclear and does not explain why LSKD itself is not compared directly. Please clarify the baseline-selection protocol.
[§4.3, Figures 5–6] The t-SNE and logit-correlation visualizations are qualitative and would be more informative with a quantitative metric (e.g., k-nearest-neighbor accuracy or mean absolute difference). Also, the figure captions do not specify which model is the teacher and which is the student for each panel.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TopKD's claims are empirical against external baselines; missing algorithm and dangling self-reference are completeness/novelty issues, not circular reductions.

full rationale

The paper's derivation chain is empirical rather than deductive. The proposed loss (Eq. 5) combines a contrastive loss (Eq. 1) and a Top-K Decoupled Loss (Eq. 4), with the teacher logits rescaled by TSM (Eq. 2). No equation is defined in terms of the result it is used to predict; the reported accuracies are comparisons against external methods (CRD, ReviewKD, SimKD, DKD, etc.) under standard settings. The selection of K=10, α=3, β=1 from validation-set ablations (Tables 5-6) is hyperparameter tuning, not a fitted input renamed as a prediction. The main caveats are the missing Algorithm ?? for computing w_i and Δ in Eq. 2, and the fact that reference [44] (the authors' own prior 'Multi-perspective Contrastive Logit Distillation') appears in the bibliography but is never cited in the text. These are completeness/attribution issues: they make the method hard to verify and could indicate that the contrastive core is repackaged prior work, but they do not exhibit the specific reduction of a derived result to its own input that the circularity criteria require. No load-bearing self-citation or uniqueness theorem is invoked to force the choice of TSM/TDL. Accordingly, the paper is not circular in the sense defined here, though it is incomplete and self-referential in ways that warrant scrutiny.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central claim depends on tunable hyperparameters (K, alpha, beta, tau) and on an incompletely specified scaling rule (wi, Delta). The method also assumes, without independent evidence, that top-K logit structure is the most informative supervision signal and that cosine/contrastive objectives transfer across datasets. No new physical entities are introduced.

free parameters (6)

K (Top-K subset size) = 10
Selected by the ablation in Table 5 on CIFAR-100 and used for all datasets; the choice controls how many teacher logits are amplified.
alpha (positive Top-K cosine weight) = 3
Selected from Table 6 grid search on CIFAR-100; upper limit 5, default alpha=3.
beta (negative Top-K cosine weight) = 1
Selected from Table 6 grid search on CIFAR-100; upper limit 2, default beta=1.
tau (temperature in Eq. 1) = not specified in main text
Controls sharpness of the contrastive similarity matrix; no value or sensitivity analysis is reported in the provided text.
wi (rank-dependent scaling factor in Eq. 2) = unspecified; Algorithm ?? missing
Central to TSM, but its exact formula is deferred to a missing appendix, so the paper's main mechanism is underdetermined.
Delta (bias proportional to Top-K vs Non-Top-K logit gap) = proportionality constant unspecified
Same as wi, the rule for Delta is in missing Algorithm ??, so this is an unquantified tuning choice.

assumptions (5)

standard math Cross-entropy over the similarity matrix with diagonal labels implements instance-level contrastive alignment (Eq. 1).
Standard SupCon formulation; accepted from SimCLR/MoCo.
domain assumption Teacher Top-K logits encode semantically meaningful inter-class structure even when Top-1 is wrong (Section 3.1, Figure 2).
Supported only by one qualitative example; no quantitative evidence.
domain assumption Cosine similarity captures structural/relational information better than KL divergence for Top-K supervision (Section 3.3).
Asserted, not demonstrated; no controlled comparison isolating this assumption.
domain assumption Scaling only the teacher's Top-K logits (Eq. 2) preserves and emphasizes the informative signal without distorting the student.
Mechanism is undefined due to missing Algorithm ??; assumed to work.
domain assumption Hyperparameters tuned on CIFAR-100 (K=10, alpha=3, beta=1) transfer to all other datasets, architectures and tasks.
No per-task tuning analysis; results rely on this transfer.

invented entities (1)

Top-K knowledge
purpose: The proposed transferable signal contained in the teacher's top-K output logits, which the method amplifies.
Defined operationally as the top-K indices of teacher logits; has no falsifiable handle beyond the method's own results.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TopKD: Top-scaled Knowledge Distillation." pith.science (2026). https://pith.science/paper/22Y5ZNEU

@misc{pith2026250804539,
  author       = {Pith},
  title        = {Pith review of: TopKD: Top-scaled Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/22Y5ZNEU}},
  note         = {Machine review of arXiv:2508.04539}
}

read the original abstract

Recent advances in knowledge distillation (KD) predominantly emphasize feature-level knowledge transfer, frequently overlooking critical information embedded within the teacher's logit distributions. In this paper, we revisit logit-based distillation and reveal an underexplored yet critical element: Top-K knowledge. Motivated by this insight, we propose Top-scaled Knowledge Distillation (TopKD), a simple, efficient, and architecture-agnostic framework that significantly enhances logit-based distillation. TopKD consists of two main components: (1) a Top-K Scaling Module (TSM), which adaptively amplifies the most informative logits, and (2) a Top-K Decoupled Loss (TDL), which offers targeted and effective supervision. Notably, TopKD integrates seamlessly into existing KD methods without introducing extra modules or requiring architectural changes. Extensive experiments on CIFAR-100, ImageNet, STL-10, and Tiny-ImageNet demonstrate that TopKD consistently surpasses state-of-the-art distillation methods. Moreover, our method demonstrates substantial effectiveness when distilling Vision Transformers, underscoring its versatility across diverse network architectures. These findings highlight the significant potential of logits to advance knowledge distillation.

Figures

Figures reproduced from arXiv: 2508.04539 by the authors.

**Figure 1.** Overall structure of our TopKD. Building upon this insight, we revisit the distribution of teacher’s logits and identify the underexplored Top-K knowledge, which captures particularly informative supervision signals. To exploit this property, we propose Top-scaled Knowledge Distillation (TopKD), a framework that explicitly emphasizes the teacher’s Top-K knowledge during distillation process. Our contributions can be… view at source ↗

**Figure 2.** An example from the CIFAR-100 training set where the teacher incorrectly predicts otter [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Comparison of contrastive loss (Dim=0) and cosine similarity loss (Dim=1) for student [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Performance comparison before and after integrating the TSM & the TDL components into [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: The t-SNE [42] visualization of features in a homogeneous structure. Our method yields more separable class distributions. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Differences between the student’s and teacher’s logit correlation matrices in a [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 44 canonical work pages

[44]

Multi-perspective Contrastive Logit Distillation

Q. Wang and J. Zhou. Multi-perspective contrastive logit distillation. arXiv preprint arXiv:2411.10693, 2024

work page Pith review arXiv 2024
[1]

T. Bai, J. Zhao, and B. Wen. Guided adversarial contrastive distillation for robust students. IEEE Transactions on Information Forensics and Security, 19:9643–9655, 2024

work page 2024
[2]

Chen, J.-P

D. Chen, J.-P. Mei, H. Zhang, C. Wang, Y . Feng, and C. Chen. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11933–11942, 2022

work page 2022
[3]

L. Chen, D. Wang, Z. Gan, J. Liu, R. Henao, and L. Carin. Wasserstein contrastive representation distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16296–16305, 2021

work page 2021
[4]

P. Chen, S. Liu, H. Zhao, and J. Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5008–5017, 2021

work page 2021
[5]

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020

2020
[6]

X. Chen, Q. Cao, Y . Zhong, J. Zhang, S. Gao, and D. Tao. Dearkd: data-efficient early knowledge distillation for vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12052–12062, 2022

work page 2022
[7]

Cheng, Z

X. Cheng, Z. Rao, Y . Chen, and Q. Zhang. Explaining knowledge distillation by quantifying the knowledge. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12925– 12935, 2020

work page 2020

Show all 53 references

[8]

J. H. Cho and B. Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4794–4802, 2019

2019
[9]

Coates, A

A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215–223. JMLR Workshop and Conference Proceedings, 2011

2011
[10]

Z. Guo, H. Yan, H. Li, and X. Lin. Class attention transfer based knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11868–11877, 2023

2023
[11]

S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015

2015
[12]

K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020

2020
[13]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016

2016
[14]

B. Heo, M. Lee, S. Yun, and J. Y . Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3779–3787, 2019

2019
[15]

B. Heo, S. Yun, D. Han, S. Chun, J. Choe, and S. J. Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11936–11945, 2021

2021
[16]

G. Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

2015 arXiv
[17]

A. G. Howard. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017

2017 arXiv
[18]

Jacob, S

B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2704–2713, 2018

2018
[19]

Y . Jin, J. Wang, and D. Lin. Multi-level logit distillation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24276–24285, 2023

2023
[20]

J. Kim, S. Park, and N. Kwak. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems, 31, 2018. 10

2018
[21]

Krizhevsky, G

A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009

2009
[22]

H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016

2016 arXiv
[23]

K. Li, R. Yu, Z. Wang, L. Yuan, G. Song, and J. Chen. Locality guidance for improving vision transformers on tiny datasets. In European Conference on Computer Vision, pages 110–127. Springer, 2022

2022
[24]

L. Li, P. Dong, Z. Wei, and Y . Yang. Automated knowledge distillation via monte carlo tree search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17413–17424, 2023

2023
[25]

Z. Li, X. Li, L. Yang, B. Zhao, R. Song, L. Luo, J. Li, and J. Yang. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1504–1512, 2023

2023
[26]

T.-Y . Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117–2125, 2017

2017
[27]

T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740–755....

2014
[28]

D. Liu, M. Kan, S. Shan, and X. CHEN. Function-consistent feature distillation. In The Eleventh International Conference on Learning Representations (ICLR), 2023

2023
[29]

J. Lv, H. Yang, and P. Li. Wasserstein distance rivals kullback-leibler divergence for knowledge distillation. Advances in Neural Information Processing Systems, 37:65445–65475, 2024

2024
[30]

S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh. Improved knowledge distillation via teacher assistant. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 5191–5198, 2020

2020
[31]

mnmoustafa and M. Ali. Tiny imagenet. https://kaggle.com/competitions/tiny-imagenet,
[32]

A. v. d. Oord, Y . Li, and O. Vinyals. Representation learning with contrastive predictive coding.arXiv preprint arXiv:1807.03748, 2018

2018 arXiv
[33]

J. Rao, L. Ding, S. Qi, M. Fang, Y . Liu, L. Shen, and D. Tao. Dynamic contrastive distillation for image-text retrieval. IEEE Transactions on Multimedia, 25:8383–8395, 2023

2023
[34]

S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2016

2016
[35]

Russakovsky, J

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bern- stein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015

2015
[36]

Simonyan and A

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014

2014 arXiv
[37]

S. Sun, W. Ren, J. Li, R. Wang, and X. Cao. Logit standardization in knowledge distillation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15731–15740, 2024

2024
[38]

Sutskever, J

I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1139–1147. PMLR, 2013

2013
[39]

Y . Tian, D. Krishnan, and P. Isola. Contrastive representation distillation. InInternational Conference on Learning Representations, 2020

2020
[40]

Touvron, M

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347–10357. PMLR, 2021

2021
[41]

Tung and G

F. Tung and G. Mori. Similarity-preserving knowledge distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1365–1374, 2019. 11

2019
[42]

Van der Maaten and G

L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008

2008
[43]

Wang and K.-J

L. Wang and K.-J. Yoon. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE transactions on pattern analysis and machine intelligence, 44(6):3048– 3068, 2021

2021
[45]

W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pages 568–578, 2021

2021
[46]

K. Wu, J. Zhang, H. Peng, M. Liu, B. Xiao, J. Fu, and L. Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European conference on computer vision, pages 68–85. Springer, 2022

2022
[47]

J. Yang, X. Shen, J. Xing, X. Tian, H. Li, B. Deng, J. Huang, and X.-s. Hua. Quantization networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7308–7316, 2019

2019
[48]

L. Yuan, Y . Chen, T. Wang, W. Yu, Y . Shi, Z.-H. Jiang, F. E. Tay, J. Feng, and S. Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF international conference on computer vision, pages 558–567, 2021

2021
[49]

Zagoruyko and N

S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv:1605.07146, 2016

2016 arXiv
[50]

Zhang, X

X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848–6856, 2018

2018
[51]

B. Zhao, Q. Cui, R. Song, and J. Liang. Dot: A distillation-oriented trainer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6189–6198, 2023

2023
[52]

B. Zhao, Q. Cui, R. Song, Y . Qiu, and J. Liang. Decoupled knowledge distillation. InProceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 11953–11962, 2022

2022
[53]

Zheng and E.-H

K. Zheng and E.-H. Yang. Knowledge distillation based on transformed teacher matching. In The Twelfth International Conference on Learning Representations (ICLR 2024), 2024. 12

2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [44]

Multi-perspective Contrastive Logit Distillation

Q. Wang and J. Zhou. Multi-perspective contrastive logit distillation. arXiv preprint arXiv:2411.10693, 2024

work page Pith review arXiv 2024

[2] [1]

T. Bai, J. Zhao, and B. Wen. Guided adversarial contrastive distillation for robust students. IEEE Transactions on Information Forensics and Security, 19:9643–9655, 2024

work page 2024

[3] [2]

Chen, J.-P

D. Chen, J.-P. Mei, H. Zhang, C. Wang, Y . Feng, and C. Chen. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11933–11942, 2022

work page 2022

[4] [3]

L. Chen, D. Wang, Z. Gan, J. Liu, R. Henao, and L. Carin. Wasserstein contrastive representation distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16296–16305, 2021

work page 2021

[5] [4]

P. Chen, S. Liu, H. Zhao, and J. Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5008–5017, 2021

work page 2021

[6] [5]

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020

2020

[7] [6]

X. Chen, Q. Cao, Y . Zhong, J. Zhang, S. Gao, and D. Tao. Dearkd: data-efficient early knowledge distillation for vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12052–12062, 2022

work page 2022

[8] [7]

Cheng, Z

X. Cheng, Z. Rao, Y . Chen, and Q. Zhang. Explaining knowledge distillation by quantifying the knowledge. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12925– 12935, 2020

work page 2020