Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Multi-perspective Contrastive Logit Distillation

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that a contrastive loss computed directly on raw teacher and student logits from three perspectives—instance, sample, and category—lets logit-only distillation match or beat feature-level distillation on four…

desk verdict MCLD is a coherent logit-only contrastive distillation method with good empirical results, but the SOTA claim rests on a baseline handicapped by using 240 instead of 480 epochs for LSKD+MLKD, so the main comparison needs revision. read the letter →

arxiv 2411.10693 v2 pith:HASTPLWB submitted 2024-11-16 cs.CV

Qi Wang , Jinjia Zhou This is my paper · ORCID
classification cs.CV MSC 68T0768T45
keywords knowledgedistillationlogitcontrastivelearningrawlogitsimageclassificationtransferVisionTransformersCIFAR-100
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Logit distillation has been underused, this paper argues, because the standard pipeline—softmax with temperature, then KL divergence—discards information and ignores the semantic structure in raw logits. The paper proposes Multi-perspective Contrastive Logit Distillation (MCLD), a loss that keeps raw logits and contrasts them from three views: each image against a queue of teacher-logit negatives, each sample against its own teacher counterpart in the batch, and each category against same-class teacher logits. On CIFAR-100, ImageNet, Tiny-ImageNet, and STL-10, MCLD is reported to reach the best accuracy among logit-only methods and to match or exceed feature-distillation methods, while requiring no extra alignment modules or sampled indices. It also transfers well to new datasets and shows strong results when distilling Vision Transformers. If these results hold, the expensive feature-matching machinery of modern distillation is unnecessary for top-tier transfer.

What carries the argument

The load-bearing object is the MCLD loss, a sum of three logit-contrast terms acting on raw logits $z_s$ and $z_t$. The instance-wise term is an InfoNCE-style contrastive cross-entropy—pull the matching student–teacher logit pair together against a queue of negative pairs—using stored logit products and a target-mask queue to avoid false negatives; the sample-wise term is a $B$-way classification over the diagonal of the student–teacher similarity matrix; the category-wise term is a supervised contrastive loss over same-class and cross-class teacher logits. A warm-up weight $\omega$ on the category term grows from 0 to 1 during training, following a schedule borrowed from earlier logit-distillation methods. Together these terms turn logit distillation into a structured ranking problem over instances, samples, and categories rather than a single per-sample KL divergence.

What would settle it

A reader could rerun the CIFAR-100 ResNet32$\times$4-to-ResNet8$\times$4 experiment with all three terms computed on L2-normalized logits; if accuracy falls well below the reported 78.19, the unnormalized scale is doing the work. A second check is to multiply teacher logits by a constant factor (say 2 or 0.5) and observe whether accuracy changes substantially, which would confirm magnitude sensitivity rather than semantic contrast.

Watch

Extended reading notes

Core claim

The paper's central claim is that logits encode high-level semantic structure that can be distilled directly with contrastive learning, without softmax or intermediate features. The MCLD objective is $L_{\text{MCLD}}=L_{\text{Inst}}+L_{\text{Samp}}+\omega L_{\text{Cate}}$, where each term is a dot-product comparison between raw student and teacher logits. Instance-wise CLD runs InfoNCE between a student logit and a queue of teacher-logit negatives, with a target-mask queue so same-class samples are treated as positives; sample-wise CLD aligns same-image pairs across the batch with a $B$-way classification on the similarity matrix; category-wise CLD uses ground-truth labels to treat all same-class teacher logits as positives. The loss needs no mapping modules and no positive/negative indexing beyond labels, and the category weight $\omega$ is ramped up through a warm-up so it acts once the student can discriminate within categories. The paper reports that this decomposition consistently outperforms logit-only baselines and matches or beats feature distillation on the standard benchmarks.

Load-bearing premise

The load-bearing premise is that raw dot products between teacher and student logits measure semantic similarity even though the two models' logits have different scales; if magnitude differences dominate, the loss compares scale rather than meaning.

Editorial extensions

If this is right

  • Logit-only distillation can match or exceed feature-based distillation, so alignment modules for intermediate feature maps are optional rather than necessary for state-of-the-art transfer.
  • Ground-truth labels replace sampled positive and negative indices, which removes a data-sampling bottleneck and makes each training batch faster.
  • The three-perspective loss transfers to Vision Transformer students, where the paper reports consistent gains over both logit and feature distillation baselines.
  • Representations distilled with MCLD transfer better to STL-10 and Tiny-ImageNet than those from several feature-based and logit-based methods.
  • MCLD reduces the 'bigger teacher, worse student' phenomenon, with accuracy improving as teacher capacity increases in the reported WRN experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication left implicit is that all three terms use unnormalized dot products, so the method's success may depend on teacher and student logit scales; L2-normalizing the logits before computing $\sigma_i$, $\eta_i$, and $\psi_p$ is a direct test of whether direction or magnitude carries the signal.
  • Because the queue makes instance-wise CLD a logit-level analogue of momentum contrastive learning, the same loss could be adapted to other output spaces, such as multimodal embeddings or structured prediction heads, where KL divergence is not well defined.
  • The warm-up on the category term suggests a curriculum on contrastive hardness; applying the same ramp to the instance and sample terms, or making it adaptive to the student's discriminative ability, could improve early-training stability.
  • The paper's evidence is mostly classification and transfer; extending MCLD to dense prediction tasks such as object detection would test whether logit-level contrast retains its advantage when spatial structure matters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes Multi-perspective Contrastive Logit Distillation (MCLD), a logit-only knowledge distillation method. Instead of KL divergence on softened probabilities, MCLD computes three contrastive losses on raw logits: instance-wise CLD using a queue of teacher logits, sample-wise CLD aligning same-sample teacher/student pairs within a batch, and category-wise CLD treating same-class samples as positives. The total loss is L_Inst + L_Samp + ω*L_Cate, with ω warmed up from 0 to 1. Experiments on CIFAR-100, ImageNet, STL-10, and Tiny-ImageNet report accuracy gains over feature- and logit-distillation baselines, together with training-efficiency and transferability analyses. The central claim is that a logit-only contrastive loss can rival and often outperform feature-level distillation and attain state-of-the-art results.

Significance. If the claims hold, this is a valuable empirical contribution to knowledge distillation: it shows that raw-logit contrastive losses can capture more than KL-based logit distillation while avoiding the extra mapping modules and sample-indexing overhead of feature distillation. The ablation in Table 4 demonstrates that each of the three loss terms contributes, and the transfer-learning and efficiency experiments are useful. However, the state-of-the-art claim is not yet fully established: the main comparison handicaps one baseline by using a shorter training schedule, the loss equations contain an internal inconsistency, and the raw-dot-product formulation weakens the semantic interpretation. The paper is within the journal's scope, and the core idea is promising, but these points require attention.

major comments (4)
  1. [Sec. 5.1, Tables 1-2 and the paragraph following Table 2] LSKD+MLKD is compared at 240 epochs although its published recipe uses 480. The text acknowledges that at 240 epochs this baseline 'exhibited significant performance degradation' and even failed to converge in some cases, and the 480-epoch results are deferred to the supplements. Since the headline claim is state-of-the-art accuracy, comparing against a knowingly handicapped baseline in the main tables does not support that claim. Please include the 480-epoch results for LSKD+MLKD in the main comparison, or state explicitly that the 480-epoch runs still fall below MCLD across the evaluated teacher-student pairs, and adjust the state-of-the-art claim if this is not the case.
  2. [Sec. 4.2.1, Eqs. (5)-(7)] Eq. (7) is not algebraically equivalent to Eq. (5). In Eq. (5), the positive similarity σi appears in the denominator as exp(σi/τ), but in Eq. (7) the positive pair has γ=0 and therefore contributes exp(0)=1 to the denominator; samples from the same class (also γ=0) likewise contribute 1, not their similarity. The text says Eq. (7) reformulates Eq. (5) as a (K+1)-way cross-entropy task, but the denominator is not the normalization over the positive and negative logits. This is load-bearing for the method definition: please correct the formula, define the actual denominator and label vector, and state which equation is used in the experiments.
  3. [Sec. 4.2.1-4.2.3, Eqs. (4), (8), (10)] All three contrastive losses use raw dot products z_s·z_t without normalization. If teacher and student logits have different magnitudes, the contrastive score is dominated by scale rather than by the semantic direction that the paper uses to motivate the method. The manuscript does not report an ablation with L2-normalized, or otherwise scale-aligned, logits, nor any analysis of logit-norm behavior during training. Please add such an ablation or analysis, or temper the claim that the method captures the 'semantic properties' of logits.
  4. [Sec. 5.1, Tables 1-3] The main accuracy tables report no error bars or significance measures. For CIFAR-100 the text states that results are averaged over three trials, but no standard deviations are given, and several margins over the second-best method are very small, for example Ours 71.90 vs LSKD+DKD 71.88 in the ResNet34/ResNet18 ImageNet row of Table 3. ImageNet results appear to be single runs. To support a state-of-the-art claim, please report standard deviations or confidence intervals, or repeated trials, at least for the closest comparisons.
minor comments (6)
  1. [Sec. 3, after Eq. (3)] The sentence 'all values below zero are lost' is imprecise: softmax preserves the ordering of the values but loses sign and relative-magnitude information. Rewording in terms of information diversity would be more accurate.
  2. [Sec. 4.2.1, Eq. (7)] The label vector yinst is described as one-hot with a 1 at position 0, but the formula does not show how yinst combines with the γ mask; please clarify the exact cross-entropy implementation and whether the positive similarity appears in the denominator.
  3. [Sec. 4.2.3, Eq. (11)] The denominator in LCate sums only over the negative set N, rather than over positives and negatives as in standard supervised contrastive learning; please justify this design choice and comment on its effect.
  4. [Sec. 5.2, Table 5] The ω end epoch is tuned on the validation set, with Table 5 searching values from 1 to 240 and selecting 155. This should be disclosed in the main text, and the sentence claiming 'no deliberate tuning of hyperparameters' should be revised.
  5. [Abstract and Sec. 5.1] The abstract claims 'state-of-the-art performance' while several results are described as 'comparable or even superior' and MCLD is not the best method in every teacher-student pair; please align the wording with the actual pattern of results.
  6. [Table 3] The WTTM Top-5 entry is listed as n/a; please either provide the value or explain why it is unavailable.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MCLD's loss terms are defined directly from logits and labels, and the reported accuracies are empirical results rather than quantities forced by the loss or by self-citation.

full rationale

The paper makes no formal derivation whose output is equivalent to its input. The central object is the composite loss L_MCLD = L_Inst + L_Samp + ω*L_Cate, each term defined directly from student/teacher logits and ground-truth labels (Eqs. 4–11). The reported top-1 accuracies are empirical measurements, not quantities derivable from the loss by construction. The warm-up endpoint ω_end is selected through a small validation sweep (Table 5), but this is ordinary hyperparameter selection, not a fitted parameter renamed as a prediction; no evaluation metric is a term of the loss. There are no self-citations: the warm-up strategy is attributed to external works [51,52], and the contrastive formulation to InfoNCE [33] and CRD [40]. The skeptical concern about LSKD+MLKD being reproduced at 240 epochs instead of its published 480 epochs (Sec. 5.1) is a legitimate baseline-fairness and empirical-validity issue for the state-of-the-art claim, but it is not circularity: an unfair comparison protocol does not make MCLD's loss equivalent to its inputs. The unnormalized dot-product similarity in Eq. (4) is an algorithmic assumption about representation scale, not a circular step. No load-bearing step reduces to a fit, a self-citation, or a definitional identity, so no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three domain assumptions: that raw logits retain more usable information than temperature-softmaxed probabilities, that dot-product contrastive learning on logits transfers semantic knowledge, and that warm-up scheduling of the category term helps. The method also depends on two free hyperparameters (queue length K and temperature τ) not specified in the main text, and on the warm-up endpoint ω that is tuned on the validation set.

free parameters (3)
  • Warm-up endpoint for ω = 155 epochs
    Selected from a sweep in Table 5 (values 1, 155, 185, 215, 240, 50, 100, 150, 180, 210) based on validation accuracy on CIFAR-100; the optimal value is data-dependent.
  • Queue length K
    Mentioned as an independently settable hyperparameter in Sec. 4.2.1, but its value is not reported in the main text, and presumably appears in the supplements.
  • Temperature τ
    Used in all three losses (Eqs. 5, 7, 9, 11), but not specified in the main text.
assumptions (3)
  • domain assumption Raw logits preserve more distillable information than temperature-scaled softmax probabilities.
    The paper's motivation (Sec. 3) argues that softmax discards negative logit values and thus reduces information diversity, an assumption about what information matters for distillation.
  • domain assumption Dot-product similarity between teacher and student logits provides a useful contrastive learning signal.
    The losses rely on raw dot products without normalization, so the scale of logits influences the training signal (Sec. 4.2.1).
  • domain assumption Warm-up scheduling for the category-wise loss improves final accuracy.
    The authors assume the model needs a warm-up period before category-wise contrastive learning becomes helpful; this is empirically motivated but not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-perspective Contrastive Logit Distillation." pith.science (2026). https://pith.science/paper/HASTPLWB

@misc{pith2026241110693,
  author       = {Pith},
  title        = {Pith review of: Multi-perspective Contrastive Logit Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HASTPLWB}},
  note         = {Machine review of arXiv:2411.10693}
}
read the original abstract

In previous studies on knowledge distillation, the significance of logit distillation has frequently been overlooked. To revitalize logit distillation, we present a novel perspective by reconsidering its computation based on the semantic properties of logits and exploring how to utilize it more efficiently. Logits often contain a substantial amount of high-level semantic information; however, the conventional approach of employing logits to compute Kullback-Leibler (KL) divergence does not account for their semantic properties. Furthermore, this direct KL divergence computation fails to fully exploit the potential of logits. To address these challenges, we introduce a novel and efficient logit distillation method, Multi-perspective Contrastive Logit Distillation (MCLD), which substantially improves the performance and efficacy of logit distillation. In comparison to existing logit distillation methods and complex feature distillation methods, MCLD attains state-of-the-art performance in image classification, and transfer learning tasks across multiple datasets, including CIFAR-100, ImageNet, Tiny-ImageNet, and STL-10. Additionally, MCLD exhibits superior training efficiency and outstanding performance with distilling on Vision Transformers, further emphasizing its notable advantages. This study unveils the vast potential of logits in knowledge distillation and seeks to offer valuable insights for future research.

Figures

Figures reproduced from arXiv: 2411.10693 by the authors.

Figure 1
Figure 1. Training time (per batch) vs. Top-1 Accuracy on CIFAR [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Differences between raw logits, class probabilities (soft [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the penultimate feature map (mean in [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: The t-SNE [43] visualization of features in a homogeneous structure. We set ResNet32×4 as the teacher model and ResNet8×4 as the student model. Our method achieves more separate classification results. (a) FCFD [29] (b) LSKD+DKD [39] (c) DOT+DKD [52] (d) WTTM [53] (e) …
Figure 6
Figure 6. Figure 6: Difference in correlation matrices between the student’s and teacher’s logits in a [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. TopKD: Top-scaled Knowledge Distillation

    cs.CV 2025-08 conditional novelty 4.0 of 10

    A logit-based distillation method that amplifies and decouples the teacher's Top-K outputs reports gains over several existing losses and can be added to other methods.

Reference graph

Works this paper leans on

53 extracted references · 30 canonical work pages · cited by 1 Pith paper

  1. [1]

    Knowledge distillation with the reused teacher classifier

    Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11933–11942, 2022. 2, 3, 4, 5, 6

    work page 2022

  2. [2]

    Learning efficient object detection mod- els with knowledge distillation

    Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Man- mohan Chandraker. Learning efficient object detection mod- els with knowledge distillation. Advances in neural informa- tion processing systems, 30, 2017. 2

    work page 2017

  3. [3]

    Wasserstein contrastive rep- resentation distillation

    Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive rep- resentation distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16296–16305, 2021. 3

    work page 2021

  4. [4]

    Distilling knowledge via knowledge review

    Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5008–5017, 2021. 1, 2, 3, 4, 5, 6

    work page 2021

  5. [5]

    A simple framework for contrastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Ge- offrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on ma- chine learning, pages 1597–1607. PMLR, 2020. 3

    2020

  6. [6]

    Dearkd: data-efficient early knowledge distillation for vision transformers

    Xianing Chen, Qiong Cao, Yujie Zhong, Jing Zhang, Shenghua Gao, and Dacheng Tao. Dearkd: data-efficient early knowledge distillation for vision transformers. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12052–12062, 2022. 8

    work page 2022

  7. [7]

    Explaining knowledge distillation by quantifying the knowl- edge

    Xu Cheng, Zhefan Rao, Yilan Chen, and Quanshi Zhang. Explaining knowledge distillation by quantifying the knowl- edge. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 12925–12935,

  8. [8]

    Xception: Deep learning with depthwise separable convolutions

    Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 1251–1258, 2017. 1

    work page 2017

Show all 53 references
  1. [9]

    An analysis of single-layer networks in unsupervised feature learning

    Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215–223. JMLR Workshop and Conference Proceedings, 2011. 7

    2011

  2. [10]

    Class attention transfer based knowledge distillation

    Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11868–11877, 2023. 3, 4, 5, 6

    2023

  3. [11]

    Learn- ing both weights and connections for efficient neural net- work

    Song Han, Jeff Pool, John Tran, and William Dally. Learn- ing both weights and connections for efficient neural net- work. Advances in neural information processing systems , 28, 2015. 1

    2015

  4. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 1

    2016

  5. [13]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9729–9738, 2020. 3, 5

    2020

  6. [14]

    Knowledge transfer via distillation of activation boundaries formed by hidden neurons

    Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI conference on artificial intelligence, pages 3779–3787,

  7. [15]

    Rethinking spa- tial dimensions of vision transformers

    Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spa- tial dimensions of vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11936–11945, 2021. 8

    2021

  8. [16]

    Distilling the knowledge in a neural net- work

    Geoffrey Hinton. Distilling the knowledge in a neural net- work. arXiv preprint arXiv:1503.02531, 2015. 1, 2, 5, 6

    2015 arXiv

  9. [17]

    Mobilenets: Efficient convolutional neu- ral networks for mobile vision applications

    Andrew G Howard. Mobilenets: Efficient convolutional neu- ral networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017. 1

    2017 arXiv

  10. [18]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceed- ings of the IEEE conference on computer vision ...

    2018

  11. [19]

    Multi-level logit distil- lation

    Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distil- lation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 24276–24285,

  12. [20]

    Supervised contrastive learning

    Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661–18673,

  13. [21]

    Paraphras- ing complex network: Network compression via factor trans- fer

    Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphras- ing complex network: Network compression via factor trans- fer. Advances in neural information processing systems, 31,

  14. [22]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 1, 2, 5, 6, 7, 8

    2009

  15. [23]

    Pruning filters for efficient convnets

    Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016. 1

    2016 arXiv

  16. [24]

    Locality guidance for improving vision trans- formers on tiny datasets

    Kehan Li, Runyi Yu, Zhennan Wang, Li Yuan, Guoli Song, and Jie Chen. Locality guidance for improving vision trans- formers on tiny datasets. In European Conference on Com- puter Vision, pages 110–127. Springer, 2022. 8

    2022

  17. [25]

    Auto- mated knowledge distillation via monte carlo tree search

    Lujun Li, Peijie Dong, Zimian Wei, and Ya Yang. Auto- mated knowledge distillation via monte carlo tree search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17413–17424, 2023. 8

    2023

  18. [26]

    Online knowledge distillation for efficient pose estimation

    Zheng Li, Jingwen Ye, Mingli Song, Ying Huang, and Zhi- geng Pan. Online knowledge distillation for efficient pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 11740–11750, 2021. 2

    2021

  19. [27]

    Curriculum tem- perature for knowledge distillation

    Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum tem- perature for knowledge distillation. In Proceedings of the 9 AAAI Conference on Artificial Intelligence , pages 1504– 1512, 2023. 5, 6

    2023

  20. [28]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceeding...

    2014

  21. [29]

    Function-consistent feature distillation

    Dongyang Liu, Meina Kan, Shiguang Shan, and Xilin CHEN. Function-consistent feature distillation. In The Eleventh International Conference on Learning Representa- tions (ICLR), 2023. 1, 2, 4, 5, 6, 7, 8

    2023

  22. [30]

    Im- proved knowledge distillation via teacher assistant

    Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Im- proved knowledge distillation via teacher assistant. In Pro- ceedings of the AAAI conference on artificial intelligence , pages 5191–5198, 2020. 1, 2

    2020

  23. [31]

    Tiny imagenet

    mnmoustafa and Mohammed Ali. Tiny imagenet. https: //kaggle.com/competitions/tiny- imagenet ,

  24. [32]

    Learning deconvolution network for semantic segmentation

    Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In Proceedings of the IEEE international conference on com- puter vision, pages 1520–1528, 2015. 1

    2015

  25. [33]

    Repre- sentation learning with contrastive predictive coding

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Repre- sentation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. 4

    2018 arXiv

  26. [34]

    You only look once: Unified, real-time object de- tection

    J Redmon. You only look once: Unified, real-time object de- tection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016. 1

    2016

  27. [35]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence, 39(6):1137–1149, 2016. 1

    2016

  28. [36]

    U- net: Convolutional networks for biomedical image segmen- tation

    Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In Medical image computing and computer-assisted intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, pa...

    2015

  29. [37]

    Berg, and Li Fei-Fei

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, San- jeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Chal- lenge. International Journal of Computer Vision ...

    2015

  30. [38]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 1

    2014 arXiv

  31. [39]

    Logit standardization in knowledge distillation

    Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xi- aochun Cao. Logit standardization in knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15731–15740, 2024. 1, 5, 6, 7, 8

    2024

  32. [40]

    Con- trastive representation distillation

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive representation distillation. In International Confer- ence on Learning Representations, 2020. 1, 2, 3, 5, 6, 7

    2020

  33. [41]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through at- tention. In International conference on machine learning , pages 10347–10357. PMLR, 2021. 8

    2021

  34. [42]

    Similarity-preserving knowl- edge distillation

    Frederick Tung and Greg Mori. Similarity-preserving knowl- edge distillation. In Proceedings of the IEEE/CVF inter- national conference on computer vision , pages 1365–1374,

  35. [43]

    Visualizing data using t-sne

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9 (11), 2008. 7, 8

    2008

  36. [44]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision , p...

    2021

  37. [45]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European con- ference on computer vision, pages 68–85. Springer, 2022. 8

    2022

  38. [46]

    Detectron2

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github. com/facebookresearch/detectron2, 2019

    2019

  39. [47]

    Quan- tization networks

    Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quan- tization networks. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 7308–7316, 2019. 1

    2019

  40. [48]

    Tokens-to-token vit: Training vision transformers from scratch on imagenet

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF in- ternational conference on computer vision , page...

  41. [49]

    Wide residual net- works

    Sergey Zagoruyko and Nikos Komodakis. Wide residual net- works. arXiv preprint arXiv:1605.07146, 2016

    2016 arXiv

  42. [50]

    Shufflenet: An extremely efficient convolutional neural net- work for mobile devices

    Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural net- work for mobile devices. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 6848–6856, 2018

    2018

  43. [51]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. arXiv preprint arXiv:2203.08679, 2022. 1, 2, 5, 6, 7, 8

    2022 arXiv

  44. [52]

    Dot: A distillation-oriented trainer

    Borui Zhao, Quan Cui, Renjie Song, and Jiajun Liang. Dot: A distillation-oriented trainer. arXiv preprint arXiv:2307.08436, 2023. 1, 2, 5, 6, 7, 8

    2023 arXiv

  45. [53]

    Knowledge distillation based on transformed teacher matching

    Kaixiang Zheng and En-Hui Yang. Knowledge distillation based on transformed teacher matching. In The Twelfth In- ternational Conference on Learning Representations (ICLR 2024), 2024. 1, 2, 5, 6, 7, 8 10

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.