Pith. sign in

REVIEW 4 major objections 6 minor 52 references

Multi-Level Decoupled Relational Distillation for Heterogeneous Architectures

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that heterogeneous knowledge distillation can transfer dark knowledge from teacher to student by decoupling logits into class-wise and sample-wise relations and aligning them separately at both logit and feature levels.

desk verdict A coherent decoupled relational distillation idea with a real problem framing, but the CIFAR-100 headline rests on inherited baselines and missing implementation details. read the letter →

arxiv 2502.06189 v1 pith:OGK53TXK submitted 2025-02-10 cs.CV

Yaoxin Yang , Peng Ye , Weihao Lin , Kangcong Li , Yan Wen , Jia Hao , Tao Chen This is my paper · ORCID
classification cs.CV
keywords knowledgedistillationheterogeneousarchitecturesrelationaldarkmulti-scalefeaturefusionclasstokenvisiontransformersMamba
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge distillation usually assumes the teacher and student share the same architecture; this paper tackles the harder setting where they do not. The authors argue that the two existing strategies for cross-architecture transfer each fail in an opposite way: sharpening the teacher's logits preserves confidence in the correct class but erases the dark knowledge encoded in the rest of the distribution, while directly applying relational distillation preserves that dark knowledge but over-smooths the student's predictions. Their solution, MLDR-KD, decouples logits into class-wise and sample-wise relations, aligns each relation with the teacher using KL divergence, and repeats the same alignment on dynamically fused multi-scale student features. If the claimed gains hold, cross-architecture distillation can be made broadly effective without hand-crafted feature-space alignment, which matters as new architectures such as Mamba arrive.

What carries the argument

Two modules carry the argument. Decoupled Finegrained Relation Alignment (DFRA) expands a logit tensor to three dimensions and forms two self-relation matrices, class-wise $R_{\text{class}}\in\mathbb{R}^{B\times N\times N}$ for each sample and sample-wise $R_{\text{sample}}\in\mathbb{R}^{N\times B\times B}$ for each class, then minimizes the KL divergence between teacher and student relations. Multi-Scale Dynamic Fusion (MSDF) splits the student into four stages, gives each stage a class token (a learned vector meant to summarize global semantic information for that stage), projects each stage's features into logit space, and uses an MLP plus softmax over the stacked class tokens to produce per-stage balancing weights; the weighted sum is a fused logit that DFRA aligns with the teacher. The decoupling is the load-bearing idea: it lets the student learn the teacher's inter-class and inter-sample structure without sacrificing confidence in the correct category.

What would settle it

On the paper's Tiny-ImageNet ResNet-18/ViT-S setup, replace the learned MSDF balancing weights with fixed equal weights and retrain; if top-1 accuracy does not drop below the reported 67.13%, the dynamic-fusion mechanism is not the source of the claimed improvement.

Watch

Extended reading notes

Core claim

The central claim is that dark knowledge can survive heterogeneous distillation if the teacher's logits are not treated as a single vector to imitate but are first decoupled into two fine-grained structures: a class-wise relation, which for each sample records similarity across all class pairs, and a sample-wise relation, which for each class records similarity across all sample pairs in the batch. Each structure is normalized and aligned to the teacher's corresponding structure by KL divergence, and the same decoupled alignment is applied to projections of the student's multi-stage features. The paper reports that this consistently beats the sharpened-logit baseline and vanilla relational distillation across CNNs, Transformers, MLPs, and Mamba-based students, with the largest improvements of 4.86% on CIFAR-100 and 2.78% on Tiny-ImageNet over the best available method.

Load-bearing premise

The framework assumes every stage of the student model can provide a class token that captures global semantic information, even though CNNs and MLPs do not natively produce class tokens, and the paper does not specify how such tokens are created for those architectures.

Editorial extensions

If this is right

  • Relational distillation can be applied across heterogeneous teacher/student pairs, not only within a single architecture family, because the decoupled relations are architecture-agnostic.
  • On CIFAR-100, the largest reported gain is 4.86% for a Transformer student with an MLP-Mixer teacher, and on Tiny-ImageNet the largest reported gain is 2.78% for a Transformer student with a Mamba teacher.
  • Feature-level distillation becomes more useful in heterogeneous settings when multi-stage student features are projected into logit space and fused with learned weights; the ablation shows accuracy rising with the number of fused stages.
  • Students from all four tested architecture families—CNNs, Transformers, MLPs, and Mambas—improve over the strongest baseline, which the paper takes as evidence of generality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural testable extension is whether the same decoupling helps in dense prediction tasks such as segmentation or detection, where the class-wise matrix could be computed over semantic categories and the sample-wise matrix over spatial positions.
  • The method's reliance on class tokens is the point most likely to limit transfer to non-ViT students; injecting a learned stage-level token rather than requiring a native one could make MSDF fully architecture-agnostic.
  • Because the relation matrices have sizes determined by batch size and class count, the class-wise component should carry more information on datasets with more classes; comparing gains across CIFAR-100 and Tiny-ImageNet could test this scaling prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MLDR-KD, a heterogeneous knowledge distillation framework that combines Decoupled Finegrained Relation Alignment (DFRA) at logit and feature levels with a Multi-Scale Dynamic Fusion (MSDF) module for the student's stage-wise features. The method is evaluated on CIFAR-100 and Tiny-ImageNet across CNN, Transformer, MLP, and Mamba architectures, with reported gains over KD, RKD, DKD, and OFA-KD baselines, including a headline +4.86% improvement on CIFAR-100. The authors claim new state-of-the-art performance in heterogeneous distillation.

Significance. If the reported gains are reproducible under controlled conditions, the method would offer a simple and architecture-agnostic way to improve heterogeneous distillation, and the extension to Mamba-based models is timely. The conceptual observation that direct relational KD over-amplifies dark knowledge at the expense of correct-class confidence is well motivated, and the planned code release would support reproducibility. However, the main empirical claims currently rest on uncontrolled baseline comparisons and missing implementation details, so the significance cannot be fully assessed from the manuscript as written.

major comments (4)
  1. [§4.1, Table 1] Most OFA-KD baseline rows in Table 1 are 'inherited from OFAKD' rather than reproduced under the same training recipe as the MLDR-KD rows. Because the abstract and Section 1 claim gains 'under the same conditions,' the headline +4.86% improvement for DeiT-T with a Mixer-B/16 teacher, and the +2.89% improvement for Swin-P with a ConvNeXt-T teacher, rest on an uncontrolled comparison. Please re-run all CIFAR-100 baselines with the same code, data augmentation, epochs, schedule, and seeds as the proposed method, and report the baselines together with the proposed method.
  2. [§3.2.2, Eq. (13)] The MSDF module assumes a class token x_i at every student stage, but the paper never specifies how these tokens are obtained for CNNs (ResNet18, MobileNetV2) and MLPs (ResMLP-S12), which have no native class-token mechanism. Without this definition, the feature-level component of MLDR-KD is underspecified for most of the claimed architecture coverage. Please provide the exact construction (e.g., global average pooling, a learned token, or a projection of the spatial feature map) and discuss whether early-stage features can provide the global semantic information that the module assumes.
  3. [§3.2.1, Eqs. (6) and (12); §4.1] The temperature T in Eq. (6) and the balance coefficient λ in Eq. (12) are never assigned values, and the projector that maps stage features to logits is not described. These are free parameters that can materially change the reported accuracy; without their values and a sensitivity analysis, the experiments cannot be reproduced and the robustness of the claimed gains to these choices is unclear.
  4. [Tables 1–5] All results are single runs with no multiple seeds, standard deviations, or significance tests. Many reported deltas are below one percentage point (e.g., 0.07, 0.09, 0.12, 0.14, 0.15, 0.20 in Tables 1 and 2), so a single-seed comparison cannot support the strong state-of-the-art claim. Please report mean±std across at least three seeds for the main comparisons and ablations.
minor comments (6)
  1. [Table 4] The header in Table 4 reads 'Acc@l' instead of 'Acc@1'.
  2. [§3.2.2] The sentence 'as shown in Figure 2' appears to be the wrong reference; the four-stage split is illustrated in Figure 3, while Figure 2 shows feature-visualization comparisons.
  3. [Table 5] The semantics of the first row of Table 5 (feature level ✓, MSDF ×, DFRA ×) are unclear, since it seems to be a feature-level baseline without any alignment loss; please explain what is trained in this row.
  4. [§3.2.1, Eq. (9)] Eq. (9) defines Rsample but then writes Rbatch ∈ R^{N×B×B}; the variable name should be consistent.
  5. [§2] The citation for FASD [13] appears inconsistent with the described method (aligning Transformer and Mamba features); please verify the reference.
  6. [Tables 1–2] The 'From Scratch' columns are not defined in §4.1; it should be stated explicitly that these are student/teacher accuracies trained without distillation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: experimental KD paper with self-contained loss design; inherited baselines are an experimental-control issue, not circularity.

full rationale

The paper is an empirical knowledge-distillation method paper. Its central claim is that the proposed DFRA and MSDF losses improve student accuracy in heterogeneous distillation, and this is supported by training runs reported in Tables 1–5. No parameter is fitted to the reported accuracy numbers and then renamed as a prediction; the loss components (Eqs. 6–15) are defined independently of the experimental outcomes. There is no self-citation chain used to justify the method, and no uniqueness theorem or ansatz is imported from the authors' own prior work to force the design. The only notable weakness is that Table 1's caption states 'most of the experimental results are inherited from OFAKD, while the additional experiments we conducted are marked with *,' meaning some baseline comparisons are not re-run under identical conditions; this weakens the controlled-comparison claim in the abstract but is not a circularity of the derivation. Similarly, the MSDF module's class-token assumption for CNNs and MLPs is underspecified, but underspecification is not circularity. The ablation studies select design choices on one teacher-student pair, which is mild post hoc selection rather than a circular step. Therefore the derivation chain is self-contained, and the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The central claim rests on empirical performance rather than a theoretical derivation. The main free choices are the temperature T and loss weight lambda (values unreported) and the ad hoc construction of class tokens for non-transformer architectures. The domain assumptions about dark knowledge and logit comparability are standard in KD but are not formally justified.

free parameters (3)
  • Temperature T in decoupled relations
    Controls the sharpness of the relation matrices in Equations 6-9; its value is not reported anywhere in the paper.
  • Balance coefficient lambda in total loss (Equation 12)
    Weights the KL distillation loss against the relation alignment losses; value not reported.
  • Number of stages in MSDF = 4
    Chosen based on ablation in Table 3 on a single teacher-student pair; not justified as optimal across all settings.
assumptions (3)
  • domain assumption The teacher's logits contain 'dark knowledge' in the relationships between classes and between samples that is transferable across heterogeneous architectures.
    This is the central motivation of the paper, stated in Section 1 and used throughout without formal justification.
  • domain assumption The relative scaling of logits across heterogeneous models is comparable after applying temperature scaling.
    Equations 6-9 assume that teacher and student logits can be directly compared after the same temperature division, despite architectural differences.
  • ad hoc to paper A 'class token' can be introduced at every stage of a student model, including CNNs and MLPs, to capture global semantic information.
    The paper introduces this mechanism in Section 3.2.2 without explaining how the token is generated for non-transformer architectures or why it should be semantically meaningful.
invented entities (1)
  • Stage-wise class token in student models (including CNNs and MLPs)
    purpose: Used in the MSDF module to compute dynamic balancing weights for fusing multi-scale feature projections.
    Class tokens are a natural concept in transformer architectures, but the paper does not provide evidence that inserting them into CNNs or MLPs yields meaningful global semantics. No external validation is offered.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Level Decoupled Relational Distillation for Heterogeneous Architectures." pith.science (2026). https://pith.science/paper/OGK53TXK

@misc{pith2026250206189,
  author       = {Pith},
  title        = {Pith review of: Multi-Level Decoupled Relational Distillation for Heterogeneous Architectures},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OGK53TXK}},
  note         = {Machine review of arXiv:2502.06189}
}
read the original abstract

Heterogeneous distillation is an effective way to transfer knowledge from cross-architecture teacher models to student models. However, existing heterogeneous distillation methods do not take full advantage of the dark knowledge hidden in the teacher's output, limiting their performance.To this end, we propose a novel framework named Multi-Level Decoupled Relational Knowledge Distillation (MLDR-KD) to unleash the potential of relational distillation in heterogeneous distillation. Concretely, we first introduce Decoupled Finegrained Relation Alignment (DFRA) in both logit and feature levels to balance the trade-off between distilled dark knowledge and the confidence in the correct category of the heterogeneous teacher model. Then, Multi-Scale Dynamic Fusion (MSDF) module is applied to dynamically fuse the projected logits of multiscale features at different stages in student model, further improving performance of our method in feature level. We verify our method on four architectures (CNNs, Transformers, MLPs and Mambas), two datasets (CIFAR-100 and Tiny-ImageNet). Compared with the best available method, our MLDR-KD improves student model performance with gains of up to 4.86% on CIFAR-100 and 2.78% on Tiny-ImageNet datasets respectively, showing robustness and generality in heterogeneous distillation. Code will be released soon.

Figures

Figures reproduced from arXiv: 2502.06189 by the authors.

Figure 1
Figure 1. Conceptual comparisons of different knowledge distil [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparisons of feature visualizations when using kinds [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the proposed MLDR-KD framework. It comprises two main components: Decoupled Finegrained Relation Align [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparisons of the averaged prediction distribution of all samples of single category among OFA-KD ((a),(d),(g)), RKD [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparisons of feature visualizations between OFA [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

52 extracted references · 43 canonical work pages

  1. [1]

    Distilling the knowledge in a neural network

    Geoffrey Hinton. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 1, 3, 6

    arXiv 2015

  2. [2]

    Patch slimming for efficient vision transformers, 2022

    Yehui Tang, Kai Han, Yunhe Wang, Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers, 2022. 1

    work page 2022

  3. [3]

    Be your own teacher: Improve the performance of convolutional neural networks via self distillation

    Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3713–3722, 2019. 1

    work page 2019

  4. [4]

    Distillation- based training for multi-exit architectures

    Mary Phuong and Christoph H Lampert. Distillation- based training for multi-exit architectures. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pages 1355–1364, 2019. 1

    work page 2019

  5. [5]

    Msd: Multi-self-distillation learning via multi- classifiers within deep neural networks.arXiv preprint arXiv:1911.09418, 2019

    Yunteng Luan, Hanyu Zhao, Zhi Yang, and Yafei Dai. Msd: Multi-self-distillation learning via multi- classifiers within deep neural networks.arXiv preprint arXiv:1911.09418, 2019. 1

    arXiv 1911

  6. [6]

    Knowledge dis- tillation by on-the-fly native ensemble

    Xiatian Zhu, Shaogang Gong, et al. Knowledge dis- tillation by on-the-fly native ensemble. Advances in neural information processing systems, 31, 2018. 1

    work page 2018

  7. [7]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023. 1

    arXiv 2023

  8. [8]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000–16009, 2022. 1

    work page 2022

Show all 52 references
  1. [9]

    Sim- mim: A simple framework for masked image model- ing

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jian- min Bao, Zhuliang Yao, Qi Dai, and Han Hu. Sim- mim: A simple framework for masked image model- ing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9653– 9663, 2022. 1

    2022

  2. [10]

    BEiT: BERT pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT pre-training of image transformers. In International Conference on Learning Representa- tions, 2022. 1

    2022

  3. [11]

    Vision mamba: Efficient visual representation learning with bidirectional state space model

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. In Forty-first Inter- national Conference on Machine Learning , 2024. 2, 7

    2024

  4. [12]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 2, 7

    2016

  5. [13]

    Un- leashing the potential of mamba: Boosting a lidar 3d sparse detector by using cross-model knowledge dis- tillation

    Rui Yu, Runkai Zhao, Jiagen Li, Qingsong Zhao, Songhao Zhu, HuaiCheng Yan, and Meng Wang. Un- leashing the potential of mamba: Boosting a lidar 3d sparse detector by using cross-model knowledge dis- tillation. arXiv preprint arXiv:2409.11018, 2024. 1, 3

    2024

  6. [14]

    Ddk: Distilling domain knowledge for efficient large language models

    Jiaheng Liu, Chenchen Zhang, Jinyang Guo, Yuanx- ing Zhang, Haoran Que, Ken Deng, Zhiqi Bai, Jie Liu, Ge Zhang, Jiakai Wang, Yanan Wu, Congnan Liu, Wenbo Su, Jiamang Wang, Lin Qu, and Bo Zheng. Ddk: Distilling domain knowledge for efficient large language models. ArXiv, abs/24...

    2024 arXiv

  7. [15]

    The mamba in the llama: Distilling and accelerating hybrid models

    Junxiong Wang, Daniele Paliotta, Avner May, Alexan- der M Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models. In The Thirty-eighth Annual Conference on Neural Informa- tion Processing Systems, 2024. 1

    2024

  8. [16]

    Deit iii: Revenge of the vit

    Hugo Touvron, Matthieu Cord, and Herv´e J´egou. Deit iii: Revenge of the vit. In European conference on computer vision, pages 516–533. Springer, 2022. 1

    2022

  9. [17]

    Repvit: Revisiting mobile cnn from vit perspective

    Ao Wang, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Repvit: Revisiting mobile cnn from vit perspective. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 15909–15920, 2024. 1

    2024

  10. [18]

    One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation

    Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu. One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation. Advances in Neural Infor- mation Processing Systems, 36, 2024. 2, 3, 6

    2024

  11. [19]

    Correlation congruence for knowledge distil- lation

    Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distil- lation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5007–5016,

  12. [20]

    A com- prehensive overhaul of feature distillation

    Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A com- prehensive overhaul of feature distillation. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019. 2

    2019

  13. [21]

    Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer

    Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. In International Conference on Learning Representations, 2022. 2

    2022

  14. [22]

    Fitnets: Hints for thin deep nets

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. 2, 3

    2014 arXiv

  15. [23]

    Knowledge transfer via distillation of activation boundaries formed by hidden neurons

    Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In AAAI Conference on Artificial Intelligence, 2018. 2

    2018

  16. [24]

    A gift from knowledge distillation: Fast opti- mization, network minimization and transfer learning

    Junho Yim, Donggyu Joo, Ji-Hoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast opti- mization, network minimization and transfer learning. 2017 IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 7130–7138, 2017. 2

    2017

  17. [25]

    Damianou, Neil D

    Sungsoo Ahn, Shell Xu Hu, Andreas C. Damianou, Neil D. Lawrence, and Zhenwen Dai. Variational in- formation distillation for knowledge transfer. 2019 IEEE/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 9155–9163, 2019. 2

    2019

  18. [26]

    Wang, Yan Feng, and Chun Chen

    Defang Chen, Jianhan Mei, Hailin Zhang, C. Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 11923–11932, 2022. 2

    2022

  19. [27]

    Distilling knowledge via knowledge review

    Pengguang Chen, Shu Liu, Hengshuang Zhao, and Ji- aya Jia. Distilling knowledge via knowledge review. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5006–5015, 2021. 2

    2021

  20. [28]

    Class attention transfer based knowledge distillation

    Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 11868– 11877, 2023. 2, 3

    2023

  21. [29]

    Online knowledge distillation for efficient pose estimation

    Zheng Li, Jingwen Ye, Mingli Song, Ying Huang, and Zhigeng Pan. Online knowledge distillation for efficient pose estimation. 2021 IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pages 11720–11730, 2021. 2

    2021

  22. [30]

    Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and G. Wang. Knowledge distillation via the target-aware trans- former. 2022 IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) , pages 10905– 10914, 2022. 2

    2022

  23. [31]

    Relational knowledge distillation

    Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 3967–3976, 2019. 2, 3, 6

    2019

  24. [32]

    Contrastive representation distillation

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint arXiv:1910.10699, 2019. 3

    1910 arXiv

  25. [33]

    Masked generative dis- tillation

    Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative dis- tillation. In European Conference on Computer Vi- sion, pages 53–69. Springer, 2022. 3

    2022

  26. [34]

    Focal and global knowledge distillation for detectors

    Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 4643– 4652, 2022. 3

    2022

  27. [35]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Ji- ajun Liang. Decoupled knowledge distillation. InPro- ceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 11953–11962,

  28. [36]

    Logit standardization in knowl- edge distillation

    Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, and Xiaochun Cao. Logit standardization in knowl- edge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 15731–15740, 2024. 3

    2024

  29. [37]

    Knowledge distillation from a stronger teacher

    Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. Advances in Neural Information Processing Systems, 35:33716–33727, 2022. 3

    2022

  30. [38]

    Cross-image re- lational knowledge distillation for semantic segmenta- tion, 2022

    Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image re- lational knowledge distillation for semantic segmenta- tion, 2022. 3

    2022

  31. [39]

    Chawla, Neil Shah, and Tong Zhao

    Zhichun Guo, William Shiao, Shichang Zhang, Yozen Liu, Nitesh V . Chawla, Neil Shah, and Tong Zhao. Linkless link prediction via relational distillation,

  32. [40]

    Cross-architecture knowledge distillation

    Yufan Liu, Jiajiong Cao, Bing Li, Weiming Hu, Jingt- ing Ding, and Liang Li. Cross-architecture knowledge distillation. In Proceedings of the Asian conference on computer vision, pages 3396–3411, 2022. 3

    2022

  33. [41]

    Cross-architecture distillation for face recognition

    Weisong Zhao, Xiangyu Zhu, Zhixiang He, Xiao-Yu Zhang, and Zhen Lei. Cross-architecture distillation for face recognition. In Proceedings of the 31st ACM International Conference on Multimedia, pages 8076– 8085, 2023. 3

    2023

  34. [42]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 6

    2009

  35. [43]

    Tiny imagenet visual recogni- tion challenge

    Ya Le and Xuan Yang. Tiny imagenet visual recogni- tion challenge. CS 231N, 7(7):3, 2015. 6

    2015

  36. [44]

    Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation

    Andrew Howard, Andrey Zhmoginov, Liang-Chieh Chen, Mark Sandler, and Menglong Zhu. Inverted residuals and linear bottlenecks: Mobile networks for classification, detection and segmentation. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pag...

    2018

  37. [45]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012–10022, 2021. 7

    2021

  38. [46]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou. Training data-efficient image transformers & distillation through attention. In International con- ference on machine learning , pages 10347–10357. PMLR, 2021. 7

    2021

  39. [47]

    Tokens-to-token vit: Training vi- sion transformers from scratch on imagenet

    Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yu- jun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vi- sion transformers from scratch on imagenet. In Pro- ceedings of the IEEE/CVF international conference on computer vision, p...

    2021

  40. [48]

    Resmlp: Feedforward networks for image classification with data-efficient training

    Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE transactions on pa...

    2022

  41. [49]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Con...

    2020

  42. [50]

    Mlp-mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems , 34:24...

  43. [51]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022. 7

    2022

  44. [52]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017. 7

    2017

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.