Pith. sign in

REVIEW 5 major objections 5 minor 80 references

Cross-Architecture Distillation Made Simple with Redundancy Suppression

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that cross-architecture knowledge distillation can be reduced to suppressing redundant architecture-exclusive information, and that a simple loss forcing the teacher–student Pearson correlation matrix toward the identity…

desk verdict Simple Barlow-Twins-style loss for cross-architecture distillation with strong but possibly over-claimed OFA gains; the MLP-student OFA baselines look broken. read the letter →

arxiv 2507.21844 v1 pith:PNGHUM5Y submitted 2025-07-29 cs.CV

Weijia Zhang , Yuehao Liu , Wu Ran , Chao Ma This is my paper · ORCID
classification cs.CV
keywords cross-architectureknowledgedistillationredundancysuppressionfeaturedecorrelationinvariancemaximizationrepresentationlearningPearsoncross-correlationvisiontransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish a simple alternative to OFA for cross-architecture knowledge distillation: instead of building architecture-tailored projectors to unify heterogeneous features, it casts the transfer as suppression of redundant, architecture-exclusive information. The proposed RSD loss drives the Pearson cross-correlation matrix between teacher and student penultimate embeddings toward the identity, aligning matched feature units while decorrelating the rest, so that the shared architecture-agnostic signal is what survives. A lightweight AAD module aligns embedding dimensions and shields the student's internal representation, letting the student keep useful architecture-specific abilities while learning from the teacher. The authors report that RSD outperforms OFA on CIFAR-100 and ImageNet-1k across CNN, ViT, and MLP teacher–student pairs using a fraction of OFA's parameter overhead, and because AAD is discarded after training, inference cost is unchanged.

What carries the argument

The central object is the Pearson cross-correlation matrix P in Eq. 4, a D×D matrix of batch-wise correlations between each teacher feature unit and each student feature unit. The optimisation target is the identity matrix T, and the RSD loss is the MSE between P and T, with off-diagonal terms weighted by κ so the decorrelation pressure is explicit. The machinery has two roles: diagonal entries enforce cross-architecture invariance between corresponding units, and off-diagonal entries suppress redundancy in the heterogeneous representation space. The AAD module, a two-layer MLP with BatchNorm and GeLU that expands then adapts the student embedding to the teacher's dimension, lets the loss operate on a decoupled embedding rather than directly on the student's internal representation; it is discarded after training.

What would settle it

Construct a teacher–student pair where the task-relevant shared knowledge lives only in nonlinear interactions among feature units, for instance a label that is the XOR of two unit activations with all pairwise Pearson correlations near zero; if RSD then transfers no more knowledge than training from scratch, the linear-correlation premise is falsified. A complementary check is to train with RSD and probe the student's off-diagonal residual components for label information.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that cross-architecture distillation does not need feature-space unification: a redundancy-suppression objective on the final embeddings is enough. For a batch of teacher and student embeddings, the method computes the Pearson cross-correlation matrix P of Eq. 4, treats the identity matrix as the target T, and minimises the MSE between P and T with a heavier weight on off-diagonal entries. This simultaneously maximises invariance between the same feature units and decorrelates different units across the heterogeneous architectures, which the paper argues extracts architecture-agnostic knowledge while suppressing teacher-exclusive and student-exclusive patterns. The paper further claims that RSD works on logits alone, that it can be composed with OFA's multi-stage features, and that it remains competitive in same-architecture distillation.

Load-bearing premise

The RSD loss assumes that the linear Pearson correlation matrix between teacher and student feature units faithfully carries the architecture-agnostic knowledge, and that forcing that matrix to the identity removes only redundant architecture-exclusive information without discarding task-relevant shared signal.

Editorial extensions

If this is right

  • If RSD works as reported, it replaces OFA as the default cross-architecture distillation baseline: it is simpler, needs no architecture-specific projectors, and uses a fraction of the extra parameters.
  • Because RSD acts on penultimate embeddings rather than multi-stage features, it removes the need for depthwise convolutions for CNN features and token/attention operations for ViT and MLP features that OFA requires.
  • The AAD module is dropped at inference, so the student network carries no extra parameters or compute from the distillation setup.
  • RSD's logit-only variant outperforms KD, DKD, and OFA's logit losses in the reported comparisons, so the same objective serves black-box distillation where intermediate features are unavailable.
  • RSD composes with OFA's projected multi-stage features, and adding stages generally improves results, suggesting the redundancy-suppression objective is complementary to existing cross-architecture machinery.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear-correlation premise holds, heterogeneous vision architectures share more linear unit-wise structure than the 'conflicting representations' narrative suggests; this could be tested by probing which information is discarded in the off-diagonal residual.
  • The same P-to-identity objective could be carried to cross-modal or cross-domain distillation by replacing 'architecture' with 'modality' or 'domain', though the current 1-D-embedding design would need a spatial analogue for dense tasks.
  • The paper notes that RSD is sensitive to λ and κ and does not use 2-D features, so its current form is unlikely to transfer directly to object detection; extending the decorrelation to spatial maps is the obvious next test.
  • The identity target is a heuristic, and a learned soft target or an adaptive diagonal/off-diagonal balance might make RSD less sensitive to hyperparameters and more robust across teacher–student pairs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. This paper proposes Redundancy Suppression Distillation (RSD), a loss for cross-architecture knowledge distillation that drives the Pearson cross-correlation matrix between teacher and student penultimate embeddings toward identity, with an off-diagonal weight κ, and adds a lightweight Architecture-Agnostic Knowledge Decoupler (AAD) to align dimensions and preserve student-exclusive knowledge. Experiments on CIFAR-100 (12 teacher-student pairs) and ImageNet-1k (15 pairs) report average gains over from-scratch training of +10.69 and +2.34, exceeding the OFA baseline gains of +7.47 and +2.20 while using a fraction of OFA's extra parameters. The paper also demonstrates that RSD works as a logit distiller and can be integrated with OFA. I note that the concern about collapsed OFA baselines in Table 1 is not supported by the manuscript: the entries 45.47 and 11.05 belong to the FitNets and DIST columns respectively, while the OFA entries for the MLP-student rows are 81.22 and 80.63, which are consistent with the logit-only OFA numbers in Table 5.

Significance. If the reported results are reproducible, RSD offers a simple, universal, and parameter-efficient objective for cross-architecture distillation: a single loss term plus a discardable two-layer module. The paper provides extensive empirical coverage (27 heterogeneous teacher-student pairs), ablations of the loss components, and a compatibility study with OFA, all of which are concrete strengths. The central limitations are the absence of error bars and the overstatement of the ImageNet advantage over OFA, since RSD is worse than OFA on five of the fifteen ImageNet pairs and the average margin is only 0.14 percentage points.

major comments (5)
  1. [Tables 1–2, §4.2–4.3] The claim that RSD 'outperforms OFA' on ImageNet-1k is too strong. RSD is below OFA on five of fifteen pairs (Mixer-B/16→MobileNetV2, ResNet50→DeiT-T, Mixer-B/16→DeiT-T, Mixer-B/16→Swin-N, ResNet50→ResMLP-S12) and the average margin is only +0.14%. Please qualify the claim as an average improvement, report the per-pair win/loss counts, or provide a paired statistical test.
  2. [§4.1 and all result tables] All experiments are single runs with no multiple seeds or error bars, and the Limitations section concedes that RSD can be sensitive to λ and κ. Given that several ImageNet comparisons differ by well under 0.2 percentage points, the observed margins cannot be distinguished from training noise without variance estimates. Please report mean±std over at least three seeds for the main tables, and include a sensitivity analysis for λ and κ.
  3. [§3.2, Eq. (4)–(8)] The concept of 'architecture-agnostic knowledge' is defined only as what the RSD objective preserves; it is never measured independently. The CKA visualizations in Figure 4 show increased overall representation similarity but do not establish that the transferred components are specifically architecture-agnostic. Please soften this conceptual claim or add a direct test, such as distilling from the RSD-trained student to a third architecture, to demonstrate transfer of architecture-agnostic knowledge.
  4. [Algorithm 1 vs Eq. (4)] Eq. (4) defines Pearson correlation with mean subtraction and standard deviation normalization, but Algorithm 1 applies a 'Normalize' operation and computes torch.mm without explicit centering. If 'Normalize' is L2 normalization over the batch, the implemented loss is cosine similarity, not Pearson correlation; if it is z-score normalization, the pseudocode should say so. The discrepancy between the mathematical definition and the implementation must be resolved for reproducibility.
  5. [§4.1 and Limitations] The paper provides no code, no seeds, and no explicit values for the hyperparameters λ and κ, and the experimental setup simply states that configurations follow OFA. Because the Limitation section itself reports sensitivity to these hyperparameters, the protocol is not sufficiently specified to reproduce the results from the manuscript alone. Please release code or provide a detailed supplementary protocol with hyperparameter values and training budgets.
minor comments (5)
  1. [Abstract and Conclusion] The abstract and conclusion state that RSD 'outperforms OFA' on both CIFAR-100 and ImageNet-1k without qualification; the ImageNet claim should be stated as an average gain, not a universal superiority.
  2. [Figure 3] The parameter-overhead numbers mentioned in the text (e.g., 28.2M versus 9.6M) should be legible in the figure itself, since the figure is the primary evidence for the 'fraction of parameters' claim.
  3. [Table 5] For Mixer-B/16→DeiT-T, the full OFA score in Table 1 (73.90) is notably higher than the logit-only OFA score in Table 5 (70.69); a brief comment on why the stage-wise projectors help substantially for this pair would help the reader interpret the comparison.
  4. [Algorithm 1] The 'Normalize' operation should be defined exactly (per-dimension mean/standard deviation, L2 normalization, or something else) to avoid ambiguity in the pseudocode.
  5. [Figure 4] The CKA visualizations are difficult to read at the printed size; enlarging the figure or separating the subplots would improve the clarity of the similarity analysis.

Circularity Check

1 steps flagged · score 4.0 of 10

RSD's empirical benchmark comparisons are independent, but its central conceptual claim—that the loss extracts 'architecture-agnostic knowledge'—is self-definitional, since that knowledge is operationalized only as the identity correlation target.

  1. self definitional [Section 3.2, Eqs. (4)-(8); Conclusion, Section 5]
    "To extract the “commonality” between the pair of heterogeneous representations, we learn zs that leads to maximised invariance between itself and zt for the same feature unit [3,11,61,72]. This translates to driving the diagonal elements of P to 1. ... This makes the redundancy suppressed within the heterogeneous representation space, which implicitly facilitates the extraction of architecture-agnostic information. ... RSD employs invariance maximisation and feature decorrelation objectives to extract arch-agnostic knowledge common to heterogeneous architectures."

    The phrase 'architecture-agnostic knowledge' has no operational definition outside the RSD objective. Eq. (7) defines L_RSD = d(P(h(zs), zt), T) with T equal to the identity (Eqs. 5-6), so minimizing the loss is exactly driving P to I. The Conclusion converts this optimization target into an empirical achievement: 'RSD employs invariance maximisation and feature decorrelation objectives to extract arch-agnostic knowledge common to heterogeneous architectures.' That sentence is true by construction—whatever content makes P approximate I is labeled arch-agnostic knowledge—rather than by independent measurement.

full rationale

The accuracy comparisons in Tables 1, 2, 5, and 6 are independent external evidence: RSD is tested against KD, DKD, DIST, OFA, and other baselines, and those numbers do not reduce to the method's definition. There is no load-bearing self-citation chain: the authors' own prior works (refs. [74]-[76], [80]) are not used to justify the core objective, and no uniqueness theorem is imported from the authors. The Barlow-Twins-style decorrelation is explicitly cited as prior art rather than smuggled in, so the 'ansatz smuggled via citation' pattern does not apply. The circular element is confined to the conceptual framing: 'architecture-agnostic knowledge' is never measured or defined independently; it is operationalized as the identity correlation target. The CKA analysis in Figure 4 measures representation similarity, not whether the transferred content is task-relevant shared semantics, so it does not break the definitional circle. The limitations section itself concedes sensitivity to hyperparameters lambda and kappa and less prominent ImageNet gains, which weakens the empirical claims but is not circularity. Separately, the OFA baseline inconsistency (Table 1 full OFA scores of 45.47 and 11.05 versus Table 5's OFA logit-only score of 78.87 on the same ConvNeXt-T-to-ResMLP-S12 pair) is a serious correctness risk for the headline comparison, but an under-tuned baseline is a reproducibility concern, not a circularity, and is not scored here. Overall, the empirical contribution is independent, while the conceptual 'arch-agnostic knowledge extraction' claim is self-definitional; a moderate score of 4 reflects that partial circularity.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The method's empirical comparisons are grounded in external baselines, but its central conceptual objects (architecture-agnostic knowledge, student-exclusive knowledge) are not independently defined. The main free parameters are lambda and kappa, whose values are not given in the main text. The assumptions above are all load-bearing for the interpretation that RSD works because it suppresses redundancy.

free parameters (2)
  • lambda = not specified in main text
    Balancing weight between CE and L_RSD in Eq. (8); the paper notes performance is sensitive to it and defers values to the supplementary.
  • kappa = not specified in main text
    Weight multiplying off-diagonal entries of the correlation matrix in Algorithm 1; sensitivity noted in Limitations.
assumptions (5)
  • domain assumption Pearson correlation between teacher and student feature units captures the shared architecture-agnostic semantic content.
    Eq. (4)-(6) construct the RSD loss entirely from linear cross-correlations; no nonlinear or higher-order statistics are used, and no evidence is given that linear correlations suffice.
  • ad hoc to paper Driving the teacher-student cross-correlation matrix to the identity suppresses only redundant architecture-exclusive information and does not discard task-relevant knowledge.
    This is the central mechanism in Section 3.2 and Figure 2; the paper motivates it by analogy to redundancy reduction in unsupervised learning rather than by a derivation.
  • domain assumption Architecture-exclusive information is redundant for the distillation task and can be safely suppressed.
    Section 3.2 frames teacher-specific and student-specific patterns as 'redundant'; this is an assumption about the nature of heterogeneous representations, supported only by citations [18,43,48].
  • domain assumption Penultimate-layer embeddings are less architecture-specific than intermediate features and therefore better suited for transferring arch-agnostic knowledge.
    Section 3.2 Discussion argues this informally; it is not measured or proven.
  • ad hoc to paper The AAD decoupler preserves beneficial student-exclusive knowledge while keeping the RSD objective effective.
    Ablation in Table 4 shows w/o AAD is worse, but there is no mechanism or analysis explaining how the two-layer MLP decouples knowledge.
invented entities (1)
  • architecture-agnostic knowledge
    purpose: The hypothesized common semantic content that RSD extracts from heterogeneous teacher and student embeddings.
    The concept is defined only through the RSD objective; no independent measurement or falsifiable handle is provided, so the claim that RSD extracts it is not separately testable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-Architecture Distillation Made Simple with Redundancy Suppression." pith.science (2026). https://pith.science/paper/PNGHUM5Y

@misc{pith2026250721844,
  author       = {Pith},
  title        = {Pith review of: Cross-Architecture Distillation Made Simple with Redundancy Suppression},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PNGHUM5Y}},
  note         = {Machine review of arXiv:2507.21844}
}
read the original abstract

We describe a simple method for cross-architecture knowledge distillation, where the knowledge transfer is cast into a redundant information suppression formulation. Existing methods introduce sophisticated modules, architecture-tailored designs, and excessive parameters, which impair their efficiency and applicability. We propose to extract the architecture-agnostic knowledge in heterogeneous representations by reducing the redundant architecture-exclusive information. To this end, we present a simple redundancy suppression distillation (RSD) loss, which comprises cross-architecture invariance maximisation and feature decorrelation objectives. To prevent the student from entirely losing its architecture-specific capabilities, we further design a lightweight module that decouples the RSD objective from the student's internal representations. Our method is devoid of the architecture-specific designs and complex operations in the pioneering method of OFA. It outperforms OFA on CIFAR-100 and ImageNet-1k benchmarks with only a fraction of their parameter overhead, which highlights its potential as a simple and strong baseline to the cross-architecture distillation community.

Figures

Figures reproduced from arXiv: 2507.21844 by the authors.

Figure 1
Figure 1. A comparison of RSD and OFA. Compared to the pi￾oneering method of OFA, RSD is simpler in design, stronger in performance, and lower in complexity. Recently, with the rise of novel vision architectures such as ViTs [13, 37] and MLPs [56, 58], it becomes in￾creasingly relevant for the community to study knowledge distillation across distinct architectures, a task known as cross-architecture knowledge distillation (CA… view at source ↗
Figure 2
Figure 2. A schematic diagram of RSD for cross-architecture knowledge distillation. RSD employs a redundancy suppression objective (i.e., RSD loss) to extract and transfer architecture-agnostic knowledge that is common to both teacher and student architectures. It uses a lightweight Architecture-Agnostic Knowledge Decoupler (AAD) module to align student representation dimension to the teacher’s, while decoupling the RSD dicta… view at source ↗
Figure 3
Figure 3. A comparison of the computational overheads. For ConvNeXt-T-to-Swin-N distillation, RSD has a 0.20% advantage over OFA, when OFA is only 0.25% higher than DIST [26], by using only 9% as many extra parameters as OFA. RSD achieves an average gain of 2.34% across the 15 heterogeneous model pairs, highest amongst all methods. 4.4. Further analysis Effect of proposed designs. We conduct ablation experi￾ments to gauge the… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Cross-architectural feature similarities measured by CKA. Brighter colours indicate higher similarity and vice versa. Top: ConvNeXt-T teacher and ResMLP-S12 student. Bottom: ViT-S teacher and MobileNetV2 student. similarity between different layers and stages in hetero…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

80 extracted references · 75 canonical work pages

  1. [1]

    Towards a theory of early visual processing

    Joseph J Atick and A Norman Redlich. Towards a theory of early visual processing. Neural computation, 2(3):308–320,

    work page

  2. [2]

    Vi- creg: Variance-invariance-covariance regularization for self- supervised learning

    Adrien Bardes, Jean Ponce, and Yann LeCun. Vi- creg: Variance-invariance-covariance regularization for self- supervised learning. ICLR, 2022. 3

    work page 2022

  3. [3]

    Unsupervised learning

    Horace B Barlow. Unsupervised learning. Neural computa- tion, 1(3):295–311, 1989. 2, 3, 4

    work page 1989

  4. [4]

    Cross-layer distillation with semantic calibration

    Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross-layer distillation with semantic calibration. In AAAI, 2021. 3

    work page 2021

  5. [5]

    Knowledge distillation with the reused teacher classifier

    Defang Chen, Jian-Ping Mei, Hailin Zhang, Can Wang, Yan Feng, and Chun Chen. Knowledge distillation with the reused teacher classifier. In CVPR, 2022. 2, 8

    work page 2022

  6. [6]

    Distilling knowledge via knowledge review

    Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowledge review. InCVPR, 2021. 1, 3, 8

    work page 2021

  7. [7]

    Dearkd: data-efficient early knowledge distillation for vision transformers

    Xianing Chen, Qiong Cao, Yujie Zhong, Jing Zhang, Shenghua Gao, and Dacheng Tao. Dearkd: data-efficient early knowledge distillation for vision transformers. In CVPR, 2022. 3

    work page 2022

  8. [8]

    Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution

    Yunpeng Chen, Haoqi Fan, Bing Xu, Zhicheng Yan, Yan- nis Kalantidis, Marcus Rohrbach, Shuicheng Yan, and Ji- ashi Feng. Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution. In ICCV, 2019. 3

    work page 2019

Show all 80 references
  1. [9]

    Xception: Deep learning with depthwise separable convolutions

    Franc ¸ois Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. 2, 4

    2017

  2. [10]

    Kd-dlgan: Data limited image generation via knowledge distillation

    Kaiwen Cui, Yingchen Yu, Fangneng Zhan, Shengcai Liao, Shijian Lu, and Eric P Xing. Kd-dlgan: Data limited image generation via knowledge distillation. In CVPR, 2023. 1

    2023

  3. [11]

    Non-linear feature extraction by redundancy reduction in an unsupervised stochastic neural network

    Gustavo Deco and L Parra. Non-linear feature extraction by redundancy reduction in an unsupervised stochastic neural network. Neural networks, 10(4):683–691, 1997. 2, 3, 4

    1997

  4. [12]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2, 6

    2009

  5. [13]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  6. [14]

    Convit: Improving vision transformers with soft convolutional inductive biases

    St ´ephane d’Ascoli, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In ICML, 2021. 5

    2021

  7. [15]

    Scalekd: Strong vision transformers could be excellent teachers

    Jiawei Fan, Chao Li, Xiaolong Liu, and Anbang Yao. Scalekd: Strong vision transformers could be excellent teachers. NeurIPS, 2024. 4

    2024

  8. [16]

    Domain-adversarial training of neural networks

    Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pas- cal Germain, Hugo Larochelle, Franc ¸ois Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 17(59):1–35, 2016. 3

    2016

  9. [17]

    On the duality between contrastive and non-contrastive self-supervised learning

    Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Naj- man, and Yann Lecun. On the duality between contrastive and non-contrastive self-supervised learning. ICLR, 2023. 3

    2023

  10. [18]

    Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness

    Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. InICLR, 2018. 1, 4, 5

    2018

  11. [19]

    Cmt: Convolutional neural networks meet vision transformers

    Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In CVPR, 2022. 4, 5

    2022

  12. [20]

    Class attention transfer based knowledge distillation

    Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. In CVPR,

  13. [21]

    Learning effi- cient vision transformers via fine-grained manifold distilla- tion

    Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang. Learning effi- cient vision transformers via fine-grained manifold distilla- tion. NeurIPS, 2022. 1, 3

    2022

  14. [22]

    One-for-all: Bridge the gap be- tween heterogeneous architectures in knowledge distillation

    Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu. One-for-all: Bridge the gap be- tween heterogeneous architectures in knowledge distillation. NeurIPS, 2024. 1, 2, 3, 4, 6, 7, 8

    2024

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR,

  16. [24]

    A comprehensive overhaul of feature distillation

    Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, No- jun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In ICCV, 2019. 3, 8

    2019

  17. [25]

    Distilling the knowlegde in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowlegde in a neural network. In arXiv:1503.02531, 2015. 1, 2, 3, 6, 7, 8

    2015 arXiv

  18. [26]

    Knowledge distillation from a stronger teacher

    Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. In NeurIPS,

  19. [27]

    Evaluation-oriented knowledge distillation for deep face recognition

    Yuge Huang, Jiaxiang Wu, Xingkun Xu, and Shouhong Ding. Evaluation-oriented knowledge distillation for deep face recognition. In CVPR, 2022. 1

    2022

  20. [28]

    Similarity of neural network representa- tions revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representa- tions revisited. In ICML, 2019. 8

    2019

  21. [29]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009. 6

    2009

  22. [30]

    Scconv: Spatial and channel reconstruction convolution for feature redundancy

    Jiafeng Li, Ying Wen, and Lianghua He. Scconv: Spatial and channel reconstruction convolution for feature redundancy. In CVPR, 2023. 3

    2023

  23. [31]

    Locality guidance for improving vision trans- formers on tiny datasets

    Kehan Li, Runyi Yu, Zhennan Wang, Li Yuan, Guoli Song, and Jie Chen. Locality guidance for improving vision trans- formers on tiny datasets. In ECCV, 2022. 3

    2022

  24. [32]

    Curriculum tempera- ture for knowledge distillation

    Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum tempera- ture for knowledge distillation. In AAAI, 2023. 3

    2023

  25. [33]

    Knowledge distil- lation via the target-aware transformer

    Sihao Lin, Hongwei Xie, Bing Wang, Kaicheng Yu, Xiaojun Chang, Xiaodan Liang, and Gang Wang. Knowledge distil- lation via the target-aware transformer. In CVPR, 2022. 2, 3

    2022

  26. [34]

    Function-consistent feature distillation

    Dongyang Liu, Meina Kan, Shiguang Shan, and Xilin Chen. Function-consistent feature distillation. In ICLR, 2023. 2

    2023

  27. [35]

    Exploring inter-channel correlation for diversity-preserved knowledge distillation

    Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang. Exploring inter-channel correlation for diversity-preserved knowledge distillation. In ICCV, 2021. 3

    2021

  28. [36]

    Cross-architecture knowledge distilla- tion

    Yufan Liu, Jiajiong Cao, Bing Li, Weiming Hu, Jingting Ding, and Liang Li. Cross-architecture knowledge distilla- tion. In ACCV, 2022. 1, 2, 4

    2022

  29. [37]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021. 1, 2, 6

    2021

  30. [38]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022. 2, 6

    2022

  31. [39]

    Domain-invariant feature exploration for domain generalization

    Wang Lu, Jindong Wang, Haoliang Li, Yiqiang Chen, and Xing Xie. Domain-invariant feature exploration for domain generalization. TMLR, 2022. 3

    2022

  32. [40]

    Im- proved knowledge distillation via teacher assistant

    Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Im- proved knowledge distillation via teacher assistant. In AAAI,

  33. [41]

    Domain generalization via invariant feature representation

    Krikamol Muandet, David Balduzzi, and Bernhard Sch¨olkopf. Domain generalization via invariant feature representation. In ICML, 2013. 3

    2013

  34. [42]

    Good teachers explain: Explanation- enhanced knowledge distillation

    Amin Parchami-Araghi, Moritz B ¨ohle, Sukrut Rao, and Bernt Schiele. Good teachers explain: Explanation- enhanced knowledge distillation. In ECCV, 2024. 3

    2024

  35. [43]

    How do vision transformers work? ICLR, 2022

    Namuk Park and Songkuk Kim. How do vision transformers work? ICLR, 2022. 1, 4, 5

    2022

  36. [44]

    Rela- tional knowledge distillation

    Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Rela- tional knowledge distillation. In IEEE TMM, 2019. 1, 3, 6, 8

    2019

  37. [45]

    Correla- tion congruence for knowledge distillation

    Baoyun Peng, Xiao Jin, Jiaheng Liu, Shunfeng Zhou, Yichao Wu, Yu Liu, Dongsheng Li, and Zhaoning Zhang. Correla- tion congruence for knowledge distillation. In CVPR, 2019. 1, 3, 6

    2019

  38. [46]

    Ef- ficient domain generalization via common-specific low-rank decomposition

    Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Ef- ficient domain generalization via common-specific low-rank decomposition. In ICML, 2020. 3

    2020

  39. [47]

    Slimconv: Reducing channel redundancy in convolutional neural networks by features recombining

    Jiaxiong Qiu, Cai Chen, Shuaicheng Liu, Heng-Yu Zhang, and Bing Zeng. Slimconv: Reducing channel redundancy in convolutional neural networks by features recombining. IEEE TIP, 30:6434–6445, 2021. 3

    2021

  40. [48]

    Do vision trans- formers see like convolutional neural networks? NeurIPS,

    Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision trans- formers see like convolutional neural networks? NeurIPS,

  41. [49]

    Fitnets: Hints for thin deep nets

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015. 1, 2, 3, 4, 6

    2015

  42. [50]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zh- moginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018. 2, 6

    2018

  43. [51]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. InICLR,

  44. [52]

    Logit standardization in knowl- edge distillation

    Shangquan Sun, Wenqi Ren, Jingzhi Li, Rui Wang, Rui Wang, and Xiaochun Cao. Logit standardization in knowl- edge distillation. In CVPR, 2024. 1, 3

    2024

  45. [53]

    Con- trastive representation distillation

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Con- trastive representation distillation. In ICLR, 2020. 1, 2, 3, 6, 8

    2020

  46. [54]

    Deep learning and the information bottleneck principle

    Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In IEEE Information The- ory Workshop, 2015. 2, 4

    2015

  47. [55]

    The information bottleneck method

    Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004057, 2000. 2, 4

    2000 arXiv

  48. [56]

    Mlp-mixer: An all-mlp architecture for vision

    Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lu- cas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. NeurIPS,

  49. [57]

    Training data-efficient image transformers & distillation through at- tention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through at- tention. In ICML, 2021. 2, 3, 6

    2021

  50. [58]

    Resmlp: Feedforward networks for image classification with data-efficient training

    Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izac- ard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al. Resmlp: Feedforward networks for image classification with data-efficient training. IEEE TPAMI , 45(4):5314–5321,

  51. [59]

    Similarity-preserving knowl- edge distillation

    Frederick Tung and Greg Mori. Similarity-preserving knowl- edge distillation. In ICCV, 2019. 3

    2019

  52. [60]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017. 2

    2017

  53. [61]

    Diffuse and dis- perse: Image generation with representation regularization

    Runqian Wang and Kaiming He. Diffuse and dis- perse: Image generation with representation regularization. arXiv:2506.09027, 2025. 3, 4

    2025 arXiv

  54. [62]

    Dis- tilling object detectors with fine-grained feature imitation

    Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Dis- tilling object detectors with fine-grained feature imitation. In CVPR, 2019. 1

    2019

  55. [63]

    Glance and focus: a dynamic ap- proach to reducing spatial redundancy in image classifica- tion

    Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, and Gao Huang. Glance and focus: a dynamic ap- proach to reducing spatial redundancy in image classifica- tion. NeurIPS, 2020. 3

    2020

  56. [64]

    Improving knowledge distilla- tion via regularizing feature norm and direction

    Yuzhu Wang, Lechao Cheng, Manni Duan, Yongheng Wang, Zunlei Feng, and Shu Kong. Improving knowledge distilla- tion via regularizing feature norm and direction. In ECCV,

  57. [65]

    Scale decoupled distillation

    Shicai Wei, Chunbo Luo, and Yang Luo. Scale decoupled distillation. In CVPR, 2024. 3, 8

    2024

  58. [66]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. InECCV. Springer,

  59. [67]

    High-fidelity 3d gan inversion by pseudo- multi-view optimization

    Jiaxin Xie, Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen. High-fidelity 3d gan inversion by pseudo- multi-view optimization. In CVPR, 2023. 1

    2023

  60. [68]

    Cross-image relational knowl- edge distillation for semantic segmentation

    Chuanguang Yang, Helong Zhou, Zhulin An, Xue Jiang, Yongjun Xu, and Qian Zhang. Cross-image relational knowl- edge distillation for semantic segmentation. In CVPR, 2022

    2022

  61. [69]

    Focal and global knowledge distillation for detectors

    Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Ze- huan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In CVPR, 2022. 1

    2022

  62. [70]

    From knowledge distillation to self- knowledge distillation: A unified approach with normalized loss and customized soft labels

    Zhendong Yang, Ailing Zeng, Zhe Li, Tianke Zhang, Chun Yuan, and Yu Li. From knowledge distillation to self- knowledge distillation: A unified approach with normalized loss and customized soft labels. In ICCV, 2023. 3

    2023

  63. [71]

    Vitkd: Feature-based knowledge distillation for vision transformers

    Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Feature-based knowledge distillation for vision transformers. In CVPR Workshop, 2024. 1, 3

    2024

  64. [72]

    Barlow twins: Self-supervised learning via redundancy reduction

    Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and St´ephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021. 3, 4

    2021

  65. [73]

    Foreground object search by distilling composite image feature

    Bo Zhang, Jiacheng Sui, and Li Niu. Foreground object search by distilling composite image feature. InICCV, 2023. 1

    2023

  66. [74]

    Cross-view consistency regularisation for knowledge distil- lation

    Weijia Zhang, Dongnan Liu, Weidong Cai, and Chao Ma. Cross-view consistency regularisation for knowledge distil- lation. In ACM MM, 2024. 2, 3

    2024

  67. [75]

    Alleviating foreground sparsity for semi-supervised monoc- ular 3d object detection

    Weijia Zhang, Dongnan Liu, Chao Ma, and Weidong Cai. Alleviating foreground sparsity for semi-supervised monoc- ular 3d object detection. In WACV, 2024. 1

    2024

  68. [76]

    Vrm: Knowledge distillation via virtual relation matching

    Weijia Zhang, Fei Xie, Weidong Cai, and Chao Ma. Vrm: Knowledge distillation via virtual relation matching. arXiv:2502.20760, 2025. 3

    2025 arXiv

  69. [77]

    Shufflenet: An extremely efficient convolutional neural net- work for mobile devices

    Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural net- work for mobile devices. In CVPR, 2018. 2

    2018

  70. [78]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, 2022. 1, 3, 6, 7, 8

    2022

  71. [79]

    Knowledge distillation based on transformed teaching matching

    Kaixiang Zheng and En-Hui Yang. Knowledge distillation based on transformed teaching matching. In ICLR, 2024. 3

    2024

  72. [80]

    Unidistill: A universal cross-modality knowl- edge distillation framework for 3d object detection in bird’s- eye view

    Shengchao Zhou, Weizhou Liu, Chen Hu, Shuchang Zhou, and Chao Ma. Unidistill: A universal cross-modality knowl- edge distillation framework for 3d object detection in bird’s- eye view. In CVPR, 2023. 1

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.