Pith. sign in

REVIEW 4 major objections 5 minor 42 references

SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Aligning teacher and student feature maps in the Fourier domain, with layers chosen by spectral intensity, lifts DeiT-Tiny by +5.2% and Swin-Tiny by +1.4% top-1 accuracy on ImageNet-1K.

desk verdict Plausible, simple spectral-alignment KD that could be useful, but the 'unified framework' is an empirical observation resting on a proxy the authors themselves admit is shaky. read the letter →

arxiv 2412.19055 v3 pith:P4L4O7U3 submitted 2024-12-26 cs.CV cs.LG

Huiyuan Tian , Bonan Xu , Shijian Li , Gang Pan This is my paper · ORCID
classification cs.CVcs.LG
keywords knowledgedistillationvisiontransformersspectralintensityFouriertransformfeaturealignmentlayerselectiondynamicsImageNet-1K
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a frequency-domain view of vision transformer feature maps is enough both to locate where a teacher network stores transferable knowledge and to design a simpler way to transfer it. Its spectral analysis of CaiT-S24 finds a U-shaped intensity profile: the first and last few layers carry the strongest spectral signal, while middle layers are relatively flat and weak. The same pattern reappears stage-wise in Swin, suggesting a shared encoding strategy across uniform and hierarchical transformers. From this the authors distill a parameter-free method that aligns student and teacher features in the Fourier domain at those high-intensity layers, and they report that it raises DeiT-Tiny from 72.2% to 77.4% and Swin-Tiny from 81.3% to 82.7% top-1 accuracy on ImageNet-1K.

What carries the argument

The load-bearing object is the model-wise spectral intensity profile $L(X)=\{\ell^{(1)}(X),\ldots,\ell^{(n)}(X)\}$, where each $\ell(X)$ averages, over batch and spatial positions, the magnitude of a 1D FFT of a layer's feature map along the channel dimension. It converts a layer's encoding pattern into a scalar that the paper treats as a proxy for information richness, and it supplies two rules: for uniform transformers, distill from high-intensity early and final layers; for all architectures, align complete feature maps so that all frequency bands are preserved. The transfer mechanism is a frequency-alignment loss $L_{\mathrm{FFT}}$ that pools student and teacher channels to a common width, applies a 2D real FFT over the spatial dimensions, stacks real and imaginary parts, and penalizes their mean squared difference; this loss is added to the standard softened KL distillation objective.

What would settle it

Distill DeiT-Tiny from CaiT-S24 using the same frequency-alignment loss but targeting the lowest-intensity middle layers, say teacher layers 10 to 13 mapped to student layers 5 to 8, with the same training budget; if accuracy matches or beats the 77.4% obtained from layers {1,2,19,20,21,22,23,24}, then the spectral-intensity layer-selection rule is not what drives the reported gain.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that ViT layers differ systematically in spectral intensity, computed as the average magnitude of a 1D FFT over the channel dimension, and that these differences identify the right layers for knowledge transfer. CaiT-S24 exhibits a U-shaped curve: early and final layers show high, near-uniform frequency energy, while middle layers show low intensity with a decay from low to high frequencies. Swin-Small's four stages reproduce the same trend, which the authors read as evidence of a convergent encoding strategy across transformer families. The associated distillation method, SpectralKD, aligns student and teacher features by 2D FFT over spatial dimensions after adaptive average pooling to a common channel count, adding a frequency-domain MSE term to the standard KD loss. Post-training analysis shows the distilled student's spectral profile moves toward the teacher's even in unaligned layers, an effect the paper names "distillation dynamics".

Load-bearing premise

The load-bearing premise is that the average spectral intensity of a layer's feature maps, measured by a 1D FFT along channels, is a valid proxy for how much information that layer carries, so that the highest-intensity layers are the ones worth distilling.

Editorial extensions

If this is right

  • For uniform transformers, feature distillation no longer needs to treat all layers alike: the first and last few layers are the highest-value distillation targets.
  • Frequency-space alignment transfers both low-frequency global structure and high-frequency detail, which logit-only or partial-channel alignment misses.
  • The same spectral-alignment loss transfers knowledge across hierarchical Swin architectures without architecture-specific changes to the loss.
  • After training, the student's spectral profile moves toward the teacher's even in layers that were not directly aligned, implying that selective layer alignment reshapes the whole network.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the spectral-intensity proxy generalizes, the same first-and-last-layer recipe should hold for other uniform transformers, and that prediction can be checked quickly by inspecting whether their $L(X)$ profiles are U-shaped.
  • The "distillation dynamics" observation suggests that spectral-intensity agreement could serve as a training-free diagnostic for when distillation has converged, and might also flag which middle layers are redundant for pruning.
  • A useful stress test would replace the 1D-channel-FFT intensity with a spatial FFT or with attention-based importance scores; if the recommended layer sets and gains change, the layer-selection effect and the alignment-loss effect could be separated.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SpectralKD, a framework that analyzes Vision Transformer (ViT) internal representations in the frequency domain and uses the resulting insights to design a knowledge-distillation (KD) loss. The analysis part computes a per-layer scalar called spectral intensity ℓ(X) by applying a 1D FFT along the channel dimension of feature maps (Section 2.1, Eqs. 1-5). From this it reports a U-shaped intensity curve for CaiT-S24 and similar stage-wise patterns for Swin-Small, leading to two guidelines: distill from early and late layers of uniform transformers, and align all channels rather than a subset. The method part (Section 3) introduces a frequency-alignment loss L_FFT computed via a 2D FFT over the spatial dimensions of adaptively-pooled student and teacher feature maps, combined with a standard KD loss. The paper reports state-of-the-art ImageNet-1K results for DeiT-Tiny (+5.2%), DeiT-Small (+2.3%), and Swin-Tiny (+1.4%), and a post-training analysis of 'distillation dynamics' showing that distilled students reproduce teacher-like spectral patterns even in non-aligned layers.

Significance. If the findings hold, the paper would contribute a simple, parameter-free feature-alignment loss with a plausible interpretability story for why certain layers are better for KD, plus a cross-architecture observation (uniform vs. hierarchical ViTs) that could interest a broad computer-vision audience. The code release, use of standard ImageNet-1K benchmarks, and comparisons to several recent KD methods are commendable. However, the central interpretive claim rests on an admitted proxy (spectral intensity as information richness) that is not yet backed by controlled evidence, and the connection between the 1D channel-FFT analysis and the 2D spatial-FFT loss is not established. The reported performance gains are plausible but the layer-selection advantage over alternative strategies is small (0.2-0.4 top-1 points) and is not accompanied by error bars or multiple-seed runs.

major comments (4)
  1. [Section 2.2.4, Eq. (4)] The paper admits that in conventional signal processing, high spectral intensity does not imply information-rich representations, and it bridges this gap only with an untested hypothesis about Layer Normalization flattening frequency energy curves. Since this proxy is the foundation for the layer-selection guidelines, the authors should provide a control experiment that isolates spectral intensity from trivial correlates, such as feature scale or variance. A direct test would be to compare layer selection based on ℓ(X) against selection based on the per-layer L2 norm or mean absolute value, plus a random-layer-selection baseline. Without such a control, the U-shaped curve in Figure 1(a) may be an artifact of feature magnitude rather than a meaningful spectral property, which would collapse the paper's theoretical framing.
  2. [Table 4, Section 4.3] The three layer-matching strategies in Table 4 differ by only 0.2-0.4 top-1 accuracy points (77.0, 77.2, 77.4), a range that is typically within run-to-run variation for ImageNet training. Moreover, the spectral-selected student layer set S={1,2,7,8,9,10,11,12} appears to have been chosen on the validation split, and the paper reports no multiple seeds, confidence intervals, or significance tests. To support the claim that spectral analysis informs optimal layer selection, the authors should report variance over at least three runs, or equivalently provide a random-layer control and a holdout validation procedure that does not optimize the layer indexes on the same split used for the final accuracy.
  3. [Section 3.2, Eqs. (8)-(13); Section 2.1] There is a disconnect between the analytical tool and the distillation loss. The analysis that motivates the method uses a 1D FFT along the channel dimension (Eq. 1), while the loss L_FFT uses a 2D FFT over spatial dimensions (Eqs. 8-9). The paper does not demonstrate that the 1D channel-frequency patterns (which drive the layer-selection and channel-alignment guidelines) are predictive of the 2D spatial-frequency alignment quality that the loss optimizes. An ablation that replaces L_FFT with a channel-FFT-based loss, or that reports the correlation between 1D channel intensity and spatial-spectrum distance, would bridge this gap. Without such evidence, the proposed loss may be effective as a generic feature-alignment regularizer while the spectral-interpretation framework is not the actual cause of the gains.
  4. [Section 5, Figure 1(c), Figure 5] The 'distillation dynamics' claim is that SpectralKD induces teacher-like spectral patterns beyond the explicitly aligned layers, for example Layer 6 in Figure 5. This observation is made by comparing the non-distilled baseline with the SpectralKD-distilled student, but there is no control that trains the student with soft-KD only (without L_FFT). Without that control, it is unclear whether the teacher-like pattern is due to the spectral alignment specifically or to any form of knowledge distillation that improves accuracy. In addition, the claim that Layer 6 is 'partially teacher-like' is based on visual inspection; a quantitative distance, such as the mean squared error between the S(X) curves of student and teacher for non-aligned layers, would make the claim testable and would strengthen the analysis.
minor comments (5)
  1. [Abstract] There is a formatting issue in the abstract: 'by+1.4%' lacks a space, and the same appears in the body text. Please also ensure consistent use of 'SpectralKD' versus 'Spectral KD' in Table 3 and elsewhere.
  2. [Section 3.1] The notation Fs and Ft is used for both the feature maps and their Fourier transforms in Eqs. (8)-(9); it would be clearer to denote the transformed quantities differently (e.g., F̂s, F̂t) to distinguish the spatial-domain and frequency-domain representations.
  3. [Section 4.1, 'Hyperparameters'] The hyperparameters α, T, β, and the chosen layer sets are all tuned, yet the abstract and conclusion describe the method as 'parameter-free'. The claim of being parameter-free is misleading because the method introduces several hyperparameters and a manually selected layer set; the absence of trainable parameters should be stated without claiming full parameter-freedom.
  4. [Section 4.1, 'Implementation Details'] The paper states that training DeiT-Tiny requires approximately 184 GPU hours, but it does not specify the cost for DeiT-Small or Swin-Tiny. Reporting the total compute budget would help readers assess reproducibility.
  5. [Table 1] The table lists several baseline methods with different teacher models (RegNetY-16GF and CaiT-S24). It would be helpful to explicitly mark which comparisons use the same teacher as SpectralKD, since the choice of teacher strongly affects the difficulty of the distillation task.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central accuracy claim is benchmarked externally, and the spectral-intensity proxy is an admitted empirical assumption rather than a derivation.

full rationale

The paper's derivation chain is: define channel-FFT intensity ℓ(X) (Eqs. 1–5); observe a U-shaped ℓ across CaiT layers and similar stage-wise patterns in Swin; adopt ℓ as a proxy for information richness (Sec. 2.2.4); distill by matching spatial FFTs of selected layers (Eq. 13); evaluate on ImageNet-1K against external KD baselines (Tables 1–4). No step makes the target result an input by construction. The claimed 'prediction' that high-ℓ early/late layers are good distillation targets is not identical to the loss: the analysis uses a 1D FFT along channels, while LFFT uses a 2D FFT over spatial dimensions, so the loss does not directly optimize ℓ(X). The ablation in Table 4 compares spectral-selected layers with early-late and middle matching, giving the selection empirical out-of-sample support (77.4 vs 77.2/77.0), albeit without error bars. The post-training spectral similarity for explicitly aligned layers is partly by construction, but the paper's non-trivial claim is that unaligned Layer 6 also changes (Fig. 5), which is not forced. Sec. 2.2.4 explicitly concedes that high spectral intensity does not imply information-richness in conventional signal processing and offers a LayerNorm hypothesis; this is an unproven assumption that threatens the interpretability framing, but it is a correctness/validity risk rather than circularity. The paper contains no load-bearing self-citations, and no fitted parameter is renamed as a prediction; β is a standard hyperparameter. Therefore score 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework relies on several assumptions about the meaning of FFT magnitudes in ViT feature maps rather than introduced physical entities. The main free parameters are the loss weight beta, the KD mixture weight alpha, temperature, and the layer-matching sets, all tuned on validation data.

free parameters (4)
  • beta = 0.2 (DeiT), 0.05 (Swin)
    Loss weight for LFFT in Equation 15, tuned per model family.
  • alpha = 0.9
    Balance between hard labels and soft KD loss in Equation 14.
  • temperature T = 1
    Softmax temperature for KD loss in Equation 14.
  • Layer selection sets = T={1,2,19-24}, S={1,2,7-12}
    Chosen by comparing several matching strategies (Table 4) on ImageNet validation; the 'spectral' strategy was selected after seeing validation accuracy.
assumptions (5)
  • domain assumption Average spectral intensity (1D FFT over channels) is a proxy for information richness of a layer
    Section 2.2.4 states that in conventional signal processing strong spectral intensity does not imply information richness, but asserts a ViT-specific phenomenon without proof.
  • domain assumption 1D FFT along the channel dimension captures meaningful frequency structure
    Section 2.1 applies FFT over channels; the ordering of channels is arbitrary, so frequency indices may not correspond to a meaningful signal.
  • ad hoc to paper Layer Normalization flattens frequency energy curves
    Section 2.2.4 hypothesizes this mechanism without evidence.
  • domain assumption High-spectral-intensity layers are the best distillation targets
    Used to justify layer selection; tested only indirectly through Table 4 with small accuracy differences.
  • domain assumption Swin stages can be mapped to CaiT layers for cross-architecture comparison
    Section 2.2.3 assumes stage-wise aggregation is comparable to layer-wise, despite different depths and resolutions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis." pith.science (2026). https://pith.science/paper/P4L4O7U3

@misc{pith2026241219055,
  author       = {Pith},
  title        = {Pith review of: SpectralKD: A Unified Framework for Interpreting and Distilling Vision Transformers via Spectral Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P4L4O7U3}},
  note         = {Machine review of arXiv:2412.19055}
}
abstract

Knowledge Distillation (KD) has achieved widespread success in compressing large Vision Transformers (ViTs), but a unified theoretical framework for both ViTs and KD is still lacking. In this paper, we propose SpectralKD, a novel unified analytical framework that offers deeper insights into ViTs and optimizes KD via spectral analysis. Our model-wise analysis reveals that CaiT concentrates information in their first and last few layers, informing optimal layer selection for KD. Surprisingly, our layer-wise analysis discovers that Swin Transformer and CaiT exhibit similar spectral encoding patterns despite their architectural differences, leading to feature map alignment guideline. Building on these insights, we propose a simple yet effective spectral alignment method for KD. Benefiting from the deeper understanding by above analysis results, even such a simple strategy achieves state-of-the-art performance on ImageNet-1K without introducing any trainable parameters, improving DeiT-Tiny by $+5.2\%$ and Swin-Tiny by $+1.4\%$ in top-1 accuracy. Furthermore, our post-training analysis reveals that distilled students can reproduce spectral patterns similar to their teachers, opening a new area we term ``distillation dynamics". Code and experimental logs are available in https://github.com/thy960112/SpectralKD.

Figures

Figures reproduced from arXiv: 2412.19055 by the authors.

Figure 1
Figure 1. Model-wise frequency intensity analysis L(X) (Equa￾tion 5), plotted across the Transformer depth for three models: (a) CaiT-S24 (teacher), (b) DeiT-Tiny without distillation, and (c) DeiT-Tiny distilled by our SpectralKD. SpectralKD clearly shifts the student’s intensities in different layers closer to those of the teacher, especially in the most information-rich layers in the early and final few layers. Both CaiT-S… view at source ↗
Figure 2
Figure 2. Histogram of the model-wise frequency intensities L(X) (Equation 5) for all 24 layers of CaiT-S24. The distribution is heav￾ily skewed: most layers have relatively low spectral intensity, while only a small number exhibit substantially higher values. These peaks point to potentially critical layers for knowledge distillation. 2.1. Analysis Method Consider a batch of intermediate feature maps X ∈ R B×C×H×W from a par… view at source ↗
Figure 3
Figure 3. Layer-wise spectral intensity distributions S(X) (Equa￾tion 3) for representative layers of CaiT-S24. The visualization reveals distinct encoding patterns across network depths. Early layers (a-b) exhibit approximately uniform intensities across fre￾quencies. Middle layers (c-d) show a marked decay from low to high frequency. Final layers (e-f) once again become relatively uniform but at distinctly higher overall in… view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: provides a more granular view by comparing spec￾tral intensities of the non-distilled baseline (top row) and the SpectralKD-distilled student (bottom row) for Layers 1, 6, and 12. While Layer 1 and Layer 12 are directly distilled, Layer 6 is not. Nevertheless, Layer 6 …
Figure 6
Figure 6. Figure 6: Spectral intensity distributions S(X) computed using Equation (3) for layers (1-6) of CaiT-S24 feature maps. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Spectral intensity distributions S(X) computed using Equation (3) for layers (7-12) of CaiT-S24 feature maps. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Spectral intensity distributions S(X) computed using Equation (3) for layers (13-18) of CaiT-S24 feature maps. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Spectral intensity distributions S(X) computed using Equation (3) for layers (19-24) of CaiT-S24 feature maps. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 28 canonical work pages

  1. [1]

    and Zuidema, W

    Abnar, S. and Zuidema, W. Quantifying attention flow in transformers. arXiv preprint arXiv:2005.00928, 2020

    arXiv 2005

  2. [2]

    and Li, Y

    Allen-Zhu, Z. and Li, Y. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning, 2023

    work page 2023

  3. [3]

    Subgraphormer: Unifying subgraph GNN s and graph transformers via graph products

    Bar-Shalom, G., Bevilacqua, B., and Maron, H. Subgraphormer: Unifying subgraph GNN s and graph transformers via graph products. In Forty-first International Conference on Machine Learning, 2024

    work page 2024

  4. [4]

    Language models are few-shot learners

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 0 1877--1901, 2020

    1901

  5. [5]

    Emerging properties in self-supervised vision transformers

    Caron, M., Touvron, H., Misra, I., J \'e gou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 9650--9660, 2021

    2021

  6. [6]

    Revisiting label smoothing and knowledge distillation compatibility: What was missing? In International Conference on Machine Learning, pp.\ 2890--2916

    Chandrasegaran, K., Tran, N.-T., Zhao, Y., and Cheung, N.-M. Revisiting label smoothing and knowledge distillation compatibility: What was missing? In International Conference on Machine Learning, pp.\ 2890--2916. PMLR, 2022

    work page 2022

  7. [7]

    Transformer interpretability beyond attention visualization

    Chefer, H., Gur, S., and Wolf, L. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 782--791, 2021

    work page 2021

  8. [8]

    Dearkd: data-efficient early knowledge distillation for vision transformers

    Chen, X., Cao, Q., Zhong, Y., Zhang, J., Gao, S., and Tao, D. Dearkd: data-efficient early knowledge distillation for vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 12052--12062, 2022

    work page 2022

Show all 42 references
  1. [9]

    A comprehensive survey on model compression and acceleration

    Choudhary, T., Mishra, V., Goswami, A., and Sarangapani, J. A comprehensive survey on model compression and acceleration. Artificial Intelligence Review, 53: 0 5113--5155, 2020

    2020

  2. [10]

    Imagenet: A large-scale hierarchical image database

    Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.\ 248--255. Ieee, 2009

    2009

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Devlin, J. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Dosovitskiy, A. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  5. [13]

    Scalekd: Strong vision transformers could be excellent teachers

    Fan, J., Li, C., Liu, X., and Yao, A. Scalekd: Strong vision transformers could be excellent teachers. arXiv preprint arXiv:2411.06786, 2024

    2024 arXiv

  6. [14]

    Gonzales, R. C. and Wintz, P. Digital image processing. Addison-Wesley Longman Publishing Co., Inc., 1987

    1987

  7. [15]

    Transformer in transformer

    Han, K., Xiao, A., Wu, E., Guo, J., Xu, C., and Wang, Y. Transformer in transformer. Advances in neural information processing systems, 34: 0 15908--15919, 2021

    2021

  8. [16]

    A survey on vision transformer

    Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45 0 (1): 0 87--110, 2022

    2022

  9. [17]

    Learning efficient vision transformers via fine-grained manifold distillation

    Hao, Z., Guo, J., Jia, D., Han, K., Tang, Y., Zhang, C., Hu, H., and Wang, Y. Learning efficient vision transformers via fine-grained manifold distillation. Advances in Neural Information Processing Systems, 35: 0 9164--9175, 2022

    2022

  10. [18]

    Distilling the knowledge in a neural network

    Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  11. [19]

    Knowledge distillation from a stronger teacher

    Huang, T., You, S., Wang, F., Qian, C., and Xu, C. Knowledge distillation from a stronger teacher. Advances in Neural Information Processing Systems, 35: 0 33716--33727, 2022

    2022

  12. [20]

    Li, X., Ding, H., Yuan, H., Zhang, W., Pang, J., Cheng, G., Chen, K., Liu, Z., and Loy, C. C. Transformer-based visual segmentation: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024

    2024

  13. [21]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 10012--10022, 2021

    2021

  14. [22]

    K., Rawat, A

    Menon, A. K., Rawat, A. S., Reddi, S., Kim, S., and Kumar, S. A statistical perspective on distillation. In International Conference on Machine Learning, pp.\ 7632--7642. PMLR, 2021

    2021

  15. [23]

    and Mikolajczyk, K

    Miles, R. and Mikolajczyk, K. Understanding the role of the projector in knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 4233--4241, 2024

    2024

  16. [24]

    Ia-red ^2 : Interpretability-aware redundancy reduction for vision transformers, 2021

    Pan, B., Panda, R., Jiang, Y., Wang, Z., Feris, R., and Oliva, A. Ia-red ^2 : Interpretability-aware redundancy reduction for vision transformers, 2021

    2021

  17. [25]

    Relational knowledge distillation

    Park, W., Kim, D., Lu, Y., and Cho, M. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 3967--3976, 2019

    2019

  18. [26]

    Frequency attention for knowledge distillation

    Pham, C., Nguyen, V.-A., Le, T., Phung, D., Carneiro, G., and Do, T.-T. Frequency attention for knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.\ 2277--2286, 2024

    2024

  19. [27]

    and Lampert, C

    Phuong, M. and Lampert, C. Towards understanding knowledge distillation. In International conference on machine learning, pp.\ 5142--5151. PMLR, 2019

    2019

  20. [28]

    Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 0 12116--12128, 2021

    Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34: 0 12116--12128, 2021

    2021

  21. [29]

    Dual expert distillation network for generalized zero-shot learning

    Rao, Z., Guo, J., Lu, X., Liang, J., Zhang, J., Wang, H., Wei, K., and Cao, X. Dual expert distillation network for generalized zero-shot learning. arXiv preprint arXiv:2404.16348, 2024

    2024 arXiv

  22. [30]

    The role of masking for efficient supervised knowledge distillation of vision transformers

    Son, S., Ryu, J., Lee, N., and Lee, J. The role of masking for efficient supervised knowledge distillation of vision transformers. In European Conference on Computer Vision, pp.\ 379--396. Springer, 2025

    2025

  23. [31]

    Training data-efficient image transformers & distillation through attention

    Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and J \'e gou, H. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp.\ 10347--10357. PMLR, 2021 a

    2021

  24. [32]

    Going deeper with image transformers

    Touvron, H., Cord, M., Sablayrolles, A., Synnaeve, G., and J \'e gou, H. Going deeper with image transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp.\ 32--42, 2021 b

    2021

  25. [33]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  26. [34]

    Pytorch image models

    Wightman, R. Pytorch image models. https://github.com/rwightman/pytorch-image-models, 2019

    2019

  27. [35]

    Self-supervised learning based on transformer for flow reconstruction and prediction

    Xu, B., Zhou, Y., and Bian, X. Self-supervised learning based on transformer for flow reconstruction and prediction. Physics of Fluids, 36 0 (2), 2024

    2024

  28. [36]

    Knowledge distillation via softmax regression representation learning

    Yang, J., Martinez, B., Bulat, A., Tzimiropoulos, G., et al. Knowledge distillation via softmax regression representation learning. International Conference on Learning Representations (ICLR), 2021

    2021

  29. [37]

    From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels

    Yang, Z., Zeng, A., Li, Z., Zhang, T., Yuan, C., and Li, Y. From knowledge distillation to self-knowledge distillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 17185--1...

    2023

  30. [38]

    Attentionviz: A global view of transformer attention

    Yeh, C., Chen, Y., Wu, A., Chen, C., Vi \'e gas, F., and Wattenberg, M. Attentionviz: A global view of transformer attention. IEEE Transactions on Visualization and Computer Graphics, 2023

    2023

  31. [39]

    and Xiang, W

    Yu, L. and Xiang, W. X-pruner: explainable pruning for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 24355--24363, June 2023

    2023

  32. [40]

    Peeling back the layers: Interpreting the storytelling of vit

    Zeng, J., Yang, Z., Yang, Q., Yang, L., and Lin, H. Peeling back the layers: Interpreting the storytelling of vit. In Proceedings of the 32nd ACM International Conference on Multimedia, pp.\ 7298--7306, 2024

    2024

  33. [41]

    S., Klein, T., and Brendel, W

    Zimmermann, R. S., Klein, T., and Brendel, W. Scale alone does not improve mechanistic interpretability in vision models. Advances in Neural Information Processing Systems, 36, 2024

    2024

  34. [42]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.