Pith. sign in

REVIEW 4 major objections 7 minor 53 references

All You Need in Knowledge Distillation Is a Tailored Coordinate System

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PCA coordinates alone carry the dark knowledge in distillation

desk verdict A cheap teacher-free distillation recipe with real efficiency gains, but the central claim needs a proper control and two equations look off. read the letter →

arxiv 2412.09388 v2 pith:GFHCCM42 submitted 2024-12-12 cs.CV cs.AI

Junjie Zhou , Ke Zhu , Jianxin Wu This is my paper · ORCID
classification cs.CVcs.AI MSC 68T0768T4562H25
keywords knowledgedistillationself-supervisedlearningprincipalcomponentanalysisfeaturefew-shotcoordinatesystemteacher-freecross-architecture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the dark knowledge in a self-supervised pretrained model is largely encoded in the linear subspace, or coordinate system, where its features lie, and that a task-specific teacher classifier is not needed for distillation. The proposed TCS method computes the teacher's principal components with a single forward pass over the target task's training data, projects the student's features into that coordinate system, and prunes irrelevant coordinates with an L1-regularized iterative mask. On CIFAR-100 and ImageNet-1K with ResNet18 students, TCS reports higher accuracy than existing KD methods while using about half the training time and GPU memory, and it transfers across CNNs, transformers, MLPs, and large teacher-student capacity gaps. If correct, distillation becomes a cheap post-hoc operation on any SSL backbone rather than a co-trained teacher-student procedure.

What carries the argument

The carrying object is the tailored coordinate system itself: the matrix $V$ of principal components obtained by SVD of the centered teacher feature matrix $X = U\Sigma V^T$, plus a trainable mask $m$ that iteratively selects the half of the coordinates most relevant to the target task. The student feature is projected as $\tilde{f}^s = (\hat{f}^s - \mu)V$ and element-wise masked, forming a linear chain that can be folded into the student's classifier so inference costs nothing extra. A modified LSH loss, eLSH, optionally makes the student mimic teacher features in that coordinate system without data augmentation, reducing the teacher's entire contribution to a single forward pass.

What would settle it

Train the same student with the same iterative feature-selection loss but replace the PCA basis with a student-learned linear projector of the same rank paired with the same mask; if accuracy matches TCS on ImageNet-1K with a ResNet18 student, then the teacher's specific coordinate system is not the source of the gain.

Watch

Extended reading notes

Core claim

The central discovery is that a task-agnostic, self-supervised pretrained model can serve as the teacher in knowledge distillation without ever being fine-tuned: one forward pass of the teacher on the target training data yields feature vectors whose PCA directions define a coordinate system, and the student's own features, after a learned linear projection and coordinate selection, are trained to be similar to the teacher's features in that coordinate system. The paper states this as a theoretical hypothesis and empirical finding: the coordinate system, not the raw feature values, carries the dark knowledge. The ablation in Table 5 is the paper's core evidence: in-domain PCA helps most, out-of-domain PCA still helps by roughly 2 percent, and random coordinates do not help at all.

Load-bearing premise

The assumption that carries the argument is that the PCA coordinate system computed from one pass over the target data is what transfers the dark knowledge, rather than the learned linear projection and feature-selection mask doing most of the work.

Editorial extensions

If this is right

  • Distillation no longer requires a task-trained teacher: any SSL backbone with available features can act as teacher.
  • Training cost approaches that of training the student from scratch, since the teacher is used only once and the extra operations are linear.
  • Large capacity gaps between teacher and student stop being a liability, and even a smaller teacher can help a larger student.
  • The same recipe transfers across heterogeneous architectures (CNN, ViT, and MLP) and to practical few-shot learning with frozen backbones.
  • In traditional KD with a task-trained teacher, feature selection can be dropped and the PCA coordinate system plus eLSH still outperforms logit-based methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the coordinate-system hypothesis holds beyond recognition, the same one-pass PCA recipe could be tried for object detection or segmentation by computing PCA on the backbone features of the target dataset and aligning student feature maps in that subspace.
  • The success of out-of-domain PCA in Table 5 hints that the teacher's coordinate system is only loosely task-specific; this suggests a testable cheap variant where PCA is computed once on a generic unlabeled pool and reused across many target tasks without a per-task teacher pass.
  • The paper leaves the theoretical foundation open; a concrete next step would be to test whether the selected coordinates correspond to cluster structure in the SSL embedding space, which would link the method to known low-rank structure of deep features.
  • Because all extra modules are linear and fold into the classifier, the same machinery could in principle be applied to progressively re-tailor the coordinate system during student training, though the paper fixes the mask via accumulated gradients.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes TCS (Tailored Coordinate System), a knowledge-distillation method that uses a self-supervised pretrained network as a task-agnostic teacher. The method computes PCA of the teacher's penultimate-layer features in a single forward pass over the target training set, projects the student features into that coordinate system, and selects a subset of coordinates with a trainable L1-regularized mask. An optional eLSH loss, a modified version of the LSH feature-mimicking loss of Wang et al. (2021), is used in traditional KD settings. The method is evaluated on CIFAR-100 and ImageNet-1K for traditional KD and on ImageNet-1K and CUB under the practical few-shot learning (pFSL) protocol, with ResNet, ViT, Swin, and MLP-Mixer backbones. The authors report accuracy gains over KD baselines and claim substantially reduced training time and GPU memory.

Significance. If the central claim is correct, this would be a notable contribution: one forward pass of an SSL-pretrained teacher, followed by PCA and coordinate selection, could distill useful dark knowledge into diverse student architectures at lower cost than existing logit- or feature-based KD. The paper has several strengths: the method is simple and computationally attractive; the evaluation spans multiple architectures, datasets, and both KD and pFSL; the ablation in Table 5 comparing in-domain, out-of-domain, and random PCA coordinate systems is a sensible first step; and the eLSH hyperparameters are taken from prior work rather than tuned. However, the support is not yet complete: the formal description of the iterative feature-selection update in Eq. (9) appears to move mask values in the wrong direction, Eq. (10) has a dimension mismatch, and the ablation does not include a no-teacher control with the same extra linear head, leaving the central coordinate-system hypothesis under-tested. The paper also does not release code, which further weakens reproducibility.

major comments (4)
  1. [Method, Eq. (9)] The update rule for the mask m in Eq. (9) does not implement the described feature-selection behavior. Since m is initialized to 1^{D_t} and Sτ is the indicator of the coordinates selected by TopDims (the ones with largest accumulated gradient), setting the selected coordinates to (1−rτ)/r while leaving the unselected coordinates at m(τ) has the opposite effect: with r=0.5 and rτ decreasing from 1 to 0.5, the selected coordinates evolve from 0 toward 1 while the unselected coordinates remain 1, so the mask converges to all-ones rather than pruning irrelevant dimensions. Interchanging Sτ and 1−Sτ does not resolve this. Please provide a corrected update rule and specify whether Sτ is a binary mask or an index set, because this mechanism is the core of the 'tailored coordinate system' claim.
  2. [Method, Eq. (10)-(11)] There is a dimension mismatch in Eq. (10) under the earlier definition of W. The matrix W in Eq. (1) is the least-squares projection from student features to teacher features, with shape D_s×D_t (or D_t×D_s depending on convention), so W^T cannot be applied to the D_t-dimensional vector (f_t−μ)V unless D_s=D_t. If W in Eq. (10) is instead intended to be a new random projection of size D_t×M that produces M hash codes, it must be defined with a different symbol. The same ambiguity affects Eq. (11), where the projected student feature \tilde f_s is D_t-dimensional. Please clarify the dimensions and notation, since the eLSH loss contributes to the reported TCS results.
  3. [Ablation studies, Table 5] The central claim that teacher dark knowledge resides in the PCA coordinate system is not isolated by the current ablation. In the pFSL setting with frozen backbones, the 'random' row already contains the extra linear head, the mask, and the classifier, while the 'N/A' row is a plain linear classifier; the 0.41-point difference between these rows shows that the added head alone has a small effect. To support the coordinate-system hypothesis, the authors need a control that trains exactly the same head (W, mask, classifier) with a teacher-independent V—for example, identity, a learned rotation, or a random rotation—so that the in-domain gain over that control can be attributed to the teacher's PCA coordinate system. The same control is missing for traditional KD, where TCS- outperforms the from-scratch student but V is never replaced by a non-teacher coordinate system while keeping the added FC layer. As the paper itself calls the coordinate-system claim a hypothesis and defers theory to future work, this ablation is load-bearing.
  4. [Abstract and Conclusion vs Tables 1-2] The efficiency claim that TCS requires 'roughly half of their training time and GPU memory costs' is not supported by the paper's own tables. In Table 2 on ImageNet-1K, TCS uses 328 s/epoch compared with 415-481 s/epoch for KD/DKD/OFA/DIST/LSHL2 (a 21-32% reduction, not half), and 5.8 GB versus 7.5 GB (a 23% reduction). Table 1 shows a similar pattern. Please rephrase the claim to 'roughly three quarters' or compare against self-distillation explicitly, or report a different baseline that makes 'half' accurate.
minor comments (7)
  1. [Method, Eq. (8)] The function TopDims(a, b) is not formally defined: it is unclear whether a is a vector of per-dimension gradient accumulations or a set, and whether the return value is a binary mask or an index set. Please define it precisely and use consistent notation in Eqs. (8) and (9).
  2. [Few-shot experiments, Table 3] For DINOv2 on ImageNet-1K with 1 shot, TCS (34.2±0.4) is below the baseline (37.0±0.7) and IbM2 (37.2±0.9). The text only says TCS is 'relatively worse' for k=1,2; this negative case should be stated explicitly, as it qualifies the claim that TCS improves few-shot learning.
  3. [Experimental details, eLSH hyperparameters] The paper says all eLSH hyperparameters (M and the loss weight) are set to the default values in Wang et al. (2021) but does not report the numerical values; please list them in the main text or supplementary material.
  4. [Reproducibility] No code release is mentioned; releasing code or detailed pseudocode would substantially improve reproducibility, especially given the ambiguities in Eqs. (9)-(11).
  5. [Ablation studies, Table 5] The label 'N/A' for the no-TCS baseline is confusing because the other rows are labeled by the data used for PCA; consider using 'no TCS' or a similar explicit label.
  6. [Method, Eqs. (10)-(11)] The symbol W is reused for both the student projection layer and the hash projection; please use distinct symbols to avoid confusion.
  7. [Figure 2] The symbols λ1 and λ2 appear in the TCS panel of Figure 2 but are not explained in the caption or the text; please clarify or remove them.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claim is empirically tested against external benchmarks, and the self-citations are motivational rather than load-bearing.

full rationale

The paper's derivation chain is empirical rather than definitional. The central claim, that dark knowledge in an SSL teacher is captured by the PCA coordinate system of its features, is explicitly introduced as a 'theoretical hypothesis and empirical finding' and is tested in Table 5 against random and out-of-domain PCA controls. The distillation losses in Eqs. (4)-(12) are standard feature regression in a fixed PCA basis; the PCA is computed once from teacher features and is not a fitted quantity later reported as a prediction. The W initialization in Eq. (1) is a least-squares fit, but it is only an initialization for the student's linear layer, not the claimed result. Self-citations (Yu and Wu 2023 for low-rank features; Wang, Ge, and Wu 2021 for the LSH loss) motivate the method or supply a borrowed loss, but the paper independently reproduces the LSHL2 baseline (marked with ♯) and ablates the coordinate system, so these citations do not carry the argument alone. The absence of an identity-V or learned-V control is an ablation gap and a correctness risk, not a circular reduction: no equation makes the reported accuracy equal to the fitted PCA or to a self-citation by construction. Score 1 reflects the presence of same-author citations in the motivational chain, not demonstrated circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on an unproven hypothesis about where dark knowledge lives, plus hyperparameters r, lambda, and eLSH defaults. The method is empirical; no new physical or mathematical entities are introduced.

free parameters (4)
  • r (target selection ratio) = 0.5
    Set to 0.5 in all experiments; it is the fraction of PCA coordinates kept after feature selection.
  • lambda (L1 regularization weight) = not reported
    Balances cross-entropy and mask sparsity in Eq. 6; no value is given in the main text.
  • M (number of hash codes in eLSH) = default from Wang et al. 2021
    eLSH hash length is taken from prior work without tuning.
  • eLSH loss weight = default from Wang et al. 2021
    Weight combining eLSH with cross-entropy is not specified in the paper.
assumptions (4)
  • ad hoc to paper The dark knowledge of a pretrained model is encoded in the linear subspace, specifically the PCA coordinate system, of its features.
    Central hypothesis stated in the Method section; no derivation is given, only ablation support.
  • domain assumption Features of deep models are low-rank, so PCA captures almost all useful information.
    Borrowed from Yu and Wu 2023 and used to justify the PCA approach.
  • domain assumption A single forward pass without data augmentation is sufficient to estimate the teacher's coordinate system.
    Needed for the claimed efficiency; it is not formally justified.
  • standard math All post-backbone operations are linear and can be fused into the classifier.
    Used to claim no inference overhead; this is true if the mask is fixed after training.

how reviews work

0 comments
Cite this review

Pith. "Pith review of All You Need in Knowledge Distillation Is a Tailored Coordinate System." pith.science (2026). https://pith.science/paper/GFHCCM42

@misc{pith2026241209388,
  author       = {Pith},
  title        = {Pith review of: All You Need in Knowledge Distillation Is a Tailored Coordinate System},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GFHCCM42}},
  note         = {Machine review of arXiv:2412.09388}
}
read the original abstract

Knowledge Distillation (KD) is essential in transferring dark knowledge from a large teacher to a small student network, such that the student can be much more efficient than the teacher but with comparable accuracy. Existing KD methods, however, rely on a large teacher trained specifically for the target task, which is both very inflexible and inefficient. In this paper, we argue that a SSL-pretrained model can effectively act as the teacher and its dark knowledge can be captured by the coordinate system or linear subspace where the features lie in. We then need only one forward pass of the teacher, and then tailor the coordinate system (TCS) for the student network. Our TCS method is teacher-free and applies to diverse architectures, works well for KD and practical few-shot learning, and allows cross-architecture distillation with large capacity gap. Experiments show that TCS achieves significantly higher accuracy than state-of-the-art KD methods, while only requiring roughly half of their training time and GPU memory costs.

Figures

Figures reproduced from arXiv: 2412.09388 by the authors.

Figure 1
Figure 1. Top-1 accuracy, training time and GPU memory [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of existing KD methods and our TCS method. (a) logits-based distillation, where the student learns only [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 28 canonical work pages

  1. [1]

    Bengio, Y.; Courville, A.; and Vincent, P. 2013. Representation Learning: A Review and New Perspectives . IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8): 1798--1828

    work page 2013

  2. [2]

    Chen, G.; Choi, W.; Yu, X.; Han, T.; and Chandraker, M. 2017. Learning efficient object detection models with knowledge distillation. In NIPS, 742--751

    work page 2017

  3. [3]

    Chen, L.; Wang, D.; Gan, Z.; Liu, J.; Henao, R.; and Carin, L. 2021 a . Wasserstein contrastive representation distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 16296--16305

    work page 2021

  4. [4]

    Chen, P.; Liu, S.; Zhao, H.; and Jia, J. 2021 b . Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5008--5017

    2021

  5. [5]

    F.; and Huang, J.-B

    Chen, W.-Y.; Liu, Y.-C.; Kira, Z.; Wang, Y.-C. F.; and Huang, J.-B. 2019. A closer look at few-shot classification. In International Conference on Learning Representations, 1--16

    work page 2019

  6. [6]

    Chen, X.; Xie, S.; and He, K. 2021. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, 9640--9649

    work page 2021

  7. [7]

    Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

    arXiv 2020

  8. [8]

    Fu, M.; and Zhu, K. 2024. Instance-based Max-margin for Practical Few-shot Recognition. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2024

Show all 53 references
  1. [9]

    Hao, Z.; Guo, J.; Han, K.; Tang, Y.; Hu, H.; Wang, Y.; and Xu, C. 2024. One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation. Advances in Neural Information Processing Systems, 36

    2024

  2. [10]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  3. [11]

    Heo, B.; Kim, J.; Yun, S.; Park, H.; Kwak, N.; and Choi, J. Y. 2019. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1921--1930

    2019

  4. [12]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531

    2015 arXiv

  5. [13]

    Hospedales, T.; Antoniou, A.; Micaelli, P.; and Storkey, A. 2022. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9): 5149--5169

    2022

  6. [14]

    X.; Li, D.; St \"u hmer, J.; Kim, M.; and Hospedales, T

    Hu, S. X.; Li, D.; St \"u hmer, J.; Kim, M.; and Hospedales, T. M. 2022. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make a difference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9068--9077

    2022

  7. [15]

    Huang, T.; You, S.; Wang, F.; Qian, C.; and Xu, C. 2022. Knowledge distillation from a stronger teacher. Advances in Neural Information Processing Systems, 35: 33716--33727

    2022

  8. [16]

    Huang, Z.; and Wang, N. 2017. Like What You Like: Knowledge Distill via Neuron Selectivity Transfer. arXiv preprint arXiv:1707.01219

    2017 arXiv

  9. [17]

    Jin, X.; Peng, B.; Wu, Y.; Liu, Y.; Liu, J.; Liang, D.; Yan, J.; and Hu, X. 2019. Knowledge Distillation via Route Constrained Optimization. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 1345--1354

    2019

  10. [18]

    Kim, J.; Park, S.; and Kwak, N. 2018. Paraphrasing complex network: Network compression via factor transfer. Advances in neural information processing systems, 31

    2018

  11. [19]

    Kim, K.; Ji, B.; Yoon, D.; and Hwang, S. 2021. Self-Knowledge Distillation with Progressive Refinement of Targets. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 6547--6556

    2021

  12. [20]

    A.; Tejankar, A.; and Pirsiavash, H

    Koohpayegani, S. A.; Tejankar, A.; and Pirsiavash, H. 2020. Compress: Self-supervised learning by compressing representations. Advances in Neural Information Processing Systems, 33: 12980--12992

    2020

  13. [21]

    Krizhevsky, A. 2009. Learning multiple layers of features from tiny images. Technical report, University of Toronto

    2009

  14. [22]

    Li, X.; Wu, J.; Fang, H.; Liao, Y.; Wang, F.; and Qian, C. 2020. Local correlation consistency for knowledge distillation. In European Conference on Computer Vision, 18--33. Springer

    2020

  15. [23]

    Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; and Guo, B. 2021. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, 10012--10022

    2021

  16. [24]

    N.; and Krishnamurthy, B

    Mangla, P.; Singh, M.; Sinha, A.; Kumari, N.; Balasubramanian, V. N.; and Krishnamurthy, B. 2020. Charting the right manifold: Manifold Mixup for few-shot learning. In IEEE Winter Conference on Applications of Computer Vision, 2207--2216

    2020

  17. [25]

    Miles, R.; Elezi, I.; and Deng, J. 2024. VkD: Improving Knowledge Distillation using Orthogonal Projections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15720--15730

    2024

  18. [26]

    I.; Farajtabar, M.; Li, A.; Levine, N.; Matsukawa, A.; and Ghasemzadeh, H

    Mirzadeh, S. I.; Farajtabar, M.; Li, A.; Levine, N.; Matsukawa, A.; and Ghasemzadeh, H. 2020. Improved knowledge distillation via teacher assistant. In Proceedings of the 34th AAAI conference on artificial intelligence, 5191--5198

    2020

  19. [27]

    Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H. V.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; Howes, R.; Huang, P.-Y.; Xu, H.; Sharma, V.; Li, S.-W.; Galuba, W.; Rabbat, M.; Assran, M.; Ballas, N.; Synnaeve, G.; Misra, I.; Jegou, H.; Maira...

    2023

  20. [28]

    Park, W.; Kim, D.; Lu, Y.; and Cho, M. 2019. Relational knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 3967--3976

    2019

  21. [29]

    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115: 211--252

    2015

  22. [30]

    Schmitt, S.; Hudson, J.; Zidek, A.; and et. al. 2018. Kickstarting Deep Reinforcement Learning. CoRR, abs/1803.03835

    2018 arXiv

  23. [31]

    Son, W.; Na, J.; Choi, J.; and Hwang, W. 2021. Densely Guided Knowledge Distillation using Multiple Teacher Assistants. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 9375--9384

    2021

  24. [32]

    Tian, Y.; Krishnan, D.; and Isola, P. 2019. Contrastive representation distillation. arXiv preprint arXiv:1910.10699

    2019 arXiv

  25. [33]

    O.; Houlsby, N.; Kolesnikov, A.; Beyer, L.; Zhai, X.; Unterthiner, T.; Yung, J.; Steiner, A.; Keysers, D.; Uszkoreit, J.; et al

    Tolstikhin, I. O.; Houlsby, N.; Kolesnikov, A.; Beyer, L.; Zhai, X.; Unterthiner, T.; Yung, J.; Steiner, A.; Keysers, D.; Uszkoreit, J.; et al. 2021. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems, 34: 24261--24272

    2021

  26. [34]

    Touvron, H.; Cord, M.; Douze, M.; Massa, F.; Sablayrolles, A.; and J \'e gou, H. 2021. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, 10347--10357. PMLR

    2021

  27. [35]

    Wah, C.; Branson, S.; Welinder, P.; Perona, P.; and Belongie, S. 2011. The Caltech-UCSD Birds-200-2011 dataset. Technical Report CNS-TR-2011-001, California Institute of Technology

    2011

  28. [36]

    Wang, G.-H.; Ge, Y.; and Wu, J. 2021. Distilling knowledge by mimicking features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11): 8183--8195

    2021

  29. [37]

    Yang, Z.; Cui, Y.; Chen, Z.; Che, W.; Liu, T.; Wang, S.; and Hu, G. 2020. T ext B rewer: A n O pen- S ource K nowledge D istillation T oolkit for N atural L anguage P rocessing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System ...

    2020

  30. [38]

    Yim, J.; Joo, D.; Bae, J.; and Kim, J. 2017. A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning. In CVPR, 7130--7138

    2017

  31. [39]

    Yu, H.; and Wu, J. 2023. Compressing Transformers: Features Are Low-Rank, but Weights Are Not! In Proceedings of the AAAI Conference on Artificial Intelligence, 11007--11015

    2023

  32. [40]

    I.; and Davis, L

    Yu, R.; Li, A.; Morariu, V. I.; and Davis, L. 2017. Visual Relationship Detection With Internal and External Linguistic Knowledge Distillation. In ICCV

    2017

  33. [41]

    E.; Li, G.; Wang, T.; and Feng, J

    Yuan, L.; Tay, F. E.; Li, G.; Wang, T.; and Feng, J. 2020. Revisiting Knowledge Distillation via Label Smoothing Regularization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 3902--3910

    2020

  34. [42]

    Yun, S.; Park, J.; Lee, K.; and Shin, J. 2020. Regularizing Class-Wise Predictions via Self-Knowledge Distillation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 13873--13882

    2020

  35. [43]

    Zagoruyko, S.; and Komodakis, N. 2016. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928

    2016 arXiv

  36. [44]

    S.; Neumann, M.; Dosovitskiy, A.; et al

    Zhai, X.; Puigcerver, J.; Kolesnikov, A.; Ruyssen, P.; Riquelme, C.; Lucic, M.; Djolonga, J.; Pinto, A. S.; Neumann, M.; Dosovitskiy, A.; et al. 2019. A large-scale study of representation learning with the Visual Task Adaptation Benchmark . arXiv preprint arXiv:1910.04867

    2019 arXiv

  37. [45]

    Zhang, L.; Song, J.; Gao, A.; Chen, J.; Bao, C.; and Ma, K. 2019. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF international conference on computer vision, 3713--3722

    2019

  38. [46]

    Zhao, B.; Cui, Q.; Song, R.; Qiu, Y.; and Liang, J. 2022. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 11953--11962

    2022

  39. [47]

    Zhou, H.; Song, L.; Chen, J.; Zhou, Y.; Wang, G.; Yuan, J.; and Zhang, Q. 2021. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102.00650

    2021 arXiv

  40. [48]

    Zhu, K.; He, Y.-Y.; and Wu, J. 2023 a . Coarse is better? a new pipeline towards self-supervised learning with uncurated images. arXiv preprint arXiv:2306.04244

    2023 arXiv

  41. [49]

    Zhu, K.; He, Y.-Y.; and Wu, J. 2023 b . Quantized feature distillation for network quantization. In Proceedings of the 37th AAAI Conference on Artificial Intelligence, 11452--11460

    2023

  42. [50]

    Zhu, K.; Zhao, L.; Ge, Z.; and Zhang, X. 2024. Self-supervised visual preference alignment. In Proceedings of the 32nd ACM International Conference on Multimedia, 291--300

    2024

  43. [51]

    Zhu, W.; Liu, J.; and Huang, Y. 2023. HNSSL: Hard Negative-Based Self-Supervised Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 4778--4787

    2023

  44. [52]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.