Pith. sign in

REVIEW 2 major objections 6 minor 84 references

Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Wasserstein distance rivals Kullback-Leibler divergence as a distillation objective, with new logit and feature losses beating KL-based baselines on ImageNet, CIFAR-100, and MS-COCO.

desk verdict Good KD method, but the central claim that WD itself beats KL is not isolated from the added inter-class cost; the mechanism story needs a control before it should be taken at face value. read the letter →

arxiv 2412.08139 v1 pith:SOSNE3N2 submitted 2024-12-11 cs.CV cs.AIcs.LG

Jiaming Lv , Haoyuan Yang , Peihua Li This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords knowledgedistillationWassersteindistanceoptimaltransportKullback-LeiblerdivergencelogitfeaturecategoryinterrelationsGaussianmodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge distillation trains a small student network to copy a large teacher network, and the standard objective has long been Kullback-Leibler divergence, which compares each category's probability separately and so cannot use similarities between categories. The paper argues that Wasserstein distance, the minimal cost of moving one probability distribution onto another, fixes both of KL-divergence's main shortcomings. For logits, the proposed WKD-L uses discrete Wasserstein distance with a transport cost derived from measured interrelations among categories, so similar classes are treated as closer. For intermediate features, WKD-F fits a Gaussian to each image's feature map and uses the closed-form Wasserstein distance between Gaussians, which respects the geometry of the distribution space. On ImageNet, CIFAR-100, and MS-COCO, both losses beat their KL-divergence counterparts, and their combination reaches 72.76% top-1 accuracy for ResNet34-to-ResNet18 distillation.

What carries the argument

The central object is the Wasserstein distance (earth mover's distance), used in two forms. In discrete form, it is an entropy-regularized optimal-transport problem between probability vectors, with transport cost $c_{ij}=1-\exp(-\kappa(1-\mathrm{IR}_T(C_i,C_j)))$ where $\mathrm{IR}_T$ is the CKA similarity between categories $C_i$ and $C_j$ computed once from the teacher; this is what enables cross-category comparison. In continuous form, it is the closed-form Wasserstein distance between Gaussians, $D_{\mathrm{WD}}(\mathcal{N}_T,\mathcal{N}_S)=\|\mu_T-\mu_S\|^2+\mathrm{tr}(\Sigma_T+\Sigma_S-2((\Sigma_T^{1/2}\Sigma_S\Sigma_T^{1/2})^{1/2}))$, with the diagonal-covariance version used in practice. The machinery carries the argument by turning category similarities and feature statistics into a true metric that respects the geometry of the underlying space, which is the property KL-divergence lacks.

What would settle it

Replace WKD-L's interrelation-based transport costs with a uniform cost on ImageNet setting (a); if top-1 accuracy stays at 72.49%, the cross-category mechanism is not the source of the gain. Alternatively, train WKD-F on a feature distribution known to be heavy-tailed, such as quantized features, and compare against KL-based feature distillation; if the gap disappears, the Gaussian modeling premise is what carries the result.

Watch

Extended reading notes

Core claim

The paper's central claim is that Wasserstein distance can serve as a strong alternative to Kullback-Leibler divergence as the workhorse of knowledge distillation. WKD-L treats the teacher's and student's softmax outputs as discrete probability measures and minimizes the entropy-regularized optimal-transport cost between them, with the transport cost built from centered kernel alignment (CKA) similarities between categories, so the student learns that dog is closer to wolf than to car. WKD-F reshapes each image's feature maps into a Gaussian with a mean and a diagonal covariance, then minimizes the closed-form Wasserstein distance between teacher and student Gaussians, balancing the mean and covariance terms with a ratio. The empirical conclusion is that WKD-L reaches 72.49% top-1 accuracy on ImageNet ResNet34-to-ResNet18 distillation versus 71.03% for classic KD and 71.96% for NKD, while WKD-F reaches 72.50% versus 71.61% for ReviewKD; combining both gives 72.76%, and the same pattern holds on CIFAR-100 and MS-COCO object detection.

Load-bearing premise

For the feature-distillation branch, the load-bearing premise is that each image's feature map can be modeled as a Gaussian distribution; if deep features depart strongly from Gaussianity, the Wasserstein distance between Gaussians loses its geometric meaning as a teaching signal.

Editorial extensions

If this is right

  • Logit distillation can exploit cross-category structure: WKD-L lifts ImageNet ResNet34-to-ResNet18 top-1 accuracy from 71.03% for classic KD and 71.96% for NKD to 72.49%.
  • Feature distillation can be cast as matching Gaussians: WKD-F reaches 72.50% on the same ImageNet setting, surpassing ReviewKD's 71.61%.
  • The two losses are complementary: combining WKD-L and WKD-F reaches 72.76% top-1 on ImageNet and outperforms DKD+ReviewKD and FCFD on MS-COCO object detection.
  • WKD also works when teacher and student share an architecture: WKD-L self-distillation on ImageNet reaches 71.35%, above 70.50% for Born-Again Networks.
  • WKD can be added to existing strong methods: WKD-F raises NKD to 72.68% on ImageNet, and WKD-L plus WKD-F improves ReviewKD and FCFD on MS-COCO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an extension, one could strip WKD-L's category-interrelation cost down to a uniform cost; if accuracy does not drop, the claimed cross-category mechanism is not what carries the gain.
  • As an extension, WKD-F's Gaussian assumption could be tested directly by fitting a non-Gaussian parametric family and comparing; the paper itself flags that what distribution deep features follow is an open problem.
  • As an extension, the same interrelation matrix could be applied to other objectives, such as label smoothing or contrastive learning, where KL-divergence to a uniform distribution is currently standard.
  • As an extension, WKD-F's diagonal-covariance choice could be compared against plain mean-and-variance matching without the Wasserstein metric, which would separate the effect of the metric from the effect of matching statistics.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes Wasserstein-distance-based knowledge distillation in two forms: WKD-L for logits, using entropy-regularized discrete optimal transport with a teacher-derived category-interrelation cost matrix; and WKD-F for intermediate features, modeling per-image feature distributions as Gaussians and minimizing their continuous Wasserstein distance. The authors evaluate WKD on ImageNet and CIFAR-100 image classification, self-knowledge distillation, and MS-COCO object detection, reporting consistent improvements over KL-divergence-based baselines and several state-of-the-art distillation methods. The manuscript includes extensive ablations of the cost construction, distribution modeling, matching strategy, distillation position, hyperparameters, and runtime.

Significance. If the central claim holds, the paper makes a useful empirical contribution by showing that optimal-transport-based losses are a practical and efficient alternative to KL-based distillation, with gains on standard benchmarks and across heterogeneous architectures. The strengths of the paper are its breadth of evaluation (ImageNet, CIFAR-100, COCO, self-KD), the inclusion of multiple ablations, the public code release, and the careful reporting of latency and complexity. The main weakness is that the headline logit-distillation comparison conflates the choice of divergence with the injection of category-interrelation information, so the paper's conceptual claim is not yet isolated.

major comments (2)
  1. [§2.1, Eqs. (3)–(5); Tables 2a–2b] The central claim that Wasserstein distance rivals KL divergence for logit distillation is not tested independently of the added category-interrelation (IR) cost. WKD-L differs from KD, DKD, and NKD in two ways: it replaces KL with entropy-regularized WD, and it inserts a teacher-derived cost c_ij = 1 - exp(-kappa(1 - IR_T(C_i,C_j))) into the transport problem. The reported gains (72.04 vs 71.03 without target separation; 72.49 vs 71.70/71.96 with separation) could therefore come entirely from the IR information rather than from the optimal-transport cross-category mechanism. Table 2b varies only the IR model inside WD; it never evaluates WD with a uniform cost matrix (e.g., c_ij = 0 for i=j and 1 otherwise) nor a KL-based loss augmented with the same pairwise IR cost. Please add both control experiments. If uniform-cost WD still outperforms KD, the title claim is supported; if not, the paper should reframe WKD-L as a method that combines WD with category interrelations rather than as evidence that the WD metric itself is superior to KL.
  2. [§2.2, Eq. (9); Section F] The geometric interpretation of WKD-F relies on the assumption that deep features are Gaussian, an assumption the authors themselves flag as an open problem in Section F. The paper should add an explicit robustness discussion noting that with diagonal covariances the loss reduces to weighted mean and variance matching, which is a well-defined moment-matching objective even when the Gaussian assumption is violated. A concrete stress test would be to compare WKD-F against a pure moment-matching baseline with the same mean and variance terms but without the Wasserstein framing; if the performance is identical, the Riemannian-metric interpretation is not doing the explanatory work claimed in the text.
minor comments (6)
  1. [Section 1 and Section C.2] Several typos remain, including 'supervisor' where 'superior' is intended in the contributions list, and 'sharping parameter' in the hyperparameter summary.
  2. [Table 4] ImageNet results are reported as single runs with margins of roughly 0.5–0.9% over strong baselines; adding multiple seeds or error bars, as already done for CIFAR-100 in Table 6, would strengthen the headline comparisons.
  3. [Sections C.2, C.5, C.6] Hyperparameters are tuned separately for each architecture and setting; the paper should state in the main text how much of the reported gain survives when a single hyperparameter configuration is fixed across settings, at least for the two ImageNet settings.
  4. [Table 12b] The choice of 18×18 RoIAlign features is justified by saturation at 28×28, but the improvement over 7×7 is small (0.27 mAP) and detection results are single-run; some variance information would help.
  5. [Equation (1)] The notation for the feature matrix is difficult to parse due to formatting issues ('k9th column' and similar artifacts); please clarify the indexing and the dimensions of the kernel matrices.
  6. [Section F] The limitation paragraph is appropriately candid about the Gaussian assumption and the open question of the true feature distribution; please also note the diagonal-covariance reduction to moment matching there, since it mitigates the concern.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning: the WKD losses are defined from optimal transport and Gaussian modeling, and the reported gains are measured against external baselines rather than derived from the paper's own assumptions.

full rationale

The derivation chain is self-contained. WKD-L is defined in Eqs. (3)-(5) as an entropy-regularized discrete Wasserstein distance between teacher and student softmax probabilities, with the transport cost c_ij built from teacher-feature CKA similarities (Eq. 1); this cost is a legitimate teacher-derived supervision signal for distillation, not a fitted surrogate for the student's final accuracy. The reported ImageNet numbers (72.49 vs 71.03, etc.) are measured against external KL-divergence baselines trained under the same protocol, so they are not re-statements of the loss definition. WKD-F models per-image feature maps as Gaussians (Eq. 6) and uses the closed-form Wasserstein distance between Gaussians (Eqs. 7-9); the Gaussian assumption is explicitly flagged as an open problem in Section F, and the diagonal-covariance version reduces to mean/variance matching, but this is a modeling assumption, not a circular derivation. The only self-citation is [33] (G2DeNet, with a co-author in common), which is used as a comparison baseline rather than as a load-bearing premise. A skeptical concern about the logit experiments is that the CKA-based cost adds inter-class information that the KL baselines lack, making the WD-vs-KL comparison confounded; however, that is an experimental-control issue (the paper never evaluates WD with a uniform cost matrix), not circularity in the sense of a prediction being equivalent by construction to its inputs.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The method relies on standard optimal transport results and several tuned hyperparameters. The main ad hoc assumption is the Gaussian model for feature distributions, which the authors themselves flag as uncertain. No new entities are introduced.

free parameters (7)
  • Temperature tau for WKD-L = 2 (ImageNet, CIFAR-100), 1 (COCO)
    Controls softmax sharpness; tuned on validation in Fig. 5.
  • Weight lambda for WKD-L = 30 (ImageNet a), 25 (ImageNet b), grid-searched on CIFAR-100
    Balances WD loss vs target loss; tuned per dataset and architecture.
  • Sharpening parameter kappa = 1 (classification), 2 (COCO RN101->RN18), 1 (RN50->MNV2)
    Shapes the CKA-similarity-to-cost transformation; chosen by validation.
  • Entropy regularization eta for Sinkhorn = 0.05
    Set for all experiments after ablation showing smooth variation.
  • Mean-cov ratio gamma for WKD-F = 2
    Balances mean and covariance terms; tuned on validation.
  • Weight of WKD-F loss = 0.02 (ImageNet a), 0.001 (b), 5e-3 (COCO), grid-searched on CIFAR-100
    Tuned per setting; varies across architectures.
  • Spatial grid for Gaussian pooling = 1x1 (ImageNet), 4x4 (COCO), searched on CIFAR-100
    Chosen by ablation; affects number of Gaussians matched.
assumptions (4)
  • standard math The 2-Wasserstein distance between two Gaussians has the closed form used in Eq. 8 (mean term plus Bures metric on covariance).
    Standard result from optimal transport, see Peyré and Cuturi (Ref [14]).
  • ad hoc to paper Per-image deep features are approximately Gaussian, so a Gaussian is an adequate model for feature distributions.
    Invoked in Section 2.2; not proven and acknowledged as open in Section F.
  • domain assumption CKA computed from teacher penultimate features measures inter-category similarity that is useful for defining transport cost between logits.
    Used in Section 2.1; supported only by ablation, not by theory.
  • standard math Entropy-regularized Sinkhorn algorithm with eta=0.05 gives a good approximation to true discrete WD for distillation.
    Standard approximation result; the choice of eta is empirical.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation." pith.science (2026). https://pith.science/paper/SOSNE3N2

@misc{pith2026241208139,
  author       = {Pith},
  title        = {Pith review of: Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SOSNE3N2}},
  note         = {Machine review of arXiv:2412.08139}
}
read the original abstract

Since pioneering work of Hinton et al., knowledge distillation based on Kullback-Leibler Divergence (KL-Div) has been predominant, and recently its variants have achieved compelling performance. However, KL-Div only compares probabilities of the corresponding category between the teacher and student while lacking a mechanism for cross-category comparison. Besides, KL-Div is problematic when applied to intermediate layers, as it cannot handle non-overlapping distributions and is unaware of geometry of the underlying manifold. To address these downsides, we propose a methodology of Wasserstein Distance (WD) based knowledge distillation. Specifically, we propose a logit distillation method called WKD-L based on discrete WD, which performs cross-category comparison of probabilities and thus can explicitly leverage rich interrelations among categories. Moreover, we introduce a feature distillation method called WKD-F, which uses a parametric method for modeling feature distributions and adopts continuous WD for transferring knowledge from intermediate layers. Comprehensive evaluations on image classification and object detection have shown (1) for logit distillation WKD-L outperforms very strong KL-Div variants; (2) for feature distillation WKD-F is superior to the KL-Div counterparts and state-of-the-art competitors. The source code is available at https://peihuali.org/WKD

Figures

Figures reproduced from arXiv: 2412.08139 by the authors.

Figure 1
Figure 1. Our methodology of Wasserstein Distance (WD) based knowledge distillation. To effectively [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. KL-Div cannot perform cross-category comparison. Com￾pare to WD in Figure 1b (left). DKL(p T ∥p S ) = X i p T i log p T i /pS i  . (2) KL-Div (2) only compares predicted probabilities correspond￾ing to the same category between the teacher and student, es￾sentially short of a mechanism to perform cross-category com￾parison, as shown in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. ⋯ ⋯ ⋯ NST ∕ ICKD-C ⋯ Teacher 𝒯 Student 𝒮 Teacher 𝒯 Student 𝒮 2nd−moment 2nd−moment WCoRD ∕ EMD-IPOT Discrete WD Frobenius norm [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Visualization of interrelations among 100 categories in feature space. The categories exhibit [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Analysis of hyper-parameters of WKD-L on ImageNet. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]
Figure 6
Figure 6. Figure 6: Analysis of hyper-parameters on WKD-F on ImageNet. [PITH_FULL_IMAGE:figures/full_fig_p019_6.png]
Figure 7
Figure 7. Figure 7: Visualization of teacher-student discrepancies for WKD-L (a) and WKD-F (b). Darker [PITH_FULL_IMAGE:figures/full_fig_p022_7.png]
Figure 8
Figure 8. Figure 8: Visualization of different models via Grad-CAM. [PITH_FULL_IMAGE:figures/full_fig_p023_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

84 extracted references · 57 canonical work pages

  1. [1]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819, 2021

    2021

  2. [2]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    arXiv 2015

  3. [3]

    Decoupled knowledge distillation

    Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11953–11962, 2022

    2022

  4. [4]

    From knowledge distillation to self- knowledge distillation: A unified approach with normalized loss and customized soft labels

    Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. From knowledge distillation to self- knowledge distillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17185– 17194, 2023

    2023

  5. [5]

    Knowledge distillation based on transformed teacher matching

    Kaixiang Zheng and EN-HUI YANG. Knowledge distillation based on transformed teacher matching. In International Conference on Learning Representations, 2024

    2024

  6. [6]

    Exploring inter-channel correlation for diversity-preserved knowledge distillation

    Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang. Exploring inter-channel correlation for diversity-preserved knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8271–8280, 2021

    work page 2021

  7. [7]

    Better teacher better student: Dynamic prior knowledge for knowledge distillation

    Martin Zong, Zengyu Qiu, Xinzhu Ma, Kunlin Yang, Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In International Conference on Learning Representations, 2023

    work page 2023

  8. [8]

    Function-consistent feature distillation

    Dongyang Liu, Meina Kan, Shiguang Shan, and CHEN Xilin. Function-consistent feature distillation. In International Conference on Learning Representations, 2023

    work page 2023

Show all 84 references
  1. [9]

    Deep residual learning for im- age recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016

    2016

  2. [10]

    The Elements of Statistial Learning

    Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistial Learning. Springer, 2009

    2009

  3. [11]

    Wasserstein generative adversarial networks

    Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214–223, 2017

    2017

  4. [12]

    Abou-Moustafa and Frank P

    Karim T. Abou-Moustafa and Frank P. Ferrie. A note on metric properties for some divergence measures: The Gaussian case. In Asian Conference on Machine Learning, pages 1–15, 2012

    2012

  5. [13]

    Wasserstein dependency measure for representation learning

    Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aäron van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. In Advances in Neural Information Processing Systems, pages 15578–15588, 2019

    2019

  6. [14]

    Computational optimal transport: With applications to data science

    Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning, 11(5-6):355–607, 2019

    2019

  7. [15]

    Wasser- stein contrastive representation distillation

    Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasser- stein contrastive representation distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16296–16305, 2021

    2021

  8. [16]

    Model compression using optimal transport

    Suhas Lohit and Michael Jones. Model compression using optimal transport. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2764–2773, 2022

    2022

  9. [17]

    Similarity of neural network representations revisited

    Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519–3529, 2019. 11

    2019

  10. [18]

    Algorithms for learning kernels based on centered alignment

    Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(28):795–828, 2012

    2012

  11. [19]

    Pattern Recognition and Machine Learning

    Christopher Bishop. Pattern Recognition and Machine Learning. Springer, 2006

    2006

  12. [20]

    Wasserstein Riemannian geometry of Gaussian densities

    Luigi Malago, Luigi Montrucchio, and Giovanni Pistone. Wasserstein Riemannian geometry of Gaussian densities. Information Geometry, 1(2):137–179, 2018

    2018

  13. [21]

    Measuring statistical dependence with hilbert-schmidt norms

    Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International Conference on Algorithmic Learning Theory, pages 63–77, 2005

    2005

  14. [22]

    Vardan Papyan, Xuemei Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020

    2020

  15. [23]

    Sinkhorn distances: Lightspeed computation of optimal transport

    Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, pages 2292–2300, 2013

    2013

  16. [24]

    Fitnets: Hints for thin deep nets

    Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations, 2015

    2015

  17. [25]

    Contrastive representation distillation

    Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020

    2020

  18. [26]

    Improved feature distillation via projector ensemble

    Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang. Improved feature distillation via projector ensemble. In Advances in Neural Information Processing Systems, pages 12084–12095, 2022

    2022

  19. [27]

    Journal of Multivariate Analysis, 88(2):365–411, 2004

    Olivier Ledoit and Michael Wolf. Journal of Multivariate Analysis, 88(2):365–411, 2004

    2004

  20. [28]

    Spatial pyramid pooling in deep convolutional networks for visual recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In European Conference on Computer Vision, pages 346–361, 2014

    2014

  21. [29]

    Distilling knowledge via knowl- edge review

    Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowl- edge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5008–5017, 2021

    2021

  22. [30]

    On information and sufficiency

    Solomon Kullback and Richard A Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79–86, 1951

    1951

  23. [31]

    An invariant form for the prior probability in estimation problems.Proceedings of the Royal Society of London

    Harold Jeffreys. An invariant form for the prior probability in estimation problems.Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 186(1007):453– 461, 1946

    1946

  24. [32]

    From sample similarity to ensemble similarity: probabilistic distance measures in reproducing kernel hilbert space

    Shaohua Kevin Zhou and Rama Chellappa. From sample similarity to ensemble similarity: probabilistic distance measures in reproducing kernel hilbert space. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(6):917–929, 2006

    2006

  25. [33]

    Deep CNNs meet global covariance pooling: Better representation and generalization

    Qilong Wang, Jiangtao Xie, Wangmeng Zuo, Lei Zhang, and Peihua Li. Deep CNNs meet global covariance pooling: Better representation and generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(8):2582–2597, 2021

    2021

  26. [34]

    A fast proximal point method for computing exact Wasserstein distance

    Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact Wasserstein distance. In Uncertainty in Artificial Intelligence, pages 433–453, 2020

    2020

  27. [35]

    Like what you like: Knowledge distill via neuron selectivity transfer

    Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv:1707.01219, 2017. 12

    2017 arXiv

  28. [36]

    Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer

    Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. InInternational Conference on Learning Representations, 2017

    2017

  29. [37]

    Knowledge distillation via adaptive instance normalization

    Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via adaptive instance normalization. arXiv preprint arXiv:2003.04289, 2020

    2003 arXiv

  30. [38]

    Positive Definite Matrices

    Rajendra Bhatia. Positive Definite Matrices. Princeton University Press, 2015

    2015

  31. [39]

    Dimensionality reduction on SPD manifolds: The emergence of geometry-aware methods

    Mehrtash Harandi, Mathieu Salzmann, and Richard Hartley. Dimensionality reduction on SPD manifolds: The emergence of geometry-aware methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(1):48–62, 2018

    2018

  32. [40]

    Lawrence, and Zhenwen Dai

    Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163–9171, 2019

    2019

  33. [41]

    ImageNet: A large- scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009

    2009

  34. [42]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  35. [43]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740–755, 2014

    2014

  36. [44]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu ...

    2019

  37. [45]

    Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H

    Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, Ievgen Redko, Antoine Rolet, Ant...

    2021

  38. [46]

    One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation

    Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang, and Chang Xu. One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation. In Advances in Neural Information Processing Systems, volume 36, pages 79570–79582, 2023

    2023

  39. [47]

    Faster R-CNN: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137–1149, 2017

    2017

  40. [48]

    Feature pyramid networks for object detection

    Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2117–2125, 2017

    2017

  41. [49]

    Detectron2

    Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. https://github.com/facebookresearch/detectron2, 2019

    2019

  42. [50]

    Distilling object detectors with fine- grained feature imitation

    Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4933–4942, 2019

    2019

  43. [51]

    Instance- conditional knowledge distillation for object detection

    Zijian Kang, Peizhen Zhang, Xiangyu Zhang, Jian Sun, and Nanning Zheng. Instance- conditional knowledge distillation for object detection. In Advances in Neural Information Processing Systems, pages 16468–16480, 2021. 13

    2021

  44. [52]

    Elementary estimators for sparse covariance matrices and other structured moments

    Eunho Yang, Aurelie Lozano, and Pradeep Ravikumar. Elementary estimators for sparse covariance matrices and other structured moments. In International Conference on Machine Learning, pages 397–405, 2014

    2014

  45. [53]

    On rényi divergence measures for continuous alphabet sources

    Manuel Gil. On rényi divergence measures for continuous alphabet sources. Master thesis, 2011

    2011

  46. [54]

    Curriculum temperature for knowledge distillation

    Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1504–1512, 2023

    2023

  47. [55]

    Class attention transfer based knowledge distillation

    Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. In Croceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11868–11877, 2023

    2023

  48. [56]

    Kd-zero: Evolving knowledge distiller for any teacher-student pairs

    Lujun Li, Peijie Dong, Anggeng Li, Zimian Wei, and Ya Yang. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. Advances in Neural Information Processing Systems, pages 69490–69504, 2023

    2023

  49. [57]

    Mobilenets: Efficient convolutional neural networks for mobile vision applications

    Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  50. [58]

    Mobilenetv2: Inverted residuals and linear bottlenecks

    Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510–4520, 2018

    2018

  51. [59]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986, 2022

    2022

  52. [60]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Con...

    2020

  53. [61]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347–10357. PMLR, 2021

    2021

  54. [62]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012–10022, 2021

    2021

  55. [63]

    Knowledge distillation from a stronger teacher

    Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. In Advances in Neural Information Processing Systems, pages 33716–33727, 2022

    2022

  56. [64]

    Correlation congruence for knowledge distillation

    Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007–5016, 2019

    2019

  57. [65]

    Relational knowledge distillation

    Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967–3976, 2019

    2019

  58. [66]

    Born again neural networks

    Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference on machine learning, pages 1607–1616. PMLR, 2018

    2018

  59. [67]

    Revisiting knowledge distillation via label smoothing regularization

    Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3903–3911, 2020. 14

    2020

  60. [68]

    Refine myself by teaching myself: Feature refinement via self-knowledge distillation

    Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, and Il-Chul Moon. Refine myself by teaching myself: Feature refinement via self-knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10664–10673, 2021

    2021

  61. [69]

    Efficient one pass self-distillation with zipf’s label smoothing

    Jiajun Liang, Linze Li, Zhaodong Bing, Borui Zhao, Yao Tang, Bo Lin, and Haoqiang Fan. Efficient one pass self-distillation with zipf’s label smoothing. In European conference on computer vision, pages 104–119. Springer, 2022

    2022

  62. [70]

    Near-linear time approximation algorithms for optimal transport via sinkhorn iteration

    Jason Altschuler, Jonathan Niles-Weed, and Philippe Rigollet. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  63. [71]

    Masked generative distillation

    Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. In European Conference on Computer Vision, pages 53–69, 2022

    2022

  64. [72]

    Knowledge diffusion for distillation

    Tao Huang, Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge diffusion for distillation. In Advances in Neural Information Processing Systems, pages 65299–65316, 2023

    2023

  65. [73]

    Multi-level logit distillation

    Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24276–24285, 2023

    2023

  66. [74]

    Randaugment: Practical automated data augmentation with a reduced search space

    Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702–703, 2020

    2020

  67. [75]

    Revisit the power of vanilla knowledge distillation: from small scale to large scale

    Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, and Yunhe Wang. Revisit the power of vanilla knowledge distillation: from small scale to large scale. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing...

    2023

  68. [76]

    Wide residual networks

    Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. InProceedings of the British Machine Vision Conference, 2016

    2016

  69. [77]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015

    2015

  70. [78]

    Shufflenet: An extremely efficient convolutional neural network for mobile devices

    Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6848–6856, 2018

    2018

  71. [79]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision, pages 618–626, 2017

    2017

  72. [80]

    Distance-iou loss: Faster and better learning for bounding box regression

    Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren. Distance-iou loss: Faster and better learning for bounding box regression. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12993–13000, 2020

    2020

  73. [81]

    Revisiting knowledge distillation via label smoothing regularization

    Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3902–3910, 2020

    2020

  74. [82]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, and et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020

    1901

  75. [83]

    GPT-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  76. [84]

    What knowledge gets distilled in knowledge distillation? In Advances in Neural Information Process- ing Systems, 2023

    Utkarsh Ojha, Yuheng Li, Anirudh Sundara Rajan, Yingyu Liang, and Yong Jae Lee. What knowledge gets distilled in knowledge distillation? In Advances in Neural Information Process- ing Systems, 2023. 15 A Implementation Details on WKD A.1 Interrelations (IRs) among Category for...

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.