Pith. sign in

REVIEW 4 major objections 3 minor 46 references

Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias

T0 review · 4 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A bias-rectification strategy lets a student model surpass its teacher in knowledge distillation, reporting results on CIFAR, ImageNet, and COCO.

desk verdict The rectification guarantee fails on a simple counterexample, and the 'first to surpass teacher' claim ignores prior work, but the bias-masking idea is concrete enough to referee. read the letter →

arxiv 2412.09874 v1 pith:LDCCABLW submitted 2024-12-13 cs.CV

Jianhua Zhang , Yi Gao , Ruyu Liu , Xu Cheng , Houxiang Zhang , Shengyong Chen This is my paper · ORCID
classification cs.CV
keywords knowledgedistillationteacher-studentbiasrectificationeliminationdynamiccurriculumlearningmodelcompressionimageclassificationobjectdetection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Knowledge distillation usually assumes a small student model can at best approach the accuracy of its large teacher. This paper argues that the real bottleneck is the teacher's own wrong predictions, which the standard distillation loss copies into the student as bias knowledge. The proposed strategy separates teacher outputs into right knowledge and biased knowledge, removes the biased part, then rectifies it with a reweighting formula so the student can learn from both, under a dynamic schedule that starts with easy samples and gradually adds hard ones. On CIFAR-10, CIFAR-100, ImageNet, and MS-COCO the authors report student models surpassing their teachers, and the module is reported to improve existing distillation methods when added on top. The core claim is that filtering and correcting teacher bias, rather than transferring all teacher output, is what allows a student to go beyond the teacher.

What carries the argument

The carrying mechanism is a three-part rectification pipeline. First, bias elimination: a binary mask built by comparing the teacher's $\arg\max$ prediction with the one-hot label separates each sample's teacher distribution into right knowledge and biased knowledge. Second, bias rectification: for a biased sample, with $t_a$ the teacher's probability for the true class, $t_b$ the teacher's largest wrong-class probability, and $t_o$ all other probabilities, the paper replaces $t_a$ and $t_b$ by $(t_a + 1)/2$ and $t_b/2$, leaves $t_o$ unchanged, and renormalizes $t_a'$ and $t_b'$ by the factor $(t_a + t_b)/(t_a' + t_b')$, so the corrected distribution is a probability distribution and, in the paper's two-class analysis, peaks at the true label. Third, dynamic learning: the loss $L_{all} = (1-\gamma)(L_{CE} + L_{easy}) + \gamma L_{hard}$ with $\gamma = e/E$ shifts the student's focus from right-knowledge tasks to rectified hard tasks over training.

What would settle it

Run the teacher over a many-class training set, apply the paper's rectification equations to every sample the teacher predicts incorrectly, and count how often the corrected distribution's largest class still disagrees with the ground-truth label; if any such samples exist, the claim that rectification converts all bias into right knowledge is violated, and retraining without those samples would test how much the gain depends on them.

Watch

Extended reading notes

Core claim

The central claim is that a student model can surpass its teacher in knowledge distillation once the teacher's biased predictions are corrected instead of being passed on as-is. The authors define right knowledge as teacher predictions whose highest-probability class matches the true label, and bias knowledge as predictions that do not. A mask built by comparing the teacher's argmax with the ground-truth label separates the two, so the student first learns only the right knowledge; then the biased samples are not discarded but transformed by Equations (4)-(7), which raise the true class probability, lower the teacher's top wrong class, keep all other classes fixed, and renormalize to a valid probability distribution. A dynamic loss with coefficient $\gamma = e/E$ makes the student learn easy right-knowledge tasks first and the rectified hard tasks later. In the reported experiments, e.g., on CIFAR-100 with a ResNet-50 teacher at 79.34%, the ResNet-18 student reaches 81.50%, with analogous teacher-surpassing results on ImageNet and MS-COCO.

Load-bearing premise

The method assumes that correcting only the teacher's top wrong class is enough to make the true class the largest probability in the corrected distribution, but when several other classes share the remaining probability the corrected true class can still lose.

Editorial extensions

If this is right

  • Student models become able to exceed the teacher's top-1 accuracy on standard classification benchmarks rather than merely approaching it.
  • The bias-correction module can be added to existing logit-based and feature-based distillation methods and is reported to improve each of them.
  • The dynamic easy-to-hard schedule is reported to reduce training time by about 25% while improving final accuracy.
  • The same strategy transfers to object detection, where student detectors reach or exceed the teacher's AP on MS-COCO.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A possible side effect not analyzed in the paper is that correcting wrong teacher predictions pushes the student's target toward the one-hot ground truth, so part of the gain may come from implicit label smoothing rather than from the specific reweighting formula.
  • The two-class rectification proof does not cover many-class cases where 'other' classes hold large probability; a direct check is to count samples on CIFAR-100 for which the corrected distribution still does not peak at the true label.
  • The recipe suggests a cheaper deployment path for stale or narrow teachers: filter and correct their errors instead of fine-tuning them, which could extend the method to continual-learning settings.
  • A sharper ablation would compare the full rectification against simply dropping biased samples and training on one-hot labels, isolating the contribution of the reweighting formula from the general act of not copying teacher mistakes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper proposes a knowledge-distillation strategy that classifies teacher predictions disagreeing with the ground-truth label as "bias," removes or rectifies those biased predictions, and schedules easy/hard knowledge during training. The central claim is that this is the first KD strategy enabling a student model to surpass its teacher, supported by experiments on CIFAR-10/100, ImageNet-1K, and MS-COCO.

Significance. If the claims were correct, the contribution would be significant: a plug-and-play module that converts teacher errors into useful training signal and lets students exceed their teachers would be valuable. The paper also compares against several KD baselines across multiple tasks. However, the load-bearing theoretical guarantee is false, and the reported empirical gains are not supported by sufficient experimental detail. The paper's core contribution therefore does not currently establish its central claim.

major comments (4)
  1. [Methodology, "Rectifying Biased Knowledge from Teacher", Eqs. (4)-(7)] The assertion after Eq. (7) that "the new predicted results align with the label" is not valid for more than two classes. The normalization in Eq. (7) rescales only t'_a and t'_b and leaves every component of t_o unchanged, so a component of t_o can remain larger than the normalized tnew_a. For example, take t_a=0.01, t_b=0.50, and a single other class with t_c=0.49. Eq. (4) gives t'_a=0.505, t'_b=0.25, t'_c=0.49; Eq. (7) rescales the first two by 0.51/0.755, yielding tnew_a≈0.341, tnew_b≈0.169, and tnew_c=0.49. The rectified distribution still assigns the largest mass to the wrong class c. The derivation in the text proves only that the sum is 1 and that tnew_a>tnew_b; it never compares tnew_a with the elements of t_o. Because L_hard in Eq. (8) is a KL loss against this "rectified" distribution, the method can reinforce a wrong class on hard samples, directly contradicting the claim that bias is rectified.
  2. [Methodology, "Rethinking Knowledge Transmission", Eq. (2)] The theoretical motivation is not rigorous. Eq. (2) writes "min(Loss) ⇒ s_a→1, s_a→t_a; s_b→0, s_b→t_b," which asks s to converge to incompatible limits. The actual minimizer of L_KL + L_CE is a compromise between the teacher distribution and the one-hot label; it does not simultaneously satisfy both limits. The subsequent sentence states that when the teacher is wrong (t_a→0, t_b→1), making s_a→t_a and s_b→t_b "actually results in s_a→1 and s_b→0," which is self-contradictory. The claim that teacher bias necessarily misleads the student is therefore not established by the provided analysis.
  3. [Experiments, Tables 1-4] The central empirical claim of students surpassing teachers is not supported by the information given. Gains such as 81.50% vs. a 79.34% teacher for a ResNet-18 student on CIFAR-100 and 42.10 AP vs. a 42.04 AP teacher on MS-COCO are far outside typical KD improvements, yet the paper reports no standard deviations, no number of runs, no data augmentation details, no training schedule details, and no per-seed results. Table 4 reports "AP70," which is not the standard COCO AP75 metric. The statement that code will be released after acceptance is not a substitute for a reviewable experimental protocol. With the current evidence, the possibility that the gains come from uncontrolled factors such as training length, augmentation, or hyperparameters cannot be excluded.
  4. [Ablation Study, Table 5; "Definitions"] The framing that "bias" is any teacher prediction not matching the label, followed by a method that removes or rectifies exactly those cases, makes the contribution partly definitional. The student receives the true label both through L_CE and through the rectified teacher distribution, so improvement over KD is expected. Table 5 ablates EBK and RBK but does not include a teacher-free baseline that trains with CE only, with or without the dynamic schedule, for the same number of epochs. Without such a baseline, it is not established that the rectified teacher distribution, rather than plain label supervision and the dynamic schedule, is what enables the student to surpass the teacher. The abstract's claim that this is "the first strategy enabling the student model to surpass the teacher model" is also overbroad; no comparison with self-distillation or born-again networks is provided.
minor comments (3)
  1. [Dynamic Learning Approach, Eq. (9)] With γ=e/E, the weight of L_CE+L_easy becomes zero in the final iteration. The paper does not explain why this schedule does not cause forgetting of easy knowledge, nor how the claimed 25% training-time reduction is measured.
  2. [Figure 3] Figure 3 shows the "others" block as a single aggregated class. Because the failure of Eq. (7) arises precisely from individual classes inside t_o, the figure should display multiple classes with individually large probabilities.
  3. [Throughout] There are several typos and formatting inconsistencies, including "mod elk" and "AN Doperation" in the bias elimination section, and inconsistent capitalization of "InDistill".

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the label-relative definition of bias is the method's intended design, and the student-surpasses-teacher claim is an empirical result rather than a fitted or renamed version of the paper's own inputs.

full rationale

I walked the derivation chain from the definition of right/biased knowledge through Equations (4)-(7) and the dynamic loss in Equations (8)-(9). The paper defines 'bias' relative to ground-truth labels and then uses those same labels to mask and rectify the teacher's predictions; this is a design choice, not a circular reduction, because the paper does not claim to derive the rectified distribution from the student's eventual accuracy. The 'student surpasses teacher' statement is supported by experimental tables on external benchmarks (CIFAR, ImageNet, MS-COCO) and is not obtained by fitting a parameter and then 'predicting' that same fitted quantity. The two self-citations in the related work (Chen et al. 2022; Li et al. 2024) are non-load-bearing survey/defense-method mentions and do not support or constrain the central derivation. The main mathematical concern lies after Eq. (7), where the claim that 'the new predicted results align with the label' is not guaranteed for arbitrary teacher distributions, e.g., t_a=0.01, t_b=0.50, t_c=0.49 leaves class c as the argmax after renormalization. That is a soundness/correctness gap rather than a circular-equivalence step, since the erroneous guarantee is not used as both input and output of the same derivation. No equation in the paper is defined in terms of the result it is invoked to establish, and no benchmark result reduces by construction to a fitted input.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No new physical or categorical entities are introduced; the paper defines 'right knowledge' and 'bias' but these are just labels for subsets of existing distributions. The primary load-bearing assumptions are the binary right/wrong split and the validity of the rectification formula, both of which are questionable. There are no fitted free parameters, as the rectification and schedule are deterministic.

assumptions (3)
  • domain assumption The teacher's prediction can be classified as entirely right or entirely wrong based solely on whether its argmax equals the ground-truth label, and all non-argmax probabilities in a 'right' sample are useful knowledge.
    This binary classification of knowledge underlies the mask in the 'Eliminating Biased Knowledge from Teacher' section. It ignores the possibility that a correct top-1 prediction may still carry misleading probabilities on other classes, and that a wrong top-1 prediction may still contain useful information for classes other than the argmax.
  • ad hoc to paper The rectification formula in Eqs. (4)-(7) always produces a probability distribution whose largest mass is on the true label.
    The paper claims this in the 'Rectifying Biased Knowledge from Teacher' section, but it is not true for all inputs (e.g., t_a=0.2, t_b=0.3, t_o=0.5 yields tnew_a=0.4 < t_o). The proof is missing and the assumed two-class scenario in Figure 3 does not generalize.
  • standard math KL divergence is an appropriate measure for both right knowledge and rectified hard knowledge.
    The loss functions Leasy and Lhard use KL divergence as in standard KD, which is a chosen modeling assumption about how the student should match the teacher. No justification is given for why KL is superior to other divergences for the rectified distribution.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias." pith.science (2026). https://pith.science/paper/LDCCABLW

@misc{pith2026241209874,
  author       = {Pith},
  title        = {Pith review of: Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LDCCABLW}},
  note         = {Machine review of arXiv:2412.09874}
}
read the original abstract

Knowledge distillation (KD) is a model compression technique that transfers knowledge from a large teacher model to a smaller student model to enhance its performance. Existing methods often assume that the student model is inherently inferior to the teacher model. However, we identify that the fundamental issue affecting student performance is the bias transferred by the teacher. Current KD frameworks transmit both right and wrong knowledge, introducing bias that misleads the student model. To address this issue, we propose a novel strategy to rectify bias and greatly improve the student model's performance. Our strategy involves three steps: First, we differentiate knowledge and design a bias elimination method to filter out biases, retaining only the right knowledge for the student model to learn. Next, we propose a bias rectification method to rectify the teacher model's wrong predictions, fundamentally addressing bias interference. The student model learns from both the right knowledge and the rectified biases, greatly improving its prediction accuracy. Additionally, we introduce a dynamic learning approach with a loss function that updates weights dynamically, allowing the student model to quickly learn right knowledge-based easy tasks initially and tackle hard tasks corresponding to biases later, greatly enhancing the student model's learning efficiency. To the best of our knowledge, this is the first strategy enabling the student model to surpass the teacher model. Experiments demonstrate that our strategy, as a plug-and-play module, is versatile across various mainstream KD frameworks. We will release our code after the paper is accepted.

Figures

Figures reproduced from arXiv: 2412.09874 by the authors.

Figure 1
Figure 1. Diagram of the knowledge transfer process in KD. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of our framework. Module [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Diagram of the rectification method. Each column corresponds to a category, [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The accuracy variation curves during the train [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 22 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Chen, K.; Zhang, J.; Liu, J.; Tong, Q.; Liu, R.; and Chen, S. 2022. Semantic visual simultaneous localization and mapping: A survey. arXiv preprint arXiv:2209.06428

    work page Pith review arXiv 2022

  4. [4]

    Chen, P.; Liu, S.; Zhao, H.; and Jia, J. 2021. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5008--5017

    work page 2021

  5. [5]

    Cho, Y.; Ham, G.; Lee, J.-H.; and Kim, D. 2023. Ambiguity-aware robust teacher (ART): Enhanced self-knowledge distillation framework with pruned teacher network. Pattern Recognition, 140: 109541

    work page 2023

  6. [6]

    J.; and Tao, D

    Gou, J.; Yu, B.; Maybank, S. J.; and Tao, D. 2021. Knowledge distillation: A survey. International Journal of Computer Vision, 129: 1789--1819

    work page 2021

  7. [7]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  8. [8]

    Heo, B.; Kim, J.; Yun, S.; Park, H.; Kwak, N.; and Choi, J. Y. 2019. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1921--1930

    work page 2019

Show all 46 references
  1. [9]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the knowledge in a neural network (2015). arXiv preprint arXiv:1503.02531, 2

    2015 arXiv

  2. [10]

    Huang, Z.; and Wang, N. 2017. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv:1707.01219

    2017 arXiv

  3. [11]

    Kim, S.; and Kim, H. 2017. Transferring Knowledge to Smaller Network with Class-Distance Loss. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings . OpenReview.net

    2017

  4. [12]

    Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

    2009

  5. [13]

    Li, Q.; Jin, S.; and Yan, J. 2017. Mimicking very efficient network for object detection. In Proceedings of the ieee conference on computer vision and pattern recognition, 6356--6364

    2017

  6. [14]

    Li, S.; Cheng, X.; Shi, F.; Zhang, H.; Dai, H.; Zhang, H.; and Chen, S. 2024. A Novel Robustness-Enhancing Adversarial Defense Approach to AI-Powered Sea State Estimation for Autonomous Marine Vessels. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 1--15

    2024

  7. [15]

    Li, Z.; Li, X.; Yang, L.; Zhao, B.; Song, R.; Luo, L.; Li, J.; and Yang, J. 2023. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 1504--1512

    2023

  8. [16]

    Liang, M.; Huang, S.; and Liu, W. 2024. Dynamic semantic structure distillation for low-resolution fine-grained recognition. Pattern Recognition, 148: 110216

    2024

  9. [17]

    Lin, T.-Y.; Doll \'a r, P.; Girshick, R.; He, K.; Hariharan, B.; and Belongie, S. 2017. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2117--2125

    2017

  10. [18]

    Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 1...

    2014

  11. [19]

    M \"u ller, R.; Kornblith, S.; and Hinton, G. E. 2019. When does label smoothing help? Advances in neural information processing systems, 32

    2019

  12. [20]

    Y.; Cha, M.-H.; Kim, D.; Han, B.; et al

    Park, D. Y.; Cha, M.-H.; Kim, D.; Han, B.; et al. 2021. Learning student-friendly teacher networks for knowledge distillation. Advances in Neural Information Processing Systems, 34: 13292--13303

    2021

  13. [21]

    Park, J.; and No, A. 2022. Prune Your Model Before Distill It. In European Conference on Computer Vision, 120--136. Springer

    2022

  14. [22]

    Park, W.; Kim, D.; Lu, Y.; and Cho, M. 2019. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3967--3976

    2019

  15. [23]

    Passalis, N.; and Tefas, A. 2018. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), 268--284

    2018

  16. [24]

    R.; and Qiu, Q

    Patel, G.; Mopuri, K. R.; and Qiu, Q. 2023. Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7786--7794

    2023

  17. [25]

    Paulin, D.; and Suneson, K. 2012. Knowledge transfer, knowledge sharing and knowledge barriers--three blurry terms in KM. Electronic Journal of Knowledge Management, 10(1): pp82--92

    2012

  18. [26]

    Ren, S.; He, K.; Girshick, R.; and Sun, J. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28

    2015

  19. [27]

    E.; Chassang, A.; Gatta, C.; and Bengio, Y

    Romero, A.; Ballas, N.; Kahou, S. E.; Chassang, A.; Gatta, C.; and Bengio, Y. 2014. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550

    2014 arXiv

  20. [28]

    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3): 211--252

    2015

  21. [29]

    Sandler, M.; Howard, A.; Zhu, M.; Zhmoginov, A.; and Chen, L.-C. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4510--4520

    2018

  22. [30]

    Sarridis, I.; Koutlis, C.; Papadopoulos, S.; and Kompatsiaris, I. 2022. InDistill: Transferring Knowledge From Pruned Intermediate Layers. arXiv preprint arXiv:2205.10003

    2022 arXiv

  23. [31]

    Simonyan, K.; and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

    2014 arXiv

  24. [32]

    Tang, R.; Liu, Z.; Li, Y.; Song, Y.; Liu, H.; Wang, Q.; Shao, J.; Duan, G.; and Tan, J. 2023. Task-balanced distillation for object detection. Pattern Recognition, 137: 109320

    2023

  25. [33]

    Tian, Y.; Krishnan, D.; and Isola, P. 2019. Contrastive representation distillation. arXiv preprint arXiv:1910.10699

    2019 arXiv

  26. [34]

    Wang, L.; and Yoon, K.-J. 2021. Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE transactions on pattern analysis and machine intelligence, 44(6): 3048--3068

    2021

  27. [35]

    Wang, T.; Yuan, L.; Zhang, X.; and Feng, J. 2019. Distilling object detectors with fine-grained feature imitation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4933--4942

    2019

  28. [36]

    Xie, Y.; Wu, H.; Lin, Y.; Zhu, J.; and Zeng, H. 2024. Pairwise difference relational distillation for object re-identification. Pattern Recognition, 152: 110455

    2024

  29. [37]

    Xu, G.; Liu, Z.; and Loy, C. C. 2023. Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup. Pattern Recognition, 138: 109338

    2023

  30. [38]

    Zagoruyko, S.; and Komodakis, N. 2016 a . Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928

    2016 arXiv

  31. [39]

    Zagoruyko, S.; and Komodakis, N. 2016 b . Wide residual networks. arXiv preprint arXiv:1605.07146

    2016 arXiv

  32. [40]

    Zhang, F.; Qu, S.; Shi, F.; and Xu, C. 2024. Overcoming the Pitfalls of Vision-Language Model for Image-Text Retrieval. In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, 2350–2359. New York, NY, USA: Association for Computing Machinery. ISBN 9798400706868

    2024

  33. [41]

    Zhang, H.; Hu, Z.; Qin, W.; Xu, M.; and Wang, M. 2021. Adversarial co-distillation learning for image recognition. Pattern Recognition, 111: 107659

    2021

  34. [42]

    Zhang, X.; Zhou, X.; Lin, M.; and Sun, J. 2018. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, 6848--6856

    2018

  35. [43]

    Zhao, B.; Cui, Q.; Song, R.; Qiu, Y.; and Liang, J. 2022. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 11953--11962

    2022

  36. [44]

    Zhou, Z.; Zhuge, C.; Guan, X.; and Liu, W. 2020. Channel distillation: Channel-wise attention for knowledge distillation. arXiv preprint arXiv:2006.01683

    2020 arXiv

  37. [45]

    Zhu, J.; Tang, S.; Chen, D.; Yu, S.; Liu, Y.; Rong, M.; Yang, A.; and Wang, X. 2021. Complementary relation contrastive distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9260--9269

    2021

  38. [46]

    Zhu, S.; Shang, R.; Yuan, B.; Zhang, W.; Li, W.; Li, Y.; and Jiao, L. 2024. DynamicKD: An effective knowledge distillation via dynamic entropy correction-based distillation for gap optimizing. Pattern Recognition, 153: 110545

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.