Progressive Class-level Distillation

Jianhua Xu; Jiayan Li; Jun Li; Zhourui Zhang

REVIEW 5 major objections 6 minor 29 references

Progressive Class-level Distillation

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hard classes first: progressive distillation beats ensemble teaching.

desk verdict Classification results are a credible, modest improvement; the detection claim is unsupported by the reported baselines. read the letter →

arxiv 2505.24310 v1 pith:R7L3ZDNK submitted 2025-05-30 cs.CV

Jiayan Li , Jun Li , Zhourui Zhang , Jianhua Xu This is my paper

classification cs.CV

keywords LogitDistillationKnowledgeStep-by-stepTransferBidirectionalStage-wiseProgressiveLearningClass-level

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that standard logit distillation, which matches teacher and student over all classes at once, lets high-confidence classes dominate and starves low-probability classes that still carry discriminative information. To fix this, it proposes Progressive Class-level Distillation (PCD), which first ranks classes by the absolute difference between teacher and student logits, treating large disagreements as 'hard' classes, and then splits distillation into stages. In each stage, only a subset of classes is aligned, starting with small fine-grained groups and expanding to the full set, after which the process runs in reverse from coarse to fine. The paper reports that this staged, bidirectional alignment consistently improves over vanilla knowledge distillation and over current logit-distillation methods on CIFAR, ImageNet, and MS-COCO detection.

What carries the argument

The two load-bearing components are Logit Difference Ranking (LDR) and Bidirectional Stage-wise Distillation (BSD). LDR computes the class indices sorted by descending $|z_t - z_s|$ for a batch, yielding a 'difficult-to-easy' sequence. BSD then partitions that sequence into $S$ stages; for stage $i$ the F2CL pass uses groups of size $C/(S - i + 1)$, so the first stage splits classes into many small groups and the last stage uses the full set, while the C2FL pass mirrors this with group size $C/i$. In each group, logits outside the group are masked to $-\infty$ before softmax, and the KL divergence between teacher and student is weighted by $\lambda = 1 - \cos(p_{i,j}, q_{i,j})$, so poorly aligned groups are up-weighted.

What would settle it

On CIFAR-100, recompute the class ordering under the two natural readings of Eq. (3) — ranking the per-sample absolute differences then taking a majority vote, versus ranking the batch-mean absolute difference — and measure how often they disagree during training. If many classes flip rank between the two readings in early epochs, the staged grouping is unstable and the reported gains should be re-examined against a fixed precomputed ordering.

Watch

Extended reading notes

Core claim

PCD's central claim is that teacher–student logit alignment is more effective when it is organized as a curriculum over classes rather than performed as an all-class ensemble. The paper identifies classes with large $|z_t - z_s|$ as difficult and deserving priority, and it routes them through a two-pass procedure: a fine-to-coarse pass (F2CL), where group sizes grow from $C/S$ to $C$, and a coarse-to-fine pass (C2FL) with the reverse schedule, with a cosine-distance weight $\lambda = 1 - \cos(p_{i,j}, q_{i,j})$ amplifying groups where teacher and student disagree. The complete loss is $L_{\mathrm{CE}} + \alpha(L_{\mathrm{F2CL}} + L_{\mathrm{C2FL}})$, and the ablation study attributes the best results to the combination of ranking, the forward pass, and the reverse pass. In the reported experiments, PCD exceeds the vanilla KD baseline by up to 3.01% Top-1 on CIFAR and outperforms DKD, CTKD, LA, RC, and LR on the heterogeneous teacher–student pairs reported.

Load-bearing premise

The load-bearing premise is that sorting classes by teacher–student logit difference, computed however the batch is aggregated, yields a stable and meaningful difficulty ordering; the paper gives no evidence that this ordering is stable across training.

Editorial extensions

If this is right

PCD changes only the distillation loss, so it can be added to any existing logit-distillation pipeline with no architectural change to teacher or student.
The optimal number of stages depends on the teacher–student capacity gap: the experiments use $S=3$ for same-architecture pairs and $S=5$ for heterogeneous pairs, so practitioners should treat $S$ as a hyperparameter rather than a constant.
The detection results on MS-COCO show the staged class curriculum transfers to dense prediction; the same loss term can be reused in detector distillation.
Because the per-group weight $\lambda = 1 - \cos(p_{i,j}, q_{i,j})$ shrinks as student and teacher align, the method self-adjusts as training progresses and does not need a separate annealing schedule.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper does not run: replace the LDR ordering with a random class permutation. If the staged loss still helps, the grouping and weighting carry the benefit; if gains vanish, the disagreement-based ranking is essential.
Eq. (3) does not specify how a batch of samples collapses into one class ranking. A robust implementation might rank per sample and aggregate by median rank; we infer the reported numbers would be stable under either reading, but the paper never demonstrates this.
Because CTKD also builds a curriculum (over temperature rather than class subsets), a head-to-head with identical hyperparameters would isolate what stage-wise class grouping adds over a temperature curriculum.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the classification half of this paper is a solid, modest improvement; the detection half is not supported by the experiments as reported. I would send the paper to review, but the authors need to tighten several things.

What's actually new: ranking classes by teacher-student logit difference (LDR) and then running stage-wise group distillation in both fine-to-coarse and coarse-to-fine directions. The ingredients are familiar — curriculum via difficulty (CTKD), decoupled target/non-target terms (DKD), masking to subsets — but this particular assembly, with the reverse refinement pass, is not in the cited literature. That is a legitimate incremental novelty. On CIFAR and ImageNet, the method consistently beats vanilla KD by 1–3% and is competitive with DKD, CTKD, RC, and LR. The ablations show each component helps, and the adaptive group weighting adds a bit more. That part is credible.

Where it goes soft:

- Eq. (6) swaps p and q. Earlier, p is the teacher and q the student; in Eq. (6) the assignments are reversed, which makes the KL direction in Eq. (8) wrong as printed. Likely a typo, but it obscures the actual objective and should be fixed.

- Eq. (3) is underspecified. Are the per-class teacher-student differences computed per sample and then averaged, or averaged over the batch before sorting? With batch size 512, those give different orderings. Because the entire staged curriculum depends on this ranking, it is a reproducibility blocker. It also raises a stability question: if the ranking shifts every batch, the stages are moving targets; the paper does not discuss that.

- No error bars or significance tests. ImageNet margins over RC (+0.12) and LR (+0.22) are within plausible run-to-run noise. The CIFAR gains are larger and more believable.

- Detection on MS-COCO: the stress-test note is right. Table 4 compares only vanilla KD and FitNet, with no DKD, CTKD, RC, or LR. So the abstract's claim of superiority for 'both classification and detection' is not supported. The method might work for detection, but the table does not show it.

- S and alpha are tuned on the same validation sets, and S=3 for homogeneous, S=5 for heterogeneous architectures. That is post-hoc selection, and Table 7 shows sensitivity to S.

Who this is for: people working on logit distillation who want the idea of per-class difficulty ranking for staged transfer. If the authors release code and address the above, it becomes a useful reference. As is, it is a promising but incompletely specified recipe.

Recommendation: send to peer review with a request for code, error bars, SOTA logit-distillation baselines on detection, and a clarification of the ranking. The classification results justify the referee time.

Referee Report

5 major / 6 minor

Summary. The paper proposes Progressive Class-level Distillation (PCD), a logit distillation method that ranks classes by teacher-student logit difference and performs bidirectional stage-wise distillation with fine-to-coarse and coarse-to-fine learning. The method is evaluated on CIFAR-100, ImageNet, and MS-COCO object detection, reporting accuracy gains over vanilla KD and several logit distillation baselines. The central claim is that progressive class-group-based distillation better transfers knowledge from low-confidence classes and achieves state-of-the-art results for both classification and detection.

Significance. If the claims hold, PCD is a simple and interesting addition to logit distillation, with a plausible motivation and a curriculum-style mechanism. The paper reports consistent gains over vanilla KD across three datasets and multiple architectures, and includes ablations and parameter analysis. However, the evidence for superiority over state-of-the-art is not fully conclusive: the detection comparison lacks strong LD baselines, and the ImageNet margins over strong baselines are small and are reported without uncertainty estimates. The method itself is clear enough to be reproduced once the details in Eqs. (3), (4), and (6) are clarified.

major comments (5)

[Section 4.2, Table 4] The detection experiments compare PCD only with vanilla KD and FitNet, neither of which is a state-of-the-art logit distillation method. Since the abstract claims superiority over state-of-the-art for both classification and detection, the detection half of the claim is unsupported. Please add recent logit distillation baselines (e.g., DKD, CTKD, RC, LR) on MS-COCO, or revise the claim to state that PCD outperforms vanilla KD and FitNet in detection.
[Section 4.2, Table 3] On ImageNet, PCD's Top-1 margin over RC and LR is only 0.12% and 0.22%, and its Top-5 accuracy (90.74) is lower than LR (90.93). The paper states that results are averaged over three trials but does not report standard deviations or significance tests. These differences are likely within run-to-run noise, so the claimed superiority over strong baselines on ImageNet is not established. Please report variance or significance, or temper the claim.
[Section 3.4, Eq. (6)] As written, Eq. (6) defines p_{i,j} from student logits z^s and q_{i,j} from teacher logits z^t, which reverses the convention used in Eq. (1) and Eq. (2). Because KL divergence is asymmetric, D_{i,j} in Eq. (8) is then KL(student || teacher), not the standard KL(teacher || student). Please state which direction is intended and correct the notation, or explicitly justify the reversed direction.
[Section 3.3, Eq. (3) and Section 3.4, Eq. (4)] The ranking in Eq. (3) is underspecified for batched training: it is unclear whether |z_t - z_s| is computed per sample and then aggregated, computed per sample and sorted individually, or computed on batch-averaged logits. Since the class ordering I directly determines the group construction in Eqs. (4)-(5), this ambiguity affects the curriculum and the reproducibility of the method. Also specify how non-integer group sizes C/(S-i+1) and C/i are rounded and how leftover classes are assigned.
[Section 4.3, Table 5] The ablation table is difficult to interpret: several rows contain a single checkmark and no numerical result, and the text states that 'merely using LDR module is equivalent to vanilla KD' while the table appears to show a different structure. Please reformat the table so each row corresponds to a distinct configuration and each configuration has a clear result.

minor comments (6)

[Throughout] 'CIF AR' appears throughout the manuscript and should be written as 'CIFAR-100' or similar.
[Abstract] 'Extension experiments' should be 'Extensive experiments'.
[Section 3.4, Eq. (9)] The notation M in Eq. (9) is used for the number of class subsets in each stage, but this number depends on the stage i and on the rounding of group sizes. Please use M_i or define M_i = ceil(C / m_i).
[Figure 2] The label 'C2FLF2CL' in the figure appears to be a concatenation of the two process names and should be clarified or separated.
[References] Reference [7] is incomplete; please provide the full citation for the CIFAR dataset.
[Conclusion] The sentence 'BSD component achieves performs multi-stage knowledge transfer' contains a typo ('achieves performs').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: PCD is an empirically tested training heuristic, and no load-bearing step reduces to its own inputs.

full rationale

The paper does not derive any prediction from an assumption that already contains the conclusion. The proposed method is a training recipe: Eq. (3) ranks classes by the absolute teacher-student logit difference, Eqs. (5)-(8) mask and weight group-wise KL divergences, and Eq. (10) combines these losses with cross-entropy. The reported accuracy and AP numbers come from actually training and evaluating models against external baselines, not from algebraically unfolding the loss definition. The use of the student's own logits in the ranking and in the cosine-distance weights is a designed feedback loop, not a circular derivation: it chooses a curriculum that is then tested, and the paper does not claim that the ranking by itself guarantees the reported improvements. The ablation study and comparisons are empirical evidence, so the central claim is not equivalent to its inputs by construction. Hyperparameter choices of S and alpha are tuned on the CIFAR-100 validation set, and the final numbers are reported on the same validation set; this is a methodological concern about selection, not a circularity of the kind where a fitted parameter is renamed as a prediction. The detection comparison in Table 4 includes only KD and FitNet, so the abstract's 'both classification and detection' superiority claim is under-supported for detection; that is a completeness/evidence issue, not circularity. The underspecification of how the batch-level ranking in Eq. (3) is aggregated is a reproducibility issue, not a circularity issue. The only self-citations, such as [23] and [28], appear in the related-work survey and are not load-bearing justifications of the central claim. Therefore the paper is not circular in any of the enumerated senses.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or conceptual entities. Its free parameters are the stage count S and the loss weight alpha, both selected on the validation set. The key domain assumptions are that logit difference indicates learning priority and that low-probability classes carry useful signal; these are plausible but unproven.

free parameters (2)

S (number of stages) = 3 for homogeneous, 5 for heterogeneous
Chosen by validation performance; Table 7 shows different best values for the two settings.
alpha (loss balance) = 1.0
Chosen by validation on one teacher-student pair (Table 8), then used for all experiments.

assumptions (3)

domain assumption Low-probability classes contain discriminating information useful for distillation
Stated in the introduction and used to justify focusing on hard classes.
domain assumption Classes with larger teacher-student logit difference are the most important to distill first
Used in Eq. (3) and Sec. 3.3, not derived or independently validated.
standard math Standard properties of softmax and KL divergence
Used throughout the loss formulations in Sec. 3.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Progressive Class-level Distillation." pith.science (2026). https://pith.science/paper/R7L3ZDNK

@misc{pith2026250524310,
  author       = {Pith},
  title        = {Pith review of: Progressive Class-level Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R7L3ZDNK}},
  note         = {Machine review of arXiv:2505.24310}
}

read the original abstract

In knowledge distillation (KD), logit distillation (LD) aims to transfer class-level knowledge from a more powerful teacher network to a small student model via accurate teacher-student alignment at the logits level. Since high-confidence object classes usually dominate the distillation process, low-probability classes which also contain discriminating information are downplayed in conventional methods, leading to insufficient knowledge transfer. To address this issue, we propose a simple yet effective LD method termed Progressive Class-level Distillation (PCD). In contrast to existing methods which perform all-class ensemble distillation, our PCD approach performs stage-wise distillation for step-by-step knowledge transfer. More specifically, we perform ranking on teacher-student logits difference for identifying distillation priority from scratch, and subsequently divide the entire LD process into multiple stages. Next, bidirectional stage-wise distillation incorporating fine-to-coarse progressive learning and reverse coarse-to-fine refinement is conducted, allowing comprehensive knowledge transfer via sufficient logits alignment within separate class groups in different distillation stages. Extension experiments on public benchmarking datasets demonstrate the superiority of our method compared to state-of-the-arts for both classification and detection tasks.

Figures

Figures reproduced from arXiv: 2505.24310 by the authors.

**Figure 2.** Illustration of our PCD which comprises two components including Logit [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** t-SNE visualizations obtained by KD (left) and our PCD (right). [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Visualized teacher-student logits difference in varying classes. Obviously, [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 19 canonical work pages

[1]

IEEE Signal Processing Letters 30, 1722–1726 (2023)

Cao, Q., Zhang, K., He, X., Shen, J.: Be an excellent student: review, preview, and correction. IEEE Signal Processing Letters 30, 1722–1726 (2023)

work page 2023
[2]

In: Proceedings of the IEEE conference on computer vision and pattern recognition

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

2016
[3]

In: Proceedings of the IEEE/CVF international conference on computer vision

Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., Choi, J.Y.: A comprehensive overhaul of feature distillation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1921–1930 (2019)

work page 2019
[4]

In: NIPS 2014 Deep Learning Workshop

Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS 2014 Deep Learning Workshop. pp. 1–9 (2015)

work page 2015
[5]

arXiv preprint arXiv:1704.04861 (2017)

Howard, A.G.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

arXiv 2017
[6]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Jin, Y., Wang, J., Lin, D.: Multi-level logit distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24276– 24285 (2023)

work page 2023
[7]

Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

2009
[8]

IEEE Transactions on Cog- nitive and Developmental Systems (2025)

Lan, W., Cheung, Y.m., Xu, Q., Liu, B., Hu, Z., Li, M., Chen, Z.: Improve knowl- edge distillation via label revision and data selection. IEEE Transactions on Cog- nitive and Developmental Systems (2025)

work page 2025

Show all 29 references

[9]

In: Proceedings of the AAAI Conference on Artificial Intelligence

Li, Z., Li, X., Yang, L., Zhao, B., Song, R., Luo, L., Li, J., Yang, J.: Curriculum temperature for knowledge distillation. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37, pp. 1504–1512 (2023)

2023
[10]

In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13

Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll´ ar, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13. pp. 740–7...

2014
[11]

In: Proceedings of the European conference on computer vision (ECCV)

Ma, N., Zhang, X., Zheng, H.T., Sun, J.: Shufflenet v2: Practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV). pp. 116–131 (2018)

2018
[12]

Journal of Machine Learning Research 9(86), 2579–2605 (2008)

van der Maaten, L., Hinton, G.: Visualizing data using t-sne. Journal of Machine Learning Research 9(86), 2579–2605 (2008)

2008
[13]

In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition

Park, W., Kim, D., Lu, Y., Cho, M.: Relational knowledge distillation. In: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3967–3976 (2019) 14 F. Author et al

2019
[14]

Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for thin deep nets (2015)

2015
[15]

International journal of computer vision 115, 211–252 (2015)

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recog- nition challenge. International journal of computer vision 115, 211–252 (2015)

2015
[16]

In: Proceedings of the IEEE conference on computer vision and pattern recognition

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: In- verted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510–4520 (2018)

2018
[17]

arXiv preprint arXiv:1409.1556 (2014)

Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

2014 arXiv
[18]

arXiv preprint arXiv:2408.07703 (2024)

Sun, W., Chen, D., Lyu, S., Chen, G., Chen, C., Wang, C.: Knowledge distillation with refined logits. arXiv preprint arXiv:2408.07703 (2024)

2024 arXiv
[19]

arXiv preprint arXiv:1910.10699 (2019)

Tian, Y., Krishnan, D., Isola, P.: Contrastive representation distillation. arXiv preprint arXiv:1910.10699 (2019)

2019 arXiv
[20]

arXiv preprint arXiv:2205.11246 (2022)

Verma, A., Gulati, P., Gupta, S.: [re] distilling knowledge via knowledge review. arXiv preprint arXiv:2205.11246 (2022)

2022 arXiv
[21]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Wei, S., Luo, C., Luo, Y.: Scaled decoupled distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15975– 15983 (2024)

2024
[22]

Neurocomputing 454, 25–33 (2021)

Wen, T., Lai, S., Qian, X.: Preparing lessons: Improve knowledge distillation with better supervision. Neurocomputing 454, 25–33 (2021)

2021
[23]

In: 2023 International Joint Conference on Neural Networks (IJCNN)

Yang, G., Tang, Y., Li, J., Xu, J., Wan, X.: Amd: Adaptive masked distillation for object detection. In: 2023 International Joint Conference on Neural Networks (IJCNN). pp. 1–8 (2023)

2023
[24]

In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

Yang, Z., Li, Z., Jiang, X., Gong, Y., Yuan, Z., Zhao, D., Yuan, C.: Focal and global knowledge distillation for detectors. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4643–4652 (2022)

2022
[25]

arXiv preprint arXiv:1612.03928 (2016)

Zagoruyko, S., Komodakis, N.: Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv:1612.03928 (2016)

2016 arXiv
[26]

arXiv preprint arXiv:1605.07146 (2016)

Zagoruyko, S., Komodakis, N.: Wide residual networks. arXiv preprint arXiv:1605.07146 (2016)

2016 arXiv
[27]

IEEE Transactions on Pattern Analysis and Machine Intelligence 45(12), 15706–15724 (2023)

Zhang, L., Ma, K.: Structured knowledge distillation for accurate and efficient object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(12), 15706–15724 (2023)

2023
[28]

Zhang, Z., Li, J., Li, J., Xu, J.: Samkd: Spatial-aware adaptive masking knowledge distillation for object detection (2025)

2025
[29]

In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition

Zhao, B., Cui, Q., Song, R., Qiu, Y., Liang, J.: Decoupled knowledge distillation. In: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. pp. 11953–11962 (2022)

2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

IEEE Signal Processing Letters 30, 1722–1726 (2023)

Cao, Q., Zhang, K., He, X., Shen, J.: Be an excellent student: review, preview, and correction. IEEE Signal Processing Letters 30, 1722–1726 (2023)

work page 2023

[2] [2]

In: Proceedings of the IEEE conference on computer vision and pattern recognition

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

2016

[3] [3]

In: Proceedings of the IEEE/CVF international conference on computer vision

Heo, B., Kim, J., Yun, S., Park, H., Kwak, N., Choi, J.Y.: A comprehensive overhaul of feature distillation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 1921–1930 (2019)

work page 2019

[4] [4]

In: NIPS 2014 Deep Learning Workshop

Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. In: NIPS 2014 Deep Learning Workshop. pp. 1–9 (2015)

work page 2015

[5] [5]

arXiv preprint arXiv:1704.04861 (2017)

Howard, A.G.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

arXiv 2017

[6] [6]

In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Jin, Y., Wang, J., Lin, D.: Multi-level logit distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24276– 24285 (2023)

work page 2023

[7] [7]

Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)

2009

[8] [8]

IEEE Transactions on Cog- nitive and Developmental Systems (2025)

Lan, W., Cheung, Y.m., Xu, Q., Liu, B., Hu, Z., Li, M., Chen, Z.: Improve knowl- edge distillation via label revision and data selection. IEEE Transactions on Cog- nitive and Developmental Systems (2025)

work page 2025