Pith. sign in

REVIEW 4 major objections 5 minor 49 references

Balance Divergence for Knowledge Distillation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adding a weighted reverse-KL term to the standard forward-KL distillation loss lets students learn the teacher's tiny non-target probabilities, yielding 1–3% top-1 accuracy gains on CIFAR-100 and ImageNet and a 4.72…

desk verdict A simple two-temperature symmetric-KL loss with decent empirical gains, but the paper's theoretical motivation is mathematically wrong and its temperature recommendations contradict its own ablation. read the letter →

arxiv 2501.07804 v1 pith:L3HKVMTZ submitted 2025-01-14 cs.CV

Yafei Qi , Chen Wang , Zhaoning Zhang , Yaping Liu , Yongmin Zhang This is my paper · ORCID
classification cs.CV
keywords KnowledgedistillationKullback-LeiblerdivergenceReverseKLSoftlabelsTemperaturecoefficientImageclassificationSemanticsegmentationModelcompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that standard knowledge distillation, which minimizes forward Kullback-Leibler divergence between teacher and student softmax outputs, systematically underweights the teacher's tiny non-target probabilities, which the authors call the negative part of dark knowledge. Its proposed fix, Balance Divergence Distillation (BDD), adds a weighted reverse-KL term with its own temperature to the usual forward-KL term, so the student is pushed to match the teacher in the low-probability tail as well as the high-probability head. If the claim is right, a simple additive change to an existing loss improves logit-based distillation by 1–3% top-1 accuracy on CIFAR-100 and ImageNet, and gives a roughly 4.7 mIoU improvement for a segmentation student on Cityscapes. This matters because logit-based distillation is the cheapest form of knowledge transfer, and a drop-in loss change could improve many lightweight vision models.

What carries the argument

The load-bearing object is the two-temperature, two-direction KL sum. Forward KL, $\mathrm{KL}_f(p(F_T/\tau_f)\|p(F_S/\tau_f))$, keeps the student close to the teacher where the teacher is confident; the added reverse term, $\alpha\,\mathrm{KL}_r(p(F_S/\tau_r)\|p(F_T/\tau_r))$, is supposed to transfer the teacher's small probabilities. The separate temperatures $\tau_f$ and $\tau_r$ are the adjustable levers: a low forward temperature sharpens the positive-class signal, while a high reverse temperature spreads the student's attention over the long tail of near-zero teacher probabilities. The paper claims this decomposition counteracts the zero-avoiding property of forward KL, which otherwise lets the student ignore classes the teacher assigns essentially zero probability.

What would settle it

Compute the exact gradient of the forward-KL distillation term with respect to the student's raw scores (not its normalized probabilities) for a class where the teacher's probability is below $10^{-8}$ while the student still assigns noticeable probability to that class; if that gradient component is not negligible, the forward-KL loss does carry information about tiny teacher probabilities, contradicting the paper's motivating claim.

Watch

Extended reading notes

Core claim

The central claim is that forward-KL distillation starves the student of the teacher's tiny non-target probabilities, so the student overfits the confident classes and never models the tail. The proposed remedy is the Balance Divergence Distillation loss, $L_{\mathrm{BDD}} = \mathrm{KL}_f(p(F_T/\tau_f)\,\|\,p(F_S/\tau_f)) + \alpha\,\mathrm{KL}_r(p(F_S/\tau_r)\,\|\,p(F_T/\tau_r))$, a weighted sum of forward and reverse KL terms with separate softmax temperatures. The paper argues that the reverse term, $\mathrm{KL}(\text{student}\|\text{teacher})$, forces the student to keep probability mass where the teacher is near zero, while the forward term preserves fidelity on high-probability classes. Experiments on CIFAR-100 and ImageNet show top-1 accuracy gains of roughly 1–3% over the KD baseline across many teacher–student pairs, and on Cityscapes the method improves a segmentation student's mIoU by about 4.7 points over the baseline and by 0.58 over the channel-wise distillation method it builds on.

Load-bearing premise

The load-bearing premise is that a near-zero teacher probability sends the student no learning signal, so those tail classes never get modeled; if the shared normalizer that couples all output probabilities is taken into account, that premise may fail.

Editorial extensions

If this is right

  • Integrating BDD into an existing logit-distillation pipeline is a one-term change: compute forward KL with one temperature, reverse KL with another, and add them with weight $\alpha$.
  • On CIFAR-100, BDD improves top-1 accuracy over the KD baseline by roughly 1–2% for same-architecture teacher–student pairs and 2–3% for heterogeneous pairs, and it can be stacked on top of multi-level logit distillation.
  • On ImageNet, the gain over the KD baseline is about 1–3% for both ResNet-style and MobileNet-style students.
  • On Cityscapes, applying BDD to channel-wise attention outputs raises mIoU by about 4.7 points for one segmentation student and about 2.7 points for another, beating channel-wise distillation alone.
  • The optimal temperature split is architecture-dependent: deeper ResNet-style students favor a lower forward temperature and higher reverse temperature, while VGG-style students favor the opposite, so the two terms should be tuned rather than set equal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The mechanism may not be exactly what the paper states: because the gradient derivation treats each softmax output as independent, the two-temperature structure might mainly act as a form of logit-space regularization or ensembling rather than specifically rescuing ignored negative classes; an ablation that fixes the total KL weight and varies only the temperature split would separate these explan
  • The same reverse-KL compensation could be applied to feature-based distillation objectives that use normalized attention maps, where long-tailed channel statistics may create a similar imbalance.
  • By making the reverse temperature a hyperparameter, the method effectively rescales the teacher's distribution; a natural extension would be to learn the temperature pair from a small validation set, turning a manual knob into a trained component.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Balance Divergence Distillation (BDD), a logit-based knowledge distillation loss that adds a weighted reverse-KL term to the standard forward-KL term, using separate temperatures for the two terms (Eq. 8). The motivation is that forward KL neglects the teacher's extremely small negative-class probabilities, and that the reverse-KL term restores balance. The method is evaluated on CIFAR-100 and ImageNet classification and on Cityscapes semantic segmentation, reporting consistent gains of 1-3% top-1 accuracy over KD baselines and 4.72 mIoU improvement for PSPNet-R18. The paper also integrates BDD with MLD and CWD and reports ablations over the weighting coefficient and temperature settings.

Significance. The empirical component is a real strength: the experiments cover multiple teacher-student architectures, two classification datasets, a dense-prediction task, and integration with existing logit- and attention-based distillation methods; the gains in Tables 1-4 are consistent. If the mechanism were correctly established, BDD would be a simple and broadly applicable improvement to logit-based distillation. However, the theoretical derivation of the alleged imbalance problem is mathematically flawed, and the temperature-setting recommendation in the text is contradicted by the paper's own ablation table. The contribution therefore currently rests on an unverified mechanism, and the paper needs substantial revision before the central claim can be accepted.

major comments (4)
  1. [3.2 (Eqs. 4-7)] The derivation treats the softmax probabilities p(F^S_c) as independent variables when differentiating the KL terms. Differentiating the forward-KL term with respect to the student logit z_j gives ∂KL_f/∂z_j = (p^S_j − p^T_j)/τ, not a quantity proportional to −p^T_j/p^S_j. For a negative class with p^T_j ≈ 0, the gradient is approximately p^S_j/τ, which is nonzero whenever the student assigns any probability to that class. The limit in Eq. (5) is an artifact of ignoring the shared softmax denominator, and the same problem affects Eq. (7). The paper's central motivation, that forward KL ignores teacher negatives, is therefore not supported by its own gradient analysis. The empirical gains may still be real, but the claimed mechanism is unverified; the authors should either redo the gradient analysis correctly or reframe the contribution as an empirical loss design and provide direct evidence (for example, gradient statistics or ablations isolating the effect of the reverse-KL term) for the proposed mechanism.
  2. [3.3 (Eq. 9)] The 'accumulate' variant is not well defined. Equation (9) writes an integral with respect to d(F^T_c/τ_f) and d(F^S_c/τ_r), but F^T_c and F^S_c are network outputs, not integration variables, and no integration range or measure is specified. Table 8 reports results for 'accumulate' without explaining how the expression is computed in practice. Since this variant is used as evidence for the BDD+ configuration, its definition must be supplied or the variant should be removed from the tables.
  3. [4.4 (Table 8)] The text and the ablation table directly contradict each other. The text states that for ResNet teachers one should set τ_f smaller and τ_r larger, and that for VGG teachers one should set τ_f larger and τ_r smaller. Table 8 shows the opposite: for ResNet32×4→ResNet8×4, the best configurations are (8.0, 2.0) and 'accumulate' (76.21 and 76.22), while (2.0, 8.0) gives 76.02; for VGG13→VGG8, the best configuration is (2.0, 8.0) at 74.74 and the worst is (8.0, 2.0) at 73.50. The proposed temperature-ratio recommendation is contradicted by the reported numbers and needs to be corrected or explained.
  4. [4.2 (Tables 1-3)] Several empirical claims lack measures of uncertainty. Tables 1 and 2 state that results are averages of five runs but report no standard deviations, and Table 3 reports no variance at all. In addition, the ImageNet KD baseline for MobileNetV2 (68.58) is below the student trained without distillation (68.87), which is a known failure mode of logit distillation and should be discussed because it affects the interpretation of the reported +3.25 gain. Adding variance estimates or at least discussing run-to-run variation would materially strengthen the central empirical claim.
minor comments (5)
  1. [Abstract and Introduction] The abstract and introduction state a 4.55% mIoU improvement for the Cityscapes semantic segmentation experiments, while Table 4 and Section 4.3 report 4.72 mIoU for PSPNet-R18; these numbers should be harmonized.
  2. [Throughout] There are several typos and infelicities: 'sementatic' (Section 4.3), 'Mehtod' (Table 5 caption), 'releation' (Section 1), 'ifp(F T ) > 0' (Section 3.2), and inconsistent capitalization of Hinton's name in Section 1.
  3. [Eq. (2)] Equation (2) writes the KL divergence with an explicit spatial sum, but the later equations in Section 3 drop or reorder the spatial indices without comment; the notation should be made uniform so that the classification and dense-prediction cases are clearly distinguished.
  4. [Table 6] The checkmark-style entries in Table 6 are visually ambiguous; replacing them with explicit 'yes'/'no' entries would improve readability.
  5. [Figure 2] The caption of Figure 2 repeats almost verbatim the caption of Figure 1; the two figures should have distinct descriptive captions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BDD is a directly defined loss whose gains are measured empirically, not derived from the fitted hyperparameters.

full rationale

The paper's central claim is that adding a weighted reverse-KL term to forward-KL distillation improves accuracy. The BDD loss is defined explicitly in Eq. (6) and Eq. (8), and the reported improvements are measured benchmark results (Tables 1-4), not quantities forced by construction from fitted parameters. The hyperparameters α, τf, and τr are tuned on validation sets and reported in ablations (Tables 7-8), which is a standard overfitting risk rather than circular derivation; no fitted input is later renamed as a prediction. The motivation in Sec. 3.2 derives the forward-KL gradient with respect to individual softmax probabilities as if they were independent (Eqs. 4-5 and 7), which is mathematically questionable because the softmax denominator couples all classes, and the true gradient with respect to a student logit is p_S_c - p_T_c. However, this is an incorrect or incomplete derivation, not a circular one: the loss is not defined in terms of the claimed imbalance result, and the empirical gains do not reduce to the paper's inputs by construction. The cited works (Hinton et al., variational inference references, and others) are external and are not used to smuggle in the target conclusion; no uniqueness theorem or load-bearing self-citation appears. The undefined integral-style 'accumulate' variant in Eq. (9) is a presentation defect, but it is not circular. Overall, the derivation chain is self-contained in the sense that the proposed loss and reported accuracies stand independently of any fitted parameter that is later presented as a prediction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method depends on four hyperparameters, two of which are architecture-dependent and tuned on validation sets. The gradient analysis rests on a simplifying independence assumption that is not flagged as such, and the 'accumulate' integral in Eq. 9 is an additional unverified construct. No new entities are introduced.

free parameters (4)
  • α = 4.0
    Weight of reverse KL term, chosen by validation ablation (Table 7) on CIFAR-100; no theoretical derivation.
  • τf = 2.0 (ResNet) or 8.0 (VGG)
    Forward temperature, set differently per architecture to maximize validation accuracy (Table 8); text guidance contradicts the table for VGG.
  • τr = 8.0 (ResNet) or 2.0 (VGG)
    Reverse temperature, set opposite to τf; also chosen by validation ablation.
  • β = unspecified for classification; 3.0 for segmentation
    Overall loss weight in Eq. 11; the classification setting is not reported, and the segmentation value follows CWD.
assumptions (4)
  • ad hoc to paper Softmax probabilities p(F_S_c) are treated as independent variables in the gradient analysis, so the partial derivative with respect to one probability does not account for the shared denominator (Eqs. 4-7).
    This simplification produces the claim that forward KL has zero gradient for negative classes; with softmax coupling, the logit gradient is p_S − p_T, which is nonzero.
  • ad hoc to paper The integral 'accumulate' form in Eq. 9 is a well-defined loss.
    The integral over d(F_T/τf) has no specified integration limits or measure; it appears to integrate over the temperature parameter, which is not standard calculus. The paper reports one number for this variant without explaining implementation.
  • domain assumption Teacher outputs contain useful 'dark knowledge' in extremely small probabilities that students should mimic.
    This is the premise of the imbalance problem; it is plausible but not directly measured beyond one qualitative histogram (Fig. 2).
  • standard math KL divergence is asymmetric, which justifies adding the reverse direction.
    Used to justify Eq. 3; standard information-theory result.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Balance Divergence for Knowledge Distillation." pith.science (2026). https://pith.science/paper/L3HKVMTZ

@misc{pith2026250107804,
  author       = {Pith},
  title        = {Pith review of: Balance Divergence for Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L3HKVMTZ}},
  note         = {Machine review of arXiv:2501.07804}
}
read the original abstract

Knowledge distillation has been widely adopted in computer vision task processing, since it can effectively enhance the performance of lightweight student networks by leveraging the knowledge transferred from cumbersome teacher networks. Most existing knowledge distillation methods utilize Kullback-Leibler divergence to mimic the logit output probabilities between the teacher network and the student network. Nonetheless, these methods may neglect the negative parts of the teacher's ''dark knowledge'' because the divergence calculations may ignore the effect of the minute probabilities from the teacher's logit output. This deficiency may lead to suboptimal performance in logit mimicry during the distillation process and result in an imbalance of information acquired by the student network. In this paper, we investigate the impact of this imbalance and propose a novel method, named Balance Divergence Distillation. By introducing a compensatory operation using reverse Kullback-Leibler divergence, our method can improve the modeling of the extremely small values in the negative from the teacher and preserve the learning capacity for the positive. Furthermore, we test the impact of different temperature coefficients adjustments, which may conducted to further balance for knowledge transferring. We evaluate the proposed method on several computer vision tasks, including image classification and semantic segmentation. The evaluation results show that our method achieves an accuracy improvement of 1%~3% for lightweight students on both CIFAR-100 and ImageNet dataset, and a 4.55% improvement in mIoU for PSP-ResNet18 on the Cityscapes dataset. The experiments show that our method is a simple yet highly effective solution that can be smoothly applied to different knowledge distillation methods.

Figures

Figures reproduced from arXiv: 2501.07804 by the authors.

Figure 1
Figure 1. Comparison of output probability fitting by normal KD and BDD. As shown in figure a is normal KD, the minuscule values can not learn well by student. Figure b is our pro￾posed Balance Divergence Distillation(BDD), we highlight the positive and negative regions of the teacher’s probability outputs through temperature coefficient scaling, and then use different KL divergences in BDD to mimic the model’s outputs. that … view at source ↗
Figure 2
Figure 2. Comparison of output probability fitting by normal KD and BDD. This image il￾lustrates the logit output fitting by KD and BDD on CIFAR100. The blue histogram shows the student’s logit output, while the orange histogram represents the teacher’s logit output. The left chart reveals that standard KD tends to overfit the positives of the teacher, neglecting the nega￾tives and resulting in suboptimal learning outcomes. I… view at source ↗
Figure 3
Figure 3. Unbalance in the output feature map of semantic segmentation. The image shows output feature maps and channel attention probability maps for a semantic segmentation task on the Cityscapes dataset. The red boxes at the top and bottom of the image represent the selected regions’ output features and their corresponding channel attention probability distributions. The vertical axis of the channel attention indicates the… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative segmentation results and channel distributions. This figure displays the output feature maps and channel attention probability maps of PSPNet-R18 for a seman￾tic segmentation task on the Cityscapes dataset. (a) raw images, (b) ground truth(GT), (c)our metho…
Figure 5
Figure 5. Figure 5: Top-1 accuracy of training on validation and train sets. The red line represents the accuracy of BDD, the blue line represents the accuracy of our DKD, and the yellow line represents the accuracy of KD. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 41 canonical work pages

  1. [22]

    Amara, N

    I. Amara, N. Sepahvand, B. H. Meyer, W. J. Gross, J. J. Clark, Bd-kd: Balancing the divergences for online knowledge distillation, arXiv preprint arXiv:2212.12965 (2022)

    arXiv 2022

  2. [1]

    S. Han, H. Mao, W. J. Dally, Deep compression: Compressing deep neural network with pruning, trained quantization and huffman coding, in: ICLR, 2016

    work page 2016

  3. [2]

    Z. Liu, J. Li, Z. Shen, G. Huang, S. Yan, C. Zhang, Learning efficient con- volutional networks through network slimming, in: Proceedings of the IEEE international conference on computer vision, 2017, pp. 2736–2744

    work page 2017

  4. [3]

    Courbariaux, I

    M. Courbariaux, I. Hubara, D. Soudry, R. El-Yaniv, Y . Bengio, Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1, arXiv preprint arXiv:1602.02830 (2016). 21

    arXiv 2016

  5. [4]

    H. Li, A. Kadav, I. Durdanovic, H. Samet, H. P. Graf, Pruning filters for efficient convnets, in: ICLR (Poster), 2017

    work page 2017

  6. [5]

    Hinton, O

    G. Hinton, O. Vinyals, J. Dean, Distilling the knowledge in a neural network, arXiv preprint arXiv:1503.02531 (2015)

    arXiv 2015

  7. [6]

    Romero, N

    A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, Y . Bengio, Fit- nets: Hints for thin deep nets, in: ICLR (Poster), 2015

    work page 2015

  8. [7]

    Y . Liu, W. Zhang, J. Wang, Adaptive multi-teacher multi-level knowledge distillation, Neurocomputing 415 (2020) 106–113

    2020

Show all 49 references
  1. [8]

    Z. Yang, Z. Li, M. Shao, D. Shi, Z. Yuan, C. Yuan, Masked generative dis- tillation, in: European Conference on Computer Vision, Springer, 2022, pp. 53–69

    2022

  2. [9]

    P. Chen, S. Liu, H. Zhao, J. Jia, Distilling knowledge via knowledge review, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, 2021, pp. 5008–5017

    2021

  3. [10]

    C. Shu, Y . Liu, J. Gao, Z. Yan, C. Shen, Channel-wise knowledge distil- lation for dense prediction, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5311–5320

    2021

  4. [11]

    Z. Li, X. Li, L. Yang, B. Zhao, R. Song, L. Luo, J. Li, J. Yang, Curricu- lum temperature for knowledge distillation, in: Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 37, 2023, pp. 1504–1512

    2023

  5. [12]

    B. Zhao, Q. Cui, R. Song, Y . Qiu, J. Liang, Decoupled knowledge distilla- tion, in: Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, 2022, pp. 11953–11962

    2022

  6. [13]

    Y . Jin, J. Wang, D. Lin, Multi-level logit distillation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 24276–24285

    2023

  7. [14]

    J. H. Cho, B. Hariharan, On the efficacy of knowledge distillation, in: Pro- ceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 4794–4802. 22

    2019

  8. [15]

    Mirzadeh, M

    S. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, H. Ghasemzadeh, Improved knowledge distillation via teacher assistant, in: AAAI, AAAI Press, 2020, pp. 5191–5198

    2020

  9. [16]

    H. Zhou, L. Song, J. Chen, Y . Zhou, G. Wang, J. Yuan, Q. Zhang, Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective, in: ICLR, 2021

    2021

  10. [17]

    B. Heo, J. Kim, S. Yun, H. Park, N. Kwak, J. Y . Choi, A comprehensive over- haul of feature distillation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 1921–1930

    2019

  11. [18]

    B. Heo, M. Lee, S. Yun, J. Y . Choi, Knowledge transfer via distillation of ac- tivation boundaries formed by hidden neurons, in: Proceedings of the AAAI Conference on Artificial Intelligence, V ol. 33, 2019, pp. 3779–3787

    2019

  12. [19]

    W. Park, D. Kim, Y . Lu, M. Cho, Relational knowledge distillation, in: Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, 2019, pp. 3967–3976

    2019

  13. [20]

    Y . Tian, D. Krishnan, P. Isola, Contrastive representation distillation, in: ICLR, 2020

    2020

  14. [21]

    Y . Wang, W. Zhou, T. Jiang, X. Bai, Y . Xu, Intra-class feature variation dis- tillation for semantic segmentation, in: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16, Springer, 2020, pp. 346–362

    2020

  15. [23]

    Zagoruyko, N

    S. Zagoruyko, N. Komodakis, Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer, in: ICLR (Poster), 2017

    2017

  16. [24]

    Z. Guo, H. Yan, H. Li, X. Lin, Class attention transfer based knowledge dis- tillation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 11868–11877. 23

    2023

  17. [25]

    C.-j. Li, Z. Qu, S.-y. Wang, A method of knowledge distillation based on fea- ture fusion and attention mechanism for complex traffic scenes, Engineering Applications of Artificial Intelligence 124 (2023) 106533

    2023

  18. [26]

    Y . Liu, X. Bai, J. Wang, G. Li, J. Li, Z. Lv, Image semantic segmentation approach based on deeplabv3 plus network with an attention mechanism, Engineering Applications of Artificial Intelligence 127 (2024) 107260

    2024

  19. [27]

    J. Gou, X. Xiong, B. Yu, Y . Zhan, Z. Yi, Channel correlation-based selective knowledge distillation, IEEE Transactions on Cognitive and Developmental Systems (2022)

    2022

  20. [28]

    Jiang, Y

    Y . Jiang, Y . Cao, W. Shen, A masked reverse knowledge distillation method incorporating global and local information for image anomaly detection, Knowledge-Based Systems 280 (2023) 110982

    2023

  21. [29]

    S. Lao, G. Song, B. Liu, Y . Liu, Y . Yang, Masked autoencoders are stronger knowledge distillers, in: Proceedings of the IEEE/CVF International Con- ference on Computer Vision, 2023, pp. 6384–6393

    2023

  22. [30]

    Zheng, R

    Z. Zheng, R. Ye, P. Wang, D. Ren, W. Zuo, Q. Hou, M.-M. Cheng, Localiza- tion distillation for dense object detection, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9407– 9416

    2022

  23. [31]

    W. Cao, Y . Zhang, J. Gao, A. Cheng, K. Cheng, J. Cheng, Pkd: Gen- eral distillation framework for object detectors via pearson correlation co- efficient, Advances in Neural Information Processing Systems 35 (2022) 15394–15406

    2022

  24. [32]

    W. Li, L. Li, H. Yang, Progressive cross-domain knowledge distillation for efficient unsupervised domain adaptive object detection, Engineering Appli- cations of Artificial Intelligence 119 (2023) 105774

    2023

  25. [33]

    R. Tang, Z. Liu, Y . Li, Y . Song, H. Liu, Q. Wang, J. Shao, G. Duan, J. Tan, Task-balanced distillation for object detection, Pattern Recognition 137 (2023) 109320

    2023

  26. [34]

    Shlens, Notes on kullback-leibler divergence and likelihood, arXiv preprint arXiv:1404.2000 (2014)

    J. Shlens, Notes on kullback-leibler divergence and likelihood, arXiv preprint arXiv:1404.2000 (2014). 24

    2014 arXiv

  27. [35]

    K. P. Murphy, Machine learning: a probabilistic perspective, MIT press, 2012

    2012

  28. [36]

    Ganguly, S

    A. Ganguly, S. W. Earp, An introduction to variational inference, arXiv preprint arXiv:2108.13083 (2021)

    2021 arXiv

  29. [37]

    Neumann, et al., Variational inference for policy search in changing sit- uations, in: Proceedings of the 28th International Conference on Machine Learning, ICML 2011, 2011, pp

    G. Neumann, et al., Variational inference for policy search in changing sit- uations, in: Proceedings of the 28th International Conference on Machine Learning, ICML 2011, 2011, pp. 817–824

    2011

  30. [38]

    Krizhevsky, et al., Learning multiple layers of features from tiny images (2009)

    A. Krizhevsky, et al., Learning multiple layers of features from tiny images (2009)

    2009

  31. [39]

    Russakovsky, J

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al., Imagenet large scale visual recognition challenge, International journal of computer vision 115 (2015) 211–252

    2015

  32. [40]

    Cordts, M

    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, B. Schiele, The cityscapes dataset for semantic urban scene understanding, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 3213–3223

    2016

  33. [41]

    Y . Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, J. Wang, Structured knowledge dis- tillation for semantic segmentation, in: Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, 2019, pp. 2604–2613

    2019

  34. [42]

    K. He, X. Zhang, S. Ren, J. Sun, Identity mappings in deep residual net- works, in: Computer Vision–ECCV 2016: 14th European Conference, Am- sterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, Springer, 2016, pp. 630–645

    2016

  35. [43]

    Zagoruyko, N

    S. Zagoruyko, N. Komodakis, Wide residual networks, in: BMVC, BMV A Press, 2016

    2016

  36. [44]

    Simonyan, A

    K. Simonyan, A. Zisserman, Very deep convolutional networks for large- scale image recognition, in: ICLR, 2015

    2015

  37. [45]

    Sandler, A

    M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, L.-C. Chen, Mobilenetv2: Inverted residuals and linear bottlenecks, in: Proceedings of the IEEE con- ference on computer vision and pattern recognition, 2018, pp. 4510–4520. 25

    2018

  38. [46]

    Zhang, X

    X. Zhang, X. Zhou, M. Lin, J. Sun, Shufflenet: An extremely efficient convo- lutional neural network for mobile devices, in: Proceedings of the IEEE con- ference on computer vision and pattern recognition, 2018, pp. 6848–6856

    2018

  39. [47]

    N. Ma, X. Zhang, H.-T. Zheng, J. Sun, Shufflenet v2: Practical guidelines for efficient cnn architecture design, in: Proceedings of the European conference on computer vision (ECCV), 2018, pp. 116–131

    2018

  40. [48]

    H. Zhao, J. Shi, X. Qi, X. Wang, J. Jia, Pyramid scene parsing network, in: Proceedings of the IEEE conference on computer vision and pattern recog- nition, 2017, pp. 2881–2890

    2017

  41. [49]

    L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, A. L. Yuille, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convo- lution, and fully connected crfs, IEEE transactions on pattern analysis and machine intelligence 40 (4) (2017) 834–848. 26

    2017

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.