Pith. sign in

REVIEW 4 major objections 5 minor 34 references

The Role of Teacher Calibration in Knowledge Distillation

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The calibration error of the teacher — how well its confidence matches reality — is a major driver of knowledge distillation performance.

desk verdict Useful empirical recipe, but the calibration story is oversold; the gains are real, the causal claim is not. read the letter →

arxiv 2508.20224 v1 pith:QUFQ3GWO submitted 2025-08-27 cs.LG cs.AIcs.CV

Suyoung Kim , Seonguk Park , Junhoo Lee , Nojun Kwak This is my paper
classification cs.LGcs.AIcs.CV
keywords knowledgedistillationteachercalibrationtemperaturescalingerroroverconfidenceadaptivemodelcompressionselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that in knowledge distillation, what makes a teacher effective is not its raw accuracy but how well calibrated its probability outputs are. It shows that teacher accuracy explains only about 56–68% of student accuracy, while the teacher's Adaptive Calibration Error explains about 90–92%. It then demonstrates that temperature-scaling the teacher's logits — a simple calibration step that does not change the teacher's accuracy — improves student accuracy across classification and object detection, and can be layered onto state-of-the-art distillation methods. If the claim is right, it gives practitioners a cheap new lever: calibrate the teacher before distilling, and a new criterion for choosing which teacher to use.

What carries the argument

Temperature scaling applied only to the teacher: dividing the teacher's logits by T > 1 before the softmax softens the output probabilities without changing the predicted class. The paper pairs this with the Adaptive Calibration Error (ACE) metric, a classwise, equal-bin variant of expected calibration error that measures confidence–accuracy mismatch. The explanatory mechanism is a decomposition of the teacher probability as p = (1-k)p_cal + k p_err, where the overconfident error term is approximated as the one-hot label y; this makes the overconfident component reinforce standard cross-entropy and shrink the effective weight of the informative distillation signal.

What would settle it

Train students with two teachers of equal validation accuracy but different calibration error — for example, a standard teacher and an accuracy-matched teacher trained with label smoothing or mixup. If student accuracy does not track teacher ACE when teacher accuracy is held constant, the calibration-causes-gains claim is falsified. Similarly, if temperature-scaling an already well-calibrated teacher still improves the student, then softening alone, not calibration, is the active ingredient.

Watch

Extended reading notes

Core claim

The paper's central claim is that the teacher's calibration error is a key determinant of how much a student learns from knowledge distillation. Using 17 pretrained teachers on CIFAR-100, the authors find that teacher accuracy is weakly correlated with student accuracy, while teacher ACE is strongly correlated (R^2 of roughly 0.90–0.92). They then show that temperature-scaling the teacher's logits, which makes the teacher less overconfident without changing its accuracy, consistently improves student performance in standard KD and in the state-of-the-art MLLD method, on CIFAR-100, ImageNet, and COCO detection. The same calibration step also makes the student itself better calibrated. The aut

Load-bearing premise

The load-bearing premise is that temperature scaling improves student performance by lowering the teacher's calibration error rather than by merely softening the probability distribution; this is not fully isolated in the paper, and the authors' own Table 5 shows a case where teacher ACE rises while the student still improves.

Editorial extensions

If this is right

  • Teacher selection can be reframed: prefer the well-calibrated teacher over the highest-accuracy teacher, using teacher ACE as an offline, cheap selection criterion.
  • Any logit-based distillation pipeline can be upgraded at near-zero cost by temperature-calibrating the teacher before training, and the gain stacks with strong methods such as MLLD.
  • Students distilled from calibrated teachers inherit better calibration themselves, which is valuable when the compressed model is deployed in safety-sensitive settings.
  • The effect transfers across tasks — classification and two-stage detection — so the principle is not tied to a single architecture or dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct causal test would hold teacher accuracy fixed while varying calibration alone; temperature scaling is close but also changes distribution sharpness, so an experiment with accuracy-matched teachers trained under different calibration objectives would isolate the mechanism.
  • If calibration is the true driver, then teachers trained with calibration-improving regularizers such as mixup or label smoothing should produce the same distillation lift including for student accuracy; the paper's Table 6 hints at this without fully separating calibration gains from accuracy gains.
  • The paper's decomposition suggests a sweet spot: because true labels act as a one-hot overconfident component, a slightly underconfident teacher can counterbalance them, which may explain why the authors observe gains even when the teacher becomes slightly underconfident at high temperatures.
  • The paper's own limitation section restricts the claim to logit distillation; whether teacher calibration also benefits feature-based KD methods remains an open and testable question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that the calibration error of the teacher model is an important factor in knowledge distillation (KD). It presents an observational study (Fig. 1) correlating teacher ACE with student accuracy across 17 teacher models, claims that ACE is more predictive than teacher accuracy, and proposes a simple intervention: temperature-scaling the teacher network before KD. Experiments on CIFAR-100, ImageNet, and MS-COCO show accuracy gains over vanilla KD and some state-of-the-art methods when teacher temperature scaling is applied. The paper also includes a theoretical decomposition (Sec. III-B, Eqs. 5–8) intended to explain why overconfident teachers weaken KD, and additional experiments with other calibration methods (Sec. IV-E). The central claim is that reducing teacher calibration error leads to better students, and that calibration error should be a teacher-selection criterion.

Significance. If the causal claim were established, the paper would provide a simple, inexpensive way to improve KD performance across tasks and a new, practically useful teacher-selection criterion. The experimental scope is broad (classification and detection, multiple teacher–student architectures, integration with feature- and logit-based KD), and the core observation that temperature scaling often helps is reproducible in principle. However, the significance is substantially undercut by identification problems: the observational correlation does not control for teacher accuracy, the mechanistic derivation is tautological under an untested approximation, and the intervention (temperature scaling) does not isolate calibration error from distribution softness. The paper's own Table 5 provides a direct counterexample to the claim that lower teacher ACE causes better students. The useful empirical finding may be that softening teacher targets improves KD, but the calibration-error interpretation is not currently supported.

major comments (4)
  1. [Figure 1 / Section III-B] The claimed correlation (R²=0.92 between teacher ACE and student accuracy) is univariate and does not control for teacher accuracy. Since teacher accuracy and calibration error are correlated in practice (as noted in ref. [4]), the observation is consistent with teacher accuracy being the underlying factor. The paper should report partial correlations controlling for teacher accuracy, or stratify teachers by accuracy. As it stands, the claim that calibration error is 'more important' than teacher accuracy is not established.
  2. [Section III-B, Eqs. (5)–(8)] The derivation assumes p_error ≈ y, i.e., the overconfident error component equals the one-hot ground-truth label. This is a strong, untested assumption; an overconfident model can concentrate its probability mass on a wrong class. With this assumption, the conclusion that overconfidence reduces the KD term is a tautology rather than a derived mechanism. The derivation also does not isolate calibration error from the softness of the teacher distribution. Table 5 (ResNet32x4→ShuffleNetV1) shows teacher ACE rising from 0.081 to 0.131 while student accuracy improves from 72.69 to 73.66; the explanation in Sec. IV-D ('slightly underconfident teacher') directly contradicts the claim that lower ACE is the causal driver.
  3. [Section IV-A] The text claims that 'we can exclusively reduce the overconfident calibration error while fixing the accuracy.' This is not supported: temperature scaling alters the entire probability distribution, can increase ACE (as in Table 5), and can induce underconfidence. The observed KD gains could therefore be due to softer targets—a known KD mechanism—rather than to reduced calibration error. The paper needs a control that varies softness independently of measured calibration error (e.g., comparing against simply increasing the standard KD temperature) and should report ACE/ECE for all teacher models used in Tables 1–4. Without such a control, the central causal claim is not identified.
  4. [Table 2, Sec. IV-C1] For ResNet32x4→ShuffleNetV2, KD+Ours gives 75.99% vs KD 76.15% (Δ = -0.16), contradicting the abstract's 'consistently achieving superior performance' and Sec. IV-C1's 'consistently yields comparable or better performance across all models.' This inconsistency should be acknowledged and explained (e.g., statistical noise, variance), or the claims should be tempered to 'on average' with per-pair statistics. Combined with Table 5, this undermines the robustness of the headline claim.
minor comments (5)
  1. [Section III-B] Typo: 'overcalibrated error' should likely read 'overconfident error.' Also, the notation p(i)_cal and p(i)_error is not defined explicitly; please clarify that these are conditional on the teacher's output.
  2. [Section IV-A] The interaction between the calibration temperature (T=1.5) and the standard KD temperature (T=4) is not clearly described. Please specify the exact forward pass: are teacher logits divided by 1.5 before softmax, and then is the student trained with the standard KD loss using T=4? This is important for reproducibility.
  3. [Table 6] The table does not indicate the student architecture used. Please specify the student model and dataset for these calibration-method comparisons.
  4. [Figure 2] The y-axis label 'Accuracy (%)' should be clarified to 'Student accuracy (%)' for consistency with the text.
  5. [Section V (Limitation)] The limitation paragraph states the study is 'limited in scope to logit distillation methods,' but Table 4 applies the method on top of ReviewKD, a feature-based method. Please reconcile this statement.

Circularity Check

1 steps flagged · score 4.0 of 10

One definitional step in the analytic mechanism; empirical findings remain independent.

  1. self definitional [Section III-B, Eqs. (5)-(8)]
    "Since overcalibrated error can be expressed by one hot vector, we can approximate p(i)_error ≈ y(i). ... After decomposing calibrated and overconfident probability, we find that the coefficient associated with the overconfident error diminishes the influence of the KD loss. Therefore, it is imperative to find non-overconfident teachers for successful KD."

    The overconfident error component is defined to be (approximately) the one-hot label y. Substituting that definition into the loss automatically places the one-hot cross-entropy term with coefficient 1-λ+λk and the calibrated-KD term with coefficient λ(1-k). Thus the conclusion that overconfidence reduces the influence of KD is an algebraic restatement of the definition, not an independent derivation. The paper uses this tautology as the mechanistic explanation for why calibration error matters, but it does not establish that temperature scaling improves students through calibration error specifically rather than through distribution softness. Table 5 even reports a case where teacher ACE increases while student accuracy improves, so the defined mechanism is not the unique explanation.

full rationale

The paper's central empirical claim—that calibrating the teacher with temperature scaling improves KD performance—is supported by external benchmark experiments (Tables 1-4, 6) and is not itself circular. However, the only analytic link between calibration error and KD loss is Section III-B, where the decomposition sets the overconfident component equal to the one-hot label. The subsequent algebra then demonstrates that this component reduces the KD term, which is a definitional consequence rather than an empirically grounded mechanism. This is a partial circularity in the 'explanation,' but not in the measured results. No load-bearing self-citation is present: references [8] and [10] include the current authors, but they are only related-work mentions. Table 5's ResNet32x4->ShuffleNetV1 result (teacher ACE 0.081 -> 0.131 while student accuracy 72.69 -> 73.66) is a substantive correctness risk for the causal interpretation, but it is a confound/contradiction rather than a circular derivation. Overall, the empirical contribution stands on its own, so the circularity score is moderate rather than high.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper does not introduce new entities. Its central claims rely on: (1) a hand-picked temperature in the calibration step, (2) the choice of ACE as the calibration metric, and (3) an analytic decomposition that assumes the overconfident part of the teacher output is one-hot. The decomposition drives the theoretical explanation but is not validated.

free parameters (1)
  • Teacher calibration temperature T = 1.5 (default)
    Selected based on empirical observations of robust improvements between 1.5 and 3 (Section IV-A); not derived and potentially tuned to the test set.
assumptions (3)
  • domain assumption ACE is the appropriate metric for teacher calibration in KD
    The paper chooses ACE over ECE because ECE ignores inter-class relationships, but provides no empirical evidence that ACE is the best predictor of KD performance.
  • ad hoc to paper Overconfident error can be approximated by the one-hot true label (p_err ≈ y)
    Introduced in Section III-B to derive Eq. 8; this decomposition drives the conclusion that overconfidence reduces KD loss, but is not validated with data.
  • domain assumption Temperature scaling changes calibration without changing other KD-relevant properties
    The paper claims temperature scaling 'exclusively' reduces overconfident error, but Table 5 shows it also increases underconfident error (ECEu), so the clean isolation fails.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Role of Teacher Calibration in Knowledge Distillation." pith.science (2026). https://pith.science/paper/QUFQ3GWO

@misc{pith2026250820224,
  author       = {Pith},
  title        = {Pith review of: The Role of Teacher Calibration in Knowledge Distillation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QUFQ3GWO}},
  note         = {Machine review of arXiv:2508.20224}
}
read the original abstract

Knowledge Distillation (KD) has emerged as an effective model compression technique in deep learning, enabling the transfer of knowledge from a large teacher model to a compact student model. While KD has demonstrated significant success, it is not yet fully understood which factors contribute to improving the student's performance. In this paper, we reveal a strong correlation between the teacher's calibration error and the student's accuracy. Therefore, we claim that the calibration of the teacher model is an important factor for effective KD. Furthermore, we demonstrate that the performance of KD can be improved by simply employing a calibration method that reduces the teacher's calibration error. Our algorithm is versatile, demonstrating effectiveness across various tasks from classification to detection. Moreover, it can be easily integrated with existing state-of-the-art methods, consistently achieving superior performance.

Figures

Figures reproduced from arXiv: 2508.20224 by the authors.

Figure 1
Figure 1. FIGURE 1 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. FIGURE 2 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 28 canonical work pages

  1. [20]

    Rethinking the Knowledge Distillation From the Perspective of Model Calibration

    L. Y ang and J. Song, ‘‘Rethinking the knowledge distillation from the perspective of model calibration,’’ arXiv preprint arXiv:2111.01684, 2021

    work page Pith review arXiv 2021

  2. [4]

    C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger, ‘‘On calibration of modern neural networks,’’ inInternational conference on machine learning. PMLR, 2017, pp. 1321–1330

    work page 2017

  3. [1]

    J. H. Cho and B. Hariharan, ‘‘On the efficacy of knowledge distillation,’’ in Proceedings of the IEEE/CVF international conference on computer vision , 2019, pp. 4794–4802

    work page 2019

  4. [2]

    M. Zong, Z. Qiu, X. Ma, K. Y ang, C. Liu, J. Hou, S. Yi, and W. Ouyang, ‘‘Better teacher better student: Dynamic prior knowledge for knowledge distillation,’’ in The Eleventh International Conference on Learning Repre- sentations, 2022

    work page 2022

  5. [3]

    Beyer, X

    L. Beyer, X. Zhai, A. Royer, L. Markeeva, R. Anil, and A. Kolesnikov, ‘‘Knowledge distillation: A good teacher is patient and consistent,’’ in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 925–10 934

    work page 2022

  6. [5]

    Nixon, M

    J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran, ‘‘Measuring calibration in deep learning.’’ in CVPR workshops, vol. 2, 2019

    work page 2019

  7. [6]

    Hinton, O

    G. Hinton, O. Vinyals, and J. Dean, ‘‘Distilling the knowledge in a neural network,’’arXiv preprint arXiv:1503.02531, 2015

    arXiv 2015

  8. [7]

    Romero, N

    A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y . Bengio, ‘‘Fitnets: Hints for thin deep nets,’’ arXiv preprint arXiv:1412.6550, 2014

    arXiv 2014

Show all 34 references
  1. [8]

    J. Kim, S. Park, and N. Kwak, ‘‘Paraphrasing complex network: Network compression via factor transfer,’’ in Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, Eds., vol. 31. Curran Associates, I...

    2018

  2. [9]

    Zhang, T

    Y . Zhang, T. Xiang, T. M. Hospedales, and H. Lu, ‘‘Deep mutual learning,’’ in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 4320–4328

    2018

  3. [10]

    Chung, S

    I. Chung, S. Park, J. Kim, and N. Kwak, ‘‘Feature-map-level online adversarial knowledge distillation,’’ inProceedings of the 37th International Conference on Machine Learning, ser. Proceedings of Machine Learning Research, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13–18 J...

    2020

  4. [11]

    S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh, ‘‘Improved knowledge distillation via teacher assistant,’’ in Proceedings of the AAAI conference on artificial intelligence , vol. 34, 2020, pp. 5191–5198

    2020

  5. [12]

    W. Son, J. Na, J. Choi, and W. Hwang, ‘‘Densely guided knowledge distil- lation using multiple teacher assistants,’’ in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2021, pp. 9395–9404

    2021

  6. [13]

    Y . Jin, J. Wang, and D. Lin, ‘‘Multi-level logit distillation,’’ in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 24 276–24 285

    2023

  7. [14]

    P . Chen, S. Liu, H. Zhao, and J. Jia, ‘‘Distilling knowledge via knowledge review,’’ inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5008–5017

    2021

  8. [15]

    Furlanello, Z

    T. Furlanello, Z. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, ‘‘Born again neural networks,’’ in International Conference on Machine Learning . PMLR, 2018, pp. 1607–1616

    2018

  9. [16]

    B. Zhao, Q. Cui, R. Song, Y . Qiu, and J. Liang, ‘‘Decoupled knowledge distillation,’’ in Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , 2022, pp. 11 953–11 962

    2022

  10. [17]

    Y . Niu, L. Chen, C. Zhou, and H. Zhang, ‘‘Respecting transfer gap in knowledge distillation,’’ Advances in Neural Information Processing Systems, vol. 35, pp. 21 933–21 947, 2022

    2022

  11. [18]

    H. Zhou, L. Song, J. Chen, Y . Zhou, G. Wang, J. Y uan, and Q. Zhang, ‘‘Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective,’’arXiv preprint arXiv:2102.00650, 2021

    2021 arXiv

  12. [19]

    A. K. Menon, A. S. Rawat, S. Reddi, S. Kim, and S. Kumar, ‘‘A statistical perspective on distillation,’’ in International Conference on Machine Learning. PMLR, 2021, pp. 7632–7642

    2021

  13. [21]

    Zadrozny and C

    B. Zadrozny and C. Elkan, ‘‘Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers,’’ in Icml, vol. 1, 2001, pp. 609–616

    2001

  14. [22]

    ——, ‘‘Transforming classifier scores into accurate multiclass probability estimates,’’ in Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining , 2002, pp. 694–699

    2002

  15. [23]

    Platt et al

    J. Platt et al. , ‘‘Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods,’’ Advances in large margin classifiers, vol. 10, no. 3, pp. 61–74, 1999

    1999

  16. [24]

    M. H. DeGroot and S. E. Fienberg, ‘‘The comparison and evaluation of forecasters,’’ Journal of the Royal Statistical Society: Series D (The Statistician), vol. 32, no. 1-2, pp. 12–22, 1983

    1983

  17. [25]

    Soudry, E

    D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro, ‘‘The implicit bias of gradient descent on separable data,’’ Journal of Machine Learning Research, vol. 19, no. 70, pp. 1–57, 2018

    2018

  18. [26]

    Y . Wu, A. Kirillov, F. Massa, W.-Y . Lo, and R. Girshick, ‘‘Detectron2,’’ https://github.com/facebookresearch/detectron2, 2019

    2019

  19. [27]

    Y . Tian, D. Krishnan, and P . Isola, ‘‘Contrastive representation distillation,’’ arXiv preprint arXiv:1910.10699, 2019

    1910 arXiv

  20. [28]

    Krizhevsky, G

    A. Krizhevsky, G. Hinton et al., ‘‘Learning multiple layers of features from tiny images,’’ 2009

    2009

  21. [29]

    Russakovsky, J

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., ‘‘Imagenet large scale visual recognition challenge,’’ International journal of computer vision , vol. 115, pp. 211–252, 2015

    2015

  22. [30]

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P . Perona, D. Ramanan, P . Dollár, and C. L. Zitnick, ‘‘Microsoft coco: Common objects in context,’’ in Com- puter Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13 . Sprin...

    2014

  23. [31]

    S. Ren, K. He, R. Girshick, and J. Sun, ‘‘Faster r-cnn: Towards real- time object detection with region proposal networks,’’ Advances in neural information processing systems, vol. 28, 2015

    2015

  24. [32]

    Zhang, M

    H. Zhang, M. Cisse, Y . N. Dauphin, and D. Lopez-Paz, ‘‘mixup: Beyond empirical risk minimization,’’ arXiv preprint arXiv:1710.09412, 2017

    2017 arXiv

  25. [33]

    Thulasidasan, G

    S. Thulasidasan, G. Chennupati, J. Bilmes, T. Bhattacharya, S. Michalak, and O. M. Training, ‘‘Improved calibration and predictive uncertainty for deep neural networks,’’ arXiv preprint arXiv:1905.11001, 2019

    1905 arXiv

  26. [34]

    Zhang, Z

    L. Zhang, Z. Deng, K. Kawaguchi, and J. Zou, ‘‘When and how mixup improves calibration,’’ in International Conference on Machine Learning . PMLR, 2022, pp. 26 135–26 160. VOLUME 13, 2025 9

    2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.