REVIEW 1 cited by
Rethinking the Knowledge Distillation From the Perspective of Model Calibration
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Recent years have witnessed dramatically improvements in the knowledge distillation, which can generate a compact student model for better efficiency while retaining the model effectiveness of the teacher model. Previous studies find that: more accurate teachers do not necessary make for better teachers due to the mismatch of abilities. In this paper, we aim to analysis the phenomenon from the perspective of model calibration. We found that the larger teacher model may be too over-confident, thus the student model cannot effectively imitate. While, after the simple model calibration of the teacher model, the size of the teacher model has a positive correlation with the performance of the student model.
Forward citations
Cited by 1 Pith paper
-
The Role of Teacher Calibration in Knowledge Distillation
Calibrating the teacher with temperature scaling before knowledge distillation gives small student accuracy gains, but the paper's causal claim that calibration error is the key driver is not cleanly established.
Discussion (0). Sign in to comment.