Training the teacher a small number of steps ahead of the student and freezing it during distillation improves student generalization by up to 3.4% on image benchmarks.
Implementation of SOTA methods with Spaced KD For traditional KD methods (BAN (Furlanello et al., 2018), TAKD (Mirzadeh et al., 2020a)) and online KD methods (DML (Zhang et al.,
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
fields
cs.LG 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation
Training the teacher a small number of steps ahead of the student and freezing it during distillation improves student generalization by up to 3.4% on image benchmarks.