REVIEW 4 cited by
Revisiting Self-Distillation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Knowledge distillation is the procedure of transferring "knowledge" from a large model (the teacher) to a more compact one (the student), often being used in the context of model compression. When both models have the same architecture, this procedure is called self-distillation. Several works have anecdotally shown that a self-distilled student can outperform the teacher on held-out data. In this work, we systematically study self-distillation in a number of settings. We first show that even with a highly accurate teacher, self-distillation allows a student to surpass the teacher in all cases. Secondly, we revisit existing theoretical explanations of (self) distillation and identify contradicting examples, revealing possible drawbacks of these explanations. Finally, we provide an alternative explanation for the dynamics of self-distillation through the lens of loss landscape geometry. We conduct extensive experiments to show that self-distillation leads to flatter minima, thereby resulting in better generalization.
Forward citations
Cited by 4 Pith papers
-
Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models
CurioSFT preserves exploration during supervised fine-tuning by distilling toward the model's own temperature-scaled distribution and adaptively increasing entropy at high-entropy tokens, improving SFT accuracy by ~2....
-
Mitigating Context Bias in Domain Adaptation for Object Detection using Mask Pooling
Mask Pooling, which pools foreground and background separately using ground-truth masks, improves object-detection robustness on domain-shift benchmarks but requires masks at inference and lacks a rigorous causal derivation.
-
Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation
Checkpoint fusion guided by a 'forget' metric, followed by distillation, lets a single model recover test points that were learned and then forgotten during training, improving accuracy, especially under label noise.
-
Smooth-Distill: A Self-distillation Framework for Multitask Learning with Wearable Sensor Data
Smooth-Distill applies EMA parameter averaging as a self-distillation teacher for multitask HAR and placement detection, and reports consistent but modest gains over multitask baselines.
Discussion (0). Sign in to comment.