REVIEW 11 cited by
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We show that ensemble/knowledge distillation in Deep Learning works very differently from traditional learning theory (such as boosting or NTKs, neural tangent kernels). To properly understand them, we develop a theory showing that when data has a structure we refer to as ``multi-view'', then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the ``dark knowledge'' is hidden in the outputs of the ensemble and can be used in distillation. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.
Forward citations
Cited by 11 Pith papers
-
Machine Understanding of Scientific Language
The thesis defines and evaluates tasks and datasets for automatic fact checking, cite-worthiness, exaggeration detection, and information change measurement in science communication, culminating in SPICED, a cross-med...
-
A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
KD sparsifies complex interactions in student LLMs; a Complex Interaction Penalty that enforces this improves multiple KD methods on in- and out-of-domain benchmarks.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Aggregate Models, Not Explanations: Improving Feature Importance Estimation
For risk-based feature importance (LOCO, SAGE), computing importance on an ensemble of models is more accurate than averaging individual models' importance scores, because model-level ensembling reduces the excess ris...
-
Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
In a linear softmax memory model, Muon equalizes learning across frequency tiers and gives exponential (noiseless) or T^{-2} (noisy power-law) convergence, versus polynomial or T^{-(1-1/β)} for gradient descent.
-
Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models
CurioSFT preserves exploration during supervised fine-tuning by distilling toward the model's own temperature-scaled distribution and adaptively increasing entropy at high-entropy tokens, improving SFT accuracy by ~2....
-
Dark Side of Modalities: Reinforced Multimodal Distillation for Multimodal Knowledge Graph Reasoning
A unimodal student model, taught by reinforcement-selected combinations of multimodal teachers via neighbor-decoupled knowledge distillation, sets new state-of-the-art results on five multimodal knowledge graph reason...
-
Distillation of atomistic foundation models across architectures and chemical domains
Distillation of atomistic foundation models via synthetic data yields 10x-100x faster student potentials with near-teacher accuracy.
-
SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation
SPRKD distills weak-teacher saddle-region information into a small student, but its only tabulated result matches scratch training and larger claimed gains are untabulated.
-
Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation
Checkpoint fusion guided by a 'forget' metric, followed by distillation, lets a single model recover test points that were learned and then forgotten during training, improving accuracy, especially under label noise.
-
Learning Causality for Modern Machine Learning
A thesis compiling six papers that use causal invariance to improve graph neural networks' out-of-distribution generalization, interpretability, and robustness.
Discussion (0). Sign in to comment.