REVIEW 25 cited by
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
We formally study how ensemble of deep learning models can improve test accuracy, and how the superior performance of ensemble can be distilled into a single model using knowledge distillation. We consider the challenging case where the ensemble is simply an average of the outputs of a few independently trained neural networks with the SAME architecture, trained using the SAME algorithm on the SAME data set, and they only differ by the random seeds used in the initialization. We show that ensemble/knowledge distillation in Deep Learning works very differently from traditional learning theory (such as boosting or NTKs, neural tangent kernels). To properly understand them, we develop a theory showing that when data has a structure we refer to as ``multi-view'', then ensemble of independently trained neural networks can provably improve test accuracy, and such superior test accuracy can also be provably distilled into a single model by training a single model to match the output of the ensemble instead of the true label. Our result sheds light on how ensemble works in deep learning in a way that is completely different from traditional theorems, and how the ``dark knowledge'' is hidden in the outputs of the ensemble and can be used in distillation. In the end, we prove that self-distillation can also be viewed as implicitly combining ensemble and knowledge distillation to improve test accuracy.
Forward citations
Cited by 25 Pith papers
-
Representations Shape Weak-to-Strong Generalization: Theoretical Insights and Empirical Predictions
Weak-to-strong performance is governed by the overlap between the weak model's unlearnable error space and the strong model's principal-representation space, quantified by ||P_s(I-P_w)||.
-
Machine Understanding of Scientific Language
The thesis defines and evaluates tasks and datasets for automatic fact checking, cite-worthiness, exaggeration detection, and information change measurement in science communication, culminating in SPICED, a cross-med...
-
Self-Improvement in Language Models: The Sharpening Mechanism
Self-improvement in language models can be understood as amortizing best-of-N inference-time selection, with minimax-optimal guarantees for SFT and provable coverage-free benefits for RL with exploration.
-
A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
KD sparsifies complex interactions in student LLMs; a Complex Interaction Penalty that enforces this improves multiple KD methods on in- and out-of-domain benchmarks.
-
V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control
V-Simba, a visual RL architecture combining layer normalization, weight decay, and a distributional critic, matches or outperforms complex baselines on 29 continuous control tasks while using less compute.
-
Multi-Turn On-Policy Distillation with Prefix Replay
ReOPD offline-distills multi-turn agentic LLMs via teacher-prefix replay plus step-decay sampling, matching online OPD accuracy at ≥4× speed with zero tool calls.
-
Aggregate Models, Not Explanations: Improving Feature Importance Estimation
For risk-based feature importance (LOCO, SAGE), computing importance on an ensemble of models is more accurate than averaging individual models' importance scores, because model-level ensembling reduces the excess ris...
-
Muon in Associative Memory Learning: Training Dynamics and Scaling Laws
In a linear softmax memory model, Muon equalizes learning across frequency tiers and gives exponential (noiseless) or T^{-2} (noisy power-law) convergence, versus polynomial or T^{-(1-1/β)} for gradient descent.
-
Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Models
CurioSFT preserves exploration during supervised fine-tuning by distilling toward the model's own temperature-scaled distribution and adaptively increasing entropy at high-entropy tokens, improving SFT accuracy by ~2....
-
Dark Side of Modalities: Reinforced Multimodal Distillation for Multimodal Knowledge Graph Reasoning
A unimodal student model, taught by reinforcement-selected combinations of multimodal teachers via neighbor-decoupled knowledge distillation, sets new state-of-the-art results on five multimodal knowledge graph reason...
-
Distillation of atomistic foundation models across architectures and chemical domains
Distillation of atomistic foundation models via synthetic data yields 10x-100x faster student potentials with near-teacher accuracy.
-
Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning
Loss deceleration, a piecewise-linear break in log-log loss curves, is attributed to zero-sum learning where per-example gradients oppose one another, and scaling helps by mitigating it.
-
Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
Fine-tuning CLIP's visual encoder to match DINOv2's kernel-based similarity structure improves its fine-grained visual perception while preserving its alignment to text.
-
Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods
Weight averaging of pretrained and fine-tuned models is shown, in a linear model and on three LLM families, to reduce overadaptation and improve both downstream and retained knowledge.
-
Model Fusion via Neuron Transplantation
A new fusion method, Neuron Transplantation, concatenates ensemble members and prunes back down to a single model's size, outperforming individual models after fine-tuning.
-
Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?
Diffusion models trained with denoising score matching can follow coarse image rules but cannot reliably reproduce fine-grained inter-feature rules, and a two-layer network analysis shows a constant error for such rules.
-
On Local Overfitting and Forgetting in Deep Neural Networks
Networks routinely forget some test points during training even while overall accuracy improves, and combining mid-training checkpoints selected by a forget-score recovers some of this lost accuracy.
-
SPRKD: Effective Knowledge Distillation for Deep Neural Networks via Saddle Region Approximation
SPRKD distills weak-teacher saddle-region information into a small student, but its only tabulated result matches scratch training and larger claimed gains are untabulated.
-
SDD: Self-Degraded Defense against Malicious Fine-tuning
SDD makes an LLM reply to harmful prompts with irrelevant high-quality text, so that malicious fine-tuning degrades general ability and the model cannot follow harmful instructions.
-
Forget Me Not: Fighting Local Overfitting with Knowledge Fusion and Distillation
Checkpoint fusion guided by a 'forget' metric, followed by distillation, lets a single model recover test points that were learned and then forgotten during training, improving accuracy, especially under label noise.
-
Right Time to Learn:Promoting Generalization via Bio-inspired Spacing Effect in Knowledge Distillation
Training the teacher a small number of steps ahead of the student and freezing it during distillation improves student generalization by up to 3.4% on image benchmarks.
-
sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging
A mixture-of-modality-experts transformer with self-distillation reports improved mouse sleep staging and enables single-channel inference after multi-channel training.
-
Efficient Logit-based Knowledge Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
Temporal-wise logits distillation with ensemble self-distillation lets one SNN, trained at T=6, be deployed at T=1 through T=6 without retraining and with competitive accuracy.
-
Multi-Branch Mutual-Distillation Transformer for EEG-Based Seizure Subtype Classification
A multi-branch Transformer with mutual distillation between raw EEG and wavelet bands achieves the best reported accuracy and balanced accuracy on CHSZ and TUSZ seizure subtype classification.
-
Learning Causality for Modern Machine Learning
A thesis compiling six papers that use causal invariance to improve graph neural networks' out-of-distribution generalization, interpretability, and robustness.
Discussion (0). Continue with ORCID to comment.