RECORD
Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks
Machine Learning · Submitted 3 Oct 2024
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Through extensive experiments, we show that our distilled sets lead to up to 13% higher accuracy than prior work, on a variety of downstream tasks, in the presence of limited labeled data.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Dataset distillation (DD) generates small synthetic datasets that can efficiently train deep networks with a limited amount of memory and compute. Despite the success of DD methods for supervised learning, DD for self-supervised pre-training of deep models has remained unaddressed. Pre-training on unlabeled data is crucial for efficiently generalizing to downstream tasks with limited labeled data. In this work, we propose the first effective DD method for SSL pre-training. First, we show, theoretically and empirically, that naive application of supervised DD methods to SSL fails, due to the high variance of the SSL gradient. Then, we address this issue by relying on insights from knowledge distillation (KD) literature. Specifically, we train a small student model to match the representations of a larger teacher model trained with SSL. Then, we generate a small synthetic dataset by matching the training trajectories of the student models. As the KD objective has considerably lower variance than SSL, our approach can generate synthetic datasets that can successfully pre-train high-quality encoders. Through extensive experiments, we show that our distilled sets lead to up to 13% higher accuracy than prior work, on a variety of downstream tasks, in the presence of limited labeled data. Code at https://github.com/BigML-CS-UCLA/MKDT.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Dataset distillation (DD) generates small synthetic datasets that can efficiently train deep networks with a limited amount of memory and compute. | background 1.00 | 0.07 | 0.04 | asserted 1.00 |
| s1 | Despite the success of DD methods for supervised learning, DD for self-supervised pre-training of deep models has remained unaddressed. | gap 1.00 | 0.07 | 0.03 | asserted 1.00 |
| s2 | Pre-training on unlabeled data is crucial for efficiently generalizing to downstream tasks with limited labeled data. | background 1.00 | 0.05 | 0.03 | asserted 1.00 |
| s3 | In this work, we propose the first effective DD method for SSL pre-training. | contribution 1.00 | 0.87 | 0.21 | asserted 1.00 |
| s4 | First, we show, theoretically and empirically, that naive application of supervised DD methods to SSL fails, due to the high variance of the SSL gradient. | result 0.81 | 0.93 | 0.05 | asserted 1.00 |
| s5 | Then, we address this issue by relying on insights from knowledge distillation (KD) literature. | method 0.86 | 0.65 | 0.04 | asserted 0.95 |
| s6 | Specifically, we train a small student model to match the representations of a larger teacher model trained with SSL. | method 0.99 | 0.87 | 0.05 | asserted 0.98 |
| s7 | Then, we generate a small synthetic dataset by matching the training trajectories of the student models. | method 1.00 | 0.94 | 0.05 | asserted 0.97 |
| s8 | As the KD objective has considerably lower variance than SSL, our approach can generate synthetic datasets that can successfully pre-train high-quality encoders. | interpretation 0.56 | 0.89 | 0.34 | asserted 0.92 |
| s9 | Through extensive experiments, we show that our distilled sets lead to up to 13% higher accuracy than prior work, on a variety of downstream tasks, in the presence of limited labeled data. | result 1.00 | 0.96 | 0.72 | asserted 1.00 |
| s10 | Code at https://github.com/BigML-CS-UCLA/MKDT. | contribution 0.53 | 0.31 | 0.02 | unclear 0.85 |
What was examined, and what was not
- arXiv listing abstract v3
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Dataset distillation (DD) generates small synthetic datasets that can efficiently train deep networks with a limited amount of memory and compute. Despite the success of DD methods for supervised learning, DD for self-supervised pre-training of deep models has remained unaddressed. Pre-training on unlabeled data is crucial for efficiently generalizing to downstream tasks with limited labeled data. In this work, we propose the first effective DD method for SSL pre-training. First, we show, theoretically and empirically, that naive application of supervised DD methods to SSL fails, due to the high variance of the SSL gradient. Then, we address this issue by relying on insights from knowledge distillation (KD) literature. Specifically, we train a small student model to match the representations of a larger teacher model trained with SSL. Then, we generate a small synthetic dataset by matching the training trajectories of the student models. As the KD objective has considerably lower variance than SSL, our approach can generate synthetic datasets that can successfully pre-train high-quality encoders. Through extensive experiments, we show that our distilled sets lead to up to 13% higher accuracy than prior work, on a variety of downstream tasks, in the presence of limited labeled data. Code at https://github.com/BigML-CS-UCLA/MKDT.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 3 sentences without a confident label
Uses the method or tool of 1 paper
-
As in previous work [ref] , we use a 3-layer ConvNet for the lower resolution CIFAR datasets and a 4-layer ConvNet for the higher resolution TinyImageNet.
Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory · Experiments · 0.89
Cites as background 3 papers
-
Existing DD methods for SL generate synthetic data by matching gradients [ref] or trajectory [ref] of training on distribution of real data [ref] , or meta-model matching by generating synthetic data such that training on it achieves low loss on real data [ref] .
Dataset Distillation · Introduction · 0.99
-
Existing DD methods for SL generate synthetic data by matching gradients [ref] or trajectory [ref] of training on distribution of real data [ref] , or meta-model matching by generating synthetic data such that training on it achieves low loss on real data [ref] .
Scaling Up Dataset Distillation to ImageNet-1K with Constant Memory · Introduction · 0.99
-
This has been extensively studied for supervised learning [ref] .
An Empirical Study of Example Forgetting during Deep Neural Network Learning · Data-efficient Learning via Subset Selection · 0.95
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.