RECORD
Follow Your Path: a Progressive Method for Knowledge Distillation
Machine Learning · Submitted 20 Jul 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Experiments on both image and text datasets show that our proposed ProKT consistently achieves superior performance compared to other existing knowledge distillation methods.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Deep neural networks often have a huge number of parameters, which posts challenges in deployment in application scenarios with limited memory and computation capacity. Knowledge distillation is one approach to derive compact models from bigger ones. However, it has been observed that a converged heavy teacher model is strongly constrained for learning a compact student network and could make the optimization subject to poor local optima. In this paper, we propose ProKT, a new model-agnostic method by projecting the supervision signals of a teacher model into the student's parameter space. Such projection is implemented by decomposing the training objective into local intermediate targets with an approximate mirror descent technique. The proposed method could be less sensitive with the quirks during optimization which could result in a better local optimum. Experiments on both image and text datasets show that our proposed ProKT consistently achieves superior performance compared to other existing knowledge distillation methods.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Deep neural networks often have a huge number of parameters, which posts challenges in deployment in application scenarios with limited memory and computation capacity. | background 1.00 | 0.06 | 0.01 | asserted 0.99 |
| s1 | Knowledge distillation is one approach to derive compact models from bigger ones. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s2 | However, it has been observed that a converged heavy teacher model is strongly constrained for learning a compact student network and could make the optimization subject to poor local optima. | background 0.55 | 0.09 | 0.04 | hedged possibility 0.61 |
| s3 | In this paper, we propose ProKT, a new model-agnostic method by projecting the supervision signals of a teacher model into the student's parameter space. | contribution 0.96 | 0.91 | 0.11 | asserted 1.00 |
| s4 | Such projection is implemented by decomposing the training objective into local intermediate targets with an approximate mirror descent technique. | method 1.00 | 0.85 | 0.03 | asserted 1.00 |
| s5 | The proposed method could be less sensitive with the quirks during optimization which could result in a better local optimum. | interpretation 0.59 | 0.45 | 0.11 | hedged possibility 1.00 |
| s6 | Experiments on both image and text datasets show that our proposed ProKT consistently achieves superior performance compared to other existing knowledge distillation methods. | result 1.00 | 0.96 | 0.86 | asserted 0.99 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Deep neural networks often have a huge number of parameters, which posts challenges in deployment in application scenarios with limited memory and computation capacity. Knowledge distillation is one approach to derive compact models from bigger ones. However, it has been observed that a converged heavy teacher model is strongly constrained for learning a compact student network and could make the optimization subject to poor local optima. In this paper, we propose ProKT, a new model-agnostic method by projecting the supervision signals of a teacher model into the student's parameter space. Such projection is implemented by decomposing the training objective into local intermediate targets with an approximate mirror descent technique. The proposed method could be less sensitive with the quirks during optimization which could result in a better local optimum. Experiments on both image and text datasets show that our proposed ProKT consistently achieves superior performance compared to other existing knowledge distillation methods.
Human review
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 22 sentences without a confident label
Uses the method or tool of 4 papers
-
To evaluate the under different distillation loss, we conduct the with standard KL divergence loss and contrastive representation distillation loss proposed by CRD [ref] .
Contrastive Representation Distillation · Image Recognition · 0.92
-
For in TinyBERT, we use the data argumentation following [ref] .
TinyBERT: Distilling BERT for Natural Language Understanding · Experimental details for text classification · 1.00
-
We use the pretrained 6 layer BERT model released by [ref] [https://github.com/google-research/bert], and fine-tune it in the training set.
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models · Text Classification · 0.86
Uses data or values from 2 papers
-
We evaluate our method for sentiment classification on SST-2 [ref] , natural language inference on MNLI [ref] and QNLI [ref] , and paraphrase similarity matching on MRPC [ref] and QQP[ https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs.].
SQuAD: 100,000+ Questions for Machine Comprehension of Text · Text Classification · 0.98
-
We evaluate our method for sentiment classification on SST-2 [ref] , natural language inference on MNLI [ref] and QNLI [ref] , and paraphrase similarity matching on MRPC [ref] and QQP[ https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs.].
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference · Text Classification · 0.98
Compares with 5 papers
-
We compare our method with following baselines: (1) BERT + Finetune, fine-tune the BERT student on training set; (2) BERT/bi-LSTM + KD, fine-tune the BERT student or train the bi-LSTM on training set using the vanilla knowledge distillation loss [ref] ; (3) Route Constrained Optimization (RCO) [ref] , use 4 un-convergent teacher checkpoints as intermediate training targets; (4) bi-LSTM: train bi-LSTM in training set; (5) TinyBERT [ref] : match the attentions and representations of student model with teacher model on the first stage and then fine-tune by the vanilla KD loss on the second stage.
TinyBERT: Distilling BERT for Natural Language Understanding · Text Classification · 0.91
-
We compare our method with following baselines: (1) BERT + Finetune, fine-tune the BERT student on training set; (2) BERT/bi-LSTM + KD, fine-tune the BERT student or train the bi-LSTM on training set using the vanilla knowledge distillation loss [ref] ; (3) Route Constrained Optimization (RCO) [ref] , use 4 un-convergent teacher checkpoints as intermediate training targets; (4) bi-LSTM: train bi-LSTM in training set; (5) TinyBERT [ref] : match the attentions and representations of student model with teacher model on the first stage and then fine-tune by the vanilla KD loss on the second stage.
Distilling the Knowledge in a Neural Network · Text Classification · 0.91
-
CRD [ref] and RCO [ref] .
Contrastive Representation Distillation · Image Recognition · 0.98
Cites as support 2 papers
-
Recently, it has been observed that employing a static target as the distillation objective would leash the effectiveness of the knowledge distillation method [ref] when the capacity gap between student and teacher model is large.
Improved Knowledge Distillation via Teacher Assistant · Introduction · 0.96
-
Besides, the ineligible gap between adjacent targets would make the training process unstable and hurt the convergence property during the optimization [ref] .
AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods · Knowledge Distillation with Dynamic Target · 0.98
Cites as background 6 papers
-
Knowledge distillation [ref] is an essential way in the field which refers to a model-agnostic method where a model with fewer parameters (student) is optimized to minimize some statistical discrepancy between its predictions distribution and the predictions of a higher capacity model (teacher).
Distilling the Knowledge in a Neural Network · Introduction · 0.99
-
Orthogonal to output matching, many works have been conducted on matching the student model and teacher by enforcing the alignment on the latent representation [ref] .
Patient Knowledge Distillation for BERT Model Compression · Related Work · 0.95
-
Orthogonal to output matching, many works have been conducted on matching the student model and teacher by enforcing the alignment on the latent representation [ref] .
TinyBERT: Distilling BERT for Natural Language Understanding · Related Work · 0.95
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.