A gradually shrinking teacher signal during knowledge distillation improves 2-bit quantized ResNet20 accuracy on CIFAR-10 and CIFAR-100 compared with fixed-coefficient distillation.
Knowledge distillation for optimization of quantized deep neural networks
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Knowledge distillation (KD) is a very popular method for model size reduction. Recently, the technique is exploited for quantized deep neural networks (QDNNs) training as a way to restore the performance sacrificed by word-length reduction. KD, however, employs additional hyper-parameters, such as temperature, coefficient, and the size of teacher network for QDNN training. We analyze the effect of these hyper-parameters for QDNN optimization with KD. We find that these hyper-parameters are inter-related, and also introduce a simple and effective technique that reduces \textit{coefficient} during training. With KD employing the proposed hyper-parameters, we achieve the test accuracy of 92.7% and 67.0% on Resnet20 with 2-bit ternary weights for CIFAR-10 and CIFAR-100 data sets, respectively.
citation-role summary
method 1
citation-polarity summary
fields
cs.LG 1years
2019 1verdicts
CONDITIONAL 1roles
method 1polarities
use method 1representative citing papers
citing papers explorer
-
Knowledge distillation for optimization of quantized deep neural networks
A gradually shrinking teacher signal during knowledge distillation improves 2-bit quantized ResNet20 accuracy on CIFAR-10 and CIFAR-100 compared with fixed-coefficient distillation.