Partial sharing of input and hidden-state weights in RNN, LSTM, and GRU yields about 50% parameter reduction with roughly unchanged perplexity on two language-modeling benchmarks.
Bayesian Compression for Natural Language Processing
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian sparsification technique for RNNs which allows compressing the RNN dozens or hundreds of times without time-consuming hyperparameters tuning. We also generalize the model for vocabulary sparsification to filter out unnecessary words and compress the RNN even further. We show that the choice of the kept words is interpretable. Code is available on github: https://github.com/tipt0p/SparseBayesianRNN
fields
cs.CL 1years
2019 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Restricted Recurrent Neural Networks
Partial sharing of input and hidden-state weights in RNN, LSTM, and GRU yields about 50% parameter reduction with roughly unchanged perplexity on two language-modeling benchmarks.