RECORD
Layer Trajectory LSTM
Computation and Language · Submitted 28 Aug 2018
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Trained with 30 thousand hours of EN-US Microsoft internal data, the proposed ltLSTM performed significantly better than the standard multi-layer LSTM and residual LSTM, with up to 9.0% relative word error rate reduction across different tasks.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
It is popular to stack LSTM layers to get better modeling power, especially when large amount of training data is available. However, an LSTM-RNN with too many vanilla LSTM layers is very hard to train and there still exists the gradient vanishing issue if the network goes too deep. This issue can be partially solved by adding skip connections between layers, such as residual LSTM. In this paper, we propose a layer trajectory LSTM (ltLSTM) which builds a layer-LSTM using all the layer outputs from a standard multi-layer time-LSTM. This layer-LSTM scans the outputs from time-LSTMs, and uses the summarized layer trajectory information for final senone classification. The forward-propagation of time-LSTM and layer-LSTM can be handled in two separate threads in parallel so that the network computation time is the same as the standard time-LSTM. With a layer-LSTM running through layers, a gated path is provided from the output layer to the bottom layer, alleviating the gradient vanishing issue. Trained with 30 thousand hours of EN-US Microsoft internal data, the proposed ltLSTM performed significantly better than the standard multi-layer LSTM and residual LSTM, with up to 9.0% relative word error rate reduction across different tasks.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | It is popular to stack LSTM layers to get better modeling power, especially when large amount of training data is available. | background 1.00 | 0.04 | 0.02 | asserted 0.98 |
| s1 | However, an LSTM-RNN with too many vanilla LSTM layers is very hard to train and there still exists the gradient vanishing issue if the network goes too deep. | background 0.86 | 0.11 | 0.02 | asserted 1.00 |
| s2 | This issue can be partially solved by adding skip connections between layers, such as residual LSTM. | background 0.98 | 0.05 | 0.02 | hedged possibility 0.73 |
| s3 | In this paper, we propose a layer trajectory LSTM (ltLSTM) which builds a layer-LSTM using all the layer outputs from a standard multi-layer time-LSTM. | contribution 0.90 | 0.88 | 0.08 | asserted 1.00 |
| s4 | This layer-LSTM scans the outputs from time-LSTMs, and uses the summarized layer trajectory information for final senone classification. | method 0.96 | 0.79 | 0.06 | asserted 1.00 |
| s5 | The forward-propagation of time-LSTM and layer-LSTM can be handled in two separate threads in parallel so that the network computation time is the same as the standard time-LSTM. | method 0.93 | 0.78 | 0.04 | asserted 0.90 |
| s6 | With a layer-LSTM running through layers, a gated path is provided from the output layer to the bottom layer, alleviating the gradient vanishing issue. | method 0.77 | 0.59 | 0.08 | asserted 0.97 |
| s7 | Trained with 30 thousand hours of EN-US Microsoft internal data, the proposed ltLSTM performed significantly better than the standard multi-layer LSTM and residual LSTM, with up to 9.0% relative word error rate reduction across different tasks. | result 1.00 | 0.97 | 0.87 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
It is popular to stack LSTM layers to get better modeling power, especially when large amount of training data is available. However, an LSTM-RNN with too many vanilla LSTM layers is very hard to train and there still exists the gradient vanishing issue if the network goes too deep. This issue can be partially solved by adding skip connections between layers, such as residual LSTM. In this paper, we propose a layer trajectory LSTM (ltLSTM) which builds a layer-LSTM using all the layer outputs from a standard multi-layer time-LSTM. This layer-LSTM scans the outputs from time-LSTMs, and uses the summarized layer trajectory information for final senone classification. The forward-propagation of time-LSTM and layer-LSTM can be handled in two separate threads in parallel so that the network computation time is the same as the standard time-LSTM. With a layer-LSTM running through layers, a gated path is provided from the output layer to the bottom layer, alleviating the gradient vanishing issue. Trained with 30 thousand hours of EN-US Microsoft internal data, the proposed ltLSTM performed significantly better than the standard multi-layer LSTM and residual LSTM, with up to 9.0% relative word error rate reduction across different tasks.
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
Papers citing it
Uses its method or tool 1 paper
-
The ASR pipeline we used for decoding included a tri-gram language model and an acoustic model consisting of six layers of 512-element layer trajectory LSTM [ref] .
An End-to-end Architecture of Online Multi-channel Speech Separation · Evaluation scheme · 1.00
What it cites · 4 sentences without a confident label
Cites as background 2 papers
-
Residual LSTM [ref] uses shortcut connections between LSTM layers, and hence provides a way to alleviate the gradient vanishing problem.
Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition · Introduction · 0.82
-
Grid LSTM [ref] is a more general LSTM which arranges the LSTM memory cells into a multidimensional grid along both time and layer axis.
Grid Long Short-Term Memory · Introduction · 0.81
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.