Pith. sign in

REVIEW 3 major objections 5 minor 27 references

Restricted Recurrent Neural Networks

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By sharing rows between the input and hidden-state weight matrices, recurrent networks can halve their parameter count while matching or improving language-modeling perplexity.

desk verdict A simple, useful LSTM parameter-sharing trick with an overstated compression formula and sloppy empirical reporting; worth reviewing. read the letter →

arxiv 1908.07724 v4 pith:YALIOWWM submitted 2019-08-21 cs.CL cs.LG

Enmao Diao , Jie Ding , Vahid Tarokh This is my paper · ORCID
classification cs.CLcs.LG
keywords RecurrentNeuralNetworksLongShort-TermMemoryGatedUnitModelCompressionParameterSharingLanguageModelingPerplexityRestrictedNetwork
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a structural compression for recurrent networks: instead of learning separate weight matrices for the current input and the previous hidden state, it makes the two matrices share a large block of rows and keeps only a small state-specific block for each. Across RNN, LSTM, and GRU, the compressed models reach roughly 50% parameter reduction at a sharing rate $r=0.5$ while producing language-modeling perplexity close to—and in several configurations better than—the classical versions. Unlike pruning or low-rank factorization, the method needs no pre-trained model and gives an explicit handle on the compression rate. The authors' central claim is that partial sharing exploits dependencies between input and hidden state, with neither full sharing nor no sharing being optimal.

What carries the argument

The construction is a row-indexing scheme over a shared parameter pool $W$ (and bias pool $b$). For each source—current input or hidden state—the restricted matrix $W^r$ selects the first $s$ rows from $W$ as a shared block and then a private block of $q$ rows at offset positions, yielding $W^r_{xh}$ and $W^r_{hh}$ whose sizes exactly match the classical matrices. The number of shared rows $s$ can differ per source; the paper sets $s = \mathrm{round}(r d)$ with $d$ the output dimension and defines the overall compression rate by counting only the parameters actually indexed. Applied gate-by-gate, the same pool mechanism compresses LSTM and GRU, and it generalizes to any recurrent model with $m$ inputs and $n$ outputs by a shared-row allocation over the input/output pairs.

What would settle it

Train an RNN on a sequence task with input dimension not equal to hidden dimension using the paper's shared-row construction literally; the mismatch forces either projection layers or a change in sharing, and measuring the actual parameter count and perplexity in that common setting would show whether the claimed compression trade-off holds there.

Watch

Extended reading notes

Core claim

The central claim is that the input-to-hidden and hidden-to-hidden weight matrices in an RNN need not be separate: they can be assembled from a common pool of rows, with a shared block plus private blocks for each input source. For equal input and hidden dimensions and sharing rate $r$, the trainable parameter count drops by the factor $C=(2-r)/2$ for a vanilla RNN and approximately $C=1-r$ for LSTM and GRU, so $r=0.5$ gives about 50% compression. On Penn Treebank and WikiText-2, the restricted LSTM with tied embeddings and dropout reaches test perplexities such as 103.5 on PTB with only 0.553M recurrent parameters, and the restricted LSTM at high sharing rates beats the unrestricted vanilla RNN while using fewer parameters. The paper also reports a phase transition in the sharing rate: most of the benefit appears once $r\ge 0.5$, supporting the view that the shared block captures common structure while the private rows absorb input-specific structure.

Load-bearing premise

The construction requires the current input and the previous hidden state to have the same number of entries; if those sizes differ, the shared rows cannot apply to both without adding extra resizing layers that change the parameter count.

Editorial extensions

If this is right

  • At $r=0.5$, an RNN, LSTM, or GRU keeps about half the recurrent parameters of its classical version, with the exact factor depending on the architecture.
  • Because the construction only rearranges weight matrices, a classical pre-trained model is not needed; restricted models train from scratch with standard SGD.
  • The restricted LSTM can be smaller than a vanilla RNN and still achieve lower perplexity, suggesting gating capacity is worth more than raw parameter count.
  • The mechanism is compatible with existing regularizers such as dropout and tied input/output embeddings.
  • The paper recommends $r=0.5$ as a default sharing rate, with $r$ tunable to a target compression-performance trade-off.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's compression formulas assume the input and hidden dimensions coincide; a natural extension is an unequal-dimension variant with projection layers, where the saving per shared row would be smaller and the break-even sharing rate would shift.
  • Because the restriction is structural and defined before training, it could be combined with post-training quantization or pruning for a multiplicative reduction in footprint, although the paper does not test that combination.
  • The observed phase transition around $r=0.5$ suggests the shared rows act as a regularizer; an untested prediction is that the benefit becomes larger on small datasets or tasks with limited training data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Restricted Recurrent Neural Networks (RRNN), a structural compression method that forces the input-to-hidden and hidden-to-hidden weight matrices in RNN, LSTM, and GRU to share a fraction of their rows, controlled by a sharing rate r. The authors derive compression-rate formulas, report parameter counts and test/validation perplexities on Penn Treebank and WikiText-2, and conclude that the restricted models reduce parameters while maintaining comparable or better language-modeling performance. Code is publicly released.

Significance. If the claims hold, the method is attractive for its simplicity: it requires no pre-training, no pruning, and no low-rank factorization, and it offers an explicit control of the parameter count while remaining compatible with standard regularizers such as dropout and tied embeddings. The experiments are standard and the released code supports reproducibility. However, the paper currently contains an algebraic error in the generalized compression-rate formula and overstates the empirical support: the 'comparable or better' claim is clearly supported only for LSTM, not for vanilla RNN or GRU. The equal-dimension assumption also limits the method's generality more than the abstract suggests. These issues are fixable, but they affect the paper's headline quantitative claims.

major comments (3)
  1. [Section III B] The simplification C = (mnd - (mn-1)s)/(mnd) ≈ 1 - s/d = 1 - r is algebraically incorrect. Since s = r d, the exact expression is C = 1 - (mn-1)r/mn. For the actual models, mn = 2 (RNN), 8 (LSTM), and 6 (GRU), so at r = 0.5 the correct compression rates are 0.75, 0.5625, and 0.5833, respectively, not 0.5. Table II itself confirms this: RLSTM at r=0.5 has 0.553/0.975 ≈ 0.567 and RGRU has 0.432/0.733 ≈ 0.589. The statement '≈ 1-r' and the abstract's 'about 50% compression rate' must be corrected, and the discrepancy with the exact RNN formula (2-r)/2 in Section III A should be reconciled.
  2. [Section IV B, Tables III and IV] The abstract's claim that the restricted architectures 'generally produce comparable results at about 50% compression rate' is not supported by the reported numbers. On PTB without tied embeddings or dropout, RRNN at r=0.5 has test perplexity 176.5 versus 154.8 for the classical RNN (r=0), and RGRU has 148.4 versus 144.9; the same ordering holds in the tied/dropout rows (RRNN 226.6 vs 221.7; RGRU 163.6 vs 156.0). Only RLSTM shows comparable or better perplexity at r=0.5 (129 vs 124.6, and 103.5 vs 107.7 with tied/dropout). The paper should either restrict the performance claim to LSTM or present the trade-off more accurately for RNN and GRU.
  3. [Section III A] The shared-row construction assumes equal input and hidden dimensions (k := kxh = khh) and, in the simplified formulas, equal sharing rates and no rounding. If the embedding size differs from the hidden size, the shared rows cannot be applied to both matrices without additional projection layers, which would change the parameter count. The paper does not state this limitation in the abstract or conclusion, although the method is presented as a general RNN compression methodology. Please make this assumption explicit and, ideally, provide a variant or analysis for the unequal-dimension case.
minor comments (5)
  1. [Section III A] In the sentence beginning 'sih and shh represent the output channel size of shared parameters', the symbols appear to be a typo for sxh and shh.
  2. [Section III and Tables II-IV] The sharing-rate notation alternates between s (in the derivation), r (in the tables), and 'sharing rate s' in the text; please unify the notation and define the relationship s = r d consistently.
  3. [Section IV B] The phrase 'phase transition' is used to describe the perplexity-versus-sharing-rate behavior, but it is never formally defined; consider replacing it with a precise description of the observed improvement pattern.
  4. [Section IV B] The recommendation 'we suggest to set r = 0.5 as the default option' is presented after inspecting the test-set results; this is a heuristic and should be labeled as such, ideally with a validation-based selection criterion.
  5. [Section II A] The sentence 'RNN, GRU and LSTM have 2x, 6x, and 8x number of parameters compared with fully connected neural networks' should clarify the baseline 'fully connected' network it refers to, since the multiplier depends on the hidden dimension and architecture.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the compression rates are counting definitions and the perplexity gains are measured, not derived from the method's own outputs.

full rationale

The paper's central claims are (i) a parameter-sharing construction that reduces parameter counts by a computable amount, and (ii) empirical language-modeling perplexities showing comparable or better performance at reduced size. Claim (i) is a direct counting argument from the definitions in Sections III A and III B: the quantities P, Sr, Pr, and C are defined in terms of the constructed weight pools, so the compression rate is a consequence of the construction rather than an independent prediction fitted to data. Claim (ii) is supported by measured test perplexities in Tables III and IV, which are external evaluation results, not quantities re-derived from the construction. The suggested default r=0.5 is explicitly presented as a post-experimental recommendation ('From various experimental studies, we suggest to set r = 0.5 as the default option'), not as a prediction derived from the construction, so it does not create circularity. The paper contains no load-bearing self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation. A separate non-circular concern is that the simplification in Section III B, 'mnd−(mn−1)s / mnd ≈ 1−s/d', is algebraically inaccurate for finite m,n because the retained factor is (mn−1)/mn; this affects the advertised 50% rate for LSTM/GRU but does not make the derivation circular.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The compression is a reparametrization of existing RNN weight matrices rather than a new modeling entity; the only free knob is the sharing rate r, which is a user-controlled hyperparameter rather than a fitted constant. The two domain assumptions above are load-bearing: they allow the shared rows to be concatenated in the parameter pool.

free parameters (1)
  • sharing rate r = default 0.5; tested at 0, 0.1, 0.3, 0.5, 0.7, 0.9, 0.95, 1
    User-selected hyperparameter controlling the number of shared rows in the parameter pool. The paper recommends r=0.5 as a default based on post hoc performance on the two test sets, and the abstract's approximately 50% compression claim depends on r approximately 0.5 for LSTM and GRU.
assumptions (3)
  • domain assumption Input and hidden dimensions are equal: kxh = khh and dxh = dhh.
    Invoked in Section III A ('Assuming the common practice that k := kxh = khh') and required for the shared row-blocks to be applied to both inputs and for the stated compression-rate formulas to hold.
  • ad hoc to paper Input and hidden state distributions are dependent enough that a shared weight block is useful.
    Motivating heuristic in Section III A: 'By assuming that the inputs are not totally independent with each other, one could imagine shared parameters are able to capture the similarities.' No formal derivation is given.
  • standard math The standard RNN, LSTM, and GRU update equations are the correct model class.
    The paper builds directly on the formulations of Hochreiter-Schmidhuber and Cho et al. from Section II A and does not question their validity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Restricted Recurrent Neural Networks." pith.science (2026). https://pith.science/paper/YALIOWWM

@misc{pith2026190807724,
  author       = {Pith},
  title        = {Pith review of: Restricted Recurrent Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YALIOWWM}},
  note         = {Machine review of arXiv:1908.07724}
}
read the original abstract

Recurrent Neural Network (RNN) and its variations such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU), have become standard building blocks for learning online data of sequential nature in many research areas, including natural language processing and speech data analysis. In this paper, we present a new methodology to significantly reduce the number of parameters in RNNs while maintaining performance that is comparable or even better than classical RNNs. The new proposal, referred to as Restricted Recurrent Neural Network (RRNN), restricts the weight matrices corresponding to the input data and hidden states at each time step to share a large proportion of parameters. The new architecture can be regarded as a compression of its classical counterpart, but it does not require pre-training or sophisticated parameter fine-tuning, both of which are major issues in most existing compression techniques. Experiments on natural language modeling show that compared with its classical counterpart, the restricted recurrent architecture generally produces comparable results at about 50\% compression rate. In particular, the Restricted LSTM can outperform classical RNN with even less number of parameters.

Figures

Figures reproduced from arXiv: 1908.07724 by the authors.

Figure 1
Figure 1. Illustration of parameter restriction in RRNN. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Perplexity vs. Number of RNN parameters for Penn Treebank dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Perplexity vs. Number of RNN parameters for WikiText2 dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 13 canonical work pages

  1. [1]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  2. [2]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation,

    K. Cho, B. Van Merri ¨enboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y . Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” arXiv preprint arXiv:1406.1078, 2014

    arXiv 2014

  3. [3]

    Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,

    S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149 , 2015

    arXiv 2015

  4. [4]

    Learning structured sparsity in deep neural networks,

    W. Wen, C. Wu, Y . Wang, Y . Chen, and H. Li, “Learning structured sparsity in deep neural networks,” in Advances in neural information processing systems (NIPS) , 2016, pp. 2074–2082

    work page 2016

  5. [5]

    A survey of model compression and acceleration for deep neural networks,

    Y . Cheng, D. Wang, P. Zhou, and T. Zhang, “A survey of model compression and acceleration for deep neural networks,” arXiv preprint arXiv:1710.09282, 2017

    arXiv 2017

  6. [6]

    On the compression of recurrent neural networks with an application to lvcsr acoustic modeling for embedded speech recognition,

    R. Prabhavalkar, O. Alsharif, A. Bruguier, and L. McGraw, “On the compression of recurrent neural networks with an application to lvcsr acoustic modeling for embedded speech recognition,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2016, pp. 5970–5974

    work page 2016

  7. [7]

    Learning compact recurrent neural networks,

    Z. Lu, V . Sindhwani, and T. N. Sainath, “Learning compact recurrent neural networks,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2016, pp. 5960–5964

    work page 2016

  8. [8]

    Compressing recurrent neural network with tensor train,

    A. Tjandra, S. Sakti, and S. Nakamura, “Compressing recurrent neural network with tensor train,” in 2017 International Joint Conference on Neural Networks (IJCNN) . IEEE, 2017, pp. 4451–4458

    work page 2017

Show all 27 references
  1. [9]

    Compression of recurrent neural networks for efficient language modeling,

    A. M. Grachev, D. I. Ignatov, and A. V . Savchenko, “Compression of recurrent neural networks for efficient language modeling,” Applied Soft Computing, vol. 79, pp. 354–362, 2019

    2019

  2. [10]

    Dropout: a simple way to prevent neural networks from over- fitting,

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut- dinov, “Dropout: a simple way to prevent neural networks from over- fitting,” The journal of machine learning research , vol. 15, no. 1, pp. 1929–1958, 2014

    1929

  3. [11]

    Mobilenets: Efficient convo- lutional neural networks for mobile vision applications,

    A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convo- lutional neural networks for mobile vision applications,” arXiv preprint arXiv:1704.04861, 2017

    2017 arXiv

  4. [12]

    Shufflenet: An extremely effi- cient convolutional neural network for mobile devices,

    X. Zhang, X. Zhou, M. Lin, and J. Sun, “Shufflenet: An extremely effi- cient convolutional neural network for mobile devices,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 6848–6856

    2018

  5. [13]

    Con- densenet: An efficient densenet using learned group convolutions,

    G. Huang, S. Liu, L. Van der Maaten, and K. Q. Weinberger, “Con- densenet: An efficient densenet using learned group convolutions,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018, pp. 2752–2761

    2018

  6. [14]

    Long short-term memory recurrent neural network architectures for large scale acoustic modeling,

    H. Sak, A. Senior, and F. Beaufays, “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,” in Fifteenth annual conference of the international speech communication association, 2014

    2014

  7. [15]

    Recurrent neural network regularization,

    W. Zaremba, I. Sutskever, and O. Vinyals, “Recurrent neural network regularization,” arXiv preprint arXiv:1409.2329 , 2014

    2014 arXiv

  8. [16]

    An empirical exploration of recurrent network architectures,

    R. Jozefowicz, W. Zaremba, and I. Sutskever, “An empirical exploration of recurrent network architectures,” in International Conference on Machine Learning (ICML) , 2015, pp. 2342–2350

    2015

  9. [17]

    Improved semantic represen- tations from tree-structured long short-term memory networks,

    K. S. Tai, R. Socher, and C. D. Manning, “Improved semantic represen- tations from tree-structured long short-term memory networks,” arXiv preprint arXiv:1503.00075, 2015

    2015 arXiv

  10. [18]

    Lstm: A search space odyssey,

    K. Greff, R. K. Srivastava, J. Koutn ´ık, B. R. Steunebrink, and J. Schmid- huber, “Lstm: A search space odyssey,” IEEE transactions on neural networks and learning systems , vol. 28, no. 10, pp. 2222–2232, 2016

    2016

  11. [19]

    Performance of three slim variants of the long short-term memory (lstm) layer,

    D. Kent and F. M. Salem, “Performance of three slim variants of the long short-term memory (lstm) layer,” arXiv preprint arXiv:1901.00525, 2019

    1901 arXiv

  12. [20]

    Bayesian compression for natural language processing,

    N. Chirkova, E. Lobacheva, and D. Vetrov, “Bayesian compression for natural language processing,” arXiv preprint arXiv:1810.10927 , 2018

    2018 arXiv

  13. [21]

    Context dependent recurrent neural network language model,

    T. Mikolov and G. Zweig, “Context dependent recurrent neural network language model,” in 2012 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2012, pp. 234–239

    2012

  14. [22]

    Regularizing and optimizing LSTM language models,

    S. Merity, N. S. Keskar, and R. Socher, “Regularizing and optimizing LSTM language models,” arXiv preprint arXiv:1708.02182 , 2017

    2017 arXiv

  15. [23]

    Recurrent neural network based language model,

    T. Mikolov, M. Karafi ´at, L. Burget, J. ˇCernock`y, and S. Khudanpur, “Recurrent neural network based language model,” in Eleventh annual conference of the international speech communication association , 2010

    2010

  16. [24]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843 , 2016

    2016 arXiv

  17. [25]

    Sgdr: Stochastic gradient descent with warm restarts,

    I. Loshchilov and F. Hutter, “Sgdr: Stochastic gradient descent with warm restarts,” arXiv preprint arXiv:1608.03983 , 2016

    2016 arXiv

  18. [26]

    Using the output embedding to improve language models,

    O. Press and L. Wolf, “Using the output embedding to improve language models,” arXiv preprint arXiv:1608.05859 , 2016

    2016 arXiv

  19. [27]

    Tying word vectors and word classifiers: A loss framework for language modeling,

    H. Inan, K. Khosravi, and R. Socher, “Tying word vectors and word classifiers: A loss framework for language modeling,” arXiv preprint arXiv:1611.01462, 2016

    2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.