{"id":"0d07da1a-417c-4a01-a4df-0ce1c0d9eec0","arxiv_id":"1908.07724","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Partial sharing of input and hidden-state weights in RNN, LSTM, and GRU yields about 50% parameter reduction with roughly unchanged perplexity on two language-modeling benchmarks.","lead":"This paper proposes a new architecture, Restricted Recurrent Neural Networks (RRNN), that makes the input and hidden-state weight matrices share a common block of rows, cutting parameter counts by up to half without any pre-training or fine-tuning. Language-modeling tests on Penn Treebank and WikiText-2 show the compressed LSTM can match or beat a standard RNN at similar or smaller size.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Compression-rate simplification C≈1−r in §III B is algebraically incorrect; the correct formula is C=1−(mn−1)r/(mn), which overstates LSTM/GRU compression and contradicts the paper's own Table II and §III A.","rationale":"The reader's weakest assumption concerned unequal embedding and hidden dimensions, but the construction in Section III A/B already handles unequal column counts via kr = max(kxh, khh) and slicing; the shared-row scheme does not require identical input and hidden dimensions, only the same output dimension, which is automatic. The reader's concern is therefore not the most load-bearing issue. The real soft spot is the algebraic error in the generalized compression-rate simplification. Section III B states C ≈ 1−r, but the exact formula immediately above it, C = (mnd − (mn−1)s)/(mnd), yields C = 1 − (mn−1)r/(mn) when s = r·d. For LSTM (mn=8) and GRU (mn=6), this is materially different from 1−r. The paper's own Table II matches the corrected formula, not the simplified one, so the advertised 'about 50% compression' is an overstatement. This is a concrete, internal inconsistency in the central quantitative claim. The empirical results (RLSTM at r=0.5 achieving 103.5 perplexity on PTB) may still stand, and the method is plausible, but the paper needs to correct the formula and adjust the abstract/conclusion claims. Hence CONDITIONAL is appropriate, consistent with the reader's overall verdict, but for a different reason than the one they emphasized.","tokens_in":10170,"tokens_out":10094,"duration_ms":85810,"concrete_test":"Recompute the parameter counts in Table II from the construction in Section III B for LSTM and GRU at r=0.5 using C = 1 − (mn−1)r/(mn), and compare with the paper's reported values and with the claimed 1−r. For RLSTM, expected ≈0.975M×0.5625 = 0.548M (reported 0.553M); for RGRU, expected ≈0.733M×0.5833 = 0.428M (reported 0.432M). If the reported values match the corrected formula, then the simplified claim C≈1−r is unsupported, and the abstract's 'about 50% compression' should be revised to 'about 44% parameter reduction for LSTM and 42% for GRU at r=0.5'.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central quantitative claim is the controllable compression rate. In Section III B, the paper derives C = (mnd − (mn−1)s)/(mnd) and then simplifies to '≈ 1−s/d = 1−r'. This is not a valid approximation: substituting s = r·d gives C = 1 − (mn−1)r/(mn), which equals 1−r only in the limit mn→∞. For the actual models, RNN has mn=2, LSTM mn=8, GRU mn=6. At r=0.5, the correct rates are: RNN C=0.75 (matches §III A's (2−r)/2), LSTM C=0.5625, GRU C=0.5833. Table II confirms the correct formula: RLSTM r=0.5 has 0.553M/0.975M = 0.567, and RGRU r=0.5 has 0.432M/0.733M = 0.589. The paper's simplified formula predicts 0.5, which is materially different, especially for LSTM/GRU. This is an internal inconsistency, not a matter of external consensus: Section III A correctly derives (2−r)/2 for RNN, but the generalized section then drops the (mn−1)/mn factor. Because the abstract advertises 'about 50% compression rate' and the reader's strongest claim repeats it, the error directly affects the paper's headline contribution. The method still compresses, but the advertised rate is overstated by 12.5% (LSTM) and 16.7% (GRU) in relative terms at r=0.5.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Restricted Recurrent Neural Networks (RRNN), a structural compression method that forces the input-to-hidden and hidden-to-hidden weight matrices in RNN, LSTM, and GRU to share a fraction of their rows, controlled by a sharing rate r. The authors derive compression-rate formulas, report parameter counts and test/validation perplexities on Penn Treebank and WikiText-2, and conclude that the restricted models reduce parameters while maintaining comparable or better language-modeling performance. Code is publicly released.","tokens_in":10500,"tokens_out":8150,"duration_ms":153730,"significance":"If the claims hold, the method is attractive for its simplicity: it requires no pre-training, no pruning, and no low-rank factorization, and it offers an explicit control of the parameter count while remaining compatible with standard regularizers such as dropout and tied embeddings. The experiments are standard and the released code supports reproducibility. However, the paper currently contains an algebraic error in the generalized compression-rate formula and overstates the empirical support: the 'comparable or better' claim is clearly supported only for LSTM, not for vanilla RNN or GRU. The equal-dimension assumption also limits the method's generality more than the abstract suggests. These issues are fixable, but they affect the paper's headline quantitative claims.","major_comments":[{"comment":"The simplification C = (mnd - (mn-1)s)/(mnd) ≈ 1 - s/d = 1 - r is algebraically incorrect. Since s = r d, the exact expression is C = 1 - (mn-1)r/mn. For the actual models, mn = 2 (RNN), 8 (LSTM), and 6 (GRU), so at r = 0.5 the correct compression rates are 0.75, 0.5625, and 0.5833, respectively, not 0.5. Table II itself confirms this: RLSTM at r=0.5 has 0.553/0.975 ≈ 0.567 and RGRU has 0.432/0.733 ≈ 0.589. The statement '≈ 1-r' and the abstract's 'about 50% compression rate' must be corrected, and the discrepancy with the exact RNN formula (2-r)/2 in Section III A should be reconciled.","section":"Section III B"},{"comment":"The abstract's claim that the restricted architectures 'generally produce comparable results at about 50% compression rate' is not supported by the reported numbers. On PTB without tied embeddings or dropout, RRNN at r=0.5 has test perplexity 176.5 versus 154.8 for the classical RNN (r=0), and RGRU has 148.4 versus 144.9; the same ordering holds in the tied/dropout rows (RRNN 226.6 vs 221.7; RGRU 163.6 vs 156.0). Only RLSTM shows comparable or better perplexity at r=0.5 (129 vs 124.6, and 103.5 vs 107.7 with tied/dropout). The paper should either restrict the performance claim to LSTM or present the trade-off more accurately for RNN and GRU.","section":"Section IV B, Tables III and IV"},{"comment":"The shared-row construction assumes equal input and hidden dimensions (k := kxh = khh) and, in the simplified formulas, equal sharing rates and no rounding. If the embedding size differs from the hidden size, the shared rows cannot be applied to both matrices without additional projection layers, which would change the parameter count. The paper does not state this limitation in the abstract or conclusion, although the method is presented as a general RNN compression methodology. Please make this assumption explicit and, ideally, provide a variant or analysis for the unequal-dimension case.","section":"Section III A"}],"minor_comments":[{"comment":"In the sentence beginning 'sih and shh represent the output channel size of shared parameters', the symbols appear to be a typo for sxh and shh.","section":"Section III A"},{"comment":"The sharing-rate notation alternates between s (in the derivation), r (in the tables), and 'sharing rate s' in the text; please unify the notation and define the relationship s = r d consistently.","section":"Section III and Tables II-IV"},{"comment":"The phrase 'phase transition' is used to describe the perplexity-versus-sharing-rate behavior, but it is never formally defined; consider replacing it with a precise description of the observed improvement pattern.","section":"Section IV B"},{"comment":"The recommendation 'we suggest to set r = 0.5 as the default option' is presented after inspecting the test-set results; this is a heuristic and should be labeled as such, ideally with a validation-based selection criterion.","section":"Section IV B"},{"comment":"The sentence 'RNN, GRU and LSTM have 2x, 6x, and 8x number of parameters compared with fully connected neural networks' should clarify the baseline 'fully connected' network it refers to, since the multiplier depends on the hidden dimension and architecture.","section":"Section II A"}],"recommendation":"major_revision","confidential_remarks":"The paper is written in a compact conference-style format, and the empirical study is limited to one small configuration (200 hidden units, three layers). The core idea is simple and potentially useful, and the released code is a plus. The algebraic error in Section III B and the overgeneralized performance claim are the main obstacles; both are fixable with a focused revision. If the authors correct the formulas, adjust the empirical claims, and state the equal-dimension limitation clearly, the paper could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis one is a genuinely simple compression trick: let the input and hidden weight matrices for each RNN gate share a row block, with a tunable sharing rate r. The construction is easy to implement, needs no pretraining, and on PTB a tied-embedding LSTM with r=0.5 gets 103.5 test perplexity at about 0.55M RNN parameters, which beats the reported baselines with fewer parameters. That result alone makes the paper worth reading.\n\nWhat's new: the partial row-sharing scheme with explicit sharing rate, applied uniformly to RNN/LSTM/GRU. It is not in the cited compression literature. The compression-rate formula for vanilla RNN, C=(2-r)/2, is correct, and Table II confirms it.\n\nThe soft spots, in rough order:\n\n1. The generalized compression-rate simplification in Section III B is algebraically wrong. They write C = (mnd - (mn-1)s)/(mnd) ≈ 1 - r, but substituting s=rd gives C = 1 - (mn-1)r/(mn). For LSTM (mn=8) at r=0.5 the correct rate is 0.5625, not 0.5; for GRU it's 0.5833. Table II matches the correct formula. The abstract's 'about 50% compression' is therefore overstated for LSTM and GRU. This is an internal inconsistency, not a fundamental flaw.\n\n2. The text says 'No parameters sharing apparently produces inferior result,' but in Tables III and IV the r=0 (no sharing) models are often the best or tied for best for RNN and GRU, and within noise of the best for LSTM. The sentence is simply contradicted by their own tables.\n\n3. All results are single runs with no error bars. The 'phase transition' narrative is asserted from these curves; with no variance estimate it is hard to know how real it is.\n\n4. The paper says it could not find WT2 baselines, but the paper it cites for WT2 (Merity et al.) gives standard baselines. That is a sloppy omission, and it weakens the comparison.\n\n5. The default recommendation r=0.5 is chosen after seeing the test numbers. That is post hoc selection, disclosed, but still post hoc.\n\nIs the method real? Yes, for LSTM. For vanilla RNN and GRU, the restricted versions are consistently worse than the unrestricted ones, so the 'comparable or better' claim in the abstract does not survive across architectures. The paper is most honest when read as an LSTM compression method.\n\nWho should read it: people working on model compression for language models, especially on-device LSTM deployment. It deserves a serious referee; a competent one will ask for the corrected formula, error bars or at least multiple seeds, and a proper WT2 baseline. Neither the algebra error nor the missing baselines kills the central idea.\n\nMy call: send it to peer review. It's a useful empirical contribution with a fixable flaw.","headline":"A simple, useful LSTM parameter-sharing trick with an overstated compression formula and sloppy empirical reporting; worth reviewing.","tokens_in":11041,"tokens_out":3556,"would_cite":false,"duration_ms":31983,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By sharing rows between the input and hidden-state weight matrices, recurrent networks can halve their parameter count while matching or improving language-modeling perplexity.","keywords":["Recurrent Neural Networks","Long Short-Term Memory","Gated Recurrent Unit","Model Compression","Parameter Sharing","Language Modeling","Perplexity","Restricted Recurrent Neural Network"],"falsifier":"Train an RNN on a sequence task with input dimension not equal to hidden dimension using the paper's shared-row construction literally; the mismatch forces either projection layers or a change in sharing, and measuring the actual parameter count and perplexity in that common setting would show whether the claimed compression trade-off holds there.","tokens_in":9935,"feed_emoji":"📉","tokens_out":9403,"duration_ms":519359,"temperature":0.7,"pith_summary":"The paper proposes a structural compression for recurrent networks: instead of learning separate weight matrices for the current input and the previous hidden state, it makes the two matrices share a large block of rows and keeps only a small state-specific block for each. Across RNN, LSTM, and GRU, the compressed models reach roughly 50% parameter reduction at a sharing rate $r=0.5$ while producing language-modeling perplexity close to—and in several configurations better than—the classical versions. Unlike pruning or low-rank factorization, the method needs no pre-trained model and gives an explicit handle on the compression rate. The authors' central claim is that partial sharing exploits dependencies between input and hidden state, with neither full sharing nor no sharing being optimal.","feed_headline":"Shared RNN weight rows cut parameters in half, keep quality","feed_subtitle":"Restricted LSTMs and GRUs share rows between input and hidden weights, reaching ~50% compression at comparable perplexity.","key_machinery":"The construction is a row-indexing scheme over a shared parameter pool $W$ (and bias pool $b$). For each source—current input or hidden state—the restricted matrix $W^r$ selects the first $s$ rows from $W$ as a shared block and then a private block of $q$ rows at offset positions, yielding $W^r_{xh}$ and $W^r_{hh}$ whose sizes exactly match the classical matrices. The number of shared rows $s$ can differ per source; the paper sets $s = \\mathrm{round}(r d)$ with $d$ the output dimension and defines the overall compression rate by counting only the parameters actually indexed. Applied gate-by-gate, the same pool mechanism compresses LSTM and GRU, and it generalizes to any recurrent model with $m$ inputs and $n$ outputs by a shared-row allocation over the input/output pairs.","core_discovery":"The central claim is that the input-to-hidden and hidden-to-hidden weight matrices in an RNN need not be separate: they can be assembled from a common pool of rows, with a shared block plus private blocks for each input source. For equal input and hidden dimensions and sharing rate $r$, the trainable parameter count drops by the factor $C=(2-r)/2$ for a vanilla RNN and approximately $C=1-r$ for LSTM and GRU, so $r=0.5$ gives about 50% compression. On Penn Treebank and WikiText-2, the restricted LSTM with tied embeddings and dropout reaches test perplexities such as 103.5 on PTB with only 0.553M recurrent parameters, and the restricted LSTM at high sharing rates beats the unrestricted vanilla RNN while using fewer parameters. The paper also reports a phase transition in the sharing rate: most of the benefit appears once $r\\ge 0.5$, supporting the view that the shared block captures common structure while the private rows absorb input-specific structure.","pith_inferences":["The paper's compression formulas assume the input and hidden dimensions coincide; a natural extension is an unequal-dimension variant with projection layers, where the saving per shared row would be smaller and the break-even sharing rate would shift.","Because the restriction is structural and defined before training, it could be combined with post-training quantization or pruning for a multiplicative reduction in footprint, although the paper does not test that combination.","The observed phase transition around $r=0.5$ suggests the shared rows act as a regularizer; an untested prediction is that the benefit becomes larger on small datasets or tasks with limited training data."],"forward_implications":["At $r=0.5$, an RNN, LSTM, or GRU keeps about half the recurrent parameters of its classical version, with the exact factor depending on the architecture.","Because the construction only rearranges weight matrices, a classical pre-trained model is not needed; restricted models train from scratch with standard SGD.","The restricted LSTM can be smaller than a vanilla RNN and still achieve lower perplexity, suggesting gating capacity is worth more than raw parameter count.","The mechanism is compatible with existing regularizers such as dropout and tied input/output embeddings.","The paper recommends $r=0.5$ as a default sharing rate, with $r$ tunable to a target compression-performance trade-off."],"supporting_citations":[{"why":"Defines the LSTM gating structure that the restriction method compresses.","marker":"[1]"},{"why":"Defines the GRU formulation used as a second target for restriction.","marker":"[2]"},{"why":"Provides the LR LSTM compressed baseline with test perplexity compared in Table I.","marker":"[9]"},{"why":"Provides the LSTM-SparseVD compressed baseline with test perplexity compared in Table I.","marker":"[20]"},{"why":"Supplies the AWD-LSTM state-of-the-art perplexity reference in Table I.","marker":"[22]"},{"why":"Supplies the Penn Treebank dataset used for the main language-modeling experiments.","marker":"[23]"},{"why":"Supplies the WikiText-2 dataset used as the second benchmark.","marker":"[24]"},{"why":"Motivates the tied-embedding setup used in the restricted models.","marker":"[26]"}],"fun_headline_variants":["RNN weight sharing halves parameters, keeps perplexity","Restricted RNNs: Shared weights, 50% fewer parameters","LSTM with shared weights beats vanilla RNN at half size","Parameter sharing in RNNs: 50% compression, no pretraining","One weight pool for input and hidden, quality intact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The construction requires the current input and the previous hidden state to have the same number of entries; if those sizes differ, the shared rows cannot apply to both without adding extra resizing layers that change the parameter count.","fun_headline_variants_meta":{"raw":{"variants":["RNN weight sharing halves parameters, keeps perplexity","Restricted RNNs: Shared weights, 50% fewer parameters","LSTM with shared weights beats vanilla RNN at half size","Parameter sharing in RNNs: 50% compression, no pretraining","One weight pool for input and hidden, quality intact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1267,"prompt_tokens":946,"completion_tokens":321,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":234}},"tokens_in":562,"tokens_out":321,"duration_ms":148407,"temperature":1.0,"reasoning_tokens":234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:58:49.797060+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an RNN on a sequence task with input dimension not equal to hidden dimension using the paper's shared-row construction literally; the mismatch forces either projection layers or a change in sharing, and measuring the actual parameter count and perplexity in that common setting would show whether the claimed compression trade-off holds there.","supporting_citations":[{"cited_title":"Compression of recurrent neural networks for efficient language modeling,","cited_arxiv_id":null,"evidence_quote":"Provides the LR LSTM compressed baseline with test perplexity compared in Table I."},{"cited_title":"Bayesian Compression for Natural Language Processing","cited_arxiv_id":"1810.10927","evidence_quote":"Provides the LSTM-SparseVD compressed baseline with test perplexity compared in Table I."},{"cited_title":"Recurrent neural network based language model,","cited_arxiv_id":null,"evidence_quote":"Supplies the Penn Treebank dataset used for the main language-modeling experiments."}],"review_version":1}