{"id":"038dd673-459d-4b65-bddc-1b8d23465a2f","arxiv_id":"1908.09982","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Low-rank factorization compresses LSTM weight matrices by up to 98% with a small accuracy loss, and the hidden (multiplicative) recurrence is generally more compressible than the input (additive) recurrence.","lead":"This paper tests low-rank matrix factorization (SVD and Semi-NMF) as a post-training method to compress LSTM networks, finding that the hidden-to-hidden matrix compresses far better than the input matrix. It also proposes matrix norms as a predictor of compressibility, though one dataset (SQuAD) shows the opposite pattern.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 4.4's nuclear-norm explanation is unnormalized and untested; without it the paper's main generalization about additive versus multiplicative recurrence is not established.","rationale":"The reader is right that the generalization is fragile, but the SQuAD contradiction is weaker than stated: at r=10 Wi is much worse than Wh on SQuAD, and the r=400 differences are around 0.2 F1 with no error bars, so they may be noise. The more fundamental problem is that Section 4.4's nuclear-norm argument has no independent predictive content: it is a post-hoc reading of the same numbers, the norms are not normalized for matrix size and scale, and 'nuclear norm approximates rank' is not mathematically accurate. The empirical compression results in Tables 1-3 are plausible and support the practical claim that low-rank factorization is effective on the tested checkpoints. They do not, by themselves, establish the general rule that additive recurrence is more important than multiplicative recurrence or that nuclear norm explains it. A conditional acceptance requiring either a normalized, held-out test of the norm explanation or a softened claim is appropriate; therefore I leave the reader's CONDITIONAL verdict unchanged.","tokens_in":11736,"tokens_out":15599,"duration_ms":160705,"concrete_test":"Compute normalized low-rankness measures for Wi and Wh in each checkpoint (PTB, WT2, ELMo forward/backward): normalized nuclear norm ||W||_*/min(m,n), stable rank ||W||_F^2/||W||_2^2, and cumulative tail energy at r=10 and r=400, all scaled by ||W||_F. Order these quantities across tasks and check whether they predict the actual PPL/F1 losses in Tables 1-3, including the SQuAD r=400 case where Wi beats Wh. If the normalized measures rank Wh below Wi in ELMo yet Wi compresses better in SQuAD, the Section 4.4 explanation is falsified; if they reverse the ordering, the concern is resolved. Also report per-run variances so the 0.2-point SQuAD differences are interpretable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 claims that Wh compresses better than Wi because Wh has a lower nuclear norm, taken as a proxy for intrinsic rank. This is the load-bearing step connecting the empirical tables to the paper's headline generalization ('additive recurrence is more important than multiplicative recurrence'). The support is not compelling for three reasons. First, raw nuclear norms of matrices with different shapes and sizes are not comparable; Wi and Wh have different dimensions (Section 3.1), so a lower raw nuclear norm may reflect scale or dimensionality, not lower effective rank. Second, nuclear norm does not approximate rank: rank is the number of nonzero singular values, while nuclear norm is scale-dependent. Third, the norm hypothesis is formed and 'confirmed' on the same PTB/WT2/ELMo experiments, with no held-out prediction. The SQuAD rows in Table 3 are not reconciled: at r=400, compressing Wi slightly beats Wh for all three methods even though Wh reportedly still has the lower nuclear norm in ELMo. Consequently, the central generalization is not established; what remains is a useful but task-specific benchmark of low-rank LSTM compression.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies post-hoc compression of LSTM weight matrices Wi and Wh using truncated SVD, Semi-NMF, and magnitude pruning, applied in two settings: AWD-LSTM language models on PTB and WikiText-2, and the ELMo biLSTM layers used in BiDAF/SQuAD, ESIM/SNLI, and BCN/SST-5. The central empirical result is that low-rank factorization can reduce the 24M-parameter AWD-LSTM to 9M parameters with roughly one perplexity point of loss after fine-tuning, and that compressing Wh (the hidden-to-hidden recurrence) degrades performance less than compressing Wi (the input-to-hidden recurrence) on the language-modeling tasks. The authors further propose a norm-based explanation: Wh has a lower nuclear norm than Wi, which they interpret as Wh having lower intrinsic rank. The paper also claims 1.5x and 2x inference-time speedups and introduces an efficiency measure E(r) to compare methods across compression ratios.","tokens_in":11921,"tokens_out":6609,"duration_ms":68247,"significance":"If the empirical findings hold, the paper provides a useful benchmark for low-rank LSTM compression across several NLP tasks, including compression of a widely used pretrained contextual representation model (ELMo). The experimental scope is broad: five datasets, two factorization methods, a pruning baseline, two ranks, and fine-tuning in the PTB setting, with the AWD-LSTM baseline reproduced. The main value is the comparative evidence that matrix factorization is often preferable to magnitude pruning at high compression ratios and that the input and hidden recurrence matrices respond very differently to compression. However, the paper's headline generalization — that additive recurrence is more important than multiplicative recurrence, explained by nuclear norms — is not established by the reported evidence. The norm argument is dimensionally unnormalized, is formed and 'confirmed' on the same experiments, and is contradicted by the SQuAD results. The speedup claims are also not backed by timing measurements. These are load-bearing issues for the paper's central claims, not merely presentational.","major_comments":[{"comment":"The nuclear-norm explanation is not valid as stated and does not support the claim that Wh is inherently lower-rank than Wi. First, Wi and Wh have different shapes (Section 3.1: Wi ∈ R^{4·ninp × 4·ndim}, Wh ∈ R^{4·ndim × 4·ndim}), so raw nuclear norms are not comparable across the two matrices; a lower value may simply reflect different dimensions or overall scale. Second, the nuclear norm is the sum of singular values, not an approximation of rank (rank is the number of nonzero singular values), so 'having a lower nuclear norm is often an indicator of low-rank' is a much weaker and scale-dependent statement than the paper needs. Third, the hypothesis is generated and confirmed on the same PTB/WT2/ELMo experiments, with no held-out test or out-of-sample prediction. Fourth, Table 3 directly contradicts the proposed mechanism: at r=400 on SQuAD, compressing Wi is consistently at least as good as compressing Wh (e.g., Semi-NMF F1 81.78 for Wi vs 81.55 for Wh; SVD 81.78 vs 81.62) even though Section 4.4 reports that Wh has the lower nuclear norm in ELMo. To keep the explanatory claim, the authors should normalize the norm measure (e.g., stable rank ||W||_F²/||W||_2², or nuclear norm divided by dimension/Frobenius norm), test whether the rank/compression-loss correlation holds on held-out matrices or tasks, and reconcile the SQuAD rows. Without this, the norm analysis should be presented only as a post-hoc observation, not as an explanation.","section":"Section 4.4, Figure 2 panels (e)-(f), and Table 3"},{"comment":"The paper claims inference-time speedups of 1.5x and 2x (and '2.13x speedup' in Section 4.2), but no wall-clock timing measurements are reported anywhere in the manuscript. Footnote 2 gives the CPU model, and Section 4.1 says the authors 'report ... inference time,' but Tables 1-3 contain only parameter counts, perplexity/accuracy/F1, and E(r). Parameter reduction does not imply wall-clock speedup, especially for factorized recurrent cells, which involve two matrix multiplications and additional operations, and the speedup claim is a central advertised contribution. The authors should either provide measured inference times (with hardware, batch size, sequence length, and implementation details) or rephrase the contribution in terms of parameter and FLOP reduction.","section":"Abstract, Section 1 contribution bullets, Section 4.2"},{"comment":"The paper states that all experiments are 'averaged over 5 runs,' but no standard deviations, confidence intervals, or significance tests are reported for any table. Many of the comparisons that support the ranking of methods are numerically close (e.g., PTB fine-tuned r=400 perplexities 57.19-58.18; SST-5 r=400 accuracies 53.93-54.66), and the claim that one compression target 'works notably better' than another requires variance information. The parenthetical '(avg)' in Table 3 is not a substitute. Please report means with standard deviations over the 5 runs and, where possible, a paired significance test between the best and runner-up configurations.","section":"Section 4, Tables 1-3"},{"comment":"The experimental protocol for the ELMo downstream tasks is ambiguous. The text says the authors 'train publicly available state-of-the-art models' with ELMo and then 'low-rank factorize the pre-trained ELMo layer only,' but it is not clear whether the downstream model is trained first and the ELMo layer is compressed afterward (post-hoc compression of a deployed model), or whether the ELMo weights are factorized before downstream training and then kept fixed. These are different regimes: post-hoc compression measures the robustness of an already-trained model, whereas pre-training factorization measures the ability to learn with the compressed representation. This distinction matters for comparing the compressed results with the uncompressed baselines and with prior compression work. Please state the exact chronology of factorization, downstream training, and any fine-tuning.","section":"Section 4.3"}],"minor_comments":[{"comment":"Several citations are incomplete or in an unusual authorless format, such as '(2008; 2013; 2014; 2014)', '(2010)', '(2016a)', and '(2017)'. These should be replaced with proper author-year citations so that the reader can identify the referenced works.","section":"Section 2 (Related Work)"},{"comment":"The matrix notation in Equation (1) is malformed: '(Wi Wh)' should be presented as a block-row matrix [Wi, Wh] that multiplies (xt; ht−1), with dimensions given for the block matrices. The current rendering obscures the structure of the LSTM update.","section":"Equation (1)"},{"comment":"The efficiency measure E(r) has a typo ('the ration') and the definition of R(a,b) is ambiguous for perplexity, where the paper reports using R(M_r, M) instead of R(M, M_r). Please define the formula cleanly for both higher-is-better and lower-is-better metrics and verify that the reported E(r) values in the tables are consistent with that definition.","section":"Section 4.1, Equation (10)"},{"comment":"The claim that pruning 'requires a special implementation of sparse matrix multiplication' is overstated: magnitude pruning is often implemented with a dense binary mask and does not inherently require sparse kernels. The comparison would be clearer if the paper stated whether pruning was evaluated with dense or sparse multiplication.","section":"Section 3.2"},{"comment":"The statement 'Nuclear norm approximates the matrix rank' is misleading. The nuclear norm is the convex envelope of rank on the unit spectral-norm ball, which is not the same as approximating rank. Please rephrase to avoid conveying that a low nuclear norm directly implies a low rank in the matrix.","section":"Section 4.4"},{"comment":"The heatmaps are discussed as showing a 'salient red line' in Wh, but no color scale or axis labels are provided, and the figures are hard to interpret in grayscale. Please add a color bar and describe the visualization protocol.","section":"Figures 3 and 4"},{"comment":"The paper contains several typographical errors ('intepretations', 'Non-Matrix Factorization' in Related Work, repeated 'modifed'), and the reference list has entries with only years and no author names. A thorough proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is potentially useful, but the paper's advertised contributions go beyond what the evidence supports. The norm-based explanation and the speedup numbers are the main obstacles; the SQuAD contradiction in Table 3 is particularly damaging to the explanatory claim. I would be willing to look at a revision that either fixes these issues with real timing data and a normalized, validated norm analysis or explicitly narrows the claims to task-specific empirical observations. The manuscript's citation style in Related Work is also unusually incomplete and should be corrected before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, what to know: the paper's main value is the benchmark, not the theory. A systematic comparison of SVD, Semi-NMF, and magnitude pruning applied separately to the input (Wi) and hidden (Wh) matrices of LSTMs, across PTB, WikiText-2, and ELMo on SQuAD/SNLI/SST-5, is useful and mostly plausible. The tables support the practical recipes: factorizing an AWD-LSTM from 24M to 9M parameters costs about one perplexity point after fine-tuning; rank-400 compression is nearly lossless; and SVD generally edges out pruning at high compression. That is a real contribution for practitioners who want post-hoc compression without retraining.\n\nThe soft spots are real but don't sink the benchmark. The headline generalization — \"additive recurrence is more important than multiplicative recurrence\" / Wh compresses better than Wi — is contradicted by the paper's own SQuAD rows, where Wi consistently beats Wh for all three methods. Section 4.4's nuclear-norm explanation is post-hoc, built on the same experiments, and compares raw nuclear norms of matrices with different shapes, so the quantity isn't interpretable as a relative rank proxy. It's also not validated on held-out data. The \"correlation\" is asserted, not quantified. If you strip out that explanatory layer, the paper becomes a solid task-specific empirical study: on PTB/WT2, Wh compresses better; on SQuAD, Wi does. That is still worth reporting.\n\nTwo more soft spots: the speedup numbers (1.5x, 2.13x) appear without any timing table or measurement description, and the paper says results are averaged over 5 runs but reports no error bars or variance. Also no code released, which limits reproducibility. None of these sink the central compression results, but they need attention.\n\nWho this is for: anyone doing post-hoc LSTM/ELMo compression or comparing matrix factorization with pruning. It deserves a serious peer review. The right review outcome is probably \"revise\": keep the empirical comparison, weaken or remove the universal claim, add variance and timing, and either fix the norm analysis or cut it. If the authors do that, this is citable. I'd send it to review rather than desk-reject.","headline":"Useful empirical benchmark for LSTM/ELMo compression, but the nuclear-norm explanation is a post-hoc story contradicted by the paper's own SQuAD results.","tokens_in":12481,"tokens_out":3407,"would_cite":true,"duration_ms":36853,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Low-rank matrix factorization compresses LSTM language models by about 60% with a loss of roughly one perplexity point.","keywords":["LSTM compression","low-rank matrix factorization","truncated SVD","semi-NMF","nuclear norm","language modeling","ELMo","model compression"],"falsifier":"Train a new LSTM on a task not in the paper, measure the nuclear norms of $W_i$ and $W_h$, and compress both to the same low rank with truncated SVD; the paper's explanation predicts the matrix with the lower nuclear norm should compress with less perplexity or accuracy loss, and if compressing $W_h$ is worse despite a lower nuclear norm, or $W_i$ is better despite a higher one, the correlation claim fails.","tokens_in":11542,"feed_emoji":"📉","tokens_out":9503,"duration_ms":78226,"temperature":0.7,"pith_summary":"This paper asks whether low-rank matrix factorization can compress the recurrence matrices of an LSTM without retraining, and how much damage different kinds of compression do. Working from a 24-million-parameter LSTM language model, the authors show that replacing the hidden-to-hidden matrix $W_h$ with a rank-10 SVD or Semi-NMF factor pair cuts the model to about 9 million parameters (roughly 60 percent reduction) at the cost of about one perplexity point on Penn Treebank, and that fine-tuning the compressed model can recover or slightly beat the original. The paper's central explanatory claim is that the hidden-to-hidden (multiplicative) recurrence is inherently more compressible than the input-to-hidden (additive) recurrence, and that this asymmetry is visible in matrix norms: $W_h$ has a lower nuclear norm, indicating lower intrinsic rank. It also argues that matrix factorization generally beats magnitude pruning except when the target matrix is already sparse. If these claims hold, practitioners can compress deployed LSTMs cheaply and predict which weight matrix to attack first.","feed_headline":"Low-rank factorization shrinks LSTMs by 60 percent with little loss","feed_subtitle":"SVD and Semi-NMF cut a 24M-parameter language model to 9M while keeping perplexity nearly the same.","key_machinery":"The machinery is low-rank matrix factorization applied separately to the two LSTM recurrence matrices, $W_i$ and $W_h$, each of which stacks the four gates' input and hidden weights. Replacing $W$ with $U V$, where $U$ is $m \\times r$ and $V$ is $r \\times n$, cuts parameter count from $mn$ to $r(m+n)$; the paper uses truncated SVD, which takes the top $r$ singular values and vectors, and Semi-NMF, which factors $W$ as $U V$ with $V$ nonnegative, at ranks $r=10,100,200,300,400$. The explanatory instrument is the nuclear norm, the sum of singular values, used as a proxy for intrinsic matrix rank: the paper observes $W_h$ has the lower nuclear norm and argues this is why it compresses more gracefully. The factorized cell replaces each $W x$ term with $U(V x)$, so only the factor matrices remain in the computation graph.","core_discovery":"The central discovery is that low-rank matrix factorization post-processing compresses LSTM recurrences far below their nominal size while keeping predictive performance close: on PTB, factorizing $W_h$ to rank 10 with SVD or Semi-NMF gives 9M parameters and perplexity in the high 70s compared with 58.3 for the 24M baseline, and with fine-tuning the compressed models reach about 58, slightly better than the baseline, with a 2.13x speedup. The authors find compressing $W_h$, the matrix multiplying the previous hidden state, which they call multiplicative recurrence, works notably better than compressing $W_i$, the input matrix or additive recurrence, and they explain this by nuclear norm: $W_h$'s nuclear norm is consistently lower, indicating that $W_h$ is closer to a low-rank matrix and therefore loses less when projected into a low-dimensional factor space. A second finding is that matrix factorization is generally a better compression strategy than magnitude pruning, with pruning winning only when the original matrix is already sparse. The paper also shows the same method compresses pretrained ELMo biLSTM layers for SNLI and SST-5 with little loss, while SQuAD results reverse the $W_i$/$W_h$ ordering, a caveat the norm analysis does not fully explain.","pith_inferences":["A pre-compression nuclear-norm scan could serve as a cheap decision rule for which matrix to factorize in any trained recurrent model, since the paper's correlation predicts the lower-nuclear-norm matrix should compress first.","The SQuAD reversal suggests the additive-versus-multiplicative ordering is task-dependent; the durable generalization may be 'compress the lower-nuclear-norm matrix' rather than 'compress $W_h$', and that stronger claim still needs testing.","The factor-pair replacement transfers to other recurrent cells such as GRUs, whose analogous input and hidden matrices could be ranked by nuclear norm, so the method is not LSTM-specific.","Because fine-tuning already reaches or passes the baseline, layering the factorization with quantization or knowledge distillation could push compression further without changing the architecture."],"forward_implications":["An LSTM language model can be cut from 24M to 9M parameters with SVD or Semi-NMF and lose only about one perplexity point, without retraining.","Fine-tuning the compressed model at rank 10 can recover the baseline's performance and give a roughly 2x inference speedup.","In language modeling, compressing the hidden-to-hidden matrix $W_h$ should be preferred over compressing the input matrix $W_i$, and the nuclear norm of a trained matrix estimates how safely it can be compressed.","For dense low-rank matrices, low-rank factorization beats magnitude pruning; for already-sparse matrices, pruning is the better choice.","The same factorization transfers to pretrained biLSTM contextualizers: ELMo layers can be compressed by over 10M parameters with near-baseline F1 or accuracy on several downstream tasks."],"supporting_citations":[{"why":"Defines the LSTM cell whose two recurrence matrices are the compression targets.","marker":"[Hochreiter and Schmidhuber1997]"},{"why":"Provides the AWD-LSTM language model, training recipe, and code used for the PTB and WT-2 experiments.","marker":"[Merity et al.2018]"},{"why":"Defines Semi-NMF, one of the two low-rank factorization algorithms evaluated.","marker":"[Ding et al.2010]"},{"why":"Supplies nuclear norm as a convex proxy for matrix rank, the basis of the norm analysis.","marker":"[Fazel2002]"},{"why":"Supplies the pretrained ELMo biLSTM layers that are factorized in the downstream NLP tasks.","marker":"[Peters et al.2018]"},{"why":"Tensor-Train LSTM compression baseline whose perplexity the factored models improve on.","marker":"[Grachev et al.2017]"},{"why":"Source of the magnitude-pruning method adapted and compared against factorization.","marker":"[Han et al.2015]"},{"why":"Provides the BiDAF model for SQuAD, the downstream task where compressing the input matrix won.","marker":"[Seo et al.2016]"}],"fun_headline_variants":["Low-rank MF cuts LSTM params 62% with near-zero loss","Factorizing hidden state shrinks LSTM 60% with tiny perplexity hit","SVD and Semi-NMF compress 24M-param LSTM to 9M, keep quality","Low-rank beats pruning for LSTM compression, 2.13x speedup","Hidden-state factorization drives LSTM compression, study finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a lower nuclear norm reliably marks a matrix as safely compressible, so the hidden-to-hidden recurrence is generally compressible before the input-to-hidden recurrence; this is inferred from the same language-model experiments it is used to explain, and the SQuAD results already go the other way.","fun_headline_variants_meta":{"raw":{"variants":["Low-rank MF cuts LSTM params 62% with near-zero loss","Factorizing hidden state shrinks LSTM 60% with tiny perplexity hit","SVD and Semi-NMF compress 24M-param LSTM to 9M, keep quality","Low-rank beats pruning for LSTM compression, 2.13x speedup","Hidden-state factorization drives LSTM compression, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000651,"raw_usage":{"total_tokens":2984,"prompt_tokens":943,"completion_tokens":2041,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":1938}},"tokens_in":559,"tokens_out":2041,"duration_ms":15761,"temperature":1.0,"reasoning_tokens":1938,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:57:41.120852+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a new LSTM on a task not in the paper, measure the nuclear norms of $W_i$ and $W_h$, and compress both to the same low rank with truncated SVD; the paper's explanation predicts the matrix with the lower nuclear norm should compress with less perplexity or accuracy loss, and if compressing $W_h$ is worse despite a lower nuclear norm, or $W_i$ is better despite a higher one, the correlation claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the AWD-LSTM language model, training recipe, and code used for the PTB and WT-2 experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Semi-NMF, one of the two low-rank factorization algorithms evaluated."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained ELMo biLSTM layers that are factorized in the downstream NLP tasks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Tensor-Train LSTM compression baseline whose perplexity the factored models improve on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the magnitude-pruning method adapted and compared against factorization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BiDAF model for SQuAD, the downstream task where compressing the input matrix won."}],"review_version":1}