{"id":"7ec82794-b374-4d2c-9e0d-fc8144039f0d","arxiv_id":"2504.19449","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A training-free inference scheme combines input-activation sparsity with weight low-rank decomposition to reach 50% model-level sparsity on modern LLMs while keeping accuracy close to the dense model.","lead":"R-Sparse is a training-free method that speeds up large language models by skipping low-magnitude input channels and approximating the skipped part with a low-rank matrix computation. On Llama-2, Llama-3, and Mistral it reports 50% model-level sparsity with accuracy close to the original model and roughly 40% faster generation with a custom kernel.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The low-rank residual path is validated only by a qualitative heatmap on 16 C4 samples; the top-r SVD subspace of W need not capture the residual from masked input channels, and no held-out approximation error is reported.","rationale":"The reader's weakest assumption matches the main concern: the low-rank residual term is the novel component of R-Sparse, and it is justified only by a qualitative importance map from a small calibration set. I agree that this is the load-bearing point. The paper has real strengths: it compares across three model families and ten tasks, includes ablations against sparse-only and low-rank-only baselines, provides a code link, and shows some robustness of the importance pattern across datasets and sample counts in Appendix B.2. These are meaningful independent supports. However, none of them measures whether the deployed low-rank subspace actually approximates the masked-channel contribution under the searched per-layer recipes. The mathematical gap is concrete: a global SVD of W is optimized for the full matrix, not for the input-dependent subset of rows selected by the magnitude threshold. A row can have large norm in small singular directions, and if such a row is masked, the residual term will be inaccurate. The reported rank values are small enough that spectral approximation error is likely nontrivial, so the burden is on the authors to show that the masked-channel energy is small or that the residual error is small on held-out data. The proposed held-out error measurement would settle the concern directly. The paper's conclusion also overstates the results as 'without any performance loss' when average accuracy drops are roughly 1.5 to 3 points, but the reader already flagged this and it is secondary to the mechanism concern. The correct verdict remains conditional: the method is promising and empirically competitive, but the central approximation needs a direct quantitative check before the claim can be fully accepted.","tokens_in":14790,"tokens_out":11621,"duration_ms":120364,"concrete_test":"On Llama-2-7B, hold out 100 C4 sequences (separate from the 16 used for recipe search), run the deployed 50%-sparsity recipe, and for every linear layer compute per-token: E_res = ||(X - sigma(X))W^T||_F / ||XW^T||_F (energy in masked channels) and E_app = ||(X - sigma(X))(W - A_r B_r)^T||_F / ||(X - sigma(X))W^T||_F (relative error of the low-rank residual). Report the mean and 90th percentile over tokens and layers, and also the end-task accuracy when the low-rank residual is replaced by the exact masked rows of W (an oracle residual with the same sparse path). If E_app is large while E_res is non-negligible, or if the oracle-residual variant is substantially more accurate than R-Sparse, the central approximation is not the source of the retained performance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the approximation in Section 3.4: Y_r = (X - sigma_t(s)(X))(A_r B_r)^T. This assumes the contribution of the masked (small-magnitude) input channels is well approximated by the top-r singular subspace of the full weight matrix W. The global SVD does not guarantee this: the set of masked channels is input-dependent, and the corresponding rows of W can have large norm along small singular directions. If those directions are not in A_r B_r, the residual term is wrong. The only evidence is the sorted heatmap in Figure 3 from 16 C4 samples; it shows concentration of S_{i,j} but not the actual approximation error of the deployed (rho, r, s) recipes. Under the reported 50% budget with rho = 0.95, r is about 37 for Llama-2-7B MLP layers, so ||W - A_r B_r||_2 is likely not small; whether the masked-channel energy is small enough to compensate is never measured. Appendix B.2 shows the importance pattern is stable across datasets, but pattern stability is weaker than an error bound. If the residual energy is non-negligible and the low-rank residual is inaccurate, the retained accuracy would be due to task robustness rather than the proposed mechanism, and the claim of a training-free rank-aware approximation would be unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes R-Sparse, a training-free activation-sparsity method for LLMs with SiLU/GELU activations. For each linear layer, small-magnitude input channels are zeroed by a threshold, and the contribution of the masked channels is approximated by a low-rank SVD factor pair (A_r, B_r) of the weight matrix, so the layer output is computed as Y_s + Y_r. The per-layer trade-off between the sparse and low-rank budgets is found by an evolutionary search that minimizes perplexity on 16 C4 samples. The method is evaluated at 40% and 50% model-level sparsity on Llama-2-7B, Llama-3-8B, and Mistral-7B across eight common-sense reasoning tasks plus WikiText-2 and XSUM, with an end-to-end kernel speedup and compatibility with 4-bit GPTQ quantization.","tokens_in":15076,"tokens_out":6788,"duration_ms":67969,"significance":"R-Sparse addresses a real bottleneck in on-device LLM inference: it avoids the expensive ReLUfication pre-training used by prior activation-sparsity methods, does not need an active-channel predictor, and extends sparsity to attention layers. The empirical coverage is solid, with three model families, ten tasks, comparisons against CATS and GRIFFIN, ablations of the sparse and low-rank components, and a released codebase. The central methodological claim, however, rests on a low-rank residual approximation that is only qualitatively illustrated, and the paper overstates its accuracy results by claiming 'no performance loss' where Table 1 shows several-point drops. The contribution would be significant if the approximation error is directly measured and the claims are qualified; at present it is a promising empirical technique whose mechanism is under-verified.","major_comments":[{"comment":"The decomposition Y ≈ σ_t(s)(X) W^T + (X − σ_t(s)(X)) (A_r B_r)^T is the load-bearing approximation of the paper, but its accuracy is never directly measured. Figure 3 shows only sorted heatmaps of S_{i,j} for 16 C4 samples; since rows and columns are sorted independently, the visual concentration in the lower-right corner is partly a consequence of the sorting. The paper should report the relative approximation error ||Y − Y_s − Y_r||_F / ||Y||_F for the actually deployed per-layer recipes on calibration data not used for the search, and should compare the chosen A_r B_r against the top-r subspace of the masked-channel rows of W. Without such a measurement, the retained accuracy at 50% sparsity could be due to downstream task robustness rather than to the proposed rank-aware mechanism.","section":"§3.4, Eq. for Y_r"},{"comment":"The claim that R-Sparse achieves 50% sparsity 'without any performance loss' is contradicted by Table 1. On Llama-2-7B at 50% sparsity, the average accuracy is 64.06 versus 65.88 for the dense model; BoolQ drops from 77.71 to 72.84 (4.87 points), HellaSwag from 57.13 to 54.26, and ARC-Challenge from 43.43 to 40.78. Similar multi-point drops appear for Llama-3-8B and Mistral-7B. The 'comparable performance' wording used in the abstract is supported, but the stronger 'without any performance loss' wording should be removed or replaced with a statistically grounded statement, for example error bars over multiple evaluation runs or seeds.","section":"§5 Conclusion and Abstract"},{"comment":"The per-layer sparsity/rank recipe is obtained by minimizing perplexity on 16 C4 samples, and the rank choices are also derived from those samples, so the method has a calibration set built in. This is a legitimate design, but the paper does not quantify how sensitive the end-task results are to the calibration choice: no standard deviation is reported for Tables 1, 2, or 4, and no ablation varies the number of calibration samples or the calibration corpus for the final benchmark results. Such a sensitivity analysis, even on a subset of tasks, is needed to distinguish a robust approximation from a recipe that overfits to C4.","section":"§3.5 and Table 4"}],"minor_comments":[{"comment":"The abstract promises 'up to 43% end-to-end efficiency improvements,' while Section 4.3 reports 42% for Llama-2-7B and 40% for Llama-3-8B; please reconcile the numbers.","section":"§4.3 and Abstract"},{"comment":"The formula for memory I/O overhead reads 'rm+n/mn + s' and is ambiguous; it should be r(n+m)/(mn) + s.","section":"§3.4"},{"comment":"There is a duplicated word: 'aligns with with recent studies.'","section":"§3.3"},{"comment":"Because the heatmaps are sorted along both axes, please state this limitation in the main text and provide at least one unsorted heatmap in the appendix to support the claimed concentrated structure.","section":"§3.4 and Figure 3"},{"comment":"The terms 'rank' and 'sparse ratio' are sometimes used interchangeably; define clearly in Section 3.4.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is strong on empirical breadth but needs to address the approximation-error evidence and qualify the accuracy claim. The main verification request is feasible: apply the deployed recipes to held-out inputs and report the relative Frobenius error, and compare the chosen residual subspace against the masked-channel-row subspace. I do not see a fatal flaw that would warrant rejection, provided the authors supply this measurement and correct the 'no performance loss' wording."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about R-Sparse. It is a real, training-free method for accelerating LLM inference: at 50% model-level sparsity it keeps average accuracy within about 1.5-3 points of the dense model on Llama-2/3 and Mistral across ten tasks, and the custom kernel gives up to ~40% end-to-end speedup. The second thing is that the abstract's 'without any performance loss' is contradicted by their own Table 1—several tasks drop 4-5 points (BoolQ on Llama-2 goes 77.7 to 72.8, for example). The average story is fine; the headline claim is overstated.\n\nThe genuinely new piece is combining input-channel magnitude sparsity with a low-rank SVD residual path, plus a per-layer evolutionary search for the sparse/rank budget. The ablation shows the combination beats either branch, and it is the first training-free activation sparsity that works on attention blocks and non-ReLU models without pretraining. That is a practical advance over CATS and GRIFFIN, and the concurrent Liu et al. work is cited and described fairly. Code is public.\n\nThe soft spots are real but not disqualifying. The load-bearing approximation—that the masked input channels' contribution is captured by the top-r singular subspace of W—is supported only by a qualitative heatmap from 16 C4 samples and a stability check across datasets. There is no held-out approximation error and no bound. I read it as a motivating observation, not a proven mechanism; the end-to-end numbers stand on their own, and the ablation supports the combination, but the mechanism section is the weakest part. Also, the per-layer recipe is fitted on those same 16 samples, and no error bars are reported anywhere, so treat the exact numbers as indicative rather than definitive.\n\nWho should read it: anyone doing efficient inference on a single GPU. The speedup is credible, the baselines are treated reasonably, and the method is simple enough to reproduce from the description plus the released code. I would send it to peer review—it deserves referee time even with these caveats. The main requests to the authors would be an error-bound or at least a held-out approximation-error plot for the low-rank residual, and error bars or multiple seeds for the main table.","headline":"A genuine training-free 40% speedup with small average accuracy loss, but the 'no performance loss' claim oversells and the low-rank residual mechanism is empirically motivated, not proven.","tokens_in":15623,"tokens_out":2837,"would_cite":true,"duration_ms":28086,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training-free activation sparsity matches full-model accuracy at 50% sparsity","keywords":["activation sparsity","large language model inference","low-rank approximation","singular value decomposition","training-free compression","input channel sparsity","LLM acceleration"],"falsifier":"Measure the residual approximation error $\\|Y - \\sigma_{t(s)}(X)W^T - (X-\\sigma_{t(s)}(X))(A_rB_r)^T\\|_F$ on out-of-distribution tokens for a fixed searched recipe; if the relative error under 50% sparsity on, say, code or multilingual text is substantially larger than on the C4 validation set, the transferability of the low-rank residual assumption is refuted.","tokens_in":14602,"feed_emoji":"⚡","tokens_out":9448,"duration_ms":80802,"temperature":0.7,"pith_summary":"R-Sparse claims that a modern LLM's linear layers can be run at 50% model-level sparsity—no retraining, no ReLU replacement, and no predictor of active channels—while keeping accuracy on ten tasks close to the dense model. The key move is to split each linear layer into two computational paths: a sparse path that keeps only the large-magnitude input channels, and a fixed low-rank path, obtained offline from the weight's SVD, that reconstructs in approximate form what the masked small channels would have contributed. This matters because activation sparsity has promised faster on-device, small-batch inference, but for SiLU/GELU transformers previous training-free methods stalled around one-third model-level sparsity or needed up to 150B tokens of continued training. If the claim holds, 50% sparsity becomes available out of the box for existing models, with up to 43% measured end-to-end speedup on a custom kernel and easy combination with 4-bit weight quantization.","feed_headline":"Skip half the LLM math with no retraining and accuracy intact","feed_subtitle":"Small-magnitude inputs get a low-rank shortcut; Llama-2/3 and Mistral stay on par at 50% sparsity.","key_machinery":"The score matrix $S_{i,j}=\\sigma_i X_j V_{[j,i]}$ is the paper's diagnostic object: it decomposes each output contribution by input channel and by singular component of the weight, and it reveals a sparse lower-right concentration. The construction that carries the argument is the two-path decomposition of each linear layer, a magnitude-thresholded sparse path through the raw weight plus a rank-$r$ SVD-residual path $A_rB_r$ (with $A_r=U_r\\Sigma_r^{1/2}$, $B_r=\\Sigma_r^{1/2}V_r^T$) applied to the masked input; an offline evolutionary search sets the per-layer balance $\\rho$ between the sparse budget and the rank budget. Since the low-rank factors are computed once from the pretrained weights and the input threshold is read off the current token's activation magnitudes, nothing in the method requires training or predicting which channels will be active.","core_discovery":"The paper's central discovery is an approximation identity for a linear layer $Y = XW^T$: writing $W = U\\Sigma V^T$ and letting $\\sigma_{t(s)}(X)$ keep only entries of $X$ above a magnitude threshold, the full computation is approximated by $Y_s + Y_r = \\sigma_{t(s)}(X)W^T + (X-\\sigma_{t(s)}(X))(A_r B_r)^T$, where $A_rB_r$ is a low-rank factor of the weight. The paper supports this with two measurements on Llama-2-7B using 16 C4 samples: the non-sparse (small-magnitude) input components behave like a few data-dependent biases whose span across thousands of tokens has stable rank about 400, and the contribution matrix $S_{i,j} = \\sigma_i X_j V_{[j,i]}$, which scores how much input channel $j$ and singular component $i$ matter to the output, concentrates its large entries in a small lower-right corner. Sparsifying the input directly removes the top-left of that score matrix, and the low-rank residual term recovers most of what was removed; no prediction of future active channels is needed because the input is already known when the layer runs. An evolutionary search over the sparse-to-low-rank ratio per layer tunes the recipe, and the paper reports that at 50% model-level sparsity the accuracy drop across Llama-2, Llama-3, and Mistral is small.","pith_inferences":["If the channel-and-singular-value importance pattern is as stable across domains as the paper's five-dataset comparison suggests, the searched sparsity recipe should transfer to new tasks and possibly to other models in the same family without rerunning the search; this is a direct, testable prediction the paper does not make.","The same two-path idea, magnitude sparsity on one factor of a bilinear map corrected by a low-rank approximation of the other factor, may apply outside transformers, for example to attention key-value computations or state-space model projections.","A per-token or per-head adaptive threshold, rather than a global percentile threshold, might push beyond 50% sparsity while controlling the residual error; the paper's fixed-threshold design leaves that margin unexplored."],"forward_implications":["At 50% model-level sparsity, R-Sparse keeps average accuracy within a few points of the dense model on eight common-sense tasks for Llama-2-7B, Llama-3-8B, and Mistral-7B, and on easy tasks like SciQ the match is exact even at 70% sparsity.","Because sparsification targets the input side, it works in both attention and MLP linear layers, raising the achievable model-level sparsity beyond the one-third ceiling of MLP-only methods.","With a customized kernel, skipping half the linear-layer I/O yields up to 43% faster token generation at generation lengths from 128 to 2048 tokens.","The method composes with 4-bit weight quantization, so sparsity and quantization can be stacked for further memory savings."],"supporting_citations":[{"why":"Supplies the C4 samples used for the importance measurements and the evolutionary search objective.","marker":"(Dodge et al., 2021)"},{"why":"Defines the Llama-2-7B model family on which the observations and main experiments are made.","marker":"(Touvron et al., 2023)"},{"why":"Provides the Llama-3-8B evaluation model.","marker":"(Dubey et al., 2024)"},{"why":"Provides the Mistral-7B evaluation model.","marker":"(Jiang et al., 2023)"},{"why":"CATS is the thresholding baseline whose output-side sparsity R-Sparse extends to the input side.","marker":"(Lee et al., 2024)"},{"why":"GRIFFIN is the MLP-only baseline that motivates per-layer recipe search and comparison at 33% and 50% sparsity.","marker":"(Dong et al., 2024)"},{"why":"ReLUfication without retraining is the baseline showing why non-ReLU models need a different sparsification mechanism.","marker":"(Mirzadeh et al., 2023)"},{"why":"Supports the observation that q_proj and k_proj layers are more amenable to low-rank compression, informing layer-wise recipes.","marker":"(Jaiswal et al., 2024)"},{"why":"GPTQ is the weight quantization method used to demonstrate compatibility with R-Sparse.","marker":"(Frantar et al., 2022)"}],"fun_headline_variants":["Training-free LLM sparsity: 50% compute cut, accuracy intact","Rank-aware sparsity zero-retrain cuts LLM compute at 50%","Low-rank trick makes LLM sparsity training-free cuts compute","Activation sparsity without retraining: 50% less LLM compute","No-retrain rank-aware sparsity keeps LLM accuracy at 50%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the small-magnitude input channels that get masked out leave behind a contribution that a fixed low-rank subspace of the original weights can approximate; if the relative importance of channels and singular values shifts on inputs unlike the 16 C4 samples used to measure it, the residual path will miss the lost computation.","fun_headline_variants_meta":{"raw":{"variants":["Training-free LLM sparsity: 50% compute cut, accuracy intact","Rank-aware sparsity zero-retrain cuts LLM compute at 50%","Low-rank trick makes LLM sparsity training-free cuts compute","Activation sparsity without retraining: 50% less LLM compute","No-retrain rank-aware sparsity keeps LLM accuracy at 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001859,"raw_usage":{"total_tokens":7401,"prompt_tokens":1149,"completion_tokens":6252,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":765,"completion_tokens_details":{"reasoning_tokens":6154}},"tokens_in":765,"tokens_out":6252,"duration_ms":40757,"temperature":1.0,"reasoning_tokens":6154,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:52:09.619566+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the residual approximation error $\\|Y - \\sigma_{t(s)}(X)W^T - (X-\\sigma_{t(s)}(X))(A_rB_r)^T\\|_F$ on out-of-distribution tokens for a fixed searched recipe; if the relative error under 50% sparsity on, say, code or multilingual text is substantially larger than on the C4 validation set, the transferability of the low-rank residual assumption is refuted.","supporting_citations":[],"review_version":1}