{"id":"fa09e048-817e-4aaa-879b-e425b4885199","arxiv_id":"2501.18980","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SymWanda expresses Wanda and RIA pruning scores as special cases of a symmetric input-output reconstruction objective, and R2-DSnoT adds modest training-free post-pruning gains.","lead":"Pruning removes weights from neural networks; this paper proposes a unified score, SymWanda, that combines input and output activations and shows Wanda and RIA are special cases of it. The authors add a training-free fine-tuning step, R2-DSnoT, reporting small perplexity gains over strong baselines.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 3.6 cannot globally recover RIA: the activation factor ||C:j||^α couples rows and columns, so no fixed matrices A,B make the SymWanda score equal RIA scores; the claimed unification of Wanda and RIA is incorrect.","rationale":"I read the paper in good faith: Lemma 3.1 is correct for a single deleted weight, and Corollary 3.2 shows Wanda is a genuine special case. The problem is the claimed extension to RIA. For fixed matrices in the symmetric objective, the score decomposes as a row-only term plus a column-only term. The RIA score contains the product ||C_:j||^α · ||W_k:||^{-1}, which couples j and k in a way that no such additive decomposition can express unless the activation norms are constant across columns. Lemma 3.6 works only by constructing different matrices for each weight pair, which is not a valid instantiation of a single global pruning objective. The reader's weakest assumption concerned the gap between single-weight exactness and simultaneous pruning of many weights; that concern is real, but the more load-bearing flaw is that even the single-weight equivalence to RIA is impossible globally. This invalidates the abstract's claim of theoretical insight into RIA and undermines the central theoretical narrative. The empirical contributions are also modest: stochRIA is often within noise of RIA, and R2-DSnoT gains are small and not statistically tested; additionally, the regularization terms γ1, γ2 in Eqs. (5)-(6) are constant with respect to the candidate index for a fixed row, so the 'regularized decision boundary' does not affect index selection. Given that the central theoretical claim is internally inconsistent, I recommend rejecting the current version; a substantial rewrite that withdraws or carefully re-scopes the RIA unification claim could be reconsidered.","tokens_in":21326,"tokens_out":17538,"duration_ms":163530,"concrete_test":"Take W = [[1,2],[3,1]] and choose C with ||C_:1||^α = 1 and ||C_:2||^α = 2. Attempt to find any A ∈ R^{2×2}, B ∈ R^{2×2} such that ||A_j:||_2 + ||B_:k||_2 = (||W_j:||^{-1} + ||W_k:||^{-1}) ||C_:j||^α for all j,k ∈ {1,2}. Subtracting the k=1 and k=2 equations for row 1 gives ||B_:2|| - ||B_:1|| = 1/12, while the same subtraction for row 2 gives ||B_:2|| - ||B_:1|| = 1/6, a contradiction. A small least-squares feasibility search on this 2×2 instance will confirm the residual cannot reach zero, settling whether Lemma 3.6 can hold globally.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.3/Lemma 3.6 claims to recover RIA, S_jk = (||W_j:||^{-1} + ||W_k:||^{-1}) ||C_:j||^α, from the symmetric objective by choosing matrices A and B. For fixed A ∈ R^{a×b} and B ∈ R^{c×d}, the score in the symmetric framework is ||A_j:||_2 + ||B_:k||_2, i.e., a function of j plus a function of k. The RIA target, however, is h(j) + w_j v_k with h(j) = ||W_j:||^{-1}||C_:j||^α, w_j = ||C_:j||^α, and v_k = ||W_k:||^{-1}. Unless w_j is constant in j (or v_k is constant in k), no functions f(j), g(k) can satisfy f(j) + g(k) = h(j) + w_j v_k for all j,k: subtracting the equations for two rows j1,j2 gives (w_{j1} - w_{j2}) v_k = constant in k, which fails when v_k varies across columns. Lemma 3.6's proof constructs A and B for a single pair (j,k) at a time; it does not produce one global pair of matrices X,Y. Therefore RIA is not a special case of the symmetric objective, and the paper's central claim that the framework 'provides theoretical insights into RIA' is not supported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a symmetric pruning objective g(What)=||X(What-W)||_F + ||(What-W)Y||_F and derives a single-weight score S_jk=|W_jk|(||X_:j||_2 + ||Y_k:||_2). It claims this framework recovers Wanda and RIA as special cases, and uses it to motivate new pruning scores (generalized ℓp-norm and stochastic RIA, called stochRIA) as well as a training-free fine-tuning procedure R2-DSnoT. Experiments on LLaMA2-7b/13b, LLaMA3-8b and OPT-1.3b at several sparsity levels compare perplexity and zero-shot accuracy with Wanda, RIA and DSnoT.","tokens_in":21730,"tokens_out":14411,"duration_ms":125908,"significance":"The derivations in Lemma 3.1 and Theorem 3.5 are correct and show that Wanda and the activation-free relative-importance (RI) score are indeed special cases of the symmetric objective; the paper also provides simple constructions for ℓp and stochastic variants. If the RIA-recovery claim were true, the paper would offer a useful unification of two popular pruning heuristics. However, Lemma 3.6 is mathematically invalid as a global recovery of RIA (see major comment), so the central theoretical contribution does not hold as stated. The empirical study covers four models and multiple sparsities, which is a useful resource, but the headline claims of superior performance are not consistent with the reported tables. No code or machine-checked proofs are provided beyond the algebraic derivations.","major_comments":[{"comment":"The construction in Lemma 3.6 does not recover RIA as a special case of the symmetric objective. For fixed matrices A and B, the score in (Sym) has the additive form |W_{jk}|(||A_{j:}||_2 + ||B_{:k}||_2). To match Eq. (2) for all (j,k), one would need ||A_{j:}||_2 + ||B_{:k}||_2 = ||W_{j:}||_1^{-1}||C_{:j}||_2^α + ||W_{:k}||_1^{-1}||C_{:j}||_2^α = h(j) + w_j v_k, with h(j)=||W_{j:}||_1^{-1}||C_{:j}||_2^α, w_j=||C_{:j}||_2^α, and v_k=||W_{:k}||_1^{-1}. Subtracting the equations for two rows j1 and j2 gives (w_{j1}-w_{j2})v_k = const for all k, which is impossible when the activation norms differ and the column norms v_k vary. In Lemma 3.6, B_{s,k} is set to ||C_{:j}||_2^α ||W_{:k}||_1^{-1} for a single row j, so B depends on j; no single pair (A,B) satisfies the identity for all (j,k). Thus Eq. (2) is recovered only for the one pair used in the construction, and Table 1's identification of RIA with Lemma 3.6 is incorrect. This undermines the abstract's claim that the framework provides theoretical insights into RIA's success.","section":"Section 3.3, Lemma 3.6"},{"comment":"The exact score S_jk is derived for deleting a single weight while all other weights are unchanged. The text then states that 'the general approach can be extended to multiple weights iteratively,' and the experiments prune large fractions by sorting the single-weight scores. Sorting ignores the fact that after one weight is removed, the optimal next choice depends on the updated weight matrix and the remaining activations. No proof, bound, or experiment isolates this gap. Therefore the statement that the resulting batch-pruning strategy is 'backed by theoretical foundations' (end of Section 3.2) is not supported by the presented analysis.","section":"Section 3.2, after Lemma 3.1"},{"comment":"The text claims stochRIA 'consistently demonstrates the benefits' and has an accumulated perplexity 0.66 lower than RIA. The tables show mixed results: at 50% sparsity, stochRIA is worse than RIA on LLaMA2-7b (6.91 vs 6.88) and LLaMA3-8b (9.46 vs 9.44), and at 70% it is worse on LLaMA2-7b (72.85 vs 68.75) and LLaMA2-13b (62.15 vs 51.96). In addition, the listed deltas in Table 2 do not obviously sum to 0.66; for example, the 2:4 LLaMA3-8b entry reports +0.15 while 23.74 - 22.89 = 0.85. The aggregate claim should be recomputed from the full table, and the conclusion should be revised to say that stochRIA is comparable but cheaper, with mixed and sometimes worse outcomes at higher sparsity.","section":"Section 4.1, Tables 2 and 4"},{"comment":"The abstract and Section 4.3 state that R2-DSnoT 'significantly outperforms strong baselines' and sets a new state of the art. The reported gains over DSnoT are small: for example, in Table 5 the RIA-based perplexity improves from 9.97 to 9.96 on LLaMA2-7b, and in Table 6 the RIA-based mean zero-shot accuracy improves from 50.2 to 50.3 on LLaMA2-7b and from 47.9 to 48.1 on LLaMA3-8b. No variance estimates or significance tests are reported. Moreover, Appendix C.5 selects hyperparameters per model via a 20-run random search and reports the best configuration, which is vulnerable to selection bias. The 'state of the art' claim is not supported by the evidence.","section":"Section 4.3, Tables 5-6 and Appendix C.5"}],"minor_comments":[{"comment":"The notation is inconsistent: Theorem 3.5's proof refers to 'the j-th row of X' while using the column notation X:j, and Lemma 3.6 alternates between row and column norms. Please align the notation with Lemma 3.1, where X:j is a column and Y_k: is a row.","section":"Appendix A.2 and Lemma 3.6"},{"comment":"The symbol S_jk is defined in Lemma 3.1 as the error value g(What), but later in Section 3.6 the notation D_{q,r} is used for a related reweighting quantity. Please clarify the relationship between the symmetric score and the R2-DSnoT growing/pruning criteria in Equations (5)-(6).","section":"Lemma 3.1 and Section 3.6"},{"comment":"The table uses footnote marker '(a)' both as a table footnote and as the dimension a in the header, which is confusing; a different symbol should be used for one of them.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"The manuscript's central theorem about RIA is, in my reading, irreparable: the activation factor multiplies both the row and column terms of the RIA score, so it cannot be represented by a fixed additive row-plus-column score of the form ||A_{j:}||_2 + ||B_{:k}||_2. The empirical claims also overstate the results. If the authors remove the RIA-unification claim and present only the valid Wanda/RI constructions together with a more restrained comparison of the new pruning criteria, the paper could be reconsidered; in its current form I cannot recommend publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a real core: rewriting the pruning objective as a sum of input and output reconstruction errors, then showing that the single-weight deletion error is |W_jk|(||X_:j||_2 + ||Y_k:||_2). That Lemma 3.1 is correct, and the diagonal constructions in Theorem 3.5 do exactly what they claim. Seeing Wanda and relative importance (RI) fall out of one score is a nice organizing observation, and the ℓp-norm and stochastic variants are reasonable attempts to extend it. The ablation on activation exponent, column vs. row sensitivity, and sampling ratio is also useful and mostly clear.\n\nThe soft spots are not minor. Lemma 3.6 is the load-bearing claim that RIA is also a special case, and it fails. For any fixed X and Y, the score is separable: f(j) + g(k). The RIA target is h(j) + w_j v_k, with w_j and v_k both varying in general. Those two forms are incompatible unless one of the factors is constant. The lemma constructs different matrices for each (j,k) pair, not one global pair, so it never actually embeds RIA into the symmetric framework. This is not a citation or a cosmetic issue; it breaks the advertised unification. The reader's stress-test is right, and the paper's Section 3.3 needs a major fix or a serious downgrade of the claim.\n\nBeyond that, the leap from single-weight error to simultaneous pruning of many weights is asserted, not justified. The stochastic RIA results are often within noise of RIA and sometimes worse; the paper itself shows a mix of blue and red deltas. R2-DSnoT does show consistent small gains over DSnoT, but calling it \"a new state of the art\" is overreach without released code, variance on the main tables, and more baselines. The hyperparameter search is done on one model, which is thin.\n\nWho should read this? Anyone working on LLM pruning will find the symmetric perspective worth a few hours, especially the clean derivation of Wanda and RI. But the paper as it stands overstates its theoretical contribution. I would send it to peer review because the core idea deserves scrutiny and the experiments are extensive, but a good referee should force a rewrite of Section 3.3 and a more measured set of claims.","headline":"The symmetric objective is a genuinely clean reformulation that recovers Wanda and RI, but the claimed recovery of RIA collapses under inspection, and the empirical gains are mostly marginal.","tokens_in":22232,"tokens_out":2469,"would_cite":false,"duration_ms":25986,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Two leading pruning methods, Wanda and RIA, are special cases of one symmetric reconstruction error, and the same identity yields cheaper stochastic variants and a training-free fine-tune.","keywords":["post-training pruning","large language models","symmetric reconstruction error","Wanda","relative importance","training-free fine-tuning","dynamic sparse training","stochastic sampling"],"falsifier":"Run the paper's exact setup on one model, say LLaMA2-7b at 50% sparsity with Wikitext-2 perplexity as the metric, two ways: one-shot pruning by globally sorting the scores $S_{jk}$, and greedy iterative pruning that recomputes the score after each deletion; if greedy is materially better, the multi-weight extrapolation of Lemma 3.1 is doing unproven work, and if the two match, the single-weight ranking is robust enough to support the method.","tokens_in":21133,"feed_emoji":"✂️","tokens_out":9557,"duration_ms":73831,"temperature":0.7,"pith_summary":"The paper tries to give a theoretical foundation for two empirically successful but poorly understood post-training pruning methods for large language models: Wanda and RIA. Its central move is to replace the usual input-only reconstruction objective with a symmetric one that also tracks what each weight does to the layer's output, and to prove that deleting a single weight $W_{jk}$ changes this objective by exactly $|W_{jk}|(\\|X_{:j}\\|_2 + \\|Y_{k:}\\|_2)$. From that identity, both Wanda (set the output term to zero) and RIA (choose calibration matrices that encode reciprocal row and column norms) fall out as special cases, which the paper presents as the first theoretical explanation of why these heuristics work. Exploiting the same parametrization, the paper derives cheaper relative-importance scores that sample only 10% of each row and column, and a training-free prune-and-grow fine-tuning scheme, R2-DSnoT, that it reports outperforms strong baselines. A sympathetic reader cares because if the identity is right, a whole family of pruning methods with a common single-weight guarantee becomes available, not just two tuned heuristics.","feed_headline":"One symmetric score explains Wanda and RIA pruning","feed_subtitle":"A single reconstruction error reduces both popular pruning heuristics to special cases, and powers cheaper variants.","key_machinery":"The load-bearing object is the single-weight identity $S_{jk} = |W_{jk}|(\\|X_{:j}\\|_2 + \\|Y_{k:}\\|_2)$, the exact change in the symmetric reconstruction error when weight $W_{jk}$ alone is zeroed; the freedom to choose the calibration matrices $X$ and $Y$ turns this one formula into a generator of pruning scores. Wanda is the corner $Y=0$, RIA is recovered by diagonal or constant-vector choices whose norms equal reciprocal $\\ell^1$ row and column norms, and new methods arise by replacing those norms with $\\ell^p$ norms or with norms over randomly sampled subsets of indices (StochRIA). The identity also drives the R2-DSnoT fine-tuning rules, which use the same relative-importance products plus an $\\ell^p$ regularizer when deciding which weights to grow and which to prune.","core_discovery":"The paper's core claim is that post-training pruning of a language model layer should minimize a symmetric reconstruction error $g(\\tilde{W}) = \\|X(\\tilde{W}-W)\\|_F + \\|(\\tilde{W}-W)Y\\|_F$, where $X$ is the input calibration matrix and $Y$ is a newly introduced output calibration matrix. Lemma 3.1 gives the exact single-weight form: zeroing $W_{jk}$ costs $|W_{jk}|(\\|X_{:j}\\|_2 + \\|Y_{k:}\\|_2)$. Because the choice of $X$ and $Y$ is free, the paper shows that Wanda's score, $|W_{jk}|\\|X_{:j}\\|_2$, is exactly the $Y=0$ corner of this formula, and that RIA's relative-importance score, $|W_{jk}|(\\|W_{j:}\\|_1^{-1} + \\|W_{:k}\\|_1^{-1})$, is recovered by choosing $X$ and $Y$ whose row and column norms are reciprocals of the weight row and column $\\ell^1$ norms. This is what the authors mean by providing theoretical insights into the empirical successes of Wanda and RIA, and it lets them generate new scores, including $\\ell^p$-norm versions, random-unit-vector scalings, and a stochastic 10%-sampled relative importance, each carrying the same single-weight guarantee. On top of the pruning scores, the paper contributes R2-DSnoT, a training-free pruning-and-growing fine-tuner that applies the same relative-importance products plus a regularized decision boundary, reporting improved perplexity and zero-shot accuracy at 60% sparsity across LLaMA2, LLaMA3, and OPT models.","pith_inferences":["Editorial inference, beyond the paper: because Lemma 3.1 is exact only for a single deletion, the paper's global-sort procedure is an untested approximation; comparing it against greedy iterative re-scoring on the same models would reveal how much of the reported performance the identity itself guarantees.","Editorial inference, beyond the paper: the theorem treats $X$ and $Y$ as free choices, so a natural unexplored variant is to set $Y$ from the actual output activations of the layer rather than from $W$ itself, which could capture downstream sensitivity that the current constructions miss.","Editorial inference, beyond the paper: the ablation finding that column-wise (output-side) relative importance matters more than row-wise suggests that output-calibrated pruning alone should already rival input-based Wanda; that prediction is directly testable on the paper's own benchmark suite."],"forward_implications":["Wanda and RIA cease to be standalone heuristics and become two points in a parametrized family of pruning scores, each with an exact single-weight reconstruction guarantee.","Any new choice of calibration matrices $X$ and $Y$ produces a new pruning rule with the same guarantee, so the framework is a generator of candidate methods rather than a single algorithm.","Estimating relative importance from 10% of each row and column suffices to match, and on two of the four tested models beat, full RIA, which makes relative-importance pruning practical at large scale.","The training-free fine-tuning step R2-DSnoT improves perplexity and zero-shot accuracy on top of pruned models without any gradient updates, which, if it holds, removes the main cost barrier to iterative refinement of sparse language models."],"supporting_citations":[{"why":"Wanda, the method whose score is recovered as the $Y=0$ corner of Lemma 3.1; it supplies the objective being explained and the primary baseline.","marker":"(Sun et al., 2023)"},{"why":"RIA (plug-and-play), the relative-importance score recovered by Theorem 3.5; it supplies the second baseline and the activation-power convention $\\alpha$.","marker":"(Zhang et al., 2024b)"},{"why":"SparseGPT, whose layer-wise input reconstruction problem defines the objective that the symmetric formulation extends.","marker":"(Frantar & Alistarh, 2023)"},{"why":"DSnoT, the training-free pruning-and-growing framework on which R2-DSnoT is built and against which it is compared.","marker":"(Zhang et al., 2023)"},{"why":"LLaMA2, the model family used for the central perplexity and zero-shot evaluations.","marker":"(Touvron et al., 2023b)"},{"why":"C4, the calibration data source used in every pruning experiment reported.","marker":"(Raffel et al., 2020)"},{"why":"Wikitext-2, the evaluation dataset behind every perplexity number in the empirical comparison.","marker":"(Merity et al., 2016)"}],"fun_headline_variants":["Symmetric error unifies Wanda and RIA pruning","One score explains Wanda and RIA pruning","New theory: Wanda and RIA are symmetric cases","Symmetric pruning theory boosts LLM efficiency","R2-DSnoT: training-free pruning sets new state of the art"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The identity $S_{jk}$ is proved only for deleting a single weight, while every reported pruning result is obtained by sorting these scores and deleting many weights at once; the paper assumes this extrapolation holds, and the entire empirical evaluation rests on it.","fun_headline_variants_meta":{"raw":{"variants":["Symmetric error unifies Wanda and RIA pruning","One score explains Wanda and RIA pruning","New theory: Wanda and RIA are symmetric cases","Symmetric pruning theory boosts LLM efficiency","R2-DSnoT: training-free pruning sets new state of the art"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000862,"raw_usage":{"total_tokens":3804,"prompt_tokens":1074,"completion_tokens":2730,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":690,"completion_tokens_details":{"reasoning_tokens":2663}},"tokens_in":690,"tokens_out":2730,"duration_ms":17712,"temperature":1.0,"reasoning_tokens":2663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T21:46:50.547812+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's exact setup on one model, say LLaMA2-7b at 50% sparsity with Wikitext-2 perplexity as the metric, two ways: one-shot pruning by globally sorting the scores $S_{jk}$, and greedy iterative pruning that recomputes the score after each deletion; if greedy is materially better, the multi-weight extrapolation of Lemma 3.1 is doing unproven work, and if the two match, the single-weight ranking is robust enough to support the method.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Wanda, the method whose score is recovered as the $Y=0$ corner of Lemma 3.1; it supplies the objective being explained and the primary baseline."}],"review_version":1}