{"id":"73e733fc-df05-4a08-b5ef-9dff8cf6c821","arxiv_id":"2506.20353","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DipSVD combines channel-weighted whitening with layer-wise compression ratios and reports better perplexity and accuracy than existing SVD-based LLM compression methods.","lead":"DipSVD is a new method for compressing large language models by trimming low-rank weight matrices while trying to preserve the most important channels and layers. The paper reports that compressed models keep lower perplexity and higher task accuracy than earlier SVD-based compressors such as SVD-LLM and ASVD.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The theoretical guarantee in Sec. 3.1.2 rests on a whitening identity that is generally false; truncating the smallest singular values of the whitened weight matrix is not shown to minimize output loss.","rationale":"The most load-bearing condition for the paper's central claim is the theoretical guarantee that truncating small singular values of the whitened weight matrix minimizes output loss. The proof of this guarantee hinges on the identity S^{-1}\\tilde{X}\\tilde{X}^T(S^{-1})^T = I in Eq. (7), which the paper asserts without proof. As the reader notes, this identity is generally false, and the equality L_i = \\sigma_i plus the root-sum-square formula in Eq. (8) are therefore unsupported. The paper's local importance protection mechanism loses its stated justification. The reader's weakest_assumption points to exactly this flaw; I agree with that identification. One minor technical caveat: the reader states that the paper's S guarantees (\\tilde{X}S)^T(\\tilde{X}S)=I, but the correct whitening matrix for that property would be U\\Sigma^{-1/2}, not \\Sigma^{-1/2}U^T. This caveat only strengthens the objection, since the paper's S is not even the standard whitening matrix. The experimental issues raised by the reader, such as calibration and evaluation overlap on WikiText-2, absence of error bars, and lack of released code, are real but secondary: they weaken the empirical evidence, while the invalid proof undermines the paper's foundational theoretical contribution. For these reasons the reader's REJECT verdict should stand, and the paper would need a corrected derivation and a more rigorous empirical validation before acceptance.","tokens_in":21215,"tokens_out":15434,"duration_ms":144648,"concrete_test":"Run a small numerical check: choose random X (e.g., 4x4) and W with matching dimensions, set D=I (or D from Eq. 2), compute S = \\Sigma^{-1/2}U^T from \\tilde{X}^T\\tilde{X} = U\\Sigma U^T per Eq. (4), and evaluate M = S^{-1}\\tilde{X}\\tilde{X}^T(S^{-1})^T. If M is not the identity, the key whitening property fails. Then compute the true single-truncation loss L_i = ||\\sigma_i u_i v_i^T S^{-1}\\tilde{X}||_F for a few singular values of WS and compare to \\sigma_i; if they differ, Eq. (7) is false. This directly tests the derivation in Sec. 3.1.2 and settles whether the theoretical guarantee holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1.2's central guarantee is unsupported. The derivation of Eq. (7) requires the identity S^{-1} \\tilde{X} \\tilde{X}^T (S^{-1})^T = I. With S = \\Sigma^{-1/2} U^T from the eigendecomposition \\tilde{X}^T\\tilde{X} = U\\Sigma U^T, this identity does not hold in general. The standard whitening property for a matrix built from this eigendecomposition is (\\tilde{X}S)^T(\\tilde{X}S) = I, which requires S = U\\Sigma^{-1/2} rather than the paper's \\Sigma^{-1/2}U^T; even with that correction, the identity needed in Eq. (7) concerns the row Gram matrix of S^{-1}\\tilde{X}, which is a different object and is not forced to be the identity. Consequently, the equality L_i = \\sigma_i and the root-sum-square formula in Eq. (8) do not follow. The central claim that truncating the smallest singular values of the whitened weight matrix provably minimizes output loss is therefore unproven. The empirical method might still be effective, but the paper's theoretical foundation, the 'direct mapping between singular values and compression loss', is invalid as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DipSVD, an SVD-based LLM compression method with two protected components: (1) a local importance mechanism that reweights input channels and applies a data-whitening transformation before truncating singular values, and (2) a global importance mechanism that allocates layer-specific compression ratios via Bayesian optimization or a heuristic combining Fisher sensitivity and effective rank. The authors claim that channel-weighted whitening provides a direct, provable mapping between singular values and output loss, and they report consistent perplexity and zero-shot accuracy improvements over ASVD, FWSVD, and SVD-LLM across five models and several compression ratios.","tokens_in":21502,"tokens_out":8011,"duration_ms":85861,"significance":"If the theoretical guarantee in Sec. 3.1.2 were valid, the paper would make a useful contribution to SVD-based LLM compression by jointly addressing local and global importance. The experimental scope is broad: five models, three model families, multiple compression ratios, and both perplexity and zero-shot benchmarks, and the authors include ablations and pseudocode. However, the central proof is invalid as written, and the empirical validation does not currently separate genuine gains from calibration and tuning effects. The idea of channel-weighted whitening and layer-wise ratio allocation is plausible and may be worth pursuing, but the present manuscript does not establish its theoretical or empirical claims at the standard required for acceptance.","major_comments":[{"comment":"The derivation of L_i = sigma_i hinges on the whitening property S^{-1} \\tilde{X} \\tilde{X}^T (S^{-1})^T = I. With S = \\Sigma^{-1/2} U^T obtained from \\tilde{X}^T \\tilde{X} = U \\Sigma U^T, this identity is generally false; the standard column-whitening property is (\\tilde{X} U \\Sigma^{-1/2})^T (\\tilde{X} U \\Sigma^{-1/2}) = I, which is a different object and does not imply the row Gram identity used here. Moreover, Eq. (6) is dimensionally problematic: X in R^{m x n} and S in R^{n x n} make S^{-1} \\tilde{X} undefined when m != n. Therefore the equality L_i = sigma_i and the root-sum-square expression in Eq. (8) are unsupported, and the central claim that truncating the smallest singular values of the whitened weight matrix provably minimizes output loss is not established. The empirical method may remain useful as a heuristic, but the 'direct mapping' theoretical foundation stated in Sec. 3.1 is invalid as written.","section":"Sec. 3.1.2, Eq. (7)"},{"comment":"The empirical comparisons are reported without error bars, confidence intervals, or significance tests, and the hyperparameters a, bar, and beta appear to be selected using the same benchmark suites on which performance is reported (Figs. 4 and Sec. 4.3). In addition, calibration data are drawn from WikiText-2 while WikiText-2 perplexity is a headline result, and Fig. 5b shows that performance is best when calibration and test distributions match. As a result, the consistent gains over baselines cannot be distinguished from calibration or tuning artifacts. The paper should report variance over calibration seeds, use a disjoint validation split for hyperparameter selection, and include significance tests or effect sizes for the main tables.","section":"Sec. 4.2-4.4 and Tables 2-4"},{"comment":"The validation of the heuristic is circular as reported: beta is chosen because it maximizes the Pearson correlation in Table 5, and then that same table is presented as evidence that the heuristic agrees with Bayesian optimization. Moreover, the paper states beta = 0.25 in Sec. 4.3 but Algorithm 3 lists the default as 0.3, so the exact configuration used in the main experiments is ambiguous. The authors should specify a fixed beta chosen on a validation set and report the sensitivity of the correlation and downstream performance to beta.","section":"Sec. 4.3, Eq. (12), Algorithm 3"}],"minor_comments":[{"comment":"Equation (11) is ambiguous for H_l in R^{B x T x D}; please specify the reshaping and which matrix is decomposed (as done in Algorithm 5).","section":"Sec. 3.2.2, Eq. (11)"},{"comment":"The caption claims average accuracy of seven datasets, but the table shows only three perplexity columns and an 'Average' column; the zero-shot accuracy columns appear to be missing.","section":"Table 3"},{"comment":"The citation for C4 is given as (Mihaylov et al., 2018b), which is the OpenbookQA paper; a proper C4 reference is needed.","section":"Sec. 4.1, Benchmarks"},{"comment":"Several typos should be corrected: 'Beyesian' (Sec. 4.3), 'Sensity-based' (Table 1), 'languge' (Sec. 5), 'theoetical' and 'detemined' (Appendix B), 'varing' (Table 12), and 'th' (Appendix C.2).","section":"Throughout"},{"comment":"The notation k_l is used both as the target compression ratio and as the result of the arg min; please use distinct symbols for the target and the selected rank.","section":"Algorithm 1, line 7"},{"comment":"The line 'SetS <- S S SetS' appears garbled and does not match Eq. (4) for constructing S; please align the pseudocode with the main text.","section":"Algorithm 2, line 8"},{"comment":"Equation (12) is numbered twice, once in Sec. 3.2.2 and once in Sec. 3.3 Step 3, with different meanings; please renumber.","section":"Sec. 3.2.2 and Sec. 3.3"},{"comment":"With m = n = d, the optimal rank from k = 0.6mn/(m+n) is 0.3d, not 0.6d as stated; the final 40% FLOP reduction is correct, but the intermediate statement should be corrected.","section":"Appendix B"}],"recommendation":"reject","confidential_remarks":"The proof flaw in Sec. 3.1.2 is not a minor typo: the claimed whitening identity cannot hold for the S defined in Eq. (4), so the paper's central theoretical guarantee is unsupported. Even setting the theory aside, the empirical section would need substantial additional work (error bars, calibration disjointness, and non-circular hyperparameter selection) before the results could be considered convincing. I would be open to a resubmission if the authors either repair or remove the theoretical claim and substantially strengthen the evaluation, but I do not think the current version meets the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the empirical method is interesting but the paper's central theory doesn't hold. The whitening proof in Section 3.1.2 relies on an identity that is false, so the claimed guarantee 'truncating the smallest singular values minimizes output loss' is unproven. That's the main thing you need to know.\n\nWhat's genuinely new: the authors combine channel-weighted whitening with a global per-layer compression allocation. The local piece is a twist on SVD-LLM's truncation-aware whitening, and the global piece—using Fisher sensitivity and effective rank to set per-layer ratios—is a practical heuristic that works reasonably well. The experiments are broad: five models, compression ratios from 20% to 50%, and both perplexity and zero-shot accuracy. Results consistently favor DipSVD over ASVD, FWSVD, and SVD-LLM, with the biggest gains at high compression ratios. That is a useful outcome, if it reproduces.\n\nThe soft spot is load-bearing. In Eq. (7) they need S^{-1} \\tilde X \\tilde X^T (S^{-1})^T = I. With S = Sigma^{-1/2} U^T from the eigendecomposition of \\tilde X^T \\tilde X, that identity isn't true. The standard whitening property is \\hat X^T \\hat X = I for \\hat X = \\tilde X S, which is the column Gram matrix, not the row Gram. So L_i = sigma_i doesn't follow, and the root-sum-square formula is unjustified. The empirical method does not need this identity, but the paper's theoretical foundation—the direct mapping between singular values and compression loss—is invalid as stated. This is not a minor typo; it is the central claim.\n\nThere are also standard empirical concerns: hyperparameters a, p, and beta were tuned on the same benchmarks they're evaluated on (WikiText-2 and the zero-shot set), calibration data was drawn from WikiText-2, Fig. 5b shows best performance when calibration and test match, and there are no error bars or released code. These don't kill the method, but they weaken the evidence.\n\nWho should read this: anyone working on post-training SVD compression for LLMs. The heuristic for layer-wise ratios is worth borrowing, and the empirical comparison is a useful data point. But I wouldn't cite the theoretical guarantee.\n\nRecommendation: send to a serious referee. The empirical results are promising enough that the paper deserves a chance to be fixed; the proof can be corrected or removed, and the experiments can be rerun with proper held-out tuning. As it stands, the central claim is unsupported.","headline":"Empirical results look solid but the central whitening proof is wrong; needs major revision before acceptance.","tokens_in":22034,"tokens_out":4342,"would_cite":false,"duration_ms":41732,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that SVD-based compression of large language models can preserve far more performance by protecting both important channels within each weight matrix and important layers across the network than existing SVD…","keywords":["LLM compression","singular value decomposition","low-rank factorization","channel-weighted whitening","layer-wise compression ratio","Fisher sensitivity","effective rank","post-training compression"],"falsifier":"Compute, on real calibration data, the matrix $S^{-1}\\tilde{X}\\tilde{X}^T(S^{-1})^T$ for any compressed layer and compare it with the identity $I$; if it is not $I$, measure the actual output perturbation caused by deleting the $i$-th singular vector and compare it with $\\sigma_i$. If the perturbation is materially different from $\\sigma_i$, the stated theoretical guarantee does not hold even if the method still works in practice.","tokens_in":21020,"feed_emoji":"🧠","tokens_out":9783,"duration_ms":94852,"temperature":0.7,"pith_summary":"This paper tries to establish that SVD-based compression of large language models can be made substantially less damaging by explicitly protecting important parts of the model at two levels. Locally, it amplifies the channels that most strongly shape the input's second-order statistics before whitening the weight matrix, so that truncation discards the singular vectors that matter least. Globally, it assigns each Transformer layer its own compression ratio, pushing more of the burden onto layers with low Fisher sensitivity and low effective rank, either by a heuristic or by Bayesian optimization. The authors claim these two additions let compressed LLaMA, Vicuna, and DeepSeek models keep lower perplexity and higher zero-shot accuracy than FWSVD, ASVD, and SVD-LLM at the same compression ratios, with the gap growing as compression gets more aggressive. This matters because SVD compression is retraining-free and hardware-friendly, so improving its accuracy at high ratios directly lowers the cost of deploying large models.","feed_headline":"Protect key channels and layers and SVD compression hurts LLMs less","feed_subtitle":"On LLaMA, Vicuna, and DeepSeek, DipSVD beats three SVD baselines on perplexity and accuracy at 20-50% ratios.","key_machinery":"The machinery is channel-weighted data whitening coupled to truncated SVD of the whitened weight. Formally, with calibration activations $X$, channel importance is $\\alpha_j = \\sqrt{x_j^T (XX^T)x_j}$; a diagonal $D$ amplifies the top $p\\%$ channels by factor $a>1$; the reweighted covariance $\\tilde{X}^T\\tilde{X} = D^T X^T X D$ is decomposed as $U_{\\tilde{X}}\\Sigma_{\\tilde{X}}U_{\\tilde{X}}^T$, and the whitening matrix is $S = \\Sigma_{\\tilde{X}}^{-1/2}U_{\\tilde{X}}^T$. The paper then decomposes $WS$, truncates the smallest singular values, and reconstructs as $U\\,\\mathrm{Trunc}^*(\\Sigma)V^T S^{-1}$. The claimed effect is that truncation loss in output space equals the root-sum-square of the removed singular values, so singular values become direct proxies for performance; that is the local-protection claim. The global-protection half is the per-layer allocation law $p_l = \\frac{Q_l}{\\sum_j Q_j}\\cdot L\\cdot(1-k)$, with $Q_l = S_l^\\beta R_l^{1-\\beta}$ from Fisher sensitivity $S_l$ and effective rank $R_l$, or the Bayesian optimization objective that maximizes cosine similarity between original and compressed outputs under the global budget.","core_discovery":"On its own terms, DipSVD's discovery is that importance-weighted whitening plus layer-wise budget allocation turns truncated SVD into a usable high-ratio LLM compressor. For a weight matrix $W$ and activation data $X$, the method reweights channels by importance $\\alpha_j = \\sqrt{x_j^T(XX^T)x_j}$, amplifies the top channels with a diagonal matrix $D$, forms a whitening matrix $S$ from the SVD of $\\tilde{X}^T\\tilde{X}$ with $\\tilde{X}=XD$, then decomposes $WS$, keeps the largest singular values, and reconstructs with $S^{-1}$. The paper argues this whitening makes the output-space loss from truncating the $i$-th singular value equal to $\\sigma_i$, so the total loss is the root-sum-square of the discarded singular values; this is the claimed guarantee that small singular values are the safe ones to drop. At the network level, the method computes a per-layer importance score $Q_l$ combining Fisher sensitivity and effective rank, or searches directly with Bayesian optimization, and sets preservation ratios proportional to $Q_l$, so sensitive layers are compressed less. The empirical claim is that this dual protection beats all three prior SVD compressors on every tested model and compression ratio, and that the cheap heuristic tracks the Bayesian version well.","pith_inferences":["The importance weighting is defined purely from calibration activations, so the same channel-amplification scheme could be dropped into other low-rank or pruning pipelines that currently rely on plain SVD; the paper does not test this transfer.","The heuristic needs only gradients and hidden states on a small calibration set, so it is likely to scale to much larger or instruction-tuned models where Bayesian optimization over the full layer space would be too costly; this is an extrapolation, not a result in the paper.","A natural extension is replacing the fixed amplification factor $a$ with a per-layer or per-channel adaptive value, which the paper's limitation section itself flags as unexplored."],"forward_implications":["At tested compression ratios from 20% to 50%, compressed LLaMA-7B/13B, Vicuna-7B/13B, and DeepSeek-7B show lower perplexity and higher zero-shot accuracy than FWSVD, ASVD, and SVD-LLM under identical conditions.","The gap over baselines widens as the compression ratio rises, so the method's advantage is largest exactly where SVD compression is otherwise least usable.","Both protection components are independently useful: the ablation shows each one alone beats SVD-LLM, while the full combination gives the best results.","The heuristic allocation reproduces the Bayesian optimization allocation with Pearson correlation above 0.64 across ratios, so the cheap method can substitute for expensive search.","The low-rank form $\\hat{X} = \\Sigma_k V_k^T X$ followed by $U_k \\hat{X}$ yields a parameter-budget-derived FLOP reduction, about 40% at 40% compression on Vicuna-7B, which translates into throughput and latency gains."],"supporting_citations":[{"why":"The Fisher-weighted SVD baseline whose local importance handling DipSVD extends and must outperform.","marker":"Hsu et al., 2022"},{"why":"The activation-aware SVD baseline that already assigns layer-wise compression ratios, providing the global-importance comparison point.","marker":"Yuan et al., 2023"},{"why":"The truncation-aware whitening SVD compressor that DipSVD most directly builds on and the strongest baseline in the experiments.","marker":"Wang et al., 2024b"}],"fun_headline_variants":["DipSVD: Protect key singular values, compress LLMs more","Dual-importance SVD: High-ratio LLM compression that works","Importance-aware SVD: Compress LLMs 20-50% with less loss","DipSVD: Whitening + layer protection for leaner LLMs","SVD that protects important parts: DipSVD high-ratio win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof that truncating singular value $\\sigma_i$ costs exactly $\\sigma_i$ in output space assumes the whitening operation makes calibration data satisfy a specific algebraic identity, but the construction as written guarantees a different, related identity, so the exact equality is a premise the paper does not actually deliver.","fun_headline_variants_meta":{"raw":{"variants":["DipSVD: Protect key singular values, compress LLMs more","Dual-importance SVD: High-ratio LLM compression that works","Importance-aware SVD: Compress LLMs 20-50% with less loss","DipSVD: Whitening + layer protection for leaner LLMs","SVD that protects important parts: DipSVD high-ratio win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001096,"raw_usage":{"total_tokens":4601,"prompt_tokens":994,"completion_tokens":3607,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":3507}},"tokens_in":610,"tokens_out":3607,"duration_ms":25110,"temperature":1.0,"reasoning_tokens":3507,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:50:32.138400+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute, on real calibration data, the matrix $S^{-1}\\tilde{X}\\tilde{X}^T(S^{-1})^T$ for any compressed layer and compare it with the identity $I$; if it is not $I$, measure the actual output perturbation caused by deleting the $i$-th singular vector and compare it with $\\sigma_i$. If the perturbation is materially different from $\\sigma_i$, the stated theoretical guarantee does not hold even if the method still works in practice.","supporting_citations":[],"review_version":1}