{"id":"5345de5d-e9ec-4c3a-86f7-f25fa19160a3","arxiv_id":"2505.17974","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GFWSVD compresses LLMs by weighting SVD with a Kronecker-factored Fisher information matrix, and reports accuracy gains over FWSVD, ASVD, and SVD-LLM at high compression rates.","lead":"GFWSVD is a new method for compressing large language models that uses a Kronecker-factorized Fisher information matrix to weight singular value decomposition, capturing parameter correlations that diagonal Fisher methods miss. If the reported results hold, it offers a better accuracy-to-size trade-off at aggressive compression rates.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rank-1 Kronecker factorization of the empirical Fisher (Theorem 1, Cond. 2) is an acknowledged idealization; without a measure of its approximation error, the claimed optimality and gains over diagonal-Fisher baselines are unsupported.","rationale":"The central contribution is a weighted-SVD compression rule proven optimal when the empirical Fisher is exactly Kronecker-factored. The method's actual computation yields only the best rank-1 Kronecker approximation (Eq. 12), so the proof's Condition 2 is not verified. The authors themselves flag this in Section 6. Without a quantitative assessment of the approximation error, the claimed 'optimal' weighting is not established. This is load-bearing because the paper's headline advantage over FWSVD is attributed to exploiting off-diagonal Fisher structure; if the rank-1 approximation is far from the true FIM, that structure is not faithfully encoded and the advantage could be an artifact of the compression schedule. The reader's weakest_assumption identifies exactly this. I additionally note that the fast matrix-vector product (Eq. 15) is dimensionally inconsistent, which independently requires correction; this reinforces, rather than replaces, the rank-1 concern. The empirical results may still hold, but the theoretical and algorithmic descriptions need stronger support, so the CONDITIONAL verdict is appropriate.","tokens_in":14379,"tokens_out":18951,"duration_ms":145097,"concrete_test":"Compute the relative Frobenius-norm error ||IF − A⊗B||_F / ||IF||_F for several LLaMA-2-7B linear layers using the paper's 1024-batch FineWeb calibration and Algorithm 1 (with either interpretation of the A/B assignment). If the median error exceeds ~0.3, Theorem 1's Condition 2 is badly violated; the theoretical optimality claim is void and the empirical gains over diagonal Fisher need an alternative explanation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 1's optimality guarantee requires the empirical Fisher to be exactly a single Kronecker product, IF = A⊗B (Condition 2). In practice the method solves Eq. 12, a rank-1 approximation, and Algorithm 1 returns the leading Kronecker factor. The paper's own Limitations (Section 6) concede that this 'assumes exact factorization, which may not hold in practice,' yet no experiment quantifies the approximation error. If ||IF − A⊗B||_F is not small relative to ||IF||_F, the weighted objective in Eq. 9 is not the true Fisher-weighted loss, so the truncated SVD of L_B^T W* L_A is optimal for the wrong problem. The reported gains over FWSVD then have no theoretical support and may instead follow from the asymmetric rank-allocation protocol (ASVD-style per-layer scores for GFWSVD, unspecified allocation for FWSVD). A further, concrete inconsistency is that the fast matrix-vector formulas (Eqs. 15 and 30) are dimensionally wrong as written: with G_i ∈ R^{n×m}, (G_i⊗G_i) vec(Z) = vec(G_i Z G_i^T) requires Z ∈ R^{m×m}, and Algorithm 1 assigns u and v to the wrong factors. These errors must be corrected before the scalable algorithm can be reproduced.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GFWSVD, a post-training low-rank compression method for transformer weight matrices that replaces the diagonal Fisher approximation of FWSVD with a Kronecker-factored empirical Fisher information matrix. The central theoretical result (Theorem 1) shows that, under an MLE loss, an exact Kronecker-structured Fisher matrix, and matrix-variate normal weights, the optimal rank-r reconstruction is obtained by a whitened SVD of the weight matrix using Cholesky factors of the Kronecker factors. To make the method tractable, the authors derive a fast matrix-vector product for the permuted Fisher matrix and report experiments on BERT (GLUE) and LLaMA-2-7B (WikiText-2, PTB, MMLU), claiming consistent improvements over FWSVD, ASVD, and SVD-LLM. The paper also proves that FWSVD is a special case of GFWSVD when the Fisher approximation is taken to be I_m ⊗ D.","tokens_in":14729,"tokens_out":7016,"duration_ms":52328,"significance":"The conceptual contribution is valuable: connecting Kronecker-factored Fisher information to weighted SVD for compression, and showing that FWSVD is a limiting case, is a clean and publishable idea. The proof of Theorem 1 is straightforward and correct under its explicit idealizations, and the reduction of Kronecker decomposition to fast matrix-matrix products is the right approach for scalability. The empirical evaluation covers both encoder and decoder models and multiple downstream tasks, which is appropriate for the claimed breadth. However, the significance is conditional on fixing the dimensional inconsistencies in the core algorithm and on demonstrating that the empirical gains are not artifacts of the rank-allocation protocol or of the idealized Kronecker assumption.","major_comments":[{"comment":"The dimensions of the permuted Fisher matrix are inconsistent across the paper. Eq. (13) defines \\tilde I_F = (1/|D|) Σ G_i⊗G_i with G_i ∈ R^{n×m}, so \\tilde I_F ∈ R^{n^2×m^2}, but Section 4.2 states that the matrix \\tilde J is in R^{m^2×n^2}; Algorithm 1 then reshapes the length-n^2 left singular vector b into B ∈ R^{m×m} and the length-m^2 right singular vector a into A ∈ R^{n×n}, which is possible only when n = m. This must be corrected and the derivation of the permuted-Fisher identity made consistent before the scalable algorithm can be reproduced.","section":"Section 4, Algorithm 1, Eqs. (13) and (15)"},{"comment":"The Kronecker-product identity stated before Eq. (15) is incorrect. The correct identity is (K⊗L) vec(C) = vec(L C K^T), so (G_i⊗G_i) vec(Z) equals vec(G_i Z G_i^T) with Z ∈ R^{m×m}, not vec(G_i^T Z G_i) with the Z ∈ R^{n×n} introduced in Eq. (14). Correspondingly, the right multiplication formula in Eq. (30) should be Σ vec(G_i^T Z G_i), not Σ vec(G_i Z G_i^T). As written, both equations are dimensionally invalid for rectangular weight matrices and would prevent a reader from implementing the proposed fast algorithm.","section":"Section 4.1, Eqs. (14)–(15) and Eq. (30)"},{"comment":"Theorem 1's optimality guarantee requires Condition 2, the exact Kronecker structure I_F = A⊗B, but the method in practice solves Eq. (12), a rank-1 Kronecker approximation, and the Limitations section concedes that 'this assumes exact factorization, which may not hold in practice.' The paper provides no experiment measuring the approximation error, e.g., ||I_F − A⊗B||_F/||I_F||_F or a spectral proxy, so the reported gains over diagonal-Fisher baselines have no direct theoretical support under the actual approximation error.","section":"Theorem 1, Eq. (12), and Section 6"},{"comment":"The LLaMA comparison is potentially confounded by an asymmetry in rank allocation. For GFWSVD the authors state that they adopted the ASVD per-layer importance scores, but for the FWSVD baseline no per-layer rank-allocation protocol is described; if FWSVD used uniform ranks or another fixed schedule while GFWSVD used a sensitivity-based schedule, the consistent improvements in Table 3 may be due to the allocation scheme rather than to the Kronecker-Fisher weighting. The paper should specify the allocation for all methods or run an ablation with a common allocation protocol.","section":"Section 5.2 and Table 3"}],"minor_comments":[{"comment":"The sentence 'at a 20 compression rate on the MMLU benchmark, our method outperforms FWSVD ... by 5 percent, SVD-LLM by 3 percent, and ASVD by 6 percent compression rate' is grammatically garbled; it should refer to 'a 20% compression rate' and report accuracy differences (percentage points), not 'compression rate' as the outcome.","section":"Abstract"},{"comment":"The text says 'Table 2 reports the empirical decomposition times for \\tilde I_F', but Table 2 in the manuscript is the GLUE performance table; the reference should point to Figure 2 itself or to a separate table of runtime results.","section":"Section 5.1, Figure 2"},{"comment":"The code link is a placeholder ('GitHub: link'); the final version should provide the actual repository URL and ideally a version tag for reproducibility.","section":"Section 5"},{"comment":"The sentence 'define G_i ∈ R^{n×m} as a weight gradients L(θ)|_{θ=W} on the i-th batch' has a grammatical error and imprecise notation; it should say 'as the gradients of L(θ) with respect to θ evaluated at θ = W'.","section":"Section 4, first paragraph"},{"comment":"Condition 3 states W ~ MN(W*, B^{-1}, A^{-1}), but the proof and Eq. (9) use I_F = A⊗B with A = L_A L_A^T and B = L_B L_B^T; the ordering of the Kronecker factors relative to the MVN covariance convention in Section 3.2 is easy to mix up and should be clarified with a short note that the algebra is invariant to the exchange.","section":"Section 3.2 and Theorem 1"}],"recommendation":"major_revision","confidential_remarks":"The paper has a clean conceptual contribution but the dimensional errors in the core algorithm (Section 4, Algorithm 1, Eqs. 15 and 30) are load-bearing and must be fixed before the method can be independently reproduced. The empirical claim also needs a matched rank-allocation protocol for the baselines and ideally a measurement of the Kronecker approximation error. These are fixable within the scope of the manuscript, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a look but not yet trustworthy as a benchmark contribution. What is actually new is the combination: feed a Kronecker-factored empirical Fisher into a generalized SVD for post-training LLM compression, with an argument that FWSVD becomes the diagonal special case. The theory is standard generalized least-squares SVD, correctly applied under the stated idealizations. Theorem 1 is fine given the assumptions. The scalable Lanczos-based rank-1 SVD is also the right move, and the claimed cubic-instead-of-quartic runtime is plausible.\n\nThe soft spots are real and partly serious. The stress-test note is right about the dimensions. With G \\in R^{n\\times m}, (G\\otimes G)\\mathrm{vec}(Z) = \\mathrm{vec}(G Z G^T) requires Z \\in R^{m\\times n}, not n\\times n as written in Eq. 15. Eq. 30 has the same problem. That is a concrete reproduction bug. The factor assignment in Algorithm 1 may also be swapped; I would want that checked.\n\nBigger picture, the LLaMA experiments compare GFWSVD (which gets ASVD-style per-layer rank allocation) against an FWSVD baseline that gets no such allocation. That alone can explain much of the gain, so the central empirical claim is not yet established. The paper itself concedes the rank-1 Kronecker exactness is an idealization, but it never measures the approximation error, so we do not know how often the method is optimizing the wrong objective. The code link is a placeholder ('link'), and there are no real error bars over seeds.\n\nWho is this for? People working on post-training compression should know the idea, but they should not build on the numbers yet. It deserves a serious referee because the direction is sensible and the theoretical derivation is mostly standard and correct. The revisions are not cosmetic: fix the formulas, release code, and run FWSVD with the same per-layer allocation. If those land, I would take the empirical claims seriously.","headline":"A sound idea in need of serious cleanup: Kronecker-factored Fisher weighting for LLM compression is a plausible extension of FWSVD, but the empirical comparison is undercut by an unfair baseline and the fast multiplication formulas are dimensionally wrong as written.","tokens_in":15228,"tokens_out":2994,"would_cite":false,"duration_ms":26955,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Kronecker-factored Fisher matrix, not its diagonal, gives optimal rank-r SVD compression of a layer under maximum-likelihood loss.","keywords":["Kronecker-factored Fisher information","low-rank compression","large language models","Fisher-weighted SVD","post-training compression","generalized SVD","off-diagonal parameter correlations","LLM compression"],"falsifier":"Measure the normalized Frobenius residual $\\|I_F - A\\otimes B\\|_F / \\|I_F\\|_F$ of the Kronecker fit on the actual calibration gradients for every compressed layer. If GFWSVD still shows its reported gains on layers where this residual is large, then the theorem's Kronecker premise is not what drives the improvement; alternatively, run GFWSVD against FWSVD on a synthetic weight matrix with a deliberately non-Kronecker Fisher matrix, since the theorem predicts GFWSVD should win only in the Kronecker case.","tokens_in":14212,"feed_emoji":"✂️","tokens_out":17107,"duration_ms":120036,"temperature":0.7,"pith_summary":"Large-model compression by SVD normally decides which parameter directions matter from the diagonal of the Fisher information matrix, which treats each weight independently. This paper proposes to factor the full Fisher information into a Kronecker product of two smaller matrices and to use the Cholesky factors of those matrices as reweighting transformations on both sides of the SVD. Its central theorem states that when the loss is maximum-likelihood and the Fisher information really is such a Kronecker product, the resulting generalized SVD gives the optimal rank-r compression. The authors add a scalable algorithm that computes the two factors from gradients, reducing the per-layer cost from quartic to cubic in the weight-matrix dimension, and report consistent gains over diagonal-Fisher and activation-based baselines on GLUE, WikiText-2, PTB, and MMLU, with the largest improvements at aggressive compression rates.","feed_headline":"Full Fisher matrix wins in LLM compression by up to 6 points","feed_subtitle":"Factoring the Fisher matrix captures correlations diagonal methods miss, preserving more accuracy at low ranks.","key_machinery":"The load-bearing object is the rank-1 Kronecker product approximation of the observed Fisher information, $I_F \\approx A\\otimes B$, computed from per-batch gradient matrices $G_i$. Its Cholesky factors $L_A$ and $L_B$ define the transformation $\\widehat W_r = L_B^{-T} \\tilde W_r L_A^{-1}$, in which the truncated SVD is taken of $\\tilde W = L_B^T W^* L_A$; the inverse Cholesky factors act as sensitivity reweighting maps that make an ordinary SVD optimal for the Fisher-weighted objective. To obtain $A$ and $B$ cheaply, the algorithm forms the permuted Fisher matrix $\\tilde I_F = \\sum_i G_i \\otimes G_i$ and computes its leading singular triplet with Lanczos iteration, implementing each left product as $\\sum_i \\mathrm{vec}(G_i^\\top Z G_i)$, so the dominant cost is cubic in the weight-matrix dimension rather than quartic.","core_discovery":"Under the paper's three conditions - MLE loss, empirical Fisher information of the form $I_F = A\\otimes B$, and weights drawn from a matrix-variate normal centered at the optimal weights - the best rank-r compression of a layer weight matrix $W^*$ is $\\widehat W_r = L_B^{-T} \\tilde W_r L_A^{-1}$, where $\\tilde W_r$ is the truncated SVD of $\\tilde W = L_B^T W^* L_A$ and $A = L_A L_A^T$, $B = L_B L_B^T$ are Cholesky factorizations. In other words, the Fisher-weighted truncation problem becomes an ordinary SVD after a change of coordinates defined by the Cholesky factors. The paper proves that the diagonal-Fisher method FWSVD is the special case where one factor is diagonal and the other is identity, and it gives a Kronecker-factorization algorithm whose matrix-vector products are evaluated as sums of gradient-matrix multiplications, reducing the per-layer cost from $O(m^2 n^2)$ to $O(m n^2 + m^2 n)$. Empirical results on BERT and LLaMA-2 show that this method matches or beats SVD, FWSVD, ASVD, and SVD-LLM on most settings, with the clearest margins at low ranks.","pith_inferences":["A natural extension the paper does not test would be to replace the single Kronecker product with a sum of several Kronecker products; the optimal solution would then involve a coupled SVD rather than a single truncation.","The same Cholesky-reweighted coordinates could plausibly carry over to quantization or pruning, since they rank directions by Fisher sensitivity rather than by Euclidean length.","The reported gains may depend on the calibration data; comparing GFWSVD against FWSVD with different calibration sample sizes would isolate the value of the off-diagonal signal.","Because the theorem covers a single linear layer, the multi-layer transformer results rest on a heuristic extension; modeling cross-layer dependencies, which the paper names as a limitation, is the most direct route to further gains."],"forward_implications":["Within the theorem's scope, no other rank-r factorization of a layer can beat the Cholesky-SVD construction, so the method is optimal rather than heuristic for a single linear layer.","FWSVD is recovered as the diagonal limit of the same construction, so any empirical gain of GFWSVD over FWSVD is attributable specifically to off-diagonal parameter correlations.","The cubic-time factorization brings full-Fisher compression into the regime of modern transformer layers with dimensions on the order of $10^3$.","The method needs only gradients from calibration data, so it applies post-training without retraining, and in the LLM experiments it is combined with per-layer importance-based rank allocation.","At the most aggressive tested compression (20 percent of parameters removed), GFWSVD is the only method that keeps MMLU accuracy at 0.32 and PTB perplexity at 50.5, while the diagonal-Fisher baseline collapses to 1523 perplexity, showing that off-diagonal sensitivity matters most when capacity is scarce."],"supporting_citations":[{"why":"supplies the diagonal-Fisher FWSVD baseline and the weighted low-rank objective that the paper proves is a special case of its construction.","marker":"Hsu et al. [2022]"},{"why":"supplies the activation-aware ASVD baseline and the per-layer importance scoring scheme adopted for the LLM compression setup.","marker":"Yuan et al. [2023]"},{"why":"supplies the truncation-aware SVD-LLM baseline compared on WikiText-2, PTB, and MMLU.","marker":"Wang et al. [2025b]"},{"why":"introduced Kronecker-factored Fisher approximations, the structural idea the algorithm adapts to post-training compression.","marker":"Grosse and Martens [2016]"},{"why":"provides the rank-1 SVD of the permuted matrix that computes the Kronecker factors in Algorithm 1.","marker":"Loan and Pitsianis [1992]"},{"why":"supplies the matrix-variate normal formalism used in the theorem's distributional assumption.","marker":"Gupta and Nagar [2018]"},{"why":"gives the generalized least-squares matrix decomposition whose GSVD solution is used in the proof of optimality.","marker":"Allen et al. [2014]"}],"fun_headline_variants":["Fisher correlations boost LLM compression by 6 points","Beyond diagonal Fisher: LLM compression gains 6 points","Low-rank LLM compression gets 6-point boost from Fisher correlations","Kronecker-factored Fisher sharpens LLM compression by 6 points","Full Fisher matrix outperforms diagonal SVD by 6 points in LLM compression"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each layer's full Fisher information matrix is exactly a single Kronecker product of a row-factor and a column-factor, with no leftover correlations; the paper's own limitation section says this exact factorization may not hold in practice, and if it fails the reweighting is optimal for the wrong objective.","fun_headline_variants_meta":{"raw":{"variants":["Fisher correlations boost LLM compression by 6 points","Beyond diagonal Fisher: LLM compression gains 6 points","Low-rank LLM compression gets 6-point boost from Fisher correlations","Kronecker-factored Fisher sharpens LLM compression by 6 points","Full Fisher matrix outperforms diagonal SVD by 6 points in LLM compression"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2803,"prompt_tokens":1027,"completion_tokens":1776,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":1684}},"tokens_in":643,"tokens_out":1776,"duration_ms":10689,"temperature":1.0,"reasoning_tokens":1684,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:37:46.847140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the normalized Frobenius residual $\\|I_F - A\\otimes B\\|_F / \\|I_F\\|_F$ of the Kronecker fit on the actual calibration gradients for every compressed layer. If GFWSVD still shows its reported gains on layers where this residual is large, then the theorem's Kronecker premise is not what drives the improvement; alternatively, run GFWSVD against FWSVD on a synthetic weight matrix with a deliberately non-Kronecker Fisher matrix, since the theorem predicts GFWSVD should win only in the Kronecker case.","supporting_citations":[{"cited_title":"Language model compression with weighted low-rank factorization, 2022","cited_arxiv_id":null,"evidence_quote":"supplies the diagonal-Fisher FWSVD baseline and the weighted low-rank objective that the paper proves is a special case of its construction."},{"cited_title":"A kronecker-factored approximate fisher matrix for convolution layers","cited_arxiv_id":null,"evidence_quote":"introduced Kronecker-factored Fisher approximations, the structural idea the algorithm adapts to post-training compression."},{"cited_title":"Approximation with kronecker products","cited_arxiv_id":null,"evidence_quote":"provides the rank-1 SVD of the permuted matrix that computes the Kronecker factors in Algorithm 1."},{"cited_title":"Matrix variate distributions","cited_arxiv_id":null,"evidence_quote":"supplies the matrix-variate normal formalism used in the theorem's distributional assumption."},{"cited_title":"A generalized least-square matrix decomposition","cited_arxiv_id":null,"evidence_quote":"gives the generalized least-squares matrix decomposition whose GSVD solution is used in the proof of optimality."}],"review_version":1}