{"id":"6bc50b80-28bc-4558-b879-dd78659c5c47","arxiv_id":"2411.17426","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Attention pairs (Q-K and V-O) are SVD-decomposed so pruning or fine-tuning touches only a small singular-factor matrix, yielding better pruning tolerance and small PEFT gains.","lead":"CLOVER rewrites the combined Query-Key and Value-Output matrices inside each attention head as orthogonal building blocks, then either deletes unimportant blocks or fine-tunes a small middle matrix. The authors report that this makes pruning far gentler on models like GPT-2 XL and gives small accuracy gains over LoRA-style fine-tuning on common reasoning benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CLOVER fine-tuning on RoPE models replaces Q-K cross-layer SVD with intra-layer Key SVD and a dense d-by-d trainable matrix; correctness of the headline full-rank cross-layer mechanism is untested on main experiments.","rationale":"The reader's rejection centers on the RoPE incompatibility, and I agree that this is the load-bearing weakness: the paper's own Section 5 admits the cross-layer Q-K decomposition 'is not feasible' with RoPE, and Section 4.2 explicitly substitutes intra-layer Key decomposition for all LLaMA fine-tuning. The strongest reported results (Table 2) therefore exercise a different mechanism than the one derived in Section 3. The dense trainable 'S' is a second, related weakness: a d-by-d trainable matrix is not a diagonal singular-value matrix, and claiming full-rank updates from tuning such a matrix is at best imprecise. However, I do not think the correct disposition is an outright reject. There is credible empirical content: GPT-2 XL pruning results in Table 1 show large perplexity improvements, the V-O and MLP-block results are still tested on LLaMA, and the method does reduce to a well-defined intra-layer orthogonalization plus dense transition-matrix PEFT. That weaker method may well be competitive, but the paper as written does not isolate it from the cross-layer story. A conditional acceptance requiring the control experiment (or an honest re-labeling of the method as intra-layer orthogonalization with dense transition fine-tuning, plus a RoPE-free cross-layer demonstration) is the most constructive verdict. My attack is deliberately narrower than the reader's broader case: the single most load-bearing concern is not the existence of RoPE itself, but that the paper's main fine-tuning experiments never run the cross-layer Q-K mechanism that the title and Sections 1 and 3 advertise as the contribution, while the implementation details blur the distinction between diagonal singular values and a dense matrix.","tokens_in":19455,"tokens_out":1893,"duration_ms":16342,"concrete_test":"Re-run the LLaMA-2-7B commonsense fine-tuning with the advertised cross-layer decomposition replaced by an explicit control: (a) implement CLOVER exactly as in Section 4.2, using intra-layer Key SVD with dense d-by-d transition matrix; (b) implement a variant that applies the cross-layer Q-K SVD to a RoPE-free model (e.g., GPT-2 XL or a BLOOM-style ALiBi model) on the same tasks. If the cross-layer variant does not match or beat the Key-SVD variant while using equal parameters, the paper's claimed mechanism, rather than merely per-layer orthogonal initialization plus a trainable dense matrix, is not responsible for the reported gains. Additionally, report the singular value spectrum of the trained dense S to check whether it remains approximately diagonal and low-rank as claimed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that treating W_QK = W_Q W_K^T and W_VO as low-rank cross-layer products, orthogonalizing via SVD, then fine-tuning only the singular values yields effective pruning and full-rank PEFT. Section 5 concedes the cross-layer Q-K factorization is invalid under RoPE ('When nonlinear operations are present between Q-K or V-O pairs (such as with the widely-used RoPE), cross-layer orthogonalization is not feasible'), and Section 4.2 states that for all LLaMA fine-tuning experiments the authors instead 'perform orthogonal decomposition in the Key layer and fine-tune the transition matrix.' Since LLaMA-2-7B and the other benchmarked LLMs use RoPE, the headline fine-tuning mechanism (updating S in W_QK = U S V^T) is never actually exercised in the main experiments. Moreover, the fine-tuning implementation is described as tuning a dense d-by-d 'transition matrix' or 'singular value matrix', not a diagonal S, and Appendix A.2 counts 32x128x128 trainable parameters per Q-K pair; a dense 128x128 matrix has rank up to 128, so the claim that CLOVER learns 'singular values' or that its advantage over LoRA comes from full-rank updates along fixed orthogonal bases is not supported by the described implementation. The GPT-2 XL pruning results and V-O/MLP-block fine-tuning may validate a weaker intra-layer orthogonalization claim, but they do not validate the cross-layer Q-K mechanism as advertised.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLOVER, which applies SVD to the merged per-head matrices W_Q W_K^T and W_V W_O^T, obtaining orthogonal bases and singular values that can guide structured pruning or serve as trainable parameters for parameter-efficient fine-tuning. The authors report large pruning gains on GPT-2 XL and Whisper-Large-v3, and fine-tuning gains over LoRA, DoRA, HiRA, and PiSSA on commonsense reasoning benchmarks for LLaMA-family models. The SVD factorization in Section 3 is mathematically valid, but the main fine-tuning experiments on RoPE-based LLaMA models do not use the cross-layer Q-K decomposition; instead, Section 4.2 switches to an intra-layer Key decomposition with a dense transition matrix. The paper therefore does not test its headline cross-layer fine-tuning mechanism, and the implemented parameterization differs from the derived singular-value tuning.","tokens_in":19731,"tokens_out":6367,"duration_ms":63777,"significance":"If the cross-layer mechanism worked as claimed, it would be a useful contribution to attention-layer compression and parameter-efficient fine-tuning, and the paper covers a broad set of models. The SVD identity in Section 3 is correct and the GPT-2 XL pruning results are a concrete positive data point. However, the significance is substantially weakened because the central fine-tuning claim is not exercised in the main benchmarks: the LLaMA experiments use RoPE, where the paper itself concedes cross-layer Q-K orthogonalization is infeasible, and the trainable middle factor is implemented as a dense matrix rather than as singular values. The paper may contain a viable weaker contribution about intra-layer orthogonalization and pruning, but that is not what the title, abstract, and Section 1 promise.","major_comments":[{"comment":"The paper's own limitation paragraph (Section 5) states that cross-layer orthogonalization of Q-K is not feasible when RoPE is present, and Section 4.2 says that for all LLaMA fine-tuning experiments the authors 'perform orthogonal decomposition in the Key layer and fine-tune the transition matrix' instead of decomposing W_Q W_K^T. Since LLaMA-2-7B and the other fine-tuned models use RoPE, Table 2 never exercises the cross-layer Q-K mechanism that the title, abstract, and Section 3 advertise. The results may support a weaker intra-layer orthogonalization claim, but they do not support the paper's central fine-tuning claim.","section":"Section 4.2 and Section 5"},{"comment":"The method described in Section 3 freezes U and V and tunes only the diagonal singular-value matrix S, whose nonzero size is at most d by d. Appendix A.2 counts 32 by 128 by 128 trainable parameters per Q-K pair, which is the size of a dense 128 by 128 matrix, not a diagonal S; Section 4.2 also calls the trainable object a 'transition matrix.' A dense matrix has rank up to 128 and does not correspond to scaling fixed orthogonal bases, so the claim that CLOVER learns 'singular values' and achieves full-rank updates through all orthogonal bases is not what the implementation actually does. This discrepancy changes the effective method relative to the derivation in Section 3.","section":"Section 3, Section 4.2, and Appendix A.2"},{"comment":"The fine-tuning comparison is not controlled: LoRA and DoRA numbers are taken from the DoRA paper, HiRA from its own paper, and PiSSA is trained with lr=2e-5 while CLOVER uses lr=1e-4; the authors also state that they omit validation every 80 iterations for PiSSA and CLOVER, unlike the DoRA protocol. Under these heterogeneous conditions, the reported gains over SOTA are not attributable to the method alone. A same-pipeline comparison with identical data, hyperparameter choices, and checkpoint selection is needed to support the superiority claims in Table 2 and the abstract.","section":"Section 4.2 and Table 2"},{"comment":"The abstract's quantitative claim that pruning 70% of the Q-K pairs with CLOVER gives perplexity similar to pruning just 8% with vanilla methods is not supported by Table 1: the closest entries are 75% CLOVER perplexity 187.4 versus 12.5% vanilla perplexity 33.76, which are not similar even on a log scale. Figure 1c uses a different 45-versus-5 vector comparison without the experimental details needed to verify it. The stated headline number should be reconciled with the table or removed.","section":"Abstract and Table 1"}],"minor_comments":[{"comment":"The paper repeatedly misspells CLOVER as 'CLVOER' in Section 5 and Appendix A.2; these typos should be corrected.","section":"Section 5 and Appendix A.2"},{"comment":"The citation for LLM-Adapters (Hu et al., 2023) used in Appendix A.3 is not included in the reference list, and the unpublished 'Anonymous, 2025' citation for HiRA makes it difficult to verify the reported baseline values.","section":"Section 4.2 and Appendix A.3"},{"comment":"The 'red dot' and 'red intersection point' mentioned in the caption and text of Figure 2 are not visible in the reproduced figure, so the claim that CLOVER's remaining vectors are consistently less important after that point is hard to check.","section":"Figure 2 and Section 4.3"},{"comment":"The pruning thresholds in Section 4.4 (||W_Q|| ||W_K|| <= 5e-3 and ||W_V|| ||W_O^T|| <= 6e-3) are presented without justification or sensitivity analysis, which weakens the training-free pruning claim for Whisper.","section":"Section 4.4"},{"comment":"The statement that parameters are reintegrated 'without increasing its parameter count' is imprecise for the fine-tuning setting, because a dense trainable transition matrix is stored during training before being merged.","section":"Abstract and Section 1"}],"recommendation":"reject","confidential_remarks":"The stress-test concern is well-founded: the main fine-tuning experiments evaluate a different mechanism from the one in the title and derivation, and the dense transition matrix contradicts the singular-value parameterization. I do not see a local fix; the paper would need substantial reframing and a new controlled experimental design to support the central claims. The pruning results alone may justify a narrower future submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a real idea, but the main fine-tuning experiments do not actually test it. The cross-layer SVD of W_Q W_K^T is a neat way to expose redundancy, and the pruning results on GPT-2 XL are genuinely impressive. But for LLaMA, which uses RoPE, the Q-K part of the method is abandoned: Section 4.2 says the authors instead perform orthogonal decomposition in the Key layer and fine-tune the transition matrix. So the advertised cross-layer Q-K fine-tuning mechanism is never validated in the model family that matters most. The paper's own limitations section admits this, which is honest, but it means the headline claim outruns the evidence.\n\nWhat is new: the cross-layer pairing of Q-K and V-O as rank-limited products, then tuning or pruning the middle factor, is not in the cited SVFT, PiSSA, LoRA, or DoRA work. The SVD identity is correct, and the parameter count analysis is transparent. The GPT-2 XL numbers are the strongest part: pruning 50% of attention parameters with CLOVER degrades perplexity far less than vanilla pruning, and fine-tuning recovers most of the gap. For non-RoPE models, this seems like a useful pruning initialization.\n\nThe soft spots are substantial. First, the trainable factor is described as singular values, but Appendix A.2 counts 32x128x128 parameters per Q-K pair, which is a dense 128x128 matrix per head, not a diagonal S. If that is right, the method is closer to a per-head full-rank adapter, and the 'fine-tuning singular values' story is misleading. The claimed 'full-rank update' then tells us little beyond the fact that a dense matrix has full rank. Second, the fine-tuning gains over PiSSA are 0.6-1%, and the experiments use a single learning rate without validation. Third, the Whisper pruning demo is one audio sample with no held-out WER. Fourth, pruning is compared only to vanilla L2-norm pruning, not to any state-of-the-art structured pruning method.\n\nOverall verdict: the pruning contribution is worth a serious look, but the fine-tuning claims need major revision. Either test the cross-layer Q-K mechanism on a non-RoPE LLM, or reframe the method as intra-layer orthogonal PEFT and drop the cross-layer fine-tuning language. The dense-matrix issue needs to be resolved before the 'singular value' claims can be trusted. I would send this to peer review, but the reviewers should demand code, multi-seed runs, and an honest split of cross-layer versus intra-layer results.\n\nFor a reading group, it is a good case study in overclaiming; I would bring it up for discussion. If I worked on orthogonal PEFT, I would cite the pruning results, not the fine-tuning claims.","headline":"A clever orthogonalization trick with solid non-RoPE pruning results, but the headline fine-tuning mechanism is not exercised on RoPE models and the trainable 'singular values' are likely dense matrices.","tokens_in":20337,"tokens_out":2856,"would_cite":true,"duration_ms":40273,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","15A18"],"pacs":[],"model":"deepseek-v4-flash","headline":"CLOVER decomposes attention heads' query–key and value–output products with one SVD each, making tiny singular directions prunable and turning singular values into a full-rank fine-tuning update that beats LoRA, DoRA, HiRA, and PiSSA.","keywords":["cross-layer orthogonalization","SVD-based pruning","attention head compression","parameter-efficient fine-tuning","full-rank update","singular-value adaptation","KV cache compression"],"falsifier":"Fine-tune LLaMA-2-7B with CLOVER, PiSSA, and LoRA using identical validation-selected checkpoints and matched learning-rate sweeps, then compute the singular-value spectrum of each method's merged update: if CLOVER's accuracy lead over PiSSA disappears, or its update's effective rank is far below the head dimension, the full-rank orthogonalization mechanism is not what produces the reported gains.","tokens_in":19194,"feed_emoji":"✂️","tokens_out":18313,"duration_ms":141753,"temperature":0.7,"pith_summary":"CLOVER claims that the two matrix pairs inside every attention head — query against key, value against output — act as one low-rank bilinear form each, so a single truncated singular value decomposition (SVD) per pair can orthogonalize the head without adding any parameters. Once orthogonalized, most of the head's behavior sits in a few singular directions and the rest is nearly zero, which makes pruning those leftover directions nearly free: the paper reports that pruning 70% of the query–key vectors in GPT-2 XL degrades perplexity about as much as an 8% vanilla prune. The same singular values are also the only trainable parameters during fine-tuning, and because tuning them rescales every basis direction, the paper measures the resulting update to be full-rank, in contrast with LoRA's low-rank updates; on eight commonsense tasks this beats LoRA, DoRA, HiRA, and PiSSA on LLaMA-7B/13B, LLaMA-2-7B, and LLaMA-3-8B at equal or fewer parameters. A sympathetic reader would care because compression and adaptation are usually separate toolchains, and CLOVER offers one orthogonalization that feeds both, with no inference-time overhead.","feed_headline":"Pruning 70% of Q–K pairs now hurts as much as pruning 8%","feed_subtitle":"Per-head SVD turns attention redundancy into prunable directions and singular values into full-rank adapters.","key_machinery":"The load-bearing object is the merged per-head bilinear form $W^h_{QK}=W^h_Q (W^h_K)^\\top\\in\\mathbb{R}^{D\\times D}$, whose rank is bounded by the head dimension $d$, together with its truncated SVD $W^h_{QK}=U^h S^h (V^h)^\\top$ keeping the top $r\\le d$ singular triples; the same construction on $W^h_{VO}=W^h_V W^h_O$ covers the value–output pair, and the MLP Up layer is handled by treating blocks of 64 consecutive dimensions as heads. The mechanism that carries the argument is treating the Q and K projections as each other's transformation matrices, so orthogonalizing the product creates no extra matrices — the factors simply take the place of the original weights. The singular-value diagonal $S^h$ is then a single small object doing double duty: its magnitudes rank the directions for pruning, and its entries are the only trainable scalars during fine-tuning, giving a per-head parameter count of a dense $d\\times d$ diagonal comparable to a rank-32 LoRA. The claimed consequence that makes fine-tuning strong is that updating $S$ rescales every basis direction simultaneously, so the induced update is not confined to a low-rank subspace, and the paper's measured singular-value spectra show it matching full fine-tuning's rank.","core_discovery":"The paper's central claim is that crossing the layers before decomposing is what makes an attention head compressible without extra machinery. Instead of factorizing each projection matrix individually — the route taken by SVFT, which needs an accompanying transformation matrix and doubles the parameter count — CLOVER forms the merged products $W^h_{QK}=W^h_Q (W^h_K)^\\top$ and $W^h_{VO}=W^h_V W^h_O$ inside each head, each a $D\\times D$ matrix of rank at most the head dimension $d$, and truncates their SVDs to the top $r\\le d$ singular triples. The orthogonal factors $U$ and $V$ replace the original weight slices, so the orthogonalization costs no extra parameters, and the diagonal $S$ is the only new object. In pruning mode, small singular values (and their vectors) are dropped; in fine-tuning mode, $U$ and $V$ are frozen and only $S$ is trained, then merged back at the end. Because $S$ scales every orthogonal basis at once, the update $U\\,\\Delta S\\,V^\\top$ can touch all $d$ directions, which the paper calls a full-rank update and shows to be the reason it beats fixed-subspace adapters such as PiSSA. The paper states one clear limit in its conclusion: a nonlinearity between the Q–K or V–O pair, most notably RoPE, makes cross-layer orthogonalization infeasible, so the LLaMA fine-tuning experiments instead orthogonalize the Key layer head-wise.","pith_inferences":["Because every LLaMA fine-tuning result uses the Key-layer fallback rather than the cross-layer Q–K decomposition, the full-rank-update wins over LoRA and PiSSA should be read as evidence for intra-head orthogonalization, while the cross-layer mechanism is directly validated only by the pruning results on RoPE-free models such as GPT-2, Whisper, ViT, and SDXL.","A natural stress test the paper does not run is continual or long-context learning, where LoRA's intrusive dimensions are known to accumulate; if CLOVER's full-rank updates avoid that accumulation, it becomes a general drop-in adapter rather than a pruning trick.","The same decomposition suggests a quantization policy the paper only gestures at: keep the large singular directions in high precision and drop or aggressively quantize the near-zero directions, which could remove the outliers that KV-cache quantizers currently fight."],"forward_implications":["Attention pruning becomes dramatically safer on models whose positional encoding does not sit between Q and K: pruning half of GPT-2 XL's attention parameters raises perplexity 1.38× with CLOVER versus 21.9× with vanilla pruning, and roughly 66 million fine-tuning tokens nearly restore the base perplexity.","Fine-tuning the singular-value diagonal alone matches LoRA rank-32 parameter counts while producing a full-rank update, and outperforms LoRA, DoRA, HiRA, and PiSSA by between 0.6 and 9 percentage points across LLaMA-7B/13B, LLaMA-2-7B, and LLaMA-3-8B on eight commonsense tasks.","Because CLOVER changes only the initialization and merges its factors back into the original weights, it can be combined with any other pruning or quantization technique without altering the model structure or adding inference cost.","Models with pronounced linear redundancy, such as Whisper-Large-v3, can shed 46.42% of their attention parameters with no fine-tuning at all and keep nearly identical output."],"supporting_citations":[{"why":"LoRA, the standard low-rank adapter CLOVER must beat and whose rank-32 parameter count it matches.","marker":"(Hu et al., 2021)"},{"why":"PiSSA, the closest competitor: it also starts from principal singular vectors, but over a fixed subset of directions, which CLOVER sets out to improve on.","marker":"(Meng et al., 2024)"},{"why":"DoRA, source of the fine-tuning hyperparameters and of the reported LoRA/DoRA baseline results on the eight commonsense tasks.","marker":"(Liu et al., 2024d)"},{"why":"HiRA, the high-rank adapter whose reported results CLOVER compares against and surpasses in the fine-tuning table.","marker":"(Anonymous, 2025)"},{"why":"SVFT, the orthogonal-decomposition method CLOVER contrasts with because factorizing each weight separately doubles parameters.","marker":"(Lingam et al., 2024)"},{"why":"RoFormer/RoPE, named by the paper as the nonlinearity between Q and K that makes cross-layer orthogonalization infeasible.","marker":"(Su et al., 2024)"},{"why":"The 'intrusive dimensions' analysis used to argue CLOVER's updates resemble full fine-tuning instead of LoRA's.","marker":"(Shuttleworth et al., 2024)"},{"why":"GPT-2, the model for the headline pruning-perplexity comparison in Table 1.","marker":"(Radford et al., 2019)"},{"why":"LLaMA-2, the principal test bed for the fine-tuning comparison table.","marker":"(AI@Meta, 2023)"}],"fun_headline_variants":["SVD on cross-layer attention pairs: prune 70% like 8%","Cross-layer SVD slashes pruning cost: 70% ≈ 8%","Prune 70% of Q-K pairs now matches 8% with CLOVER","Full-rank SVD fine-tuning beats LoRA, DoRA, HiRA, PiSSA","No extra params: SVD-based pruning and full-rank fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that an attention head's query–key interaction really is a fixed bilinear product of its Q and K projection matrices, with no position-dependent rotation (such as RoPE, rotary position embeddings) inserted between them, a condition the paper itself says fails for the RoPE-based models it fine-tunes.","fun_headline_variants_meta":{"raw":{"variants":["SVD on cross-layer attention pairs: prune 70% like 8%","Cross-layer SVD slashes pruning cost: 70% ≈ 8%","Prune 70% of Q-K pairs now matches 8% with CLOVER","Full-rank SVD fine-tuning beats LoRA, DoRA, HiRA, PiSSA","No extra params: SVD-based pruning and full-rank fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001689,"raw_usage":{"total_tokens":6808,"prompt_tokens":1172,"completion_tokens":5636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":788,"completion_tokens_details":{"reasoning_tokens":5528}},"tokens_in":788,"tokens_out":5636,"duration_ms":40050,"temperature":1.0,"reasoning_tokens":5528,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:08:10.114236+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune LLaMA-2-7B with CLOVER, PiSSA, and LoRA using identical validation-selected checkpoints and matched learning-rate sweeps, then compute the singular-value spectrum of each method's merged update: if CLOVER's accuracy lead over PiSSA disappears, or its update's effective rank is far below the head dimension, the full-rank orthogonalization mechanism is not what produces the reported gains.","supporting_citations":[],"review_version":1}