{"id":"8f380800-7257-4b59-b626-fe54f8c757b1","arxiv_id":"2505.23099","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Fine-tuning mostly amplifies and reorients the top singular directions of weight matrices, and SpecLoRA learns to rescale a top-left block plus LoRA to improve PEFT performance.","lead":"This paper studies how fine-tuning changes the singular values and directions of a model's weight matrices, and proposes SpecLoRA, a parameter-efficient method that rescales a small block of the weight matrix and adds a low-rank LoRA update. It reports modest accuracy gains over LoRA and other baselines on language, commonsense, and vision benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The efficient implementation (Γ⊙W)+AB does not implement the SVD-based update of Eq. (6); scaling a top-left k×k block of W does not rescale top-k singular directions, so the method's advertised spectral mechanism is unsupported.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: the efficient implementation (Eqs. 7–8) does not implement the spectral update (Eq. 6). I agree, and this is the central issue because the paper's novelty claim rests on being 'spectral-directed.' The mathematical mismatch is concrete: Δ_impl has nonzeros only in the top-left k×k block of W, whereas Δ_spec is dense across the top k rows (all m columns) and depends on the global SVD. For a generic full-rank weight matrix, the two updates differ, and the learnable vector d cannot bridge the gap. Thus the method's advertised mechanism is not realized, and the empirical improvements do not support the spectral interpretation. The paper's Section 3 spectral analysis is interesting but only qualitative and, in any case, does not establish that Eq. (7) modulates singular directions. The paper's stated limitation (fixed k) is not the main problem; the main problem is that the implementation is not the proposed spectral update. The reader's REJECT verdict is appropriate, and my stress-test does not change it. I grant that the empirical results (Tables 1–3) could be real, but a correct paper would need to either prove the equivalence, provide quantitative verification that the mask's effect on singular vectors matches Eq. (6), or reframe the method as an empirical block-scaling trick without the spectral claim. Since none of these are present, the central claim fails.","tokens_in":12453,"tokens_out":5947,"duration_ms":59882,"concrete_test":"Take a fixed pretrained linear weight (e.g., a q_proj matrix from LLaMA3-8B, or a random Gaussian matrix as a control). Fix k and a constant d (e.g., d_i = 1.1). Compute W_spec = [\\tilde U_{1:k}, U_{k+1:n}]ΣV^T as in Eq. (6) and W_impl = Γ⊙W as in Eqs. (7)–(8) with the same d. Report the relative Frobenius difference ‖W_spec − W_impl‖_F / ‖W‖_F and the principal angle between the top-k left singular subspaces of W_impl and the intended U_{1:k}. If the relative difference is not close to 0 (e.g., > 1%) and the subspace overlap is low, the claimed equivalence is false. Repeat with d = 1+ε for small ε and with k = 32 and k = 200 to match the paper's settings. This directly checks whether the efficient implementation realizes the spectral update.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 defines a spectral update (Eq. 6) that replaces the top-k left singular vectors of W with a rescaled version. Section 4.3 claims Eq. (7), W → (Γ⊙W)+AB with Γ from Eq. (8), is an efficient implementation of the same update. This equivalence is false. The update implemented by Γ is Δ_impl = ((Γ-1)⊙W), which has nonzeros only in the top-left k×k principal block: (Δ_impl)_{ij} = (d_i-1)W_{ij} for i≤k, j≤k, and 0 otherwise. The update in Eq. (6) is Δ_spec = (D-I_k) U_{1:k}^{(1:k)} Σ_{1:k} V_{1:k}^T (plus AB), where U_{1:k}^{(1:k)} is the top-left k×k block of the left singular vector matrix and V_{1:k} is the full m×k matrix of top right singular vectors. Δ_spec has rank ≤ k but is generally nonzero across all m columns of the top k rows (because V_{1:k}^T is dense), and its entries depend on the global SVD, not just W's first k rows and columns. For a generic weight matrix, Δ_impl ≠ Δ_spec; no choice of d makes them equal unless W has very special structure (e.g., W is already aligned with coordinate axes). Consequently, the method's advertised mechanism—direct rescaling of the top singular directions—is not what is implemented. The empirical gains in Tables 1–3 may come from the learnable mask, the LoRA term, or their interaction, but they do not validate the spectral hypothesis. The paper provides no proof, no quantitative check, and no ablation verifying that the implementation actually modifies the top singular subspace as claimed. This is an internal inconsistency in the core construction, not a disagreement with external consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes singular value/vector changes during full fine-tuning of LLaMA3-8B on Commonsense170K, claiming that fine-tuning mainly amplifies the top singular values while leaving the rest largely intact and reorients the dominant singular vectors toward task-specific directions. Based on this, it proposes SpecLoRA, a PEFT method defined as W -> (Gamma ⊙ W) + AB with a block mask Gamma, and claims this is an efficient implementation of an SVD-based rescaling of the top-k left singular directions. The paper reports consistent average improvements over strong baselines on GLUE, commonsense reasoning, and VTAB-1K benchmarks.","tokens_in":12896,"tokens_out":11329,"duration_ms":108333,"significance":"If the claimed mechanism were correct, the paper would provide a useful bridge between spectral analysis of fine-tuning and PEFT design, and the proposed mask is simple, adds negligible parameters, and shows consistent average improvements over strong baselines (e.g., +1.35 on GLUE over LoRA, +0.5 on commonsense reasoning over DoRA, +1.0 on VTAB-1K over SSF). However, the paper provides no code, no quantitative spectral summaries, no error bars, and its central implementation claim is algebraically false. The empirical gains therefore do not validate the spectral hypothesis, and the contribution reduces to an empirical heuristic whose advantage over existing methods is modest and possibly not statistically significant. The paper does not ship machine-checked proofs, reproducible code, parameter-free derivations, or falsifiable predictions.","major_comments":[{"comment":"The claimed equivalence between the SVD-based update in Eq. (6) and the Hadamard-mask implementation in Eq. (7) is false for a generic weight matrix. With Δ_impl = (Γ-1)⊙W, the update has nonzero entries only in the top-left k×k block, whereas the update in Eq. (6) equals (tilde U_{1:k} - U_{1:k}) Σ_{1:k} V_{1:k}^T, which is generally nonzero in all m columns of the first k rows because V_{1:k}^T is dense, and its entries depend on the global SVD of W. No choice of the learnable vector d makes these updates equal unless W has special structure such as being coordinate-aligned. Consequently, the implemented method does not rescale the top singular directions, and the empirical gains in Tables 1-3 cannot be attributed to the advertised spectral mechanism. The paper provides neither a proof of the equivalence nor a numerical check that the mask approximately reproduces the SVD update.","section":"Sec. 4.3, Eqs. (6)-(8)"},{"comment":"The spectral analysis is purely qualitative and is not sufficient to support the paper's central empirical claim. The results are based on a single model (LLaMA3-8B) and a single dataset (Commonsense170K), with figures showing 'randomly selected layers' and no aggregate statistics, confidence intervals, or quantification of how much the top singular values are amplified. The cosine-similarity analysis in Fig. 2 also ignores the sign ambiguity and potential degeneracy of singular vectors, which can make 'corresponding singular vectors' ill-defined. Without rigorous quantification, the claimed regularities remain anecdotal, and the motivation for SpecLoRA is correspondingly weak.","section":"Sec. 3, Figs. 1-2"},{"comment":"The 'Bottom' variant is not defined precisely. Since the mask Γ in Eq. (8) operates on the top-left k×k block of W in the implemented method, it is unclear what 'applying the same mechanism to the bottom singular directions' means, or whether the variant actually targets the bottom singular directions at all. The ablation therefore does not establish that modifying top singular directions is better than modifying bottom ones, and it also inherits the ambiguity from the first major comment.","section":"Sec. 5.4.3, Table 4"}],"minor_comments":[{"comment":"The notation Γ = [[d d ... d] with k copies; 1 blocks] is ambiguous: it is not clear whether d is a row or column vector, or how the k×k block is tiled into an n×m matrix. Please define Γ explicitly, e.g., Γ_{ij} = d_i for i ≤ k and j ≤ k, and 1 otherwise.","section":"Eq. (8)"},{"comment":"PISSA [35] and MiLoRA [51] appear in Table 2 but are not discussed in Section 2; since they are directly relevant SVD-based PEFT methods, a brief discussion would help position the contribution.","section":"Related Work"},{"comment":"Table 3 is difficult to read because the numeric columns are not aligned with the task names; please reformat it so that each method row is clearly separated and the average is easy to verify.","section":"Table 3"},{"comment":"In Section 1, 'with the form as' should be 'with the form'; also, the sentence beginning 'This formulation inherently aligns...' is vague and should be made precise.","section":"Sec. 1"},{"comment":"The paper does not report whether the results in Tables 1-3 are single runs or averages over multiple seeds; please state this explicitly and, if possible, provide standard deviations or error bars.","section":"Experiments"}],"recommendation":"reject","confidential_remarks":"The central issue is an algebraic error: the implemented update does not correspond to the SVD-based rescaling the paper describes. This is not a matter of a missing proof; it is a false claim. The paper could be reworked as an empirical heuristic, but that would substantially change its scope and would require new ablations to understand what the mask actually does. I therefore recommend rejection. The spectral analysis in Section 3 would need to be made quantitative before any resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's central mechanistic claim does not survive inspection. The claimed efficient implementation in Eq. (7) is not equivalent to the spectral update in Eq. (6). Scaling the top-left kx k block of W does not rescale the top-k left singular vectors. For a generic matrix, the change in Eq. (7) is confined to the first k rows and columns, while the true spectral update would affect all columns of those rows. The paper offers no proof or numerical verification of the equivalence. This is load-bearing because Section 4.3 explicitly says the Hadamard mask implements the SVD-directed rescaling.\n\nWhat the paper does well: SpecLoRA is a new parameterization—a learnable row-scaling on the top-left block plus LoRA—and the experiments are reasonably thorough. Across GLUE, Commonsense170K, and VTAB-1k, it beats LoRA by 1–2 points on average, and the ablation comparing top vs bottom scaling supports that location matters. That is a useful empirical result, even if the spectral interpretation is wrong. The spectral analysis itself (Figures 1–2) is qualitative, based on one model and one dataset, but it is consistent with earlier SVD-based observations from PiSSA and MiLoRA.\n\nSoft spots: besides the false equivalence, the paper gives no quantitative validation of the claimed spectral mechanism. No error bars, no check that the implemented update actually modifies top singular directions. The method would work just as well described as a structured scaling trick plus LoRA; the spectral story is not needed. The self-citations are heavy but not a problem when the cited results are relevant.\n\nWho is this for? PEFT practitioners who want a cheap variant that improves on LoRA. The paper deserves a serious referee because the empirical gains are solid and the flaw is fixable by reframing, but the current version should not be accepted as is. I would ask the authors to either prove the equivalence, drop the spectral claim, or provide evidence that the mask actually reshapes the top singular subspace.","headline":"The SVD-to-mask equivalence is false, and the paper's mechanism claim hinges on it; the empirical variant is real but the spectral story needs replacing.","tokens_in":13427,"tokens_out":4323,"would_cite":false,"duration_ms":41810,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","15A18"],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning amplifies the top singular values of weight matrices, and rescaling those directions with a learnable mask outperforms LoRA on three benchmarks.","keywords":["singular value decomposition","parameter-efficient fine-tuning","LoRA","spectral analysis","low-rank adaptation","foundation models","SpecLoRA"],"falsifier":"Take a generic pre-trained weight matrix $W$, compute its SVD, apply the SpecLoRA mask with a fixed nonzero $d$, and compare the top singular vectors of $\\Gamma \\odot W$ with those of the explicitly SVD-rescaled matrix $\\begin{bmatrix} D U_{1:k} & U_{k+1:n}\\end{bmatrix}\\Sigma V^\\top$; if they diverge for a typical $W$, the assertion that the efficient implementation reproduces spectral rescaling is refuted.","tokens_in":12286,"feed_emoji":"📊","tokens_out":13429,"duration_ms":105968,"temperature":0.7,"pith_summary":"This paper argues that when a large pre-trained model is fine-tuned on a downstream task, the weight matrices change in a highly structured way: the top singular values are amplified and the top singular vectors rotate toward task-specific directions, while the rest of the spectrum and the lower singular vectors stay almost unchanged. On the basis of this observation, the paper proposes SpecLoRA, a parameter-efficient fine-tuning method that multiplies the pre-trained weight matrix by a learnable mask that scales a small top-left block, and adds a low-rank LoRA update. It reports consistent gains over strong baselines: 89.48 average on GLUE with DeBERTaV3-base, 85.5 on Commonsense170K with LLaMA3-8B, and 76.7 on VTAB-1K with ViT-B/16, all at a parameter budget on par with LoRA. If the spectral claim is right, it explains why low-rank adaptation works and points to a cheaper, more targeted way to adapt foundation models.","feed_headline":"Rescaling top singular directions beats LoRA at same budget","feed_subtitle":"A spectral analysis of fine-tuned weights shows task knowledge lives in a few directions; SpecLoRA rescales them.","key_machinery":"The central object is the singular value decomposition $W = U\\Sigma V^\\top$ of each pre-trained weight matrix, used both as an analytical lens and as the motivation for a spectral modulation mask. The paper's key move is to replace explicit SVD reconstruction with the Hadamard product $W \\to (\\Gamma \\odot W) + AB$, where $\\Gamma$ is a learnable $n \\times m$ matrix that scales the first $k$ rows of the first $k$ columns (a corner of the matrix) and leaves the rest unchanged. This formulation is claimed to modulate the dominant singular directions while preserving the global structure, and the low-rank term $AB$ is added to capture residual task-specific directions. The mask's learnable scaling vector $d \\in \\mathbb{R}^k$ is the only new parameter beyond LoRA.","core_discovery":"The central discovery is that fully fine-tuning a pre-trained weight matrix $W$ changes it almost entirely through its top singular components: the singular value spectrum of the fine-tuned matrix overlaps with the pre-trained one except for an amplification of the largest values, and the corresponding left singular vectors become nearly orthogonal to the original while the remaining singular vectors stay closely aligned. This is interpreted as evidence that task-specific knowledge is injected into a low-dimensional subspace. Based on this, SpecLoRA modifies the forward pass to $W \\to (\\Gamma \\odot W) + AB$, where $\\Gamma$ is a learnable scaling mask on a small corner of $W$ and $AB$ is the usual LoRA update; the mask is meant to rescale the top-$k$ singular directions without a full SVD. The paper claims this mechanism outperforms LoRA, DoRA, and other PEFT baselines on natural language understanding, commonsense reasoning, and vision tasks.","pith_inferences":["An implication the paper leaves implicit is that the spectral analysis is done on only a handful of selected layers and matrices; a natural extension is to test whether the amplification-and-reorientation pattern holds across all layers and across different model families, which would determine how universal the claim is.","Because the mask only scales a fixed corner of $W$, the method may in effect be rescaling coordinate-wise magnitudes rather than true singular directions; if so, the empirical gains could stem from a different mechanism, such as per-feature input scaling, and the spectral interpretation would need to be revised.","The near-orthogonality of top singular vectors between pre-trained and fine-tuned weights could be inflated by SVD instability when the top singular values are close together; computing canonical correlations between the principal subspaces, rather than pairwise vector cosines, would provide a more robust test of the reorientation claim."],"forward_implications":["If fine-tuning only amplifies the top singular values, then a parameter-efficient method that rescales those values should recover most of the benefit of full fine-tuning; the reported benchmark results are consistent with that.","SpecLoRA improves over LoRA at the same parameter budget on GLUE, commonsense reasoning, and VTAB-1K, suggesting that spectral guidance adds signal beyond the low-rank update alone.","The method is compatible with existing PEFT pipelines because it only adds a mask multiply before the LoRA update, so it can be combined with adapters or prompts.","The paper reports larger relative gains on low-resource tasks such as RTE and CoLA, indicating that spectral rescaling is especially helpful when training data is scarce."],"supporting_citations":[{"why":"Provides the LoRA update $W + AB$ that SpecLoRA builds on as its baseline and comparison.","marker":"[23]"},{"why":"PiSSA, the direct antecedent that also adapts principal singular values; SpecLoRA's spectral rescaling is compared against it in the commonsense table.","marker":"[35]"},{"why":"Intrinsic dimensionality of fine-tuning, cited to ground the low-dimensional subspace interpretation of the spectral findings.","marker":"[1]"},{"why":"DoRA, a strong weight-decomposed LoRA baseline that SpecLoRA must surpass in the commonsense and GLUE experiments.","marker":"[32]"},{"why":"Supplies the Commonsense170K training corpus used for the LLaMA3-8B commonsense reasoning evaluation.","marker":"[24]"},{"why":"VTAB-1K, the vision benchmark used for the VTAB-1K experiments and the comparison against visual PEFT baselines.","marker":"[57]"},{"why":"GLUE benchmark, the NLU evaluation suite with eight tasks used for the DeBERTaV3-base experiments.","marker":"[50]"}],"fun_headline_variants":["Task knowledge lives in top singular directions","SpecLoRA rescales singular values to beat LoRA","Fine-tuning's impact is concentrated in top singulars","Rescaling top singular directions: efficient adaptation","Spectral insight leads to SpecLoRA's edge over LoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the efficient mask $\\Gamma \\odot W$ is a faithful way to rescale the top-$k$ singular directions of $W$, even though scaling a fixed corner of a generic matrix does not generally alter its singular vectors in that way.","fun_headline_variants_meta":{"raw":{"variants":["Task knowledge lives in top singular directions","SpecLoRA rescales singular values to beat LoRA","Fine-tuning's impact is concentrated in top singulars","Rescaling top singular directions: efficient adaptation","Spectral insight leads to SpecLoRA's edge over LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000563,"raw_usage":{"total_tokens":2663,"prompt_tokens":930,"completion_tokens":1733,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1657}},"tokens_in":546,"tokens_out":1733,"duration_ms":12421,"temperature":1.0,"reasoning_tokens":1657,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:53:56.852611+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a generic pre-trained weight matrix $W$, compute its SVD, apply the SpecLoRA mask with a fixed nonzero $d$, and compare the top singular vectors of $\\Gamma \\odot W$ with those of the explicitly SVD-rescaled matrix $\\begin{bmatrix} D U_{1:k} & U_{k+1:n}\\end{bmatrix}\\Sigma V^\\top$; if they diverge for a typical $W$, the assertion that the efficient implementation reproduces spectral rescaling is refuted.","supporting_citations":[{"cited_title":"The visual task adaptation benchmark","cited_arxiv_id":null,"evidence_quote":"VTAB-1K, the vision benchmark used for the VTAB-1K experiments and the comparison against visual PEFT baselines."}],"review_version":1}