{"id":"3e7aceab-9ef2-41b3-b1ce-27a70923fa61","arxiv_id":"2505.14350","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"OSoRA fine-tunes LLMs by updating only singular values and one output-dimension vector, using frozen singular vectors from an SVD of the pretrained weights.","lead":"This paper introduces OSoRA, a parameter-efficient method that fine-tunes large language models by learning only singular values and one output-dimension scaling vector on top of an SVD of the pretrained weights. The authors claim it matches or beats LoRA and VeRA with far fewer trainable parameters, which could make fine-tuning cheaper on limited hardware.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) cancels the learnable OSoRA term in Eq. (4), so the forward pass is exactly W0x for every training step; as written, OSoRA implements no adaptation.","rationale":"The reader's verdict is REJECT and I agree with that verdict, but my load-bearing concern differs from the reader's stated weakest assumption. The reader's weakest_assumption treats the suitability of the frozen top-singular-vector subspace as the main empirical risk; the paper's own Limitations section concedes that risk. The more decisive defect is in the method's own equations: Eq. (5) subtracts the exact adaptive term that Eq. (4) adds, leaving y = W0 x identically. Because the paper provides no code and no formal verification, the equation-level cancellation cannot be blamed on a typo in an appendix; it sits at the center of the proposed method and makes every reported accuracy gain unreproducible as written. The strongest independent support would be a released implementation and a corrected Eq. (5); absent that, the central claim of 'updating only singular values and a single output-dimension vector is sufficient' is not supported by the paper's formulation. I would therefore keep a REJECT verdict, with a clear path to revision: fix Eq. (5), release code, and re-check the empirical tables. I marked agreement as partial because the reader did flag 'the math error in Eq. (5)' in the rationale even though the formal weakest_assumption field points to the different, secondary subspace question.","tokens_in":12325,"tokens_out":5327,"duration_ms":51634,"concrete_test":"Substitute Eq. (5) into Eq. (4) analytically for any d, k, r: every term cancels and y = W0 x. To make this empirical, implement Eqs. (4)-(5) on a toy linear layer with random input, initialize O as ones and Sr at the top-r singular values, take one gradient step, and compare outputs before and after; a faithful implementation will show no change. If the authors intended W'_0 = W0 - U_r Sigma_r V_r^T with fixed initial Sigma_r, Eq. (5) and the surrounding text must be corrected and the experiments re-run against the corrected formulation before the claims can be assessed.","verdict_should_be":"REJECT","load_bearing_attack":"In Section 3.2, Eq. (4) defines the adapted forward pass as y = W'_0 x + Lambda_O U_r Lambda_Sr V_r^T x, and Eq. (5) defines W'_0 = W0 - Lambda_O U_r Lambda_Sr V_r^T. Substituting Eq. (5) into Eq. (4) gives y = W0 x identically, regardless of the values of the trainable vectors O and Sr. Because Lambda_O and Lambda_Sr are exactly the learnable quantities being optimized, the residual in Eq. (5) is not frozen and does not exclude the adapted contribution; it removes precisely the term that Eq. (4) adds back. The paper states that W'_0 is the frozen component after excluding the top-r singular contribution, which suggests the intended definition should use the initial singular values (e.g., W'_0 = W0 - U_r Sigma_r V_r^T), but that is not the equation given. Taken literally, the method has zero effect on model output, so the benchmark improvements in Tables 1, 2, 4, 5, and 6 cannot be produced by the stated method. This is an internal inconsistency in the central construction, not a disagreement with empirical consensus, and it blocks any faithful reproduction. The manuscript's own Limitations section also notes the fixed-subspace premise is unproven, but that concern is secondary: the equations must first define a non-vacuous update.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OSoRA, a parameter-efficient fine-tuning method that performs SVD on pretrained weight matrices, freezes the top-r singular vector matrices, and trains only a singular-value vector S_r and an output-dimension vector O. The authors claim that OSoRA matches or exceeds LoRA, VeRA, DoRA, and PiSSA on commonsense-reasoning and mathematics benchmarks while using only r+d trainable parameters. The paper includes parameter-count analysis, gradient formulas, rank-robustness experiments, and ablation studies.","tokens_in":12611,"tokens_out":5289,"duration_ms":52548,"significance":"If the method were correctly specified, OSoRA would be a useful contribution to PEFT: it combines the parameter efficiency of VeRA with an SVD-based initialization inspired by PiSSA, and the experimental evaluation covers multiple model families and benchmarks without circularity. The paper also ships honest limitations, including the unproven fixed-subspace premise. However, as written, the central formulation is internally inconsistent: Equation (5) defines the frozen residual in terms of the trainable matrices, which makes the forward pass exactly W0x and the method vacuous. This is a load-bearing error that must be corrected before the reported results can be attributed to the stated method.","major_comments":[{"comment":"The definition of W'_0 in Eq. (5) uses the trainable matrices Λ_O and Λ_Sr. Substituting Eq. (5) into Eq. (4) gives y = W0x identically, for any values of O and S_r, so no adaptation occurs as written. The accompanying text says W'_0 is the frozen component after excluding the top-r singular contribution, which indicates the intended definition should use the initial singular values, e.g., W'_0 = W0 − U_r Σ_r V_r^T. This is not a typo in an isolated formula: the same definition propagates to the merged-weight Eq. (7) and the DoRA combination Eq. (12), and it determines what the benchmark tables in Section 4 claim to measure. Please correct Eq. (5) (and any downstream uses) and state explicitly that W'_0 is frozen with respect to the initial SVD values.","section":"§3.2, Eqs. (4)–(5)"},{"comment":"Equation (10) as printed has a dimension mismatch: ∂L/∂∆W is d×k, U_r is d×r, Λ_Sr is r×r, and V_r^T is r×k, so the product is d×k and its diag is undefined. The correct chain-rule expression for ∂L/∂O should be diag( ∂L/∂∆W V_r Λ_Sr U_r^T ), up to ordering and transposes. This formula is part of the theoretical justification for the optimization dynamics, so it needs to be fixed or removed if it is only illustrative.","section":"§3.6, Eq. (10)"},{"comment":"The experimental results cannot be interpreted as evidence for OSoRA until the Eq. (5) issue is resolved, because the stated forward pass is identical to the frozen pretrained model. The authors should either confirm that the implementation used W'_0 = W0 − U_r Σ_r V_r^T (or an equivalent frozen residual) and add a numerical check that the merged weight at initialization equals W0 exactly, or provide code. The Limitations section already concedes that the fixed-subspace premise is unproven, but that is secondary to the need for a non-vacuous formulation.","section":"§4, Tables 1–6 and §6"}],"minor_comments":[{"comment":"Figure 3 includes a VeRA bar in the MATH/GSM8K comparison, but the text describes only three OSoRA variants; please clarify the figure legend and the comparison intent.","section":"§4.4, Figure 3"},{"comment":"In Table 7, multiple methods (LoRA, DoRA, PiSSA) share merged cells for r and learning rate, which makes it difficult to see which hyperparameters apply to each method; please separate the rows or use a clearer layout.","section":"Appendix A, Table 7"},{"comment":"The citation of Abacha et al. (2025), a medical-error-detection paper, to motivate the computational cost of fine-tuning large models, seems off-topic; a general LLM-scaling citation would be more appropriate.","section":"§1"},{"comment":"The paper uses both 'OSoRAG' and 'OSoRA G' for the Gaussian-initialized variant; please standardize the notation.","section":"§4.4, Table 4 and surrounding text"},{"comment":"The parameter-ratio equation is correct, but the intermediate step would be easier to follow if the factorization (r+d)/r(d+k) were shown as 1/(d+k) + d/[r(d+k)] explicitly before the limit statement.","section":"§3.5, Eq. (8)"}],"recommendation":"major_revision","confidential_remarks":"The central Eq. (5) error is fixable, but it is fundamental: as submitted, the method is vacuous. I would require the corrected residual definition, a corrected gradient formula, and ideally a small numerical or code-level sanity check showing that the implemented forward pass matches the intended one. If the authors can confirm that the reported results were obtained with the intended frozen residual, the paper may be suitable after revision; otherwise the empirical claims are unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Name],\n\nHere's the short version: Eq. (5) cancels the adaptation term in Eq. (4), so as written the forward pass is exactly y = W0 x. OSoRA does nothing. That's a load-bearing flaw, not a stylistic quibble.\n\nWhat's actually new: the paper marries VeRA's frozen-random-matrix + scaling-vector trick with PiSSA's SVD initialization. That's a reasonable synthesis and gives a linear r+d parameter count, which is genuine. The MATH/GSM8K numbers are pretty good—OSoRA beats VeRA by 3-5 points on LLaMA3-8B, and the ablation in Figure 3 supports the claim that both Sr and O matter. The Gaussian-init comparison is a nice piece of evidence for the ones-init choice.\n\nNow the soft spots. The central equations are internally inconsistent: if you substitute Eq. (5) into Eq. (4), the Λ_O U_r Λ_Sr V_r^T term cancels exactly, leaving y = W0 x. The learnable parameters have zero effect. The intended W'_0 should probably be W0 - U_r Σ_r V_r^T, using the original singular values, but that's not what's written. So the experiments cannot be reproduced from this formulation. The common-sense results are also not consistently superior—on Qwen2.5-32B, OSoRA averages 83.23 vs LoRA's 84.86. And the memory-footprint claim in Eq. (6) is overstated: storing the frozen singular vectors costs dr+kr parameters, same order as LoRA itself. No code or data is provided, which makes it hard to check whether the authors used the corrected (but unwritten) version.\n\nThe paper does cite the relevant prior work and includes a limitations section, so it's not dismissive. But the internal contradiction is fatal for the current version. I'd still send it to a serious referee, because the idea is sound enough to be salvageable with a corrected residual and honest memory accounting. A good referee will catch the error in five minutes. The paper as submitted is not publishable; it needs major revision.\n\nRecommendation: send to peer review, but expect heavy revision.","headline":"Eq. (5) cancels the adaptation term, so OSoRA as written is a no-op; the underlying idea is salvageable but the paper needs major revision.","tokens_in":13105,"tokens_out":3560,"would_cite":false,"duration_ms":32001,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OSoRA: adapting a language model by tuning only singular values and one output-dimension vector matches or beats LoRA, VeRA, and PiSSA with r+d trainable parameters.","keywords":["parameter-efficient fine-tuning","low-rank adaptation","singular value decomposition","learnable scaling vectors","large language models","SVD initialization","LoRA extension","model merging"],"falsifier":"A direct falsifier is to take OSoRA and replace the SVD-derived $U_r, V_r$ in Equation (4) with a random orthogonal basis, as VeRA does, while keeping the same initialization of $S_r$ from $W_0$'s singular values and the same all-ones $O$. If this random-basis variant matches OSoRA's accuracy on the reported benchmarks, then the top-singular-subspace assumption is not doing the work the paper assigns to it.","tokens_in":12131,"feed_emoji":"🔧","tokens_out":18867,"duration_ms":138658,"temperature":0.7,"pith_summary":"OSoRA is a parameter-efficient fine-tuning method for large language models that claims a full low-rank adaptation can be driven by just $r+d$ learnable scalars: the top-$r$ singular values of a pretrained weight matrix plus one output-dimension scaling vector, with the corresponding singular vectors frozen. The paper argues that the top singular subspace of the original weights already carries the directions a downstream task needs, so during training only the strengths of those directions and their per-output scaling have to be adjusted. On commonsense reasoning and mathematics benchmarks across LLaMA2-13B, Qwen1.5-7B, Qwen2.5-32B, Mistral-7B, and LLaMA3-8B, OSoRA reports accuracy comparable to or better than LoRA, VeRA, DoRA, and PiSSA at a fraction of the trainable parameter count. Ablations support the joint training of both learnable vectors, and the paper shows the trained vectors can be merged back into a single weight matrix so inference adds no overhead. The appeal is that it makes fine-tuning larger models feasible on limited hardware without giving up accuracy.","feed_headline":"Fine-tune with one vector plus singular values: OSoRA matches LoRA","feed_subtitle":"OSoRA trains only r+d trainable parameters per layer and still beats or ties LoRA, VeRA, and PiSSA on reasoning benchmarks","key_machinery":"The central object is the SVD-based reparameterization of Equation (4): $y = W'_0 x + \\Lambda_O U_r \\Lambda_{S_r} V_r^\\top x$. Here $U_r \\in \\mathbb{R}^{d\\times r}$ and $V_r \\in \\mathbb{R}^{k\\times r}$ are the frozen left and right singular vectors belonging to the $r$ largest singular values of the pretrained weight $W_0$, and $\\Lambda_O$ and $\\Lambda_{S_r}$ are diagonal matrices built from the two trainable vectors: $O \\in \\mathbb{R}^d$ scales each output dimension, while $S_r \\in \\mathbb{R}^r$ weights each rank component. Because the singular vectors are fixed, the parameter count is $r+d$ regardless of the input dimension $k$, growing linearly rather than exponentially in $r$. The same vectors that are frozen during training are also what makes merging possible: after training, $\\Lambda_O U_r \\Lambda_{S_r} V_r^\\top$ is a rank-$r$ matrix that can be added into $W'_0$, so inference is just the original model with a modified weight matrix.","core_discovery":"The central claim is that adapting a large language model does not require learning a full low-rank update matrix. After computing the SVD of a pretrained weight matrix $W_0 = U\\Sigma V^\\top$, OSoRA freezes the top-$r$ singular vectors $U_r$ and $V_r$ and learns only the corresponding singular values $S_r \\in \\mathbb{R}^r$ and an output-dimension scaling vector $O \\in \\mathbb{R}^d$. The adapted forward pass is $y = W'_0 x + \\Lambda_O U_r \\Lambda_{S_r} V_r^\\top x$, with $W'_0$ the original weight minus the top-$r$ component, so the entire task-specific update lives in the fixed subspace spanned by the top singular vectors. The paper reports that this $r+d$-parameter scheme matches or exceeds LoRA, VeRA, PiSSA, and DoRA on commonsense reasoning and mathematical reasoning benchmarks, and that ablations show both $S_r$ and $O$ must be trained jointly for best results.","pith_inferences":["Inference: OSoRA is effectively a spectral gating operation—it removes the top-$r$ component of $W_0$ and re-inserts it with rescaled singular values—so tasks whose desired updates align with the principal subspace should benefit most, while tasks needing orthogonal directions should hit a performance ceiling; replacing $U_r$, $V_r$ with a random orthogonal basis while keeping the same $S_r$ initi","Inference: Since $O$ has one scalar per output dimension, OSoRA can express per-output rescaling that LoRA cannot with the same rank; on attention projections this may amount to reweighting attention heads, a hypothesis the paper does not test.","Inference: By freezing $U_r$ and $V_r$, the effective hypothesis space is a rank-$r$ manifold with fixed tangent directions; the method should be most attractive when the task's required update is concentrated in the top-$r$ singular subspace of the pretrained weights, and a cheap predictor of degradation is the energy of the task update orthogonal to that subspace, $\\mathrm{tr}(\\Delta W^\\top (I -","Inference: The paper's ablation that fixing $O$ (OSoRA**) hurts more than fixing $S_r$ suggests the output-dimension vector carries much of the adaptation signal; a testable extension is to initialize $O$ from a trained LoRA's row norms rather than all-ones and check whether that closes the remaining gap."],"forward_implications":["OSoRA makes the parameter cost of fine-tuning independent of the input dimension $k$, so the same budget covers very wide matrices; for example, a rank-512 adaptation on a 7B model uses roughly $r+d$ instead of $r(d+k)$ trainable parameters.","After training, the learned vectors merge into a single weight matrix, so deployed checkpoints are identical in size to the original model and inference adds no extra memory or computation.","Because only $S_r$ and $O$ are optimized, optimizer memory is minimal, which permits larger ranks (512, 1024) before memory constraints bite.","The ablations imply that a PEFT method can spend its entire trainable budget on spectral scaling plus per-output rescaling, and that initializing $O$ with ones is a necessary ingredient.","Composing OSoRA with DoRA improves MATH and GSM8K further with roughly 360 thousand trainable parameters versus DoRA's roughly 7 million, indicating the method composes with other PEFT decompositions."],"supporting_citations":[{"why":"Introduces LoRA, the low-rank adaptation formulation OSoRA extends and the primary baseline for parameter-count comparisons.","marker":"Hu et al., 2022"},{"why":"Introduces VeRA, the vector-scaling method that already achieves r+d parameters and is OSoRA's direct efficiency baseline.","marker":"Kopiczko et al., 2024"},{"why":"Introduces PiSSA, the SVD-based principal-component initialization that OSoRA builds on and compares against.","marker":"Meng et al., 2024"},{"why":"Introduces DoRA, the magnitude-direction decomposition used as a baseline and in the OSoRA+DoRA integration.","marker":"Liu et al., 2024"},{"why":"Provides MetaMathQA, the training set used for the mathematical reasoning experiments.","marker":"Yu et al., 2024"},{"why":"Supplies the MATH benchmark used to evaluate mathematical reasoning.","marker":"Hendrycks et al., 2021"},{"why":"Supplies the GSM8K benchmark used to evaluate mathematical reasoning.","marker":"Cobbe et al., 2021"}],"fun_headline_variants":["OSoRA: fine-tune LLMs with just r+d parameters per layer","One vector plus singular values: OSoRA matches LoRA","OSoRA cuts trainable params to r+d, beats LoRA","SVD-based PEFT: OSoRA trains only r+d per layer","OSoRA: frozen singular vectors, learn r+d parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the dominant directions of the pretrained weights are already the directions a new task needs; if the task requires changing the model along directions those dominant directions do not cover, OSoRA has no way to express the change.","fun_headline_variants_meta":{"raw":{"variants":["OSoRA: fine-tune LLMs with just r+d parameters per layer","One vector plus singular values: OSoRA matches LoRA","OSoRA cuts trainable params to r+d, beats LoRA","SVD-based PEFT: OSoRA trains only r+d per layer","OSoRA: frozen singular vectors, learn r+d parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000509,"raw_usage":{"total_tokens":2501,"prompt_tokens":993,"completion_tokens":1508,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1413}},"tokens_in":609,"tokens_out":1508,"duration_ms":19774,"temperature":1.0,"reasoning_tokens":1413,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:35:09.508716+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct falsifier is to take OSoRA and replace the SVD-derived $U_r, V_r$ in Equation (4) with a random orthogonal basis, as VeRA does, while keeping the same initialization of $S_r$ from $W_0$'s singular values and the same all-ones $O$. If this random-basis variant matches OSoRA's accuracy on the reported benchmarks, then the top-singular-subspace assumption is not doing the work the paper assigns to it.","supporting_citations":[],"review_version":1}