{"id":"11294a98-df99-4d9b-a869-19175925a1d4","arxiv_id":"2505.12433","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SRLoRA replaces low-importance LoRA pairs with unused SVD directions during fine-tuning, keeping the trainable parameter count constant and aiming for faster, better adaptation.","lead":"This paper proposes a modified LoRA training scheme that periodically throws away the least useful small components, folds them into the frozen model, and creates new components from unused directions of the original weights. The idea is interesting, but the reported test results show only mixed improvements despite the paper's claim of consistent gains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stale SVD directions and missing p_r update make the subspace-recomposition claim unverifiable; results do not support 'consistent' gains.","rationale":"The Reader's Weakest Assumption is exactly the most load-bearing concern: Eq. 11 reinitializes from the SVD of W0, while Eq. 12 subtracts the new projection from the current W; this only preserves diversity if W0's singular vectors remain relevant after W has accumulated many fused low-importance components. The paper provides no measurement (e.g., subspace distance or overlap with the active subspace) that would justify this assumption. Even stronger than that internal gap is the missing p_r update in Algorithm 1: p_r labels 'last used singular direction' but no step increments it, so Eqs. 11–12 are formally underspecified and the method cannot be reproduced exactly as written. This internal incompleteness is sufficient for rejection, independent of empirical disagreements. On evidence: Table 3 shows SRLoRA best only on SST-2 and STS-B among six GLUE tasks and trails on MRPC and RTE; Table 4 shows a 4.06-point regression on MNIST. The paper's own Section 4.2 admits gains are 'less pronounced on simpler or saturated tasks,' softening the abstract's 'consistently achieves faster convergence and improved accuracy' claim. No error bars (single runs per Table 4) and no ablations isolate the fusion ratio, switch frequency, r_target, or freshly initialized directions. The mechanism is genuinely different from AdaLoRA and DyLoRA, and the CIFAR-100 result is a positive signal, but the central claim and the method specification are not yet supported. I retain the reader's REJECT verdict unchanged: a revised version with a complete algorithm, controlled comparisons to adaptive-rank baselines, and honest scalar claims could warrant a different verdict.","tokens_in":9707,"tokens_out":3049,"duration_ms":27707,"concrete_test":"Re-run SRLoRA on the reported GLUE subset and on MNIST with a fully specified p_r update rule (e.g., p_r += r' after each switch) plus the exact switch schedule, with at least 3 seeds, and report per-task mean/std accuracy; additionally, measure the dot product between the newly initialized B_new*A_new directions and the current active subspace at each switch. If, on the tasks where SRLoRA currently loses (RTE, MRPC, MNIST), the loss persists or the new directions are far from orthogonal to the active subspace (cosine similarity above 0.3), then the stale-SVD concern is confirmed and the 'consistent improvement' claim fails; if SRLoRA wins or ties on all tasks under the corrected specification and orthogonality holds, the concern is resolved in the method's favor.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that SRLoRA consistently beats LoRA by refreshing its low-rank subspace—rests on two assumptions that are neither proven nor fully specified. First, reinitializing replaced pairs with the next unused singular vectors of the original W0 (Eq. 11) assumes those directions still form a useful exploratory basis after repeated fusion/subtraction cycles have substantially modified the frozen weights through Eqs. 9–12; as W changes, new directions may be redundant or harmful, yet no experiment or analysis checks this. Second, Algorithm 1 uses a pointer p_r for the last used direction but never defines how p_r advances across switch events, so the method as written is not fully implementable and the claimed mechanism cannot be independently reproduced. These are internal gaps, not disagreements with consensus. Empirically, the headline claim of consistency is contradicted by the paper's own Table 3, where SRLoRA trails on MRPC and RTE and is mixed on CoLA and QNLI when metrics are read jointly, and by Table 4, where MNIST drops from 98.89 to 94.83. Only CIFAR-100 shows a clear gain. The paper also lacks error bars, ablations, a defined pointer update, and a comparison to the adaptive-rank baselines it cites (AdaLoRA, DyLoRA). The mechanism is interesting and the CIFAR-100 result is encouraging, but the evidence does not support the central claim as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SRLoRA, a modification of LoRA that periodically recomposes the low-rank update subspace without changing the parameter budget. At fixed intervals, rank-1 components of the LoRA update are scored by a smoothed gradient-weight product; the lowest-scoring pairs are fused into the frozen weight matrix and reinitialized using the next unused left and right singular vectors of the original pretrained weight. The new projection is subtracted from the frozen weights to avoid duplication. The method is evaluated on six GLUE tasks with DeBERTa-v3-base and on CIFAR-100, STL-10, and MNIST with ViT-B/16, and the paper reports loss curves and final accuracy tables. The abstract claims consistent faster convergence and improved accuracy over standard LoRA.","tokens_in":9970,"tokens_out":11454,"duration_ms":112215,"significance":"The core idea, refreshing the low-rank subspace under a fixed parameter budget, is a reasonable direction for PEFT, and the CIFAR-100 gain (92.51 vs. 90.06) is a positive signal. The paper also builds on known components such as PiSSA initialization and sensitivity-based importance, which makes the method easy to compare with prior work. However, the evidence in the manuscript does not support the headline claim: final results are mixed on GLUE and clearly worse on MNIST (94.83 vs. 98.89), and there are no error bars, ablations, or comparisons with AdaLoRA or DyLoRA. The algorithm is incompletely specified, because the pointer p_r is never updated, and one importance formula contradicts its own description. As presented, the contribution is a promising but unverified mechanism rather than a validated method.","major_comments":[{"comment":"The claim that SRLoRA 'consistently achieves faster convergence and improved accuracy over standard LoRA' is contradicted by the paper's own final-accuracy tables. In Table 3, SRLoRA is worse than LoRA on MRPC (90.3/86.6 vs. 90.8/87.5), CoLA (65.1 vs. 65.4), and QNLI (93.4 vs. 94.0); in Table 4, MNIST accuracy drops from 98.89 to 94.83. The Conclusion's statement that 'gains are less pronounced on simpler datasets' does not account for a 4-point drop on MNIST. Because no error bars or repeated-seed results are reported, the claimed consistency cannot be assessed.","section":"Abstract; §4.2, Tables 3 and 4"},{"comment":"The method is under-specified because the pointer p_r, described in Eq. (11) as 'the index of the last used singular direction', is never updated in Algorithm 1 or in the text. Step 8 uses p_r to reinitialize components, but no line advances p_r after a switch, so a reader cannot determine which directions are used at the second and subsequent switches. The algorithm should specify p_r ← p_r + r′ after each switch and state the condition p_r + r′ ≤ d.","section":"Algorithm 1; Eq. (11)"},{"comment":"The importance formula and its verbal description are inconsistent. Eq. (5) defines \\bar{U} as an EMA of the absolute deviation, i.e., uncertainty; Eq. (6) computes s = \\bar{I}\\bar{U}. The text says this prioritizes parameters that are 'stable and reliable (indicated by a low deviation captured by \\bar{U})', but a low \\bar{U} value makes s smaller, not larger. Either the formula or the description is wrong, and this matters because the low-scoring components are exactly the ones chosen for fusion.","section":"Section 3.1, Eq. (6)"},{"comment":"The reinitialization mechanism assumes that singular vectors of the original W0 remain a useful exploration basis after W has been altered by repeated fusion and subtraction steps. Eq. (12) subtracts the new projection from the current W, but this only removes the initialization at the switch moment; after gradient updates the active subspace drifts, so the 'unused' directions may become redundant or harmful. No experiment or analysis checks the overlap between the reinitialized components and the current frozen weights. This is a load-bearing assumption for the claimed subspace-recomposition benefit.","section":"Eqs. (10)–(12), §3.2"},{"comment":"The experimental validation is incomplete for the paper's novelty claims. There is no comparison with AdaLoRA or DyLoRA, despite these being cited as the adaptive-rank baselines that SRLoRA is meant to improve upon, and there are no ablations of the fusion ratio γ, the switching schedule, or the target rank r_target. Without these, it is unclear whether the observed CIFAR-100 gain comes from the subspace recomposition mechanism or from PiSSA initialization and the particular hyperparameter choices.","section":"§4.1 and §4.2"}],"minor_comments":[{"comment":"The second paragraph contains the duplicated phrase 'from from the singular value decomposition'.","section":"Section 1"},{"comment":"Eq. (3) uses I(w_ij) while Eqs. (4)–(6) use I^{(t)}(w_ij); the notation should be consistent throughout.","section":"Eqs. (3)–(6)"},{"comment":"Step 9 writes B_new1 and A_new1 without the matrix subscripts used in Eq. (11); the notation should be aligned.","section":"Algorithm 1"},{"comment":"The SRLoRA row is misformatted ('1.33M/184M96.190.3/86.6'), and the caption should state explicitly that MRPC and STS-B report two metrics (F1/Acc and Pearson/Spearman, respectively).","section":"Table 3"},{"comment":"The captions introduce 'active intervals' and 'candidate rank pairs' without definitions in the Method section; these terms should be defined or removed.","section":"Figures 4 and 5"},{"comment":"PiSSA initialization is presented without stating that the frozen weight is replaced by the residual W0 − BA; this is important for understanding Eqs. (10)–(12).","section":"Eq. (2)"},{"comment":"Vision training hyperparameters such as SGD learning rate, batch size, and data augmentation are not reported for the image experiments; these should be provided for reproducibility.","section":"Table 4"}],"recommendation":"reject","confidential_remarks":"The depth of the experiments is below the level expected for the claimed contribution, and the mismatch between the abstract and Tables 3 and 4 is the main obstacle. I would be open to seeing a revised version with corrected claims, repeated seeds, ablations, and comparisons with adaptive-rank baselines, but the current manuscript is not ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The mechanism is genuinely new, but the paper's own tables undercut its headline claim of consistent improvement. The idea is to periodically fuse low-importance rank-1 LoRA pairs (a column of B and the corresponding row of A) into the frozen weight, then reinitialize those ranks with the next unused singular vectors of the original W0 and subtract the new projection from the frozen weight. That specific combination doesn't appear in AdaLoRA, PiSSA, or LoRA-GA, and it keeps the trainable parameter count fixed, which is appealing. The loss curves do show faster early convergence, the CIFAR-100 gain is substantial (90.06 vs 92.51), and the active-interval analysis is a nice diagnostic. The related-work discussion is fair and positions the novelty correctly.\n\nThe soft spot is the evidence. Table 3 shows SRLoRA worse than LoRA on MRPC, CoLA, and QNLI; Table 4 shows a 4-point drop on MNIST and a small drop on STL-10. That's not 'consistent'. There are no error bars or repeated seeds, so even the wins aren't clearly beyond noise. Algorithm 1 uses a pointer p_r but never defines how it advances, so the method as written isn't fully reproducible. And the core assumption that the original W0's singular vectors stay useful after repeated fusions/subtractions is unexamined; subtracting the new projection from W doesn't guarantee orthogonality to the active LoRA subspace. The per-task target rank and fusion ratio are extra knobs that could explain the results without ablations. There's also no comparison to AdaLoRA or DyLoRA, the adaptive-rank methods the paper explicitly builds on.\n\nThis paper deserves a serious referee, because the mechanism is novel and potentially useful. But it needs major revision: honest claims, error bars, code, a defined pointer update, ablations, and adaptive-rank baselines. I'd send it to review rather than desk reject, expecting heavy revision. For a reading group, it's a good discussion piece on how to evaluate PEFT claims.","headline":"The subspace-recomposition mechanism is novel and worth a serious look, but the paper's own tables contradict its 'consistent improvement' claim; it deserves a serious referee, not as-is.","tokens_in":10539,"tokens_out":3838,"would_cite":false,"duration_ms":36310,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SRLoRA claims LoRA can escape its fixed low-rank subspace during training, without adding trainable parameters, by fusing low-importance pairs into the frozen weights and reopening them along unused singular directions.","keywords":["SRLoRA","low-rank adaptation","parameter-efficient fine-tuning","importance-based fusion","subspace recomposition","singular value decomposition","GLUE benchmark","vision transformers"],"falsifier":"Disable the fusion step but keep the reinitialization schedule, and compare final accuracy and the effective rank of the accumulated update; if the advantage persists without fusion, then importance-based fusion is not the driver. As a direct observable, compute the overlap between each newly chosen singular direction of $W_0$ and the span of the current frozen weight $W^{(t)}$ at each switch: high overlap means the subtractive reinitialization is recycling directions that are already represented, which would undercut the claimed subspace expansion.","tokens_in":9469,"feed_emoji":"🔄","tokens_out":10380,"duration_ms":92203,"temperature":0.7,"pith_summary":"The paper's claim is that LoRA's real limitation is not its rank but the fact that its update subspace is fixed at initialization. SRLoRA addresses this by periodically scoring each rank-1 component of the LoRA update, fusing the least important components into the frozen weight matrix, and reinitializing them with singular directions of the original pretrained weights that have not been used yet. The number of trainable parameters stays exactly the same as in standard LoRA throughout. On GLUE tasks with DeBERTa-v3-base and on ViT image classification, the paper reports that SRLoRA converges faster and reaches higher accuracy than LoRA, with the clearest advantage on tasks that need a broader adaptation subspace.","feed_headline":"SRLoRA swaps weak LoRA directions mid-training and beats baseline","feed_subtitle":"It fuses weak LoRA pairs into frozen weights, then reopens them using unused singular directions at no extra cost.","key_machinery":"The load-bearing object is the rank-1 LoRA pair, one column of $B$ and the corresponding row of $A$. Periodic switch events decide which pairs to keep by the importance score $S_k^{(t)} = \\frac{1}{m}\\sum_i s(B_{ik}) + \\frac{1}{n}\\sum_j s(A_{kj})$, where each entry's score is the product of its exponentially smoothed sensitivity $\\bar{I}$ and its smoothed uncertainty $\\bar{U}$; low-scoring pairs are fused into the frozen weight, discarded, and replaced by the next unused SVD triplets of the original pretrained matrix. The subtractive step $W \\leftarrow W - B_{\\mathrm{new}}A_{\\mathrm{new}}$ is what makes the replacement orthogonal to the active subspace, and the pointer $p_r$ is the bookkeeping device for next unused. The whole fuse, reinitialize, subtract cycle is the mechanism that is supposed to expand the effective subspace at constant parameter count.","core_discovery":"SRLoRA's central claim is that a fixed-parameter LoRA update can be made exploratory by treating each rank-1 pair $B_{\\cdot k} A_{k\\cdot}$ as a disposable component. At scheduled steps, the method ranks these components by an importance score $S_k$ derived from smoothed gradient-weight sensitivities and their uncertainties, fuses the low-scoring fraction into the frozen weight via $W \\leftarrow W + \\sum_{k \\in I_{\\mathrm{low}}} B_{\\cdot k} A_{k\\cdot}$, deletes those pairs, and rebuilds them from the next unused singular directions of the pretrained matrix $W_0$ using $B_{\\mathrm{new}} = U_{[:,p_r:p_r+r']}\\Sigma^{1/2}$ and $A_{\\mathrm{new}}=\\Sigma^{1/2}V^{\\top}_{[:,p_r:p_r+r']}$. The new projection is then subtracted from the current weight, $W \\leftarrow W - B_{\\mathrm{new}}A_{\\mathrm{new}}$, so the fresh directions start out orthogonal to the already-explored space. The paper argues this loop lets LoRA explore a larger effective subspace than static LoRA, at identical parameter count, and backs the argument with faster training-loss reduction on RTE, QNLI, SST-2 and CoLA plus higher CIFAR-100 accuracy.","pith_inferences":["The fixed basis from $W_0$ is a design choice, not a necessity; comparing SRLoRA against a variant that re-SVDs the current updated weight $W^{(t)}$ at each switch would show whether the original singular vectors or simply the act of switching carries the benefit.","The same fuse-and-reinitialize loop could be lifted to other PEFT modules, such as adapters, prefixes, or vector layers, wherever a component can be scored and a fresh orthogonal direction can be supplied; the paper only demonstrates it for SVD-initialized LoRA.","An adaptive switch scheduler, triggered by flattening of the importance-score distribution or by loss stagnation instead of a fixed step count, is the natural next test and could reduce sensitivity to the timing of the swaps.","One implication the paper leaves implicit is that the subtraction in Eq. (12) can overwrite parts of the frozen weight that earlier fusions already placed there; tracking how much of the fused content gets removed by later subtractions would quantify interference between fusions and reinitializations."],"forward_implications":["SRLoRA can be added to an existing LoRA pipeline with a one-time SVD of each target weight plus periodic importance computation, and no extra trainable parameters are needed, so memory and deployment costs stay at LoRA's level.","Convergence should speed up relative to LoRA in the early phase of fine-tuning, because the first fusion-and-reinitialization event replaces low-importance directions with large singular directions of the pretrained weights.","The benefit is conditional on the task: when the initial $r$ directions already cover what the task needs, or when the dataset is simple enough that LoRA saturates, the recomposition loop has little room to help, as the paper's own STL-10 and MNIST numbers show.","A longer training schedule with more switch events permits more subspace exploration at the same parameter budget, so SRLoRA's relative advantage should grow with training length up to the chosen target-rank ceiling $r_{\\mathrm{target}}$."],"supporting_citations":[{"why":"Defines the LoRA update $\\Delta W = BA$ that SRLoRA modifies; the fixed-subspace baseline the paper compares against.","marker":"[5]"},{"why":"Supplies the PiSSA SVD-based initialization that sets SRLoRA's starting LoRA subspace.","marker":"[10]"},{"why":"Introduces the gradient-weight product sensitivity score underlying the importance metric.","marker":"[11]"},{"why":"Provides the EMA smoothing and uncertainty estimates that stabilize the importance scores.","marker":"[17]"},{"why":"Contributes the sensitivity-based importance framework and the default smoothing coefficients adopted here.","marker":"[18]"},{"why":"Provides DeBERTa-v3-base, the pretrained backbone used in all GLUE experiments.","marker":"[3]"}],"fun_headline_variants":["SRLoRA recycles weak LoRA pairs to explore new directions","SRLoRA: dynamic LoRA subspace, same params, better accuracy","SRLoRA swaps out weak LoRA directions mid-training for better fit","SRLoRA: fuse-and-refresh LoRA to beat static LoRA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"SRLoRA's load-bearing assumption is that the singular vectors of the original pretrained weight matrix $W_0$ remain a good exploratory basis even after the frozen weights have been modified by fusions and subtractions, so the next unused directions really are new and useful rather than redundant or harmful.","fun_headline_variants_meta":{"raw":{"variants":["SRLoRA recycles weak LoRA pairs to explore new directions","SRLoRA: dynamic LoRA subspace, same params, better accuracy","SRLoRA swaps out weak LoRA directions mid-training for better fit","SRLoRA: fuse-and-refresh LoRA to beat static LoRA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1516,"prompt_tokens":1066,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":682,"completion_tokens_details":{"reasoning_tokens":367}},"tokens_in":682,"tokens_out":450,"duration_ms":4767,"temperature":1.0,"reasoning_tokens":367,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:34:05.530346+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Disable the fusion step but keep the reinitialization schedule, and compare final accuracy and the effective rank of the accumulated update; if the advantage persists without fusion, then importance-based fusion is not the driver. As a direct observable, compute the overlap between each newly chosen singular direction of $W_0$ and the span of the current frozen weight $W^{(t)}$ at each switch: high overlap means the subtractive reinitialization is recycling directions that are already represented, which would undercut the claimed subspace expansion.","supporting_citations":[{"cited_title":"LoRA: Low-rank adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"Defines the LoRA update $\\Delta W = BA$ that SRLoRA modifies; the fixed-subspace baseline the paper compares against."},{"cited_title":"PiSSA: Principal singular values and singular vectors adaptation of large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the PiSSA SVD-based initialization that sets SRLoRA's starting LoRA subspace."},{"cited_title":"PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance","cited_arxiv_id":"2206.12562","evidence_quote":"Provides the EMA smoothing and uncertainty estimates that stabilize the importance scores."},{"cited_title":"Adaptive budget allocation for parameter-efficient fine-tuning","cited_arxiv_id":null,"evidence_quote":"Contributes the sensitivity-based importance framework and the default smoothing coefficients adopted here."},{"cited_title":"DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing","cited_arxiv_id":null,"evidence_quote":"Provides DeBERTa-v3-base, the pretrained backbone used in all GLUE experiments."}],"review_version":1}