{"id":"d9be415b-14a6-4903-932c-0e50b9076236","arxiv_id":"2602.00722","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Balancing the singular values of LoRA task updates, plus orthogonality to past gradients, reduces backward and forward forgetting in continual adaptation of vision-language models.","lead":"Low-rank model updates in continual learning usually concentrate most of their adaptation energy in a few dominant directions; this paper claims that imbalance itself drives forgetting, and proposes EBLoRA, which spreads energy evenly across all update directions and steers new updates away from previously sensitive gradients. On two vision-language continual-learning benchmarks, EBLoRA reports much less forgetting and better forward transfer than prior low-rank baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal claim confounded: smoothing experiments that isolate spectral balance also shrink update magnitude; no equal-norm control, so reduced interference may be an energy effect.","rationale":"The reader's verdict is CONDITIONAL, and I agree with that verdict; the method is well-specified and the headline gains are sizable, but the causal mechanism is not yet established. The most load-bearing premise is the one the reader identified first: the smoothing experiments are magnitude-confounded. This is not a minor detail: the abstract's 'thereby' makes spectral imbalance the cause of forgetting, and the entire design principle (equal singular values) is derived from it. Since the confound is mathematically guaranteed (Cauchy–Schwarz), the onus is on the authors to provide an equal-norm control. I am not raising this as a rejection of the method: EBO/EBLoRA may still be an effective regularizer. But the paper's central explanatory claim should be conditional on this check. The GPM subspace concern is real but secondary; the EBO ablation already shows balance-related gains without GO, though that comparison also lacks norm matching. Decision: keep the verdict conditional; no change from the reader.","tokens_in":22531,"tokens_out":9429,"duration_ms":109647,"concrete_test":"Re-run the Fig. 2b and Fig. 4-left merging experiments with an equal-norm control: after smoothing each task update, rescale the entire matrix by c = ‖ΔW‖_F / ‖ΔW_smooth‖_F so that its Frobenius norm equals the original unsmoothed update, then merge and recompute NAI. If the NAI improvement at α=1 vanishes or reverses, the effect is energy, not balance; if it persists, the spectral-balance mechanism is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"At the heart of the paper is the claim that imbalanced singular value spectra cause forgetting. The direct evidence is the smoothing experiment in §2.2/Fig. 2b and its continuation in §4.2/Fig. 4-left. In both, a LoRA update ΔW = U diag(σ) V^T is modified by replacing each σ_i with (1−α)σ_i + ασ̄, where σ̄ = (1/r)Σ_j σ_j. This changes two things at once: the spectrum becomes flatter, and the Frobenius norm shrinks. Indeed, ‖ΔW_smooth‖_F^2 = r σ̄^2 ≤ Σ_i σ_i^2 = ‖ΔW‖_F^2, with strict inequality whenever the spectrum is not already flat. Thus the 'balanced' adapters are also strictly smaller perturbations. The higher NAI after smoothing could therefore be caused by smaller update energy rather than by balance; the experiments include no control that renormalizes the smoothed update to the original norm (or compares against a uniform scaling of the original LoRA). The same confound applies to the EBO vs LoRA-FT comparison in §4.3/Table 3: EBO decouples magnitude into a learned scalar s_t, and the paper does not report whether the final EBO updates have the same Frobenius norm as LoRA-FT updates. Without such a control, the causal attribution in the abstract is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that catastrophic forgetting in low-rank continual adaptation is caused by imbalanced singular value spectra of LoRA updates, and proposes EBLoRA, which factorizes each task update as ΔW_t = s_t U_t V_t^T with orthonormal U_t, V_t and enforces G^T_{t-1} U_t = 0 to avoid previously sensitive gradient directions. The constrained problem is solved by projected first-order optimization on a restricted Stiefel manifold. The authors report strong gains on UCIT and MLLM-DCL, with ablations (EBO, GO, IL) attributed to energy balance, gradient orthogonality, and depth-aware initialization. Theoretical results (Props. 3.1–3.2) prove optimality of the proposed tangent projection and retraction.","tokens_in":22658,"tokens_out":3196,"duration_ms":37101,"significance":"If the causal claim is established, the paper offers a simple and actionable design principle: balanced low-rank updates plus gradient orthogonality are sufficient to reduce both backward and forward forgetting. The theoretical parts are sound: Props. 3.1–3.2 are correct, and the appendix proofs are valid. The paper also ships detailed per-step tables (Tables 4–6), from which the headline MFN/BWT/FWT values recompute, and provides a public code link. The main empirical claim, however, currently rests on experiments that conflate spectral balance with update magnitude; the causal interpretation therefore needs additional controls before the paper's central message can be accepted.","major_comments":[{"comment":"The smoothing experiment replaces each singular value σ_i with (1−α)σ_i + ασ̄, where σ̄ = (1/r)Σ_j σ_j. This changes two quantities at once: the spectrum becomes flatter, and the Frobenius norm shrinks, since ‖ΔW_smooth‖_F^2 = rσ̄² ≤ Σ_i σ_i², with strict inequality whenever the spectrum is not flat. The increased NAI after smoothing could therefore be caused by smaller update energy rather than by balance. No control is reported that renormalizes the smoothed update to the original norm or that applies a uniform scaling to the unsmoothed update. Please add such a magnitude-matched control (e.g., scale the smoothed adapter back to the original Frobenius norm before merging, or compare against c·ΔW for c<1). This is load-bearing because the abstract and Section 2.2 attribute reduced interference specifically to spectral balance.","section":"§2.2 and §4.2 (Fig. 2b and Fig. 4 left)"},{"comment":"The ablation isolating 'energy-balanced optimization' (EBO) removes gradient orthogonality and depth-aware initialization, but it still differs from LoRA-FT in both the factorization (s_t U_t V_t^T) and the learned magnitude s_t. The paper does not report the final Frobenius norms of EBO updates versus LoRA-FT updates, so the improvement of EBO over LoRA-FT (70.2 MFN vs 61.4) could reflect a different update scale rather than spectral balance. The statement in §4.2 that Fig. 4 right uses 'equal-norm perturbations' is about the injected noise, not about matching the norms of the compared target updates. Please provide a norm-matched comparison — for example, rescale each LoRA-FT update per layer to the same Frobenius norm as the EBO update (or vice versa) and report the resulting metrics.","section":"§4.2 and Table 3 (EBO ablation)"},{"comment":"The entire gradient-orthogonality mechanism depends on the snapshot G_t and the stored subspace G_{t-1}. Algorithm 2 says only 'Sample a small subset of mini-batches' to form G_t, without specifying the number of mini-batches, the number of gradient steps, or whether the snapshot is taken from the initial model or after some training. Appendix C fixes the energy threshold ε=0.95 but gives no sensitivity analysis. Since Eq. (4), the initialization in lines 7–8, and the retraction in Algorithm 1 all rely on this subspace estimate, the method is not fully reproducible and the robustness of the reported gains to this choice is unknown. Please specify the snapshot protocol and report results for a range of ε (and snapshot sizes).","section":"Algorithm 2 (lines 4–8) and Appendix C"}],"minor_comments":[{"comment":"Typo: 'LLaV A' should be 'LLaVA'.","section":"§4.1"},{"comment":"The proof of Prop. 3.2 assumes G^T G = I (used in the step G^T G G^T = G^T). This holds for the GPM construction, but the paper should state explicitly that G always has orthonormal columns, since the projector P^⊥_G = I − G G^T is only an orthogonal projector under that condition.","section":"Appendix A.2"},{"comment":"The reference 'Tab. 4a and 6' is slightly confusing; consider clarifying which rows of Table 4 correspond to LoRA-FT.","section":"Fig. 5 caption"}],"recommendation":"major_revision","confidential_remarks":"The main causal claim is currently confounded with update magnitude, but the issue is fixable with norm-matched controls. The method itself is empirically strong and the theoretical optimality results are correct, so I see this as a major-revision rather than reject. One additional point for the editor: the strongest baseline, KeepLoRA (Luo et al., 2026), appears to share authors with the current paper; the authors should be asked to clarify that the baseline was trained under identical protocols and hyperparameters."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the balanced sUV^T parameterization is a genuine new construction and the benchmark gains are large, but the paper's central causal claim is under-supported because the key experiments confound spectral balance with update magnitude. The method still deserves a serious referee.\n\nWhat is actually new: formulating the LoRA update as s_t U_t V_t^T with orthonormal U,V — equal singular values enforced, so spectral balance becomes an explicit design choice — and treating it as optimization on a restricted Stiefel manifold. The projection/retraction proofs (Props 3.1–3.2) are correct, and Algorithms 1–2 are specific enough to reimplement. I recomputed FWT and BWT from the per-step tables; they are internally consistent. The reported gains on UCIT (MFN 72.8 vs 61.4 for LoRA-FT) and MLLM-DCL are substantial, and the EBO ablation shows balance alone delivers most of the gain. That is a strong empirical result even if the mechanism is not fully pinned down.\n\nSoft spots. The smoothing experiments in Fig 2b and Fig 4-left replace each singular value with (1−α)σ_i + ασ̄. That does not just flatten the spectrum; it strictly shrinks the Frobenius norm of the update unless the spectrum is already flat. So the improvement could simply be a smaller perturbation, and there is no control that renormalizes the smoothed adapter to the original norm. The equal-norm perturbation comparison in Fig 4-right is the right idea but does not cleanly separate balance from the parameterization. The causal claim in the title is therefore not established. Also, the plasticity claim is overstated: Table 4 shows EBLoRA's immediate accuracy on CLEVR is 66.9 vs LoRA-FT's 77.9; it forgets less but also initially learns less. Final MFN is higher, so the method is not hurting, but the honest framing matters. No error bars, and the GPM subspace energy threshold (ε=0.95) and subspace size are not sensitivity-analyzed. One more minor point: the strongest baseline, KeepLoRA, is from the same group; not a red flag, but the comparison should be scrutinized for parity.\n\nWho this is for: anyone working on parameter-efficient continual learning or model merging. The flat-spectrum idea is worth citing on its own, independent of the causal story.\n\nRecommendation: send to peer review. The method is well-specified and the gains are large. The causal claim needs a magnitude-matched control and the plasticity claim needs softening. Both are fixable in revision.","headline":"The balanced sUV^T parameterization is genuinely new and the gains are real, but the causal claim is confounded by update magnitude; the method still deserves peer review.","tokens_in":23387,"tokens_out":3450,"would_cite":true,"duration_ms":37553,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Catastrophic forgetting in low-rank continual adaptation is caused by imbalanced singular-value spectra, and forcing equal-strength components plus gradient-orthogonal update directions largely eliminates it.","keywords":["continual learning","catastrophic forgetting","low-rank adaptation","singular value spectrum","energy-balanced updates","Stiefel manifold optimization","gradient orthogonality","vision-language models"],"falsifier":"Merge LoRA adapters with singular values smoothed to the mean, then rescale the update so its total energy (sum of squared singular values) exactly matches the unsmoothed version: if the rescaling erases the interference benefit, then reduced energy, not balance, was causing the improvement. A second check: run the full method with the gradient-orthogonality constraint disabled while keeping the balanced factorization; if backward transfer degrades sharply, the orthogonality constraint, not the spectral balance, is doing the work.","tokens_in":22200,"feed_emoji":"⚖️","tokens_out":12845,"duration_ms":136767,"temperature":0.7,"pith_summary":"This paper tries to establish a causal account of catastrophic forgetting in continual learning with low-rank adapters: when a task update is a low-rank matrix, its singular values are highly imbalanced, so a handful of components carry most of the adaptation energy, and it is exactly those dominant directions that later tasks overwrite. If that is right, forgetting is a structural consequence of imbalanced updates rather than merely an optimization failure, and the fix is to regulate the internal shape of each update instead of only defending old knowledge. The proposed method, EBLoRA (Energy-Balanced Low-Rank Adaptation), rewrites every task update as a scalar magnitude times two orthonormal bases, making all components carry equal energy, and requires the update basis to be orthogonal to a stored set of previous-task gradient directions. Across two vision-language continual learning benchmarks this reduces backward forgetting from around −15 to −2 points and raises final accuracy by more than ten points, without sacrificing the model's ability to keep learning new tasks. The reason to care is that the recipe is a design principle — balance the spectrum, orthogonalize against past gradients — rather than replay buffers, stored data, or task-specific architectures.","feed_headline":"Equalized update components nearly eliminate forgetting","feed_subtitle":"Decoupling update size and direction, plus orthogonality to past gradients, lifts final accuracy by more than ten points.","key_machinery":"The load-bearing object is the factorization ΔW_t = s_t U_t V_t^T — a scalar magnitude times two orthonormal bases — which decouples how much adaptation happens from which directions it takes, and makes the update's singular values exactly equal by construction, i.e., energy-balanced. Optimization happens on the restricted Stiefel manifold M_t = {U : U^T U = I_r, G_{t-1}^T U = 0}, where the two constraints are orthonormality of the update basis and orthogonality against a stored subspace G_{t-1} of gradient directions gathered from earlier tasks. Two geometric facts carry the method: the tangent-space projection and the whitening retraction are each the unique Frobenius-norm-closest map onto","core_discovery":"Central claim: spectral imbalance, not interference with earlier updates per se, drives forgetting in low-rank continual adaptation. LoRA updates show long-tailed singular value spectra whose variance grows during training, and merging experiments show that flattening singular values toward their mean, with singular vectors fixed, reduces cross-task interference. The paper concludes that a balanced update is both less disruptive to prior knowledge and less vulnerable to later overwriting. The design: factor each update as ΔW_t = s_t U_t V_t^T (scalar magnitude, orthonormal bases), so all r singular values are equal; enforce G^T_{t-1} U_t = 0 to avoid previously sensitive directions. A projec","pith_inferences":["If the spectral story is right, the balancing recipe should transfer beyond continual learning to other low-rank parameter-efficient regimes — model merging, task arithmetic, and sequential fine-tuning in domains other than vision-language — wherever imbalanced components would produce the same overwriting pattern.","A cheap probe would be to keep the sU V^T factorization but drop the gradient-orthogonality constraint: if most of the forgetting reduction survives, then balance alone carries the effect and the geometric machinery matters only for the residual gap.","The paper's equal-norm comparison contrasts two end-to-end training paradigms (EBO vs standard LoRA); a tighter test would randomize only the singular values of otherwise identical adapters while holding total energy fixed, to separate 'balance helps' from 'smaller perturbation helps.'"],"forward_implications":["If spectral imbalance is the cause, then balancing the singular values of an update should reduce forgetting for any low-rank continual method, regardless of whether it uses replay, regularization, or architecture changes.","The stability–plasticity trade-off is looser than commonly believed: backward transfer goes from −15.4 to −2.0 on UCIT and from −11.2 to −0.7 on MLLM-DCL while final accuracy rises by more than ten points on both benchmarks.","The gains hold across heterogeneous task types — image classification, captioning, visual question answering, and multiple-choice — with no access to past data, so the principle applies where storing or replaying old samples is infeasible.","Forward transfer also improves (34.6 vs 26.8 on UCIT), indicating that balanced components generalize better to tasks never seen during training."],"fun_headline_variants":["Balancing singular spectra eliminates forgetting in continual learning","Equalizing update singular values prevents catastrophic forgetting","Spectral balance in low-rank updates curbs forgetting","Flattening singular value spectra reduces both forward and backward forgetting","Decoupling update magnitude and direction fights forgetting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The argument's load-bearing premise is that the experiments isolating balance actually isolate balance — replacing singular values with their mean also lowers the update's total energy whenever the spectrum is skewed, so the measured drop in interference could come from smaller updates rather than from equal-strength components; a second pillar is that the stored gradient subspace, whose size and sampling the paper leaves unspecified, faithfully captures every direction previ","fun_headline_variants_meta":{"raw":{"variants":["Balancing singular spectra eliminates forgetting in continual learning","Equalizing update singular values prevents catastrophic forgetting","Spectral balance in low-rank updates curbs forgetting","Flattening singular value spectra reduces both forward and backward forgetting","Decoupling update magnitude and direction fights forgetting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000161,"raw_usage":{"total_tokens":1051,"prompt_tokens":701,"completion_tokens":350,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":445,"completion_tokens_details":{"reasoning_tokens":277}},"tokens_in":445,"tokens_out":350,"duration_ms":4219,"temperature":1.0,"reasoning_tokens":277,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T05:57:10.864114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Merge LoRA adapters with singular values smoothed to the mean, then rescale the update so its total energy (sum of squared singular values) exactly matches the unsmoothed version: if the rescaling erases the interference benefit, then reduced energy, not balance, was causing the improvement. A second check: run the full method with the gradient-orthogonality constraint disabled while keeping the balanced factorization; if backward transfer degrades sharply, the orthogonality constraint, not the spectral balance, is doing the work.","supporting_citations":[],"review_version":1}