{"id":"d39e13e2-9803-413b-a498-c39d9b80f891","arxiv_id":"2505.15875","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A data-free LoRA merging framework that decouples weight magnitude from direction and orthogonalizes directions to reduce task interference, outperforming existing merging methods across vision, language and multimodal models.","lead":"Researchers propose DO-Merging, a training-free recipe that merges multiple LoRA-adapted models by separating each layer's weights into magnitude and direction, then orthogonalizing the directions with gradient descent. It reports consistent gains over existing merging methods on vision, language and multimodal benchmarks, and claims near free-lunch improvements when plugged into other merging pipelines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sec. 3.4's claim that orthogonalizing LoRA factors A_i and B_i separately is equivalent to orthogonalizing the product W_i=B_iA_i is false; Alg.","rationale":"The reader's weakest-assumption analysis identifies the same core mathematical error: separate orthogonality of A and B does not imply orthogonality of the product matrices used in the merge. This is the most load-bearing concern because the paper explicitly advertises theoretical guarantees for the orthogonalization component, and Algorithm 1 executes that component on A and B separately before constructing the full-rank directions. My analysis agrees with the reader's concern and with the conditional verdict: the empirical results are plausible and consistently positive, but the theoretical justification is unsound as written and the experimental evidence is not independently auditable. I do not see a reason to move to reject, because the empirical method could still work; the correct action is to require the authors to correct or remove the equivalence claim, release code and hyperparameters, and supply repeated-run statistics. Hence the reader's CONDITIONAL verdict should stand unchanged.","tokens_in":21774,"tokens_out":13946,"duration_ms":120529,"concrete_test":"Obtain the LoRA checkpoints and exact hyperparameters used for the ViT-B/32 eight-task experiment, then re-run Algorithm 1. After Step 1, compute the layer-wise off-diagonal product inner products tr((B_i A_i)^T (B_j A_j)) before and after the separate A/B orthogonalization. If the off-diagonal products are not substantially reduced toward zero, the orthogonalization step is not performing the mechanism the theory claims. In parallel, repeat the full merge with at least five seeds and report mean/std; if the 1.3–1.8 point average gains persist and the product inner products are near zero, the equivalence issue is a wording defect rather than a load-bearing flaw.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The low-cost version of the method hinges on the sentence in Sec. 3.4: 'we apply orthogonality separately on A and B, which is equivalent to orthogonality on the product matrix.' This equivalence is not true as stated. With W_i = B_i A_i, the Frobenius inner product of two products is tr((B_i A_i)^T B_j A_j) = tr(A_i^T B_i^T B_j A_j). Minimizing the pairwise inner products of the A_i and of the B_i separately controls tr(A_i^T A_j) and tr(B_i^T B_j) only; it does not force the product inner product to vanish. (The much stronger condition B_i^T B_j ≈ 0 would suffice, but Eq. 4 does not state a squared-norm objective that would drive the full matrix to zero.) Algorithm 1 therefore does not, as claimed, produce orthogonal full-rank direction matrices W_i. Consequently the theoretical support advertised around Theorem 3.3 and the statement in Sec. 3.2 that 'conclusions remain unaffected' do not follow from the presented mathematics. What remains is an empirical heuristic supported by single-run tables with no code, no error bars, and margins over the best baselines of roughly 1.3–1.8 points in Tables 1–2, which could be within run-to-run variation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DO-Merging, a data-free framework for merging LoRA adapters. It decomposes each LoRA weight matrix into per-column magnitudes and normalized directions, merges magnitudes by simple averaging, and merges directions after applying a layer-wise gradient-descent orthogonalization step. Theoretical claims are made both for the decoupling step (Theorems 3.1 and 3.2) and for the orthogonalization step (Theorem 3.3). Experiments are reported across ViT, T5, LLaMA-3, Qwen-14B, and Qwen2-VL benchmarks, with ablations on the two components and on combinations with existing methods.","tokens_in":22096,"tokens_out":2887,"duration_ms":23835,"significance":"If the empirical results are reproducible, the paper addresses a practical and under-studied problem: merging LoRA adapters without access to training data. The framework is simple, modular, and computationally light, and the paper demonstrates consistent gains over several existing merging baselines on a broad set of vision, language, and multimodal tasks. The ablation study showing that both the decoupling and orthogonalization components contribute to the final performance is a strength, as is the demonstration that the components can be combined with other merging methods. The theoretical guarantees, however, are the main advertised contribution, and several of them are not supported by the presented arguments; this makes the scope of the paper's technical claims substantially weaker than the text suggests.","major_comments":[{"comment":"The statement 'we apply orthogonality separately on A and B, which is equivalent to orthogonality on the product matrix' is false in general. With W_i = B_i A_i, the Frobenius inner product of two products is tr((B_i A_i)^T B_j A_j) = tr(A_i^T B_i^T B_j A_j). Separately minimizing the pairwise inner products of the A_i and of the B_i controls tr(A_i^T A_j) and tr(B_i^T B_j) only; it does not force the product inner product to vanish. Consequently, the full-rank direction matrices W_i are not orthogonal as claimed, and the theoretical justification for the low-cost variant in Sec. 3.2 ('conclusions remain unaffected') and the support for Theorem 3.3 do not follow from the presented mathematics. The authors should either prove the claimed equivalence under additional assumptions on A_i and B_i, or revise the algorithm and claims to reflect what the separate orthogonalization actually controls.","section":"Sec. 3.4 and Algorithm 1 (lines 2–3)"},{"comment":"The proof of Theorem 3.3 does not establish the stated claim. The theorem asserts that smaller ||W_i^T W_j|| leads to less conflict during merging, but the proof only analyzes a single matrix entry and argues that gradient descent on the loss L_o pushes one entry toward another; it never shows that a smaller matrix-level inner product reduces the number of sign conflicts. Moreover, Eq. (4) defines L to contain exactly the quantity ||W_i^T W_j|| (via the cross terms), so showing that gradient descent reduces this quantity is close to assuming the conclusion. The theorem should either be restated as an empirical motivation, or a complete proof connecting the loss to a well-defined notion of conflict should be supplied.","section":"Sec. 3.4, Theorem 3.3 and Appendix B.3"},{"comment":"The proof of Theorem 3.2 contains algebraic steps that are not substantiated. In Case 2, the expansion from Eq. (24) to Eq. (26) drops terms that do not vanish under the stated assumptions: the expectations involving products of entries of W1 and W2 are not handled, and the coefficient (1/8)(alpha1+alpha2)^2 appears without derivation. The resulting expression for E(L2) is therefore not trustworthy, and the claimed inequality E(L2) < E(L1) for ||alpha1|| != ||alpha2|| is not established. The theorem is load-bearing for the decoupling contribution, so this needs a corrected proof or a downgrade of the claim.","section":"Sec. 3.3, Theorem 3.2 and Appendix B.2"},{"comment":"Theorem 3.1 is built on a loss function, Eq. (1), that is hand-chosen to penalize norm mismatch: the weights on the two distance terms are inversely proportional to ||alpha_i||^2, and the theorem essentially restates that this particular loss is minimized when the two norms are equal. This provides limited support for the paper's broader observational claim in Fig. 1(c). Additionally, Assumption 3.1 writes W_i = alpha_i * W_i with alpha_i nonnegative, but the proof operates with scalar norms ||alpha_1|| and ||alpha_2|| without clarifying how a vector alpha is reduced to a scalar; the notation should be made precise.","section":"Sec. 3.2, Theorem 3.1 and Eq. (1)"},{"comment":"All experimental results appear to be reported from a single run, with no standard errors, confidence intervals, or multiple random seeds. The average improvements over the strongest baselines are about 1.3–1.8 points in Tables 1 and 3, which could be within run-to-run variation for large model merges. For the paper to support the claim of 'significantly higher performance,' the authors should provide variance estimates or at least multiple runs, and they should make code available to allow reproduction.","section":"Tables 1–6 and Sec. 4"}],"minor_comments":[{"comment":"The text refers to 'Fig. 4(c)' when discussing full-finetune merging, but Fig. 4(c) is captioned 'The Impact of LoRA Rank'; the referenced result appears to be in Fig. 5, so the cross-reference should be corrected.","section":"Sec. 4.5, 'Transfer to Full-Finetune Merging'"},{"comment":"The same symbol W (and alpha) is used both for the decomposed matrix in Eq. (2) and for the random matrix in Assumption 3.1, with only font differences that are easy to miss; clearer notation or a statement of overloading would help.","section":"Eq. (2) and Assumption 3.1"},{"comment":"The loss in Eq. (4) does not specify whether the sum over i and j includes the diagonal terms i=j; if it does, the regularization term ||delta_i||^2 is not the only term preventing trivial solutions, and the objective should be stated more explicitly.","section":"Sec. 3.4, Eq. (4)"},{"comment":"The definition of Magnitude Distribution Variance applies variance over the n task index for each layer's absolute entries, but the notation var(|W_i^k|_{i=1}^n) is ambiguous and should be written as an explicit sum over columns and rows.","section":"Appendix C.4, Eq. (38)"},{"comment":"The open-source weight links for Qwen2-VL tasks are given as a single footnote with task names but no URLs or HF identifiers; the links should be resolved for reproducibility.","section":"Appendix C.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a practical LoRA-merging method with broad empirical coverage, but the theoretical guarantees—which are prominently advertised in the abstract and main text—are not reliable. The false equivalence between separate A/B orthogonalization and product-matrix orthogonalization is a load-bearing error for the low-cost version of the method. This is fixable revision-wise if the authors either prove the equivalence under realistic assumptions or substantially soften the theoretical claims and reposition the method as an empirical heuristic. I would also encourage the editor to ask for code and multi-seed results, since single-run tables with 1–2 point gaps are not yet convincing evidence of 'significantly higher performance.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on arXiv 2505.15875. The empirical recipe is worth a look; the theory as written does not hold up.\n\nWhat's genuinely new: the combination of decoupling column-norm magnitudes from directions and applying data-free, layer-wise orthogonalization to the direction components before merging LoRA adapters. The experiments are broad — ViT-B/32, B/16, L/14, T5-base/large, LLaMA-3-8B, Qwen-14B, Qwen2-VL — and the reported gains over Task Arithmetic, Ties, TSVM, KNOTS, CoPA are consistent, roughly 1–4 points. The ablations and flexible-combination studies are useful and suggest both components contribute.\n\nThe soft spots are real. The killer is Sec. 3.4: the claim that applying orthogonality separately on A and B is equivalent to orthogonality on the product matrix is false. Orthogonalizing A_i and B_i separately does not make the products B_i A_i orthogonal; the inner product tr((B_i A_i)^T B_j A_j) = tr(A_i^T B_i^T B_j A_j) is not controlled by tr(A_i^T A_j) and tr(B_i^T B_j) alone. The algorithm's advertised guarantees and the \"conclusions remain unaffected\" claim rest on this equivalence, so they don't follow. The theoretical support in Theorems 3.1–3.3 is also shaky: the loss in Eq. 1 already penalizes norm mismatch, so Theorem 3.1 is close to built into the loss; Theorem 3.3's proof does not establish the stated claim about fewer sign conflicts; and the orthogonalization objective Eq. 4 is the very quantity Theorem 3.3 claims beneficial, making it close to assuming the conclusion.\n\nOn the empirical side, there is no code, no error bars, and no repeated-run statistics. Several margins over the strongest baseline are 1.3–1.8 points, which could be run-to-run variation. The paper's own limitations section only mentions model selection, leaving the theoretical and statistical gaps unaddressed.\n\nIf the method works, it is a useful practical tool and a reasonable contribution to LoRA merging. But as it stands, the paper overclaims: the theory is not \"guarantees,\" and the low-cost variant's justification collapses. The right path is major revision: release code with hyperparameters, add repeated runs, correct or remove the A/B equivalence claim, and either fix the theorems or present the method as a heuristic.\n\nVerdict: this deserves a serious referee, not a desk reject. The empirical recipe is plausible and the evaluation breadth is real. But I'd want to see the code and the corrected claims before trusting the numbers.","headline":"A practically plausible LoRA merging recipe undercut by an incorrect equivalence claim and thin statistics.","tokens_in":22591,"tokens_out":1887,"would_cite":false,"duration_ms":15911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Separating LoRA weights into magnitude and direction, then orthogonalizing directions data-free, lifts merged-model accuracy by up to 4.8 percent over existing methods.","keywords":["model merging","LoRA","low-rank adaptation","parameter decoupling","orthogonalization","data-free optimization","task interference","magnitude variance"],"falsifier":"Take two fine-tuned LoRA modules, run the paper's orthogonalization on $A_1,B_1$ and $A_2,B_2$ separately, and compute the Frobenius inner product $|\\langle B_1A_1, B_2A_2\\rangle_F|$. If this stays as large as it was before orthogonalization even though $\\langle A_1,A_2\\rangle$ and $\\langle B_1,B_2\\rangle$ are near zero, then the Sec. 3.4 equivalence fails and the low-cost variant loses its stated guarantee.","tokens_in":21585,"feed_emoji":"🧩","tokens_out":11718,"duration_ms":85177,"temperature":0.7,"pith_summary":"Existing model-merging methods designed for full fine-tuning fail when applied to LoRA, and the paper argues this is because LoRA modules trained on different tasks have much larger variance in parameter magnitude than full fine-tuned weights. To fix this, the paper proposes DO-Merging, which splits each LoRA weight matrix into a column-norm magnitude vector and a normalized direction matrix, merges the two parts separately, and applies a data-free layer-wise orthogonalization to reduce task interference in the directions. The merged update takes the form $W_{\\mathrm{out}} = W_{\\mathrm{pre}} + \\lambda(\\sum_i \\alpha_i)(\\sum_j \\bar{W}_j)$, and the paper provides theoretical arguments that both the decoupling and the orthogonalization reduce expected merging loss. Experiments across vision, language, and multi-modal models report consistent gains over existing methods, with the two components also improving other merging methods when used as plug-ins.","feed_headline":"Data-free LoRA merging boosts accuracy up to 4.8%","feed_subtitle":"The method splits weights into magnitude and direction, then aligns directions without any training data.","key_machinery":"The central object is the decouple-and-orthogonalize decomposition of each LoRA update $W = BA$ into a magnitude vector $\\alpha$, whose entries are the column norms, and a direction matrix $\\bar{W}$, whose columns are unit-norm. The paper's orthogonalization step is a data-free layer-wise gradient descent on the loss $\\mathcal{L} = \\sum_{i,j} (W_i + \\delta_i)^\\top (W_j + \\delta_j) + \\sum_i \\|\\delta_i\\|^2$, which pushes the task vectors toward mutual orthogonality while keeping the perturbations small; for LoRA this is applied to $A$ and $B$ separately to cut cost. Merging then follows the decoupled formula $W_{\\mathrm{out}} = W_{\\mathrm{pre}} + \\lambda(\\sum_i \\alpha_i)(\\sum_j \\bar{W}_j)$. The theoretical engines are Theorems 3.1–3.3, which relate magnitude imbalance to expected merging loss and orthogonality to conflict reduction.","core_discovery":"The paper's central claim is that the poor performance of existing merging methods on LoRA is caused by magnitude disparity, not by the low-rank structure itself. DO-Merging establishes that by (i) extracting each column's norm as a magnitude vector and merging magnitudes by averaging, and (ii) applying gradient descent on the direction matrices with an orthogonality penalty, the merged model $W_{\\mathrm{out}} = W_{\\mathrm{pre}} + \\lambda(\\sum_i \\alpha_i)(\\sum_j \\bar{W}_j)$ preserves each task's output mapping better than merging raw weights. The theoretical part shows that when two tasks' magnitude norms differ, the expected loss of the decoupled merge is strictly smaller than that of the naive average, and that reducing the inner product between direction matrices reduces parameter conflicts as the perturbation size shrinks. The authors verify this with experiments on vision transformers, T5, LLaMA-3, Qwen, and Qwen2-VL, reporting average gains of roughly 2–3% over the strongest baselines at negligible extra cost.","pith_inferences":["A natural extension not tested in the paper is to compute the magnitude vector from activation statistics or per-output-neuron norms; the output-alignment argument in Sec. 3.3 suggests this could outperform the fixed column-norm choice.","The data-free orthogonalization could be used as a generic preprocessing step before any merging or interpolation scheme, since it requires no labels and no forward passes; the paper only demonstrates it inside DO-Merging and on two external baselines.","The diagnosis that magnitude variance drives merging failure suggests that LoRA fine-tuning procedures that constrain column-norm growth could produce models that are easier to merge, an implication the authors do not pursue.","Theorem 3.1 is proven for two tasks; extending the analysis to $n$ tasks with per-layer magnitudes would give a quantitative prediction relating variance to expected loss that could be tested across a wider benchmark suite."],"forward_implications":["Users can merge LoRA-tuned models without any training data or additional fine-tuning, using only the fine-tuned weights and a cheap layer-wise gradient step.","The decoupling and orthogonalization components can be added to existing merging algorithms such as Ties-Merging and Breadcrumbs, giving near-free improvements of almost 4% on the paper's benchmarks.","The same pipeline transfers to fully fine-tuned models, where magnitude variance is smaller, so the method is not limited to LoRA.","Larger LoRA ranks show larger gains under DO-Merging, so the advantage over baselines grows with rank.","The theoretical link between magnitude variance and merging loss implies that column-norm spread across tasks can serve as a cheap predictor of merging difficulty."],"supporting_citations":[{"why":"Defines the LoRA low-rank decomposition W = BA that the merging framework operates on.","marker":"[14]"},{"why":"Defines task arithmetic, the baseline merging formula that DO-Merging is compared against and modifies.","marker":"[18]"},{"why":"Ties-Merging is the main conflict-resolution baseline shown to underperform on LoRA.","marker":"[41]"},{"why":"Shows that imposing orthogonality between LoRA modules during fine-tuning improves merging, motivating the data-free orthogonalization step.","marker":"[48]"},{"why":"Supplies the weight-disentanglement idea that the paper adapts to a no-training decoupling.","marker":"[45]"},{"why":"A LoRA-specific merging baseline that also notes LoRA parameter discrepancies but does not address magnitude variance.","marker":"[33]"},{"why":"The task-singular-vector method that the paper's orthogonalization extends and the experiments must beat.","marker":"[10]"}],"fun_headline_variants":["Decouple and orthogonalize: the data-free fix for LoRA merging","LoRA merging fails on magnitude variance; new data-free method fixes it","Split LoRA magnitudes and directions for better zero-data merging","Data-free LoRA merge: decouple and orthogonalize for up to 4.8% gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that orthogonalizing the LoRA factors $A$ and $B$ separately makes the full product matrices $BA$ orthogonal to each other, which is not generally true.","fun_headline_variants_meta":{"raw":{"variants":["Decouple and orthogonalize: the data-free fix for LoRA merging","LoRA merging fails on magnitude variance; new data-free method fixes it","Split LoRA magnitudes and directions for better zero-data merging","Data-free LoRA merge: decouple and orthogonalize for up to 4.8% gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001059,"raw_usage":{"total_tokens":4471,"prompt_tokens":1003,"completion_tokens":3468,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":3386}},"tokens_in":619,"tokens_out":3468,"duration_ms":24531,"temperature":1.0,"reasoning_tokens":3386,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:12:12.449569+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two fine-tuned LoRA modules, run the paper's orthogonalization on $A_1,B_1$ and $A_2,B_2$ separately, and compute the Frobenius inner product $|\\langle B_1A_1, B_2A_2\\rangle_F|$. If this stays as large as it was before orthogonalization even though $\\langle A_1,A_2\\rangle$ and $\\langle B_1,B_2\\rangle$ are near zero, then the Sec. 3.4 equivalence fails and the low-cost variant loses its stated guarantee.","supporting_citations":[{"cited_title":"Model merging with svd to tie the knots","cited_arxiv_id":null,"evidence_quote":"A LoRA-specific merging baseline that also notes LoRA parameter discrepancies but does not address magnitude variance."}],"review_version":1}