REVIEW 3 major objections 6 minor 27 references
Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that initializing each LoRA adapter's projection matrix along the lowest-variance directions of other tasks' features before fine-tuning removes most cross-task interference in merged models.
desk verdict Cheap, apparently effective LoRA-initialization trick for merging, but the causal story is unverified and the post-hoc extension is badly oversold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the cross-task output shift $B_2A_2h_1$ in the merged weight $W_m = W_0 + B_1A_1 + B_2A_2$. The paper replaces the common goal of orthogonal task vectors with a data-aware objective: minimize $\lVert A_2H_1^\top\rVert_F$ subject to $A_2A_2^\top = I$, where $H_1$ is the matrix of task-1 latent features across the network. Since $H_1$ is full-rank, exact annihilation is impossible, so the analytical solution picks the last $r$ eigenvectors of the sample covariance $S = H_1^\top H_1/(k-1)$, the directions where task-1 features have the least variance; the orthonormal-row constraint removes the scale ambiguity between $B_2$ and $A_2$ while leaving $B_2$ free to fit task 2.
What would settle it
Measure the realized interference term $\lVert B_2A_2H_1^\top\rVert_F$ on held-out features before and after fine-tuning and check whether merged accuracy tracks it. If merged accuracy rises while this realized norm does not fall, or if an equally small post-training rotation from a random initialization produces the same gains, the claimed mechanism is not the cause.
Extended reading notes
Core claim
At the heart of the paper is the observation that when two LoRA models are merged as $W_m = W_0 + B_1A_1 + B_2A_2$, a task-1 feature $h_1$ produces $W_mh_1 = W_1h_1 + B_2A_2h_1$; the second term is the unintended shift caused by task 2's adapter. Existing methods orthogonalize or align task vectors without looking at data, and the paper argues this misses the real source of damage. Its proposed OSRM instead minimizes $\lVert A_2H_1^\top\rVert_F$ over orthonormal-row matrices $A_2$, using latent features $H_1$ from every other task, and proves a closed-form solution that initializes $A_2$ to the eigenvectors of the other task's feature covariance with the smallest eigenvalues. After fine-tuning, when $A_2$ is allowed to move, the paper shows via an orthogonal Procrustes distance that the drift is marginal, and that merging accuracy improves across five language models and all tested merging algorithms.
Load-bearing premise
The load-bearing premise is that the input-projection matrix inside each LoRA adapter stays close, after fine-tuning, to the initialization that was chosen to avoid other tasks' data directions, and that the companion B matrix does not amplify whatever residual overlap remains.
Editorial extensions
If this is right
- Any existing merging algorithm that averages or manipulates LoRA weights can be combined with OSRM, and the paper reports higher average accuracy for task arithmetic, RegMean, Fisher, TIES, and EMR across all tested backbones.
- Because OSRM acts before fine-tuning, the per-task models are not retrained or re-routed at merge time; single-task accuracy is essentially unchanged, with an average gap below 1%.
- The benefit grows with the number of merged tasks: on RoBERTa-large the advantage over baselines becomes clear once more than five tasks are merged, which is the regime that makes model storage and deployment expensive.
- The method is insensitive to the merging scaling coefficient, so practitioners can set task-arithmetic-style hyperparameters coarsely rather than tuning them per merge.
- For externally obtained LoRA checkpoints, OSRM can still be applied post hoc by re-decomposing each learned weight into an OSRM-style $\tilde{A}$ and a least-squares $B$; merged performance is largely preserved, though this post hoc route drops individual-task accuracy by a large margin.
Reading between the lines
- A stronger test of the mechanism than the one in the paper is to measure the realized interference norm $\lVert B_2A_2H_1^\top\rVert_F$ on validation features before and after fine-tuning, rather than only the rotation of $A$; the paper's Procrustes evidence does not directly show that the interference term itself shrank.
- Because the method needs only a few dozen to a hundred latent feature vectors per task, it transfers naturally to settings where raw training data cannot be pooled, such as federated or privacy-restricted training; the paper lists privacy as a motivation but does not evaluate that scenario.
- The non-monotonic dependence on the sample count $k$ suggests that smallest-eigenvalue directions estimated from very large samples start capturing shared cross-task knowledge, so $k$ behaves as a regularization knob rather than a simple sample-size knob; modifying the objective to exclude shared directions explicitly could improve the large-$k$ regime.
- The same data-aware initialization idea could in principle be applied to other parameter-efficient adapters whose updates factor into two matrices, but the paper does not test such extensions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes OSRM, a pre-fine-tuning initialization strategy for LoRA that aims to reduce interference when merging task-specific models. The authors observe that the merged-model output on a task receives a perturbation from other tasks' LoRA updates, and they propose to initialize each LoRA's A matrix in the subspace of smallest variance of the other tasks' features, derived from the analytical solution of a constrained Frobenius-norm minimization. The method is designed to be compatible with existing merging algorithms. Experiments on eight GLUE tasks with RoBERTa-large, T5-large, Llama3.2-1B, Llama3.2-3B, and Llama3-8B report average improvements over several merging baselines, with a relaxation that allows A to be updated during fine-tuning.
Significance. If the proposed mechanism is validated, the contribution is practically valuable: it is a simple, plug-and-play initialization that can be combined with existing merging methods and requires only a small number of latent feature samples. The analytical solution in Appendix A is correct for the stated surrogate objective, and the paper provides code, which supports reproducibility. However, the causal claim connecting Eq. (3) to the empirical merging gains is not sufficiently established: the post-fine-tuning verification is weak, the objective is a surrogate that ignores B, and the experiments are single-seed with several small or negative differences. The significance of the result is therefore contingent on additional evidence.
major comments (3)
- [§4.4, Fig. 3] The paper's central mechanism is that initializing A via Eq. (3) reduces the cross-task interference term B2 A2 H1^T in the merged model. Section 4.4 relaxes the orthogonality constraint and allows A to be updated, and the only evidence that the initialization remains valid is the normalized orthogonal Procrustes distance in Fig. 3. This metric is not tied to the actual interference objective: a small rotational distance does not imply a small change in ||A_ft H1^T||_F, especially when the feature covariance is ill-conditioned, because a rotation can mix high-variance directions into the row space. Moreover, the true interference is ||B_ft A_ft H1^T||_F, and B is unconstrained and may rescale any residual. The authors should directly measure the post-fine-tuning interference term (or a faithful proxy) with and without OSRM, and ideally ablate frozen-A versus updated-A to separate the initialization's effect from altered training dynamics. Without such evidence, the mechanism proposed in Section 4.2 remains a plausible hypothesis rather than the demonstrated cause of the observed gains.
- [§5.2, Tables 1–5] All merging and single-task results are reported from a single seed with no error bars or significance tests. Several of the claimed improvements are below one percentage point (e.g., Table 4, RegMean average 45.53 vs. 45.77; Table 3, Fisher average 61.06 vs. 59.64 where OSRM is worse; Table 5, TIES average 53.01 vs. 52.69 where OSRM is worse). The abstract's statement that OSRM "consistently outperforms" baselines is thus not statistically supported. In addition, Table 6 shows strong sensitivity to the sample size k, with the RegMean average ranging from 47.66 (k=2) to 70.42 (k=10) and 69.81 (k=100); this non-monotonic behavior indicates that the robustness claim needs variance reporting and multiple runs.
- [§4.2, Eq. (2)] The optimization in Eq. (2) minimizes ||A2 H1^T||_F, but the actual interference term in Eq. (1) is B2 A2 h1. Because B2 is left unconstrained, the surrogate objective does not by itself guarantee low interference; a learned B2 of large norm can amplify any residual overlap between A2 and H1. The paper discusses the scale ambiguity between B and A but does not provide evidence that the learned B matrices remain well-behaved under OSRM. The authors should measure the relative norms of ||A_ft H1^T||_F and ||B_ft A_ft H1^T||_F after fine-tuning, or otherwise validate that reducing the A-side norm is sufficient to reduce the full interference term.
minor comments (6)
- [§4.4] The phrase "validly orthogonal to the latent features" is not defined precisely; the authors should state what quantitative condition would falsify the relaxation.
- [Fig. 3] The y-axis label "Change of A (%)" is not self-contained; the caption should specify that the metric is the normalized orthogonal Procrustes distance D/||A_init||_F, and the units should be explained.
- [§5.4, Table 16] The post-hoc extension degrades individual performance dramatically (average 35.07 versus 88.05 for the original models), and the paper acknowledges this is a future challenge; this limitation should be stated in the main text rather than only in the appendix, since the method is advertised as "plug-and-play" and users may expect it to work on existing LoRA checkpoints.
- [References] Several references have incomplete or inconsistent formatting (e.g., "Loshchilov, 2017" lacks the author's first name, and entries such as "Yue Gang, Jianhong Shun, and Mu Qing. 2025" are not in the standard style used elsewhere).
- [§5.2] The model name is written inconsistently as "Llama3.2-3B" and "LLaMA3-8B" and "Llama3-8B"; please standardize the capitalization.
- [Algorithm 1] In line 5, "Average sample-wise features to get {H̄_t}" is ambiguous: it should clarify that the average produces a single mean vector per layer (or a matrix of per-sample averages), and how that relates to the covariance computation in Eq. (3).
Circularity Check
No significant circularity: the OSRM initialization is the provable minimizer of an objective defined on other tasks' features, and no benchmark-derived constant is hidden in the derivation; the Fig. 3 Procrustes check is a proxy validation gap, not a reduction.
full rationale
The paper's derivation chain is self-contained and not circular. The central precursor, Eq. (2)'s objective min ||A H^T||_F subject to A A^T = I, is minimized by Eq. (3) (eigenvectors of the other task's feature covariance with the smallest eigenvalues), as proven in Appendix A; this is a genuine mathematical reduction whose only inputs are the other tasks' latent features, with no benchmark metric or merged-accuracy value entering the initialization. Improvement over baselines is therefore an empirical finding, not a fitted quantity: hyperparameters (k=100 samples for OSRM, lambda=0.3 for TA) are shared across OSRM and baseline, and Fig. 4/Table 6 show gains across the lambda and k ranges. The one soft spot is the justification for the relaxed constraint: Section 4.4 validates the fine-tuned A via the normalized orthogonal Procrustes distance (Fig. 3), which measures deviation from the initialization rather than the actual interference term ||B_ft A_ft H^T||_F; this is a validation gap that leaves the causal mechanism imperfectly verified, but it is not a circular reduction, since the claimed conclusion (effective orthogonality to other tasks' features) is not defined in terms of that metric. The paper is candid about its limits: Section 4.4 admits freezing A helps merging but degrades single-task accuracy; Section 5.4/Table 16 shows post-hoc OSRM fixes A to the Eq. (3) solution by construction, trivially preserving the Eq. (2) property, yet individual and merged performance collapse to about 35%, which the paper reports honestly - demonstrating that the 'property' is not being passed off as the target result. No load-bearing self-citations exist: all references (LoRA, TA/TIES/EMR, KnOTS, and the Gower & Dijksterhuis Procrustes textbook) are external, and Section 7 states the scope limitations. Accordingly, no equation reduces to its own input by construction; the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (5)
- number of samples k =
100
- LoRA rank r =
8
- LoRA scaling factor alpha =
16
- merging scaling coefficient lambda =
0.3 for TA, 1.0 for TIES
- RegMean non-diagonal multiplier for T5-large =
0.1
assumptions (4)
- standard math Ky Fan / Poincare separation theorem: minimizing tr(A S A^T) with orthonormal rows selects eigenvectors with smallest eigenvalues.
- domain assumption The learned LoRA matrices B and A are mostly full-rank, so A can be assumed orthonormal without loss of generality.
- domain assumption The merged model's interference is dominated by the cross term B_j A_j h_i, and B_j being unconstrained will not undo the orthogonality of A_j.
- domain assumption Averaging sample-wise features per task (Eq. 4) is a valid proxy for the full concatenated feature matrix when merging many tasks.
Cite this review
Pith. "Pith review of Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging." pith.science (2026). https://pith.science/paper/4JYV5KLW
@misc{pith2026250522934,
author = {Pith},
title = {Pith review of: Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging},
year = {2026},
howpublished = {\url{https://pith.science/paper/4JYV5KLW}},
note = {Machine review of arXiv:2505.22934}
}
read the original abstract
Fine-tuning large language models (LMs) for individual tasks yields strong performance but is expensive for deployment and storage. Recent works explore model merging to combine multiple task-specific models into a single multi-task model without additional training. However, existing merging methods often fail for models fine-tuned with low-rank adaptation (LoRA), due to significant performance degradation. In this paper, we show that this issue arises from a previously overlooked interplay between model parameters and data distributions. We propose Orthogonal Subspaces for Robust model Merging (OSRM) to constrain the LoRA subspace *prior* to fine-tuning, ensuring that updates relevant to one task do not adversely shift outputs for others. Our approach can seamlessly integrate with most existing merging algorithms, reducing the unintended interference among tasks. Extensive experiments on eight datasets, tested with three widely used LMs and two large LMs, demonstrate that our method not only boosts merging performance but also preserves single-task accuracy. Furthermore, our approach exhibits greater robustness to the hyperparameters of merging. These results highlight the importance of data-parameter interaction in model merging and offer a plug-and-play solution for merging LoRA models.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[4]
arXiv preprint arXiv:2407.21783
The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Yue Gang, Jianhong Shun, and Mu Qing
-
[5]
arXiv preprint arXiv:2403.08994
Ethos: Rectifying lan- guage models in orthogonal parameter space. arXiv preprint arXiv:2403.08994. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan
-
[6]
arXiv preprint arXiv:2408.13656
Localize-and-stitch: Efficient model merging via sparse task arithmetic. arXiv preprint arXiv:2408.13656. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen
-
[8]
arXiv preprint arXiv:2405.17461
Emr-merging: Tuning-free high-performance model merging. arXiv preprint arXiv:2405.17461. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Worts- man, Suchin Gururangan, Ludwig Schmidt, Han- naneh Hajishirzi, and Ali Farhadi
-
[9]
arXiv preprint arXiv:2212.04089
Edit- ing models with task arithmetic. arXiv preprint arXiv:2212.04089. Shankar Iyer, Nikhil Dandekar, Kornél Csernai, et al
-
[10]
arXiv preprint arXiv:2212.09849
Dataless knowledge fu- sion by merging weights of language models. arXiv preprint arXiv:2212.09849. So Kuroki, Taishi Nakamura, Takuya Akiba, and Yujin Tang
-
[11]
arXiv preprint arXiv:2410.14735
Agent skill acquisition for large language models via cycleqd. arXiv preprint arXiv:2410.14735. Yinhan Liu
-
[13]
arXiv preprint arXiv:1711.05101
Decoupled weight decay regulariza- tion. arXiv preprint arXiv:1711.05101. Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dan- gyang Chen, and Yu Cheng
Show all 27 references
-
[14]
arXiv preprint arXiv:2406.15479
Twin-merging: Dynamic integration of modular expertise in model merging. arXiv preprint arXiv:2406.15479. Michael S Matena and Colin A Raffel
-
[15]
arXiv preprint arXiv:2410.13025
Lora soups: Merging loras for practical skill compo- sition tasks. arXiv preprint arXiv:2410.13025. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al
-
[16]
Nevertheless, the merged performance remains largely preserved, demonstrating the effectiveness of our method
When OSRM is applied after fine-tuning, the origi- nal LoRA weight matrices cannot be perfectly re- covered, resulting in a significant drop in the indi- vidual performance of the models. Nevertheless, the merged performance remains largely preserved, demonstrating the effecti...
2025
-
[17]
arXiv preprint arXiv:2410.19735
Model merging with svd to tie the knots. arXiv preprint arXiv:2410.19735. Derek Tam, Mohit Bansal, and Colin Raffel
-
[18]
Transactions on Machine Learning Research
Merging by matching models in task parameter sub- spaces. Transactions on Machine Learning Research. Anke Tang, Li Shen, Yong Luo, Yibing Zhan, Han Hu, Bo Du, Yixin Chen, and Dacheng Tao. 2023a. Pa- rameter efficient multi-task model fusion with partial linearization. arXiv pr...
-
[19]
arXiv preprint arXiv:2411.18607
Task arithmetic through the lens of one-shot federated learning. arXiv preprint arXiv:2411.18607. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al
-
[20]
arXiv preprint arXiv:2307.09288
Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman
-
[21]
In the Pro- ceedings of ICLR
GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In the Pro- ceedings of ICLR. Ke Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jimenez, Francois Fleuret, and Pas- cal Frossard. 2024a. Lines: Post-training layer scal- in...
-
[23]
arXiv preprint arXiv:2409.16167
Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering. arXiv preprint arXiv:2409.16167. Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen
-
[24]
arXiv preprint arXiv:2406.11385
Metagpt: Merging large language mod- els using model exclusive task arithmetic. arXiv preprint arXiv:2406.11385. A Proof. We prove that Eq. (3) is an analytical solution to the problem Eq. (2). Proof. Let S = 1 k−1 H ⊤ 1 H1 be the covariance ma- trix of H1, which is symmetric ...
-
[25]
All the experiments are conducted on eight NVIDIA RTX A6000 GPUs
Similar to (Liu, 2019), we use a grid search for the optimal hyper- parameters. All the experiments are conducted on eight NVIDIA RTX A6000 GPUs. E More Experimental Results E.1 Averaged Results We show the averaged performance of each model across all datasets in Tables 11 to...
2019
-
[27]
Individual
and LLM agents (Kuroki et al., 2024). We also believe Table 8: Dataset details in the GLUE benchmark. Acc. and cc. mean accuracy and correlation coefficient, respectively. Dataset #Train (K) #Val (K) #Test (K) Metric CoLA 8.55 1.04 1.06 Matthews cc. MNLI 393 9.82 9.8 Acc. MRPC...
2024
-
[2017]
arXiv preprint arXiv:1708.00055
Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055. MohammadReza Davari and Eugene Belilovsky
2017 arXiv
-
[2018]
arXiv preprint 1805.12471
Neural network acceptability judgments. arXiv preprint 1805.12471. Adina Williams, Nikita Nangia, and Samuel R. Bow- man
-
[2019]
arXiv preprint arXiv:1907.11692,
Roberta: A robustly opti- mized bert pretraining approach. arXiv preprint arXiv:1907.11692,
1907 arXiv
-
[2021]
arXiv preprint arXiv:2106.09685
Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.09685. Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xi- angyu Yue, and Wanli Ouyang
-
[2022]
arXiv preprint arXiv:2209.04836
Git re-basin: Merging models modulo permutation symmetries. arXiv preprint arXiv:2209.04836. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia
-
[2023]
arXiv preprint arXiv:2305.03053
Zipit! merging models from different tasks without training. arXiv preprint arXiv:2305.03053. George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman
-
[2024]
arXiv preprint arXiv:2410.02396
Parameter com- petition balancing for model merging. arXiv preprint arXiv:2410.02396. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.