Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Separating LoRA weights into magnitude and direction, then orthogonalizing directions data-free, lifts merged-model accuracy by up to 4.8 percent over existing methods.

desk verdict A practically plausible LoRA merging recipe undercut by an incorrect equivalence claim and thin statistics. read the letter →

arxiv 2505.15875 v1 pith:N5DDY4IN submitted 2025-05-21 cs.CV cs.LG

Shenghe Zheng , Hongzhi Wang , Chenyu Huang , Xiaohui Wang , Tao Chen , Jiayuan Fan , Shuyue Hu , Peng Ye This is my paper · ORCID
classification cs.CVcs.LG
keywords modelmergingLoRAlow-rankadaptationparameterdecouplingorthogonalizationdata-freeoptimizationtaskinterferencemagnitudevariance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing model-merging methods designed for full fine-tuning fail when applied to LoRA, and the paper argues this is because LoRA modules trained on different tasks have much larger variance in parameter magnitude than full fine-tuned weights. To fix this, the paper proposes DO-Merging, which splits each LoRA weight matrix into a column-norm magnitude vector and a normalized direction matrix, merges the two parts separately, and applies a data-free layer-wise orthogonalization to reduce task interference in the directions. The merged update takes the form $W_{\mathrm{out}} = W_{\mathrm{pre}} + \lambda(\sum_i \alpha_i)(\sum_j \bar{W}_j)$, and the paper provides theoretical arguments that both the decoupling and the orthogonalization reduce expected merging loss. Experiments across vision, language, and multi-modal models report consistent gains over existing methods, with the two components also improving other merging methods when used as plug-ins.

What carries the argument

The central object is the decouple-and-orthogonalize decomposition of each LoRA update $W = BA$ into a magnitude vector $\alpha$, whose entries are the column norms, and a direction matrix $\bar{W}$, whose columns are unit-norm. The paper's orthogonalization step is a data-free layer-wise gradient descent on the loss $\mathcal{L} = \sum_{i,j} (W_i + \delta_i)^\top (W_j + \delta_j) + \sum_i \|\delta_i\|^2$, which pushes the task vectors toward mutual orthogonality while keeping the perturbations small; for LoRA this is applied to $A$ and $B$ separately to cut cost. Merging then follows the decoupled formula $W_{\mathrm{out}} = W_{\mathrm{pre}} + \lambda(\sum_i \alpha_i)(\sum_j \bar{W}_j)$. The theoretical engines are Theorems 3.1–3.3, which relate magnitude imbalance to expected merging loss and orthogonality to conflict reduction.

What would settle it

Take two fine-tuned LoRA modules, run the paper's orthogonalization on $A_1,B_1$ and $A_2,B_2$ separately, and compute the Frobenius inner product $|\langle B_1A_1, B_2A_2\rangle_F|$. If this stays as large as it was before orthogonalization even though $\langle A_1,A_2\rangle$ and $\langle B_1,B_2\rangle$ are near zero, then the Sec. 3.4 equivalence fails and the low-cost variant loses its stated guarantee.

Watch

Extended reading notes

Core claim

The paper's central claim is that the poor performance of existing merging methods on LoRA is caused by magnitude disparity, not by the low-rank structure itself. DO-Merging establishes that by (i) extracting each column's norm as a magnitude vector and merging magnitudes by averaging, and (ii) applying gradient descent on the direction matrices with an orthogonality penalty, the merged model $W_{\mathrm{out}} = W_{\mathrm{pre}} + \lambda(\sum_i \alpha_i)(\sum_j \bar{W}_j)$ preserves each task's output mapping better than merging raw weights. The theoretical part shows that when two tasks' magnitude norms differ, the expected loss of the decoupled merge is strictly smaller than that of the naive average, and that reducing the inner product between direction matrices reduces parameter conflicts as the perturbation size shrinks. The authors verify this with experiments on vision transformers, T5, LLaMA-3, Qwen, and Qwen2-VL, reporting average gains of roughly 2–3% over the strongest baselines at negligible extra cost.

Load-bearing premise

The load-bearing premise is that orthogonalizing the LoRA factors $A$ and $B$ separately makes the full product matrices $BA$ orthogonal to each other, which is not generally true.

Editorial extensions

If this is right

  • Users can merge LoRA-tuned models without any training data or additional fine-tuning, using only the fine-tuned weights and a cheap layer-wise gradient step.
  • The decoupling and orthogonalization components can be added to existing merging algorithms such as Ties-Merging and Breadcrumbs, giving near-free improvements of almost 4% on the paper's benchmarks.
  • The same pipeline transfers to fully fine-tuned models, where magnitude variance is smaller, so the method is not limited to LoRA.
  • Larger LoRA ranks show larger gains under DO-Merging, so the advantage over baselines grows with rank.
  • The theoretical link between magnitude variance and merging loss implies that column-norm spread across tasks can serve as a cheap predictor of merging difficulty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not tested in the paper is to compute the magnitude vector from activation statistics or per-output-neuron norms; the output-alignment argument in Sec. 3.3 suggests this could outperform the fixed column-norm choice.
  • The data-free orthogonalization could be used as a generic preprocessing step before any merging or interpolation scheme, since it requires no labels and no forward passes; the paper only demonstrates it inside DO-Merging and on two external baselines.
  • The diagnosis that magnitude variance drives merging failure suggests that LoRA fine-tuning procedures that constrain column-norm growth could produce models that are easier to merge, an implication the authors do not pursue.
  • Theorem 3.1 is proven for two tasks; extending the analysis to $n$ tasks with per-layer magnitudes would give a quantitative prediction relating variance to expected loss that could be tested across a wider benchmark suite.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DO-Merging, a data-free framework for merging LoRA adapters. It decomposes each LoRA weight matrix into per-column magnitudes and normalized directions, merges magnitudes by simple averaging, and merges directions after applying a layer-wise gradient-descent orthogonalization step. Theoretical claims are made both for the decoupling step (Theorems 3.1 and 3.2) and for the orthogonalization step (Theorem 3.3). Experiments are reported across ViT, T5, LLaMA-3, Qwen-14B, and Qwen2-VL benchmarks, with ablations on the two components and on combinations with existing methods.

Significance. If the empirical results are reproducible, the paper addresses a practical and under-studied problem: merging LoRA adapters without access to training data. The framework is simple, modular, and computationally light, and the paper demonstrates consistent gains over several existing merging baselines on a broad set of vision, language, and multimodal tasks. The ablation study showing that both the decoupling and orthogonalization components contribute to the final performance is a strength, as is the demonstration that the components can be combined with other merging methods. The theoretical guarantees, however, are the main advertised contribution, and several of them are not supported by the presented arguments; this makes the scope of the paper's technical claims substantially weaker than the text suggests.

major comments (5)
  1. [Sec. 3.4 and Algorithm 1 (lines 2–3)] The statement 'we apply orthogonality separately on A and B, which is equivalent to orthogonality on the product matrix' is false in general. With W_i = B_i A_i, the Frobenius inner product of two products is tr((B_i A_i)^T B_j A_j) = tr(A_i^T B_i^T B_j A_j). Separately minimizing the pairwise inner products of the A_i and of the B_i controls tr(A_i^T A_j) and tr(B_i^T B_j) only; it does not force the product inner product to vanish. Consequently, the full-rank direction matrices W_i are not orthogonal as claimed, and the theoretical justification for the low-cost variant in Sec. 3.2 ('conclusions remain unaffected') and the support for Theorem 3.3 do not follow from the presented mathematics. The authors should either prove the claimed equivalence under additional assumptions on A_i and B_i, or revise the algorithm and claims to reflect what the separate orthogonalization actually controls.
  2. [Sec. 3.4, Theorem 3.3 and Appendix B.3] The proof of Theorem 3.3 does not establish the stated claim. The theorem asserts that smaller ||W_i^T W_j|| leads to less conflict during merging, but the proof only analyzes a single matrix entry and argues that gradient descent on the loss L_o pushes one entry toward another; it never shows that a smaller matrix-level inner product reduces the number of sign conflicts. Moreover, Eq. (4) defines L to contain exactly the quantity ||W_i^T W_j|| (via the cross terms), so showing that gradient descent reduces this quantity is close to assuming the conclusion. The theorem should either be restated as an empirical motivation, or a complete proof connecting the loss to a well-defined notion of conflict should be supplied.
  3. [Sec. 3.3, Theorem 3.2 and Appendix B.2] The proof of Theorem 3.2 contains algebraic steps that are not substantiated. In Case 2, the expansion from Eq. (24) to Eq. (26) drops terms that do not vanish under the stated assumptions: the expectations involving products of entries of W1 and W2 are not handled, and the coefficient (1/8)(alpha1+alpha2)^2 appears without derivation. The resulting expression for E(L2) is therefore not trustworthy, and the claimed inequality E(L2) < E(L1) for ||alpha1|| != ||alpha2|| is not established. The theorem is load-bearing for the decoupling contribution, so this needs a corrected proof or a downgrade of the claim.
  4. [Sec. 3.2, Theorem 3.1 and Eq. (1)] Theorem 3.1 is built on a loss function, Eq. (1), that is hand-chosen to penalize norm mismatch: the weights on the two distance terms are inversely proportional to ||alpha_i||^2, and the theorem essentially restates that this particular loss is minimized when the two norms are equal. This provides limited support for the paper's broader observational claim in Fig. 1(c). Additionally, Assumption 3.1 writes W_i = alpha_i * W_i with alpha_i nonnegative, but the proof operates with scalar norms ||alpha_1|| and ||alpha_2|| without clarifying how a vector alpha is reduced to a scalar; the notation should be made precise.
  5. [Tables 1–6 and Sec. 4] All experimental results appear to be reported from a single run, with no standard errors, confidence intervals, or multiple random seeds. The average improvements over the strongest baselines are about 1.3–1.8 points in Tables 1 and 3, which could be within run-to-run variation for large model merges. For the paper to support the claim of 'significantly higher performance,' the authors should provide variance estimates or at least multiple runs, and they should make code available to allow reproduction.
minor comments (5)
  1. [Sec. 4.5, 'Transfer to Full-Finetune Merging'] The text refers to 'Fig. 4(c)' when discussing full-finetune merging, but Fig. 4(c) is captioned 'The Impact of LoRA Rank'; the referenced result appears to be in Fig. 5, so the cross-reference should be corrected.
  2. [Eq. (2) and Assumption 3.1] The same symbol W (and alpha) is used both for the decomposed matrix in Eq. (2) and for the random matrix in Assumption 3.1, with only font differences that are easy to miss; clearer notation or a statement of overloading would help.
  3. [Sec. 3.4, Eq. (4)] The loss in Eq. (4) does not specify whether the sum over i and j includes the diagonal terms i=j; if it does, the regularization term ||delta_i||^2 is not the only term preventing trivial solutions, and the objective should be stated more explicitly.
  4. [Appendix C.4, Eq. (38)] The definition of Magnitude Distribution Variance applies variance over the n task index for each layer's absolute entries, but the notation var(|W_i^k|_{i=1}^n) is ambiguous and should be written as an explicit sum over columns and rows.
  5. [Appendix C.1] The open-source weight links for Qwen2-VL tasks are given as a single footnote with task names but no URLs or HF identifiers; the links should be resolved for reproducibility.

Circularity Check

2 steps flagged · score 4.0 of 10

Theorems 3.1/3.2 and 3.3 restate the paper's chosen loss/objective rather than providing independent guarantees; the benchmark comparisons themselves are not circular.

  1. self definitional [Sec. 3.4, Eq. (4), Theorem 3.3 (Appendix B.3)]
    "Thus, for each layer of task vectors, we construct a loss function as follows: L=Σ_i Σ_j (W_i+δ_i)^T (W_j+δ_j)+Σ_i ||δ_i||^2 = L_o+L_r. (4) ... Theorem 3.3. As ||δ_i||^2 →0, smaller values of ||W_i^T W_j|| lead to less conflict during merging."

    The object minimized in Eq. (4) is exactly the quantity whose benefit Theorem 3.3 asserts: L_o is the sum of pairwise inner products of the perturbed direction matrices, and in the δ→0 limit its minimizers are precisely directions with small ||W_i^T W_j||. The theorem says that smaller values of ||W_i^T W_j|| lead to less conflict, i.e., the conclusion is the objective itself. The proof in Appendix B.3 defines conflict as opposite signs and then analyzes gradient descent on L_o, concluding it moves (W2)_ij closer to (W1)_ij; that is an analysis of the same loss, not independent evidence.

  2. self definitional [Sec. 3.2, Assumption 3.1, Eq. (1), Theorems 3.1–3.2 (Appendix B.2)]
    "Assumption 3.1. Consider two matrices W1, W2 ∈ R^{m×n}, and assume Wi = α_i × Wi, where α_i ∈ R^{1×n} is the magnitude and Wi[:, j] ∼ N(0,1). We assume that the merged matrix W preserves features when close to the original matrices, with performance negatively correlated with the loss: L= (||α1||^2+||α2||^2)/||α1||^2 ||W−W1||^2 + (||α1||^2+||α2||^2)/||α2||^2 ||W−W2||^2."

    The assumption that 'performance is negatively correlated with L' is doing all the work. The loss in Eq. (1) is chosen with magnitude-ratio weights: the coefficient on ||W−W_i||^2 is inversely proportional to ||α_i||^2, so the loss is large precisely when the magnitudes are unbalanced and the smaller-magnitude matrix is poorly reproduced. Theorem 3.1 then 'proves' that unbalanced magnitudes raise this loss, and Theorem 3.2 'proves' that decoupling, which removes the α_i from the direction factors, lowers it. Both conclusions are contained in the definition of L; no external, measured link between this weighted loss and real task accuracy is established.

full rationale

The circularity in this paper is confined to the theoretical wrapper. The decoupling theorems prove properties of a loss defined in Eq. (1) whose weighting already encodes the claim that magnitude imbalance hurts merging, and the orthogonalization theorem asserts that the exact inner-product quantity minimized in Eq. (4) is beneficial, with a proof that analyzes that same objective. These are not independent first-principles results. The empirical section, however, evaluates DO-Merging against external baselines on vision, language, and multimodal benchmarks; those accuracy numbers are not fitted from the theory and would be independent evidence if the paper provided code, seeds, and error bars. I do not count the Sec. 3.4 sentence claiming orthogonality on A and B separately is equivalent to orthogonality on the product matrix as circularity: that claim is mathematically false (the product inner product tr(A_i^T B_i^T B_j A_j) is not controlled by tr(A_i^T A_j) and tr(B_i^T B_j) alone), but it is a correctness risk, not a reduction of a result to its inputs. The self-citations in the paper are used for experimental configuration and general motivation, not as load-bearing uniqueness theorems, so they do not raise the score. Overall, the advertised theoretical guarantees reduce to the construction of the loss, giving a 4; the benchmark evidence prevents a higher score because the central empirical claim is not itself forced by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The method rests on several stated but unproved modeling choices: a Gaussian generative model for directions, a redundancy assumption that allows data-free perturbation, an orthogonality-implies-non-interference assumption, the column-norm decomposition, and a false equivalence between low-rank and full-rank orthogonality. No new physical or model entities are introduced. The main hyperparameters are not reported.

free parameters (2)
  • Merging coefficient lambda = not reported
    Global scaling of the merged LoRA update in Eq. (5); no value or tuning procedure is reported.
  • Orthogonalization gradient step size and iteration count = not reported
    Gradient descent on Eq. (4) requires a learning rate and iteration count; without them the perturbation magnitude delta is uncontrolled.
assumptions (5)
  • domain assumption Fine-tuned parameters are highly redundant, so small perturbations do not hurt single-task performance.
    Sec. 3.4 states this to justify data-free orthogonalization; if false, the orthogonalization step damages individual tasks before merging.
  • domain assumption Orthogonality between task vectors implies minimal task interference during merging.
    Sec. 3.4 and Theorem 3.3 rely on this premise, citing prior work [10].
  • ad hoc to paper Orthogonalizing A and B separately is equivalent to orthogonalizing the product BA.
    Sec. 3.4 asserts this without proof. It is false in general because (B1A1)^T(B2A2) = A1^T B1^T B2 A2 does not vanish when A and B are separately orthogonal.
  • ad hoc to paper Assumption 3.1: Wi = alpha_i times Wi with direction columns i.i.d. standard normal and nonnegative magnitudes.
    Used in Theorems 3.1 and 3.2. Actual LoRA matrices after column normalization have unit-norm columns, so the Gaussian assumption is a simplification that may not reflect true distributions.
  • domain assumption Column norms are the appropriate magnitude decomposition, and averaging them preserves output characteristics.
    Sec. 3.3 motivates column-wise normalization, but no proof is given that this decomposition preserves task information during merging.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging." pith.science (2026). https://pith.science/paper/N5DDY4IN

@misc{pith2026250515875,
  author       = {Pith},
  title        = {Pith review of: Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N5DDY4IN}},
  note         = {Machine review of arXiv:2505.15875}
}
read the original abstract

With more open-source models available for diverse tasks, model merging has gained attention by combining models into one, reducing training, storage, and inference costs. Current research mainly focuses on model merging for full fine-tuning, overlooking the popular LoRA. However, our empirical analysis reveals that: a) existing merging methods designed for full fine-tuning perform poorly on LoRA; b) LoRA modules show much larger parameter magnitude variance than full fine-tuned weights; c) greater parameter magnitude variance correlates with worse merging performance. Considering that large magnitude variances cause deviations in the distribution of the merged parameters, resulting in information loss and performance degradation, we propose a Decoupled and Orthogonal merging approach(DO-Merging). By separating parameters into magnitude and direction components and merging them independently, we reduce the impact of magnitude differences on the directional alignment of the merged models, thereby preserving task information. Furthermore, we introduce a data-free, layer-wise gradient descent method with orthogonal constraints to mitigate interference during the merging of direction components. We provide theoretical guarantees for both the decoupling and orthogonal components. And we validate through extensive experiments across vision, language, and multi-modal domains that our proposed DO-Merging can achieve significantly higher performance than existing merging methods at a minimal cost. Notably, each component can be flexibly integrated with existing methods, offering near free-lunch improvements across tasks.

Figures

Figures reproduced from arXiv: 2505.15875 by the authors.

Figure 1
Figure 1. Key Observations on LoRA Merging. (a) Existing methods work well for full fine-tuning [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. DO-Merging Framework. Left: Large magnitude differences in LoRA across tasks [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Key observations on orthogonalization. (a) Average Norm Performance change of task [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Discussion on Key Properties of DO-Merging. (a). Both components of DO-Merging [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Applying DO-Merging to the merg￾ing of fully fine-tuned models is also effective. Experiments are conducted on ViT-B/32. The Impact of LoRA Rank. Most experiments in the previous sections used a LoRA rank of 16, a com￾mon configuration. Here, we evaluate our method’s e…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MED-DSLC: Multi-Expert-Domain Classification via Domain Supervision and Logit Calibration

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Domain-supervised MoE routing plus per-domain temperature scaling restores global logit calibration when merging frozen LoRA experts, cutting cross-domain interference in multi-domain VLM classification.

  2. Online Data Selection Is Implicit Alignment

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Online SFT data selection acts as an implicit preference model, shifting refusal rates, verbosity, and sycophancy in directions predictable from the selected data's attribute mixture.

  3. The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giants

    cs.CL 2025-05 reject novelty 5.0 of 10

    Clustering-based routing plus self-consistency voting among ten 7B open models reportedly outranks GPT-4.1 and GPT-4.5 on average over 15 diverse benchmarks.

Reference graph

Works this paper leans on

53 extracted references · 9 canonical work pages · cited by 3 Pith papers

  1. [1]

    Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609, 2023

    arXiv 2023

  2. [2]

    Iteris: Iterative inference- solving alignment for lora merging.arXiv preprint arXiv:2411.15231, 2024

    Hongxu Chen, Runshi Li, Bowei Zhu, Zhen Wang, and Long Chen. Iteris: Iterative inference- solving alignment for lora merging.arXiv preprint arXiv:2411.15231, 2024

    arXiv 2024

  3. [3]

    Are we on the right way for evaluating large vision- language models? InThe Thirty-eighth Annual Conference on Neural Information Processing Systems

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision- language models? InThe Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page

  4. [4]

    Model breadcrumbs: Scaling multi-task model merging with sparse masks.arXiv preprint arXiv:2312.06795, 2023

    MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks.arXiv preprint arXiv:2312.06795, 2023

    arXiv 2023

  5. [5]

    BERT: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors,Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volu...

    2019

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representations

  7. [7]

    Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment.arXiv preprint arXiv:2312.09979, 4(7), 2023

    Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. Loramoe: Revolutionizing mixture of experts for maintaining world knowledge in language model alignment.arXiv preprint arXiv:2312.09979, 4(7), 2023

    arXiv 2023

  8. [8]

    Parameter competition balancing for model merging

    Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, et al. Parameter competition balancing for model merging. arXiv preprint arXiv:2410.02396, 2024

    arXiv 2024

Show all 53 references
  1. [9]

    Model swarms: Col- laborative search to adapt llm experts via swarm intelligence.arXiv preprint arXiv:2410.11163, 2024

    Shangbin Feng, Zifeng Wang, Yike Wang, Sayna Ebrahimi, Hamid Palangi, Lesly Miculicich, Achin Kulshrestha, Nathalie Rauschmayr, Yejin Choi, Yulia Tsvetkov, et al. Model swarms: Col- laborative search to adapt llm experts via swarm intelligence.arXiv preprint arXiv:2410.11163, 2024

    arXiv

  2. [11]

    Task singular vectors: Reducing task interference in model merging.arXiv preprint arXiv:2412.00081, 2024

    Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodolà. Task singular vectors: Reducing task interference in model merging.arXiv preprint arXiv:2412.00081, 2024

    2024 arXiv

  3. [12]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  4. [13]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations

  5. [14]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  6. [15]

    Dota: Weight-decomposed tensor adaptation for large language models.arXiv preprint arXiv:2412.20891, 2024

    Xiaolin Hu, Xiang Cheng, Peiyu Liu, Wei Liu, Jian Luan, Bin Wang, and Yong Liu. Dota: Weight-decomposed tensor adaptation for large language models.arXiv preprint arXiv:2412.20891, 2024. 10

    2024 arXiv

  7. [16]

    Lorahub: Efficient cross-task generalization via dynamic lora composition

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition. InFirst Conference on Language Modeling

  8. [17]

    Emr-merging: Tuning-free high-performance model merging.arXiv preprint arXiv:2405.17461, 2024

    Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free high-performance model merging.arXiv preprint arXiv:2405.17461, 2024

    2024 arXiv

  9. [18]

    Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  10. [19]

    Dataless knowledge fusion by merging weights of language models.arXiv preprint arXiv:2212.09849, 2022

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models.arXiv preprint arXiv:2212.09849, 2022

    2022 arXiv

  11. [20]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. InThe 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  12. [21]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 3214–3252, 2022

    2022

  13. [22]

    Dora: Weight-decomposed low-rank adaptation

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024

    2024

  14. [23]

    Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean conference on computer vision, pages 216–233. Springer, 2024

    2024

  15. [24]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. InThe Twelfth International Conference on Learning ...

  16. [25]

    Twin-merging: Dynamic integration of modular expertise in model merging.arXiv preprint arXiv:2406.15479, 2024

    Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging.arXiv preprint arXiv:2406.15479, 2024

    2024 arXiv

  17. [26]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    2022

  18. [27]

    K-lora: Unlocking training-free fusion of any subject and style loras.arXiv preprint arXiv:2502.18461, 2025

    Ziheng Ouyang, Zhen Li, and Qibin Hou. K-lora: Unlocking training-free fusion of any subject and style loras.arXiv preprint arXiv:2502.18461, 2025

    2025 arXiv

  19. [28]

    Bbq: A hand-built bias benchmark for question answering

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thomp- son, Phu Mon Htut, and Samuel Bowman. Bbq: A hand-built bias benchmark for question answering. InFindings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, 2022

    2022

  20. [29]

    Lora soups: Merging loras for practical skill composition tasks

    Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks. InProceedings of the 31st International Conference on Computational Linguistics: Industry Track, pages 644–655, 2025

    2025

  21. [30]

    Less is more: Efficient model merging with binary task switch.arXiv preprint arXiv:2412.00054, 2024

    Biqing Qi, Fangyuan Li, Zhen Wang, Junqi Gao, Dong Li, Peng Ye, and Bowen Zhou. Less is more: Efficient model merging with binary task switch.arXiv preprint arXiv:2412.00054, 2024

    2024 arXiv

  22. [31]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020. 11

    2020

  23. [32]

    Donald Shenaj, Ondrej Bohdal, Mete Ozay, Pietro Zanuttigh, and Umberto Michieli. Lora. rar: Learning to merge loras via hypernetworks for subject-style conditioned image generation. arXiv preprint arXiv:2412.05148, 2024

    2024 arXiv

  24. [33]

    Model merging with svd to tie the knots

    George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. InThe Thirteenth International Conference on Learning Representations

  25. [34]

    Fusionbench: A compre- hensive benchmark of deep model fusion.arXiv preprint arXiv:2406.03280, 2024

    Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Fusionbench: A compre- hensive benchmark of deep model fusion.arXiv preprint arXiv:2406.03280, 2024

    2024

  26. [35]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  27. [36]

    Qwen2-vl: Enhancing vision- language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model’s p...

    2024 arXiv

  28. [37]

    Huggingface’s transformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771, 2019

    T Wolf. Huggingface’s transformers: State-of-the-art natural language processing.arXiv preprint arXiv:1910.03771, 2019

    1910 arXiv

  29. [38]

    Mixture of lora experts

    Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. InThe Twelfth International Conference on Learning Representations

  30. [39]

    Grok 3: The most powerful AI in the world is here, February 2025

    xAI. Grok 3: The most powerful AI in the world is here, February 2025

    2025

  31. [40]

    Multi-task model merging via adaptive weight disentanglement.arXiv preprint arXiv:2411.18729, 2024

    Feng Xiong, Runxi Cheng, Wang Chen, Zhanqiu Zhang, Yiwen Guo, Chun Yuan, and Ruifeng Xu. Multi-task model merging via adaptive weight disentanglement.arXiv preprint arXiv:2411.18729, 2024

    2024 arXiv

  32. [41]

    Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36, 2024

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36, 2024

    2024

  33. [42]

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportu- nities.arXiv preprint arXiv:2408.07666, 2024

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportu- nities.arXiv preprint arXiv:2408.07666, 2024

    2024 arXiv

  34. [43]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  35. [44]

    Lora-composer: Leveraging low-rank adaptation for multi-concept customization in training-free diffusion models.arXiv preprint arXiv:2403.11627, 2024

    Yang Yang, Wen Wang, Liang Peng, Chaotian Song, Yao Chen, Hengjia Li, Xiaolong Yang, Qinglin Lu, Deng Cai, Boxi Wu, et al. Lora-composer: Leveraging low-rank adaptation for multi-concept customization in training-free diffusion models.arXiv preprint arXiv:2403.11627, 2024

    2024 arXiv

  36. [45]

    Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement.arXiv preprint arXiv:2408.03092, 2024

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement.arXiv preprint arXiv:2408.03092, 2024

    2024 arXiv

  37. [46]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InForty-first International Conference on Machine Learning, 2024

    2024

  38. [47]

    Parameter efficient merging for multimodal large language models with complementary parameter adaptation.arXiv preprint arXiv:2502.17159, 2025

    Fanhu Zeng, Haiyang Guo, Fei Zhu, Li Shen, and Hao Tang. Parameter efficient merging for multimodal large language models with complementary parameter adaptation.arXiv preprint arXiv:2502.17159, 2025

    2025

  39. [48]

    Lori: Reducing cross-task interference in multi-task low-rank adaptation.arXiv preprint arXiv:2504.07448, 2025

    Juzheng Zhang, Jiacheng You, Ashwinee Panda, and Tom Goldstein. Lori: Reducing cross-task interference in multi-task low-rank adaptation.arXiv preprint arXiv:2504.07448, 2025. 12

    2025 arXiv

  40. [49]

    Nature-inspired population-based evolution of large language models.arXiv preprint arXiv:2503.01155, 2025

    Yiqun Zhang, Peng Ye, Xiaocui Yang, Shi Feng, Shufei Zhang, Lei Bai, Wanli Ouyang, and Shuyue Hu. Nature-inspired population-based evolution of large language models.arXiv preprint arXiv:2503.01155, 2025

    2025 arXiv

  41. [50]

    Dlp-lora: Efficient task-specific lora fusion with a dynamic, lightweight plugin for large language models.arXiv preprint arXiv:2410.01497, 2024

    Yuxuan Zhang and Ruizhe Li. Dlp-lora: Efficient task-specific lora fusion with a dynamic, lightweight plugin for large language models.arXiv preprint arXiv:2410.01497, 2024

    2024 arXiv

  42. [51]

    Model-glue: Democratized llm scaling for a large model zoo in the wild.Advances in Neural Information Processing Systems, 37:13349–13371, 2024

    Xinyu Zhao, Guoheng Sun, Ruisi Cai, Yukun Zhou, Pingzhi Li, Peihao Wang, Bowen Tan, Yexiao He, Li Chen, Yi Liang, et al. Model-glue: Democratized llm scaling for a large model zoo in the wild.Advances in Neural Information Processing Systems, 37:13349–13371, 2024

    2024

  43. [52]

    Merging loras like playing LEGO: pushing the modularity of lora to extremes through rank-wise clustering.CoRR, abs/2409.16167, 2024

    Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. Merging loras like playing LEGO: pushing the modularity of lora to extremes through rank-wise clustering.CoRR, abs/2409.16167, 2024

    2024 arXiv

  44. [53]

    Free-merging: Fourier transform for model merging with lightweight experts.arXiv preprint arXiv:2411.16815, 2024

    Shenghe Zheng and Hongzhi Wang. Free-merging: Fourier transform for model merging with lightweight experts.arXiv preprint arXiv:2411.16815, 2024

    2024 arXiv

  45. [54]

    Hm3: Hierarchical multi- objective model merging for pretrained models.arXiv preprint arXiv:2409.18893, 2024

    Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. Hm3: Hierarchical multi- objective model merging for pretrained models.arXiv preprint arXiv:2409.18893, 2024. 13 Appendix for DO-Merging A Notations Table 9: Notations. Notations Descriptions θpre Pre-trained model pa...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.