Pith. sign in

REVIEW 3 major objections 4 minor 37 references

LoRA adapter merging gains from per-task RMS scales instead of copied singular values.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 20:20 UTC pith:Q4CMBSG6

load-bearing objection Useful incremental LoRA-merging method with clean ablations, but the stated energy-preservation story for the coefficient rule doesn't survive contact with the polar projection. the 3 major comments →

arxiv 2607.20561 v1 pith:Q4CMBSG6 submitted 2026-07-18 cs.LG cs.CV

CT-Merging: Consensus Directions and Task-Level Scaling for LoRA Adapter Merging

Keumseo Ryum , Joonhyuk Kang This is my paper
classification cs.LG cs.CV
keywords LoRA mergingmodel mergingSVD-based mergingtask-level scalingconsensus directionsadapter mergingmulti-task learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LoRA adapters are small updates that specialize one pretrained model to many tasks, but storing one per task becomes unwieldy. Merging combines them into a single multi-task update without retraining. The paper identifies a subtle bug in SVD-based merging: recomposed directions are paired with singular values copied from each task's original decomposition, and those magnitudes are miscalibrated after projection. CT-Merging fixes this by building a common basis from average task subspace projectors and assigning each task a single RMS scale based on its residual energy. On CLIP adapter benchmarks it beats existing methods, including gains of 2.56 points over DC-Merge on ViT-B/32 and 1.51 on ViT-L/14.

Core claim

The paper's central claim is that coefficient assignment, not just direction construction, is a bottleneck in SVD-based LoRA merging. After expressing task updates in a shared recomposition basis, inherited singular values preserve component order (median Spearman correlation above 0.99) but exhibit substantial magnitude mismatch (median relative vector distance 0.35–0.39), largest at the leading ranks. The paper's fix is a task-level RMS rule: every residual direction of a task receives the same coefficient, set to the RMS of that task's original singular values, preserving per-task squared energy while discarding unreliable component-wise magnitudes. Combined with a consensus basis compute

What carries the argument

The two load-bearing pieces are the consensus basis and the task-level RMS coefficient rule. The consensus basis is the top-k eigenbasis of the average task subspace projector P_U = (1/T) Σ_t U_t U_t^T, which is invariant to sign and rotation ambiguities in each task's singular vectors and selects directions with repeated support across tasks. The coefficient rule replaces copied singular values s_{t,j} with a single RMS scale per task, ρ_t = sqrt( (1/r_res) Σ_j s_{t,j}^2 ), so that the total squared energy of each task is retained while rank-wise magnitudes are removed. A final polar orthonormalization of the concatenated left and right directions yields the recomposition basis.

Load-bearing premise

The load-bearing premise is that per-task singular-value energy computed in each task's isolated SVD basis remains the right total coefficient budget after the directions are projected into the consensus-plus-residual basis and polar-orthonormalized; if that energy does not transfer, the RMS scales mis-weight tasks.

What would settle it

Recompute per-task coefficients after projection by expressing each task update in the final polar-orthonormalized basis (the paper's induced coefficients d_{t,j}) and use those induced magnitudes instead of RMS scales while keeping CT-Merging's directions; if that variant matches or beats CT-Merging, the energy-transfer assumption is not needed. Alternatively, construct a task whose energy is concentrated in the consensus subspace so its residual energy is near zero, and check whether CT-Merging under-weights it compared to the projected-coefficient variant.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If CT-Merging is correct, data-free multi-task LoRA deployment improves simply by recalibrating coefficients after direction construction, with no extra training data.
  • The task-level RMS rule preserves per-task scale differences while avoiding sign and rotation artifacts of individual singular vectors.
  • Consensus directions from average projectors are a better common basis than the SVD of the summed update or random directions, improving worst-task accuracy by 2.6–3.0 points.
  • Larger gains on KnOTS checkpoints (2.56 on ViT-B/32, 1.51 on ViT-L/14) suggest the improvement transfers across pretrained checkpoints.
  • The rank budget can be split between common and residual directions; moderate common ranks are stable, while k=0 underperforms.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same direction-versus-coefficient decoupling may generalize to other spectral merging settings, including language-model LoRAs, where per-task energy may vary just as widely.
  • Editorial inference: because rank order survives projection, a cheaper recipe could reuse the original rank ordering for direction ranking and only recalibrate magnitudes by task energy, potentially removing the need for the projector.
  • Editorial inference: selecting the common rank k from the spectral gap of the average projector, rather than fixing it, could improve results across task counts and backbones.
  • Editorial inference: a testable extension is to replace the flat RMS profile with a per-task spectrum shaped by a smooth function of the original singular values, interpolating between copied magnitudes and uniform RMS; the paper's analysis suggests this preserves task energy while reducing rank-wise noise.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. CT-Merging proposes a data-free SVD-based LoRA adapter-merging algorithm. It constructs a consensus basis from the average of task left-singular subspace projectors, projects each task's retained SVD directions away from that consensus subspace, polar-orthonormalizes the concatenated directions, and assigns RMS coefficients per task instead of copying component-wise singular values. On the DC-Merge CLIP LoRA benchmark and KnOTS checkpoints, it reports the best average normalized accuracy in most settings, with gains over DC-Merge of up to 2.56 points. The paper also contributes a diagnostic showing that inherited SVD coefficients preserve rank order but not magnitudes after recomposition, and ablations on the consensus source, common rank, and merge scale.

Significance. The problem addressed is real and timely: after projecting and realigning LoRA SVD directions, the coefficients inherited from isolated task SVDs are not calibrated, and Section 3 quantifies this mismatch usefully. The projector-based consensus construction is simple, and the consensus-source ablation in Table 4 is a good design. If the algorithm is corrected and the gains are reproducible, CT-Merging would be a solid contribution to SVD-based LoRA merging. However, the manuscript's central coefficient-assignment mechanism has two technical inconsistencies that affect what the algorithm actually computes, and the evaluation protocol selects hyperparameters on the target benchmark; these issues must be resolved before the empirical claims can be accepted.

major comments (3)
  1. [§4.4, Eq. (12)] The residual RMS coefficient is computed from E_t = Σ_j s_{t,j}^2, the SVD energy in the isolated task basis. After Eq. (8), the residual left directions are U_⊥_t = (I - U_c U_c^T) U_t, so the energy actually available in the residual component is E_t^proj = Σ_j s_{t,j}^2 ||u_{⊥,t,j}||^2. Since each projected column has norm ≤ 1, E_t^proj ≤ E_t. Tasks whose SVD directions lie mostly inside the consensus subspace are therefore assigned coefficient energy exceeding their projected residual component, and near-null residual directions are amplified. If the goal is to preserve per-task residual energy after removing common content, Eq. (12) should use the projected energy; as written, the method can systematically over-weight consensus-aligned tasks.
  2. [§4.3–4.4, Eqs. (10), (15)] Polar(U_rec) returns an orthonormal basis for the column space of the concatenated matrix; it is a global rotation/reflection and does not preserve the block structure [U_com | U_1^⊥ | ... | U_T^⊥]. The columns of eU and eV are linear combinations of all tasks' directions, and their ordering is determined by the singular values of U_rec, not by the concatenation blocks. The blockwise coefficient vector s_new in Eq. (15) therefore does not assign coefficients to task-specific directions, and the claimed 'separate residual energy budget for each task' is not realized by the algorithm as written. Please specify a blockwise orthonormalization step, or justify that polar projection preserves the intended block ordering; otherwise the reported results test a different, effectively global spectral-shaping rule.
  3. [§5.1–5.2] The evaluation protocol selects hyperparameters on the target benchmark. The text states that the global merge scale γ is swept over the same grid for all methods and the best value is reported, and k is set per architecture (k=8 for ViT-B, k=16 for ViT-L) apparently informed by the benchmark accuracy. No validation split or pre-registered selection rule is described. This inflates the headline numbers in Tables 1–3 and makes the comparison with baselines, whose hyperparameters are taken from their original papers, difficult to interpret. Please report results for fixed γ and k, or a validation-based selection, and state the selected values explicitly.
minor comments (4)
  1. [Abstract] 'KnoTS-trained checkpoints' should be 'KnOTS-trained checkpoints' for consistency with the method name.
  2. [§3] The diagnostic uses 'the recomposition basis' but does not specify how that basis is constructed. Please state the exact basis (e.g., the DC-Merge basis or the CT-Merging basis) and report the spread across layers and tasks, not only medians.
  3. [Table 1] On the ViT-B/32 8-task setting, DC-Merge (72.86) exceeds CT-Merging (72.55). The text's 'best result on eight of nine' is accurate, but the abstract's 'superior average normalized accuracy' should be qualified to avoid overstating the result.
  4. [§5.3.1] Figures 4 and 5 would be easier to read with point markers or per-setting annotations; the curves are sparse and the y-axis ranges differ.

Circularity Check

0 steps flagged

No significant circularity: the coefficient rule is a stated construction rather than a fitted prediction, and the benchmark comparisons are external.

full rationale

CT-Merging's core construction is an explicit algorithm: consensus directions are top eigenvectors of average task-subspace projectors (Eqs. 5-6), residual directions are projected left singular vectors (Eq. 8), and coefficients are RMS values of per-task SVD energies (Eqs. 11-15). None of these steps is obtained by fitting the benchmark accuracies or by importing a uniqueness/optimality claim from the authors; the coefficient rule is a proposed heuristic, not a derived prediction. Section 3's diagnostic compares inherited coefficients s_{t,l,j} with coefficients induced by expressing Delta_t in a recomposition basis, but the paper uses that diagnostic only as motivation and does not claim it is an independent confirmation of the final accuracy numbers. The empirical claims are evaluated on released DC-Merge/KnOTS checkpoints against external baselines, with the same gamma sweep for all methods; this is standard model selection rather than a 'fitted parameter renamed as prediction.' The potential concern raised by the reader, that Eq. (12) preserves coefficient energy rather than matrix energy after polar projection, is a modeling/calibration question about the heuristic, not a circular reduction: the paper never derives the RMS rule from the target accuracies. No self-citation chain is load-bearing, and no uniqueness theorem is invoked. I therefore find no step where a stated prediction is equivalent by construction to an input.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on four assumptions and three tuned hyperparameters. No new physical entities are introduced; the 'consensus directions' are a mathematical construction from existing adapter weights. The weakest link is the energy-transfer assumption: it is plausible but not proven, and the empirical ablation is the only support.

free parameters (3)
  • gamma (global merge scale) = not specified (swept; best value reported)
    Multiplies the entire merged update in Eq. (16); selected per method/benchmark by sweeping a grid and reporting the best normalized accuracy.
  • k (common rank) = 8 for ViT-B backbones; 16 for ViT-L/14
    Number of consensus directions; chosen by the authors per architecture, not derived, and shown in ablation (Fig. 4) to affect accuracy.
  • r_res (residual rank per task) = implied by total budget 128: r_res = (128 - k) / T
    Residual rank per task. The total rank budget k + T*r_res = 128 is fixed by hand; this constraint determines r_res but is a design choice.
axioms (4)
  • domain assumption Task update Delta_t is rank-r_res truncated without losing task-critical information
    Algorithm 1 keeps only top r_res singular components per task before merging; if lower-rank components carry task-specific knowledge, the merged model loses it. Invoked in Sec. 4.1 (Eq. 4) and Algorithm 1.
  • domain assumption The average projector P_U = T^{-1} sum U_t U_t^T exposes a consensus subspace whose top eigenvectors capture shared task directions
    CT-Merging builds the common basis from the top eigenvectors of P_U (Eqs. 5-6). No proof that repeated support equals semantic commonality; validated only by ablation in Table 4.
  • standard math Polar projection preserves the span of the concatenated directions while yielding orthonormal left/right bases
    Polar decomposition is a standard linear-algebra fact; used in Eq. (10) without proof.
  • ad hoc to paper Per-task residual energy E_t from isolated task SVD is the appropriate coefficient budget after recomposition
    Eqs. (11)-(12) replace component-wise singular values by a per-task RMS scale; this is the paper's core design assumption and is only empirically motivated by Sec. 3 diagnostics.

pith-pipeline@v1.3.0-alltime-deepseek · 10363 in / 12478 out tokens · 116847 ms · 2026-08-01T20:20:25.073350+00:00 · methodology

0 comments
read the original abstract

LoRA adapters provide an efficient way to specialize a pretrained model for many downstream tasks, but deploying one adapter per task requires adapter storage and task selection at inference time. Model merging addresses this issue by combining independently trained adapters into one multi-task adapter. Recent SVD-based LoRA merging methods mainly focus on constructing shared or task specific directions, while the coefficients assigned to the final directions are often directly from the original task SVD. On a fixed merged basis, inherited coefficients preserve component order with high rank correlation, yet their magnitudes differ substantially from the coefficients induced by the task updates. To address this mismatch, we propose CT-Merging, a LoRA-aware merging algorithm that estimates consensus directions from average task subspace projectors and assigns task-level RMS coefficient scales in the final update. CT-Merging uses repeated support across task SVD subspaces to construct the common basis, while reducing reliance on rank wise SVD magnitudes after direction construction. On the DC-Merge CLIP adapter benchmark, CT-Merging achieves superior average normalized accuracy compared to state-of-the-art merging methods and further improves over DC-Merge by 2.56 points on ViT-B/32 and 1.51 points on ViT-L/14 KnoTS-trained checkpoints.

Figures

Figures reproduced from arXiv: 2607.20561 by Joonhyuk Kang, Keumseo Ryum.

Figure 1
Figure 1. Figure 1: Overview of CT-Merging. CT-Merging estimates consensus directions from task SVD subspaces, projects task residual directions [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Rank wise coefficient error on CLIP LoRA adapters. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Effect of common rank k under fixed rank budget under [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 2 linked inside Pith

  1. [1]

    Remote sens- ing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sens- ing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017. 5

    2017

  2. [2]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014. 5

    2014

  3. [3]

    An analysis of single-layer networks in unsupervised feature learning

    Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. InProceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 215–223, Fort Lauderdale, FL, USA, 2011. PMLR. 5

    2011

  4. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conference on Learning Representa- tions, 2021. 5

    2021

  5. [5]

    Task singular vectors: Reducing task in- terference in model merging

    Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodol`a. Task singular vectors: Reducing task in- terference in model merging. In2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18695–18705, 2025. 1, 2

    2025

  6. [6]

    Ian J. Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, Yingbo Zhou, Chetan Ramaiah, Fangxiang Feng, Ruifan Li, Xiaojie Wang, Dimitris Athanasakis, John Shawe-Taylor, Maxim Milakov, John Park, Radu Ionescu, Marius Popescu, Cristian Grozea, James Bergstra, Ji...

  7. [7]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 5

    2019

  8. [8]

    Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark

    Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. InInternational Joint Conference on Neural Networks, number 1288, 2013. 5

    2013

  9. [9]

    Parameter-efficient transfer learning for nlp

    Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. InInternational Conference on Machine Learning (ICML), 2019. 1

    2019

  10. [10]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. InIn- ternational Conference on Learning Representations, 2022. 1

    2022

  11. [11]

    Lorahub: Efficient cross-task gen- eralization via dynamic lora composition

    Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task gen- eralization via dynamic lora composition. InConference on Language Modeling (COLM), 2024. 1

    2024

  12. [12]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. InThe Eleventh Inter- national Conference on Learning Representations, 2023. 1, 2, 4, 5

    2023

  13. [13]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. InInternational Conference on Learning Representations (ICLR), 2023. 2

    2023

  14. [14]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Com- puter Vision (ICCV) Workshops, 2013. 5

    2013

  15. [15]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. pages 32–33, 2009. 5

    2009

  16. [16]

    Lecun, L

    Y . Lecun, L. Bottou, Y . Bengio, and P. Haffner. Gradient- based learning applied to document recognition.Proceed- ings of the IEEE, 86(11):2278–2324, 1998. 5

    1998

  17. [17]

    Adarank: Adaptive rank pruning for enhanced model merging

    Chanhyuk Lee, Jiho Choi, Chanryeol Lee, Donggyun Kim, and Seunghoon Hong. Adarank: Adaptive rank pruning for enhanced model merging. InThe Fourteenth International Conference on Learning Representations, 2026. 1, 2

    2026

  18. [18]

    When shared knowledge hurts: Spectral over- accumulation in model merging

    Yayuan Li, Ze Peng, Jian Zhang, Jintao Guo, Yue Duan, and Yinghuan Shi. When shared knowledge hurts: Spectral over- accumulation in model merging. InForty-third International Conference on Machine Learning, 2026. 2

    2026

  19. [19]

    Bagdanov, and Joost van de Weijer

    Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D. Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task-specific subspaces. InForty-second International Conference on Machine Learning, 2025. 1, 2, 5

    2025

  20. [20]

    Merging models with fisher-weighted averaging

    Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. InAdvances in Neural Informa- tion Processing Systems, 2022. 2

    2022

  21. [21]

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bis- sacco, Bo Wu, and Andrew Y . Ng. Reading digits in natural images with unsupervised feature learning. InNIPS Work- shop on Deep Learning and Unsupervised Feature Learning 2011, 2011. 5

    2011

  22. [22]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729, 2008. 5

    2008

  23. [23]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. InAdvances in Neural Infor- mation Processing Systems (NeurIPS), 2023. 1

    2023

  24. [24]

    Parkhi, Andrea Vedaldi, Andrew Zisserman, and C

    Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V . Jawahar. Cats and dogs. InIEEE Conference on Com- puter Vision and Pattern Recognition, 2012. 5

    2012

  25. [25]

    Lora soups: Merg- ing loras for practical skill composition tasks.arXiv preprint arXiv:2410.13025, 2024

    Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merg- ing loras for practical skill composition tasks.arXiv preprint arXiv:2410.13025, 2024. 1

    Pith/arXiv arXiv 2024

  26. [26]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InProceedings of the 38th International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. 5

    2021

  27. [27]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the 2013 Confer- ence on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA, 2013. Associ- ation for Comput...

    2013

  28. [28]

    Model merging with svd to tie the knots.ICLR, 2025

    George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots.ICLR, 2025. 1, 2, 5, 6

    2025

  29. [29]

    Localizing task infor- mation for improved model merging and compression

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, Franc ¸ois Fleuret, and Pascal Frossard. Localizing task infor- mation for improved model merging and compression. InIn- ternational Conference on Machine Learning (ICML), 2024. 1, 2

    2024

  30. [30]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Re- becca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. InProceedings of the 39th International Conference on...

    2022

  31. [31]

    Fashion- mnist: a novel image dataset for benchmarking machine learning algorithms, 2017

    Han Xiao, Kashif Rasul, and Roland V ollgraf. Fashion- mnist: a novel image dataset for benchmarking machine learning algorithms, 2017. 5

    2017

  32. [32]

    Ehinger, James Hays, Antonio Torralba, and Aude Oliva

    Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. SUN database: Exploring a large collection of scene categories.International Journal of Com- puter Vision (IJCV), 119(1):3–22, 2016. 5

    2016

  33. [33]

    TIES-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. TIES-merging: Resolving interference when merging models. InThirty-seventh Conference on Neu- ral Information Processing Systems, 2023. 1, 2, 5

    2023

  34. [34]

    Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities.arXiv preprint arXiv:2408.07666, 2024

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xi- aocun Cao, Jie Zhang, and Dacheng Tao. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities.arXiv preprint arXiv:2408.07666, 2024. 1

    Pith/arXiv arXiv 2024

  35. [35]

    Adamerging: Adap- tive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adap- tive model merging for multi-task learning. InInternational Conference on Learning Representations (ICLR), 2024. 1, 2

    2024

  36. [36]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InProceedings of the 41st International Conference on Machine Learning, pages 57755–57775, 2024. 1, 2

    2024

  37. [37]

    Dc-merge: Improving model merging with directional consistency

    Han-Chen Zhang, Zi-Hao Zhou, Mao-Lin Luo, Shimin Di, Min-Ling Zhang, and Tong Wei. Dc-merge: Improving model merging with directional consistency. InCVPR, 2026. 1, 2, 5

    2026