Pith. sign in

REVIEW 3 major objections 4 minor 82 references

Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A merging method stores each task's unique weights as a compressed signature—roughly 1% extra per task—and reconstructs them on demand, preserving near-individual accuracy while letting one base model serve many tasks.

desk verdict The DTS compression idea and the difference-vector representation are worth engaging with, but the headline claim that DTS beats all baselines at 1% extra storage is not supported by the reported numbers. read the letter →

arxiv 2512.01461 v2 pith:I5NC3CBU submitted 2025-12-01 cs.LG cs.CV

Kuangpu Guo , Aijing Yu , Jian Liang , Yuhe Ding , Zilei Wang , Ran He , Tieniu Tan This is my paper
classification cs.LGcs.CV
keywords modelmergingmulti-tasklearningtaskvectordifferencesingularvaluedecompositionlow-rankapproximationdata-freegeneralizationpersonalized
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Model merging aims to combine several fine-tuned models into one, but straightforward averaging loses per-task accuracy even when the tasks are similar. This paper argues that the loss comes from discarding task-specific information and proposes Decomposition, Thresholding, and Scaling (DTS): keep a low-rank singular value decomposition of each task's weight delta, threshold the singular vectors into four signed groups, assign each group a scaling factor, and reconstruct the per-task weights only at inference. The central claim is that this compressed signature—about 1% extra storage per task—recovers near-individual-model accuracy on vision, language, and generation benchmarks. A variant weights these signatures by semantic similarity of task characteristics, allowing a merged model to generalize to unseen tasks without any training data. If correct, the method turns a single shared backbone into a multi-task system whose per-task personality costs almost nothing to store.

What carries the argument

The central object is the approximate per-task weight reconstruction θ̂_n = θ_0 + Û_n Σ_n V̂_n. Û_n and V̂_n come from a truncated SVD of the task vector or difference vector, with each singular vector quantized into four signed groups via median thresholds; group scalars are L2 norms of the original group elements. These pieces are stored as three 1-bit masks per matrix plus eight scalars per layer, which is what yields the roughly 1% storage budget. The mechanism's job is to make the reconstructed per-task model close to the individually fine-tuned model without storing full weights and without mixing parameters across tasks.

What would settle it

Compute the exact reconstruction error ||θ̂_n − θ_n|| for each task under DTS and correlate it with the per-task accuracy drop. The central claim would be falsified if a task with small reconstruction error still shows a large accuracy drop, or if a task whose important directions lie outside the top-r singular vectors performs near-individually—since that would indicate the reported gains come from something other than faithful reconstruction.

Watch

Extended reading notes

Core claim

The paper's central claim is that per-task accuracy can be preserved during model merging without storing full task-specific models. DTS decomposes each task vector (θ_n − θ_0) or difference vector (θ_n − θ_m) with a truncated SVD, keeps the top-r singular values, and quantizes each singular vector by thresholding its elements into four groups—positive large, positive small, negative large, negative small—each represented by a 1-bit mask and a scalar scaling factor. At inference, per-task weights are rebuilt as θ̂_n = θ_0 + Û_n Σ_n V̂_n, so each task gets its own reconstruction and parameter conflicts between tasks are avoided entirely. Experiments on vision, language, and generation benchma

Load-bearing premise

The load-bearing premise is that a truncated SVD plus four-group median thresholding and per-group scaling preserves the task-relevant directions well enough that the reconstructed weights recover near-individual performance; the paper provides empirical ablations but no theoretical bound on the reconstruction error.

Editorial extensions

If this is right

  • Deploying one backbone for many tasks becomes practical at about 1% extra memory per task, since each task's unique behavior is reconstructed only when that task is queried.
  • The difference-vector variant keeps working when the original pretrained checkpoint is unavailable, because the reference model can be a simple average of the fine-tuned models.
  • The semantic-similarity variant provides a data-free route to unseen tasks: the same per-task signatures are recombined with cosine-similarity weights derived from task names or descriptions.
  • On the reported benchmarks, the gap between merged and individually fine-tuned models shrinks to well under one percentage point on average, suggesting the merged model can substitute for separate checkpoints in many deployment scenarios.
  • Because storage is controlled by a single tunable sparsity coefficient, practitioners can trade a little accuracy for an even smaller footprint, or relax the budget for more headroom.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test the paper leaves open is whether the 1-bit masks plus group scalars behave like a structured codebook under further entropy coding across tasks, which could shave the storage cost even below 1% without changing the method.
  • The semantic-similarity weighting assumes class-name embeddings are a faithful proxy for parameter-space task proximity; a direct extension would validate this on tasks whose names are uninformative or misleading.
  • The method's strong results on the studied tasks suggest a testable transfer to multilingual or multimodal task suites, where task-specific information might concentrate in different layers than in the vision and language settings reported here.
  • If the low-rank plus thresholding approximation is truly near-lossless, DTS could serve as a compressed memory of past tasks in continual or federated learning, replaying each task's signature during later merging rounds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper proposes DTS, a personalized model merging method that stores per-task information as low-rank SVD components of task or difference vectors, quantizes the singular vectors into four groups via sign and magnitude thresholds with per-group scaling factors, and reconstructs task weights at inference. Two variants are introduced: DTS-T on task vectors and DTS-D on difference vectors, plus an extension for unseen tasks that weights stored components by cosine similarity between task embeddings. Experiments on CLIP, RoBERTa, GPT-2, and Qwen-14B report high accuracies at claimed low storage, with ablations supporting the main design choices.

Significance. The contribution is a compact, data-free representation of task-specific information for personalized merging. If the storage-accuracy claims are reproducible, the method offers a useful cost-performance point: reconstruction from truncated, thresholded SVD components recovers most individually fine-tuned performance at a small fraction of full-model storage. The paper ships code, includes ablations for each component, and covers a wide range of backbones and tasks, including an unseen-task setting. The main concerns are that the headline claims are not fully supported by the tabulated numbers.

major comments (3)
  1. [Abstract and §4.1, Tables 1-3] The abstract states that DTS achieves strong performance 'while requiring only 1% additional storage per task.' Tables 1, 2, and 3 list AMR for DTS-T and DTS-D (the default r=0.3 variants) as 3.68%, 3.81%, and 3.57%, respectively; only the starred variants, introduced specifically to cap storage below 1%, meet that budget. As written, the primary method does not operate at 1% storage. Please qualify the storage claim in the abstract and conclusions to distinguish the default and low-storage configurations.
  2. [Tables 1 and 7; §4.1] The claim that DTS 'consistently outperforms state-of-the-art baselines' at ≤1% storage is not supported by the reported numbers. In Table 1, DTS-T* achieves 90.08 vs. T-Switch's 90.15, and DTS-D* achieves 90.17. In Table 7, DTS-T* ties T-Switch at 94.16 and DTS-D* is 94.14, below T-Switch. The non-star DTS variants do beat T-Switch, but at 2.95–3.81% AMR. Since no seeds, error bars, or significance tests are reported, differences of 0.01–0.2 points are within typical fine-tuning noise. I request multiple-run results and a significance test, and a more nuanced statement of where DTS improves over T-Switch.
  3. [§4.3, Table 6] The text states that, under the same adaptive weighting mechanism, 'our method consistently achieves the best performance' on unseen tasks. Table 6 contradicts this: DTS-T averages 59.06 and DTS-T* averages 59.35, both below Twin-Merging's 59.38. Only the DTS-D variants exceed Twin-Merging. The claim should be corrected to refer to the difference-vector variants, or the table/claim must be reconciled.
minor comments (4)
  1. [Table 14] The RoBERTa row at r=1.0 is listed as '90.4', which is implausible for this benchmark and inconsistent with the surrounding values; it should likely be ~84.98.
  2. [Supplementary §7.1] EMR-Merging is cited as [52] in the baseline description, but it is reference [19] in the main list. Also, WEMOE and EMR-Merging are both numbered [52], causing confusion.
  3. [§3.2 and Supplementary §6] The main text says 'only three masks per U or V matrix,' while the supplementary says 'six mask matrices and eight scaling factors for each task.' These are consistent (three per matrix, six per task) but the wording could be clarified to avoid ambiguity about the total per-task storage.
  4. [§4.1] The statement 'WEMOE requires an extra 58.80% of model size per task' appears to refer to Table 7, while Table 1 reports 49.95% AMR for the same method. Specify which configuration/table is referenced.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: DTS is a lossy compression scheme whose reconstruction is defined from the same data it approximates; no prediction reduces to a fitted target.

full rationale

The derivation chain is self-contained. DTS takes each task vector tau_n, applies SVD_r (Eq. 2), thresholds singular-vector entries into four groups using medians (Eqs. 3-4), and computes per-group scaling factors from the L2 norms of the same U/V entries being approximated (Eqs. 5-6). Reconstruction theta_hat_n = theta_0 + U_hat_n Sigma_n V_hat_n (Eq. 7) is therefore a lossy approximation of tau_n by construction, but no accuracy target is fitted and no parameter is tuned against test performance. The only free knob, the sparsity ratio r, is swept in Table 14 as a storage/performance trade-off. The difference-vector variant is a representation choice (d_n = theta_n - theta_m), and the unseen-task weighting (Eq. 8) uses normalized cosine similarities of held-out task descriptors rather than fitted outputs. The paper's sole overlapping-author citation, [11] in the related-work list, is not load-bearing for any central claim. The skeptic's observation that DTS-* variants sometimes tie or trail T-Switch and that no error bars are reported is an empirical-evidence and significance concern, not a circularity concern.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. It relies on four domain assumptions: SVD truncation preserves task information, task ID is known at inference, text-encoder embeddings capture task similarity, and the base merged model is a good reference for difference vectors. The only free numeric parameter is the sparsity coefficient r.

free parameters (1)
  • sparsity coefficient r = 0.3 default; adaptively adjusted for * variants to keep AMR < 1%
    Controls the proportion of singular values retained. Table 14 shows a sensitivity analysis; the choice of r=0.3 and the adaptive scheme for the * variants are tuned against the storage/performance trade-off, not derived.
assumptions (4)
  • domain assumption Low-rank SVD truncation of task-specific parameter matrices is a lossy but sufficient approximation (Eq. 2).
    The method retains only top-r singular values/vectors and relies on this retaining task-relevant information. This is empirically validated by ablations but not proven.
  • domain assumption The test task ID is available at inference time.
    Sec. 3.1 states: 'we also assume that the information of the test task (e.g., task ID) is available during evaluation.' The per-task reconstruction depends on this.
  • domain assumption Class names or task descriptions, encoded by a text encoder, capture the semantic similarity between tasks.
    Sec. 3.3, Eq. 8 computes merging weights as normalized cosine similarity between mean class-name embeddings; this assumes the embeddings are a valid proxy for task relatedness.
  • domain assumption A base merged model θ_m (e.g., from Ties-Merging) is a meaningful reference so the difference vector d_n = θ_n - θ_m contains useful task-specific information.
    Introduced in Sec. 3.1 and analyzed in Sec. 9.5. The usefulness of DTS-D depends on this reference choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging." pith.science (2026). https://pith.science/paper/I5NC3CBU

@misc{pith2026251201461,
  author       = {Pith},
  title        = {Pith review of: Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I5NC3CBU}},
  note         = {Machine review of arXiv:2512.01461}
}
read the original abstract

Model merging has emerged as a promising paradigm for enabling multi-task capabilities without additional training. However, traditional basic merging methods often experience performance degradation due to parameter conflicts, even when applied to similar tasks. While recent personalized merging frameworks successfully preserve task-specific information to maintain performance, they typically incur storage overhead. In this paper, we propose Decomposition, Thresholding, and Scaling (DTS), an approximation-based personalized merging framework that pushes task-specific storage efficiency. DTS first applies singular value decomposition to the task-specific information and retains only a small subset of singular values and vectors. It then introduces a novel thresholding strategy that partitions singular vector elements into groups and assigns a scaling factor to each group. To enable generalization to unseen tasks, we further extend DTS with a variant that fuses task-specific information in a data-free manner based on the semantic similarity of task characteristics. Extensive experiments demonstrate that DTS consistently outperforms state-of-the-art baselines while requiring only 1\% extra storage per task. Furthermore, experiments on unseen tasks show that the DTS variant achieves significantly better generalization performance. Our code is available at https://github.com/krumpguo/DTS.

Figures

Figures reproduced from arXiv: 2512.01461 by the authors.

Figure 1
Figure 1. Performance of the pairwise merged model: we pair [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Framework overview. In the approximation stage, we first apply singular value decomposition to the task-specific information. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. A toy example of our thresholding strategy. Thresholding [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance (%) of merged model and additional memory [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

82 extracted references · 9 linked inside Pith

  1. [1]

    Git re-basin: Merging models modulo permutation symmetries

    Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srini- vasa. Git re-basin: Merging models modulo permutation symmetries. InProc. ICLR, 2023. 2

    2023

  2. [2]

    Qwen technical report.arXiv preprint arXiv:2309.16609,

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xi- aodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report.arXiv preprint arXiv:2309.16609,

    arXiv

  3. [3]

    Multitask learning.Machine learning, 28(1): 41–75, 1997

    Rich Caruana. Multitask learning.Machine learning, 28(1): 41–75, 1997. 1

    1997

  4. [4]

    Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation

    Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation, 2017. 5

    2017

  5. [5]

    Fw-merging: Scaling model merging with frank-wolfe optimization

    Hao Mark Chen, Shell Xu Hu, Wayne Luk, Timothy Hospedales, and Hongxiang Fan. Fw-merging: Scaling model merging with frank-wolfe optimization. InProc. ICCV, 2025. 2

    2025

  6. [6]

    Quora question pairs, 2018

    Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2018. 5, 7

    2018

  7. [7]

    Remote sensing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, pages 1865–1883, 2017. 5, 7

    2017

  8. [8]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. InProc. CVPR, 2014. 5

    2014

Show all 82 references
  1. [9]

    Model breadcrumbs: scalable upcycling of finetuned foundation models via sparse task vectors merging

    MohammadReza Davari and Eugene Belilovsky. Model breadcrumbs: scalable upcycling of finetuned foundation models via sparse task vectors merging. InProc. ICML,

  2. [10]

    The mnist database of handwritten digit images for machine learning research.IEEE signal processing magazine, pages 141–142, 2012

    Li Deng. The mnist database of handwritten digit images for machine learning research.IEEE signal processing magazine, pages 141–142, 2012. 1, 4, 5

    2012

  3. [11]

    Harmonizing and merging source mod- els for clip-based domain generalization.arXiv preprint arXiv:2506.09446, 2025

    Yuhe Ding, Jian Liang, Bo Jiang, Zi Wang, Aihua Zheng, and Bin Luo. Harmonizing and merging source mod- els for clip-based domain generalization.arXiv preprint arXiv:2506.09446, 2025. 2

    2025 arXiv

  4. [12]

    Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.arXiv:2002.06305, 2020

    Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping.arXiv:2002.06305, 2020. 2

    2002 arXiv

  5. [13]

    Automatically constructing a corpus of sentential paraphrases

    Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. InThird international workshop on paraphrasing, 2005. 5

    2005

  6. [14]

    Task singular vectors: Reducing task interference in model merging

    Antonio Andrea Gargiulo, Donato Crisostomi, Maria Sofia Bucarelli, Simone Scardapane, Fabrizio Silvestri, and Emanuele Rodola. Task singular vectors: Reducing task interference in model merging. InProc. CVPR, 2025. 2

    2025

  7. [15]

    The third pascal recognizing textual entail- ment challenge

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entail- ment challenge. InProceedings of the ACL-PASCAL work- shop on textual entailment and paraphrasing, 2007. 5, 7

    2007

  8. [16]

    Gradient reweighting: Towards imbalanced class-incremental learning

    Jiangpeng He. Gradient reweighting: Towards imbalanced class-incremental learning. InProc. CVPR, 2024. 2

    2024

  9. [17]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pages 2217–2226, 2019. 5

    2019

  10. [18]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Man- tas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InProc. ICLR,

  11. [19]

    Emr-merging: Tuning-free high- performance model merging

    Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free high- performance model merging. InProc. NeurIPS, 2024. 3, 5, 6, 7, 8, 2, 4

    2024

  12. [20]

    Multi-granular spatio-temporal token merging for training-free acceleration of video llms

    Jeongseok Hyun, Sukjun Hwang, Su Ho Han, Taeoh Kim, Inwoong Lee, Dongyoon Wee, Joon-Young Lee, Seon Joo Kim, and Minho Shim. Multi-granular spatio-temporal token merging for training-free acceleration of video llms. InProc. ICCV, 2025. 1

    2025

  13. [21]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. InProc. ICLR, 2023. 1, 2, 3, 5, 6, 7, 4

    2023

  14. [22]

    Pytorch.Programming with TensorFlow: so- lution for edge computing applications, pages 87–104, 2021

    Sagar Imambi, Kolla Bhanu Prakash, and GR Kanagachi- dambaresan. Pytorch.Programming with TensorFlow: so- lution for edge computing applications, pages 87–104, 2021. 1

    2021

  15. [23]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. InProc. ICLR, 2023. 2, 5, 6, 7, 1, 3, 4

    2023

  16. [24]

    Repair: Renormalizing permuted activations for interpolation repair

    Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renormalizing permuted activations for interpolation repair. InProc. ICLR, 2023. 2

    2023

  17. [25]

    Task vector quantization for memory- efficient model merging

    Youngeun Kim, Seunghwan Lee, Aecheon Jung, Bogon Ryu, and Sungeun Hong. Task vector quantization for memory- efficient model merging. InProc. ICCV, 2025. 2

    2025

  18. [26]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proc. ICCV, 2013. 5

    2013

  19. [27]

    Singular value decompo- sition.Numerical analysis for statisticians, pages 129–142,

    Kenneth Lange and Kenneth Lange. Singular value decompo- sition.Numerical analysis for statisticians, pages 129–142,

  20. [28]

    Mitigating parameter interference in model merging via sharpness-aware fine-tuning

    Yeoreum Lee, Jinwook Jung, and Sungyong Baik. Mitigating parameter interference in model merging via sharpness-aware fine-tuning. InProc. ICLR, 2025. 3

    2025

  21. [29]

    Map: Low-compute model merging with amortized pareto fronts via quadratic approximation

    Lu Li, Tianyu Zhang, Zhiqi Bu, Suyuchen Wang, Huan He, Jie Fu, Yonghui Wu, Jiang Bian, Yong Chen, and Yoshua Bengio. Map: Low-compute model merging with amortized pareto fronts via quadratic approximation. InProc. ICLR,

  22. [30]

    Model merging in pre-training of large language models

    Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu, Mengzhao Chen, Minrui Wang, Shiyi Zhan, et al. Model merging in pre-training of large language models. InProc. NeurIPS, 2025. 1

    2025

  23. [31]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. InProc. ACL, 2022. 5

    2022

  24. [32]

    1bit-merging: Dynamic quantized merging for large language models

    Shuqi Liu, Han Wu, Bowei He, Zehua Liu, Xiongwei Han, Mingxuan Yuan, and Linqi Song. 1bit-merging: Dynamic quantized merging for large language models. arXiv:2502.10743, 2025. 2

    2025 arXiv

  25. [33]

    Roberta: A robustly optimized bert pretraining approach.arXiv:1907.11692, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv:1907.11692, 2019. 5

    1907 arXiv

  26. [34]

    Twin-merging: Dynamic integration of modular expertise in model merging

    Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. InProc. NeurIPS, 2024. 3, 5, 6, 7, 8, 2, 4

    2024

  27. [35]

    No task left behind: Isotropic model merging with common and task-specific subspaces

    Daniel Marczak, Simone Magistri, Sebastian Cygert, Bartłomiej Twardowski, Andrew D Bagdanov, and Joost van de Weijer. No task left behind: Isotropic model merging with common and task-specific subspaces. InProc. ICML,

  28. [36]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. InProc. NeurIPS, 2022. 2, 5, 7, 1, 3, 4

    2022

  29. [37]

    Soft merging of experts with adaptive routing.arXiv:2306.03745,

    Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing.arXiv:2306.03745,

    arXiv

  30. [38]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. InProc. NeurIPS,

  31. [39]

    Dawin: Training-free dynamic weight interpolation for robust adaptation

    Changdae Oh, Yixuan Li, Kyungwoo Song, Sangdoo Yun, and Dongyoon Han. Dawin: Training-free dynamic weight interpolation for robust adaptation. InProc. ICLR, 2025. 3

    2025

  32. [40]

    Accurate and efficient low-rank model merging in core space

    Aniello Panariello, Daniel Marczak, Simone Magistri, An- gelo Porrello, Bartłomiej Twardowski, Andrew D Bagdanov, Simone Calderara, and Joost van de Weijer. Accurate and efficient low-rank model merging in core space. InProc. NeurIPS, 2025. 1

    2025

  33. [41]

    Bbq: A hand-built bias benchmark for question answering

    Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Pad- makumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R Bowman. Bbq: A hand-built bias benchmark for question answering. InProc. ACL, 2022. 5

    2022

  34. [42]

    Less is more: Efficient model merging with binary task switch

    Biqing Qi, Fangyuan Li, Zhen Wang, Junqi Gao, Dong Li, Peng Ye, and Bowen Zhou. Less is more: Efficient model merging with binary task switch. InProc. CVPR, 2025. 1, 2, 3, 4, 5, 6, 7, 8

    2025

  35. [43]

    Mingle: Mixtures of null- space gated low-rank experts for test-time continual model merging

    Zihuan Qiu, Yi Xu, Chiyuan He, Fanman Meng, Linfeng Xu, Qingbo Wu, and Hongliang Li. Mingle: Mixtures of null- space gated low-rank experts for test-time continual model merging. InProc. NeurIPS, 2025. 1

    2025

  36. [44]

    Language models are unsuper- vised multitask learners.OpenAI blog, page 9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsuper- vised multitask learners.OpenAI blog, page 9, 2019. 5

    2019

  37. [45]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InProc. ICML, 2021. 5

    2021

  38. [46]

    Squad: 100,000+ questions for machine comprehen- sion of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehen- sion of text. InProc. EMNLP, 2016. 5

    2016

  39. [47]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProc. EMNLP, 2013. 5, 7

    2013

  40. [48]

    The german traffic sign recognition benchmark: a multi-class classification competition

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Chris- tian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. InThe 2011 interna- tional joint conference on neural networks, 2011. 5

    2011

  41. [49]

    Model merging with svd to tie the knots

    George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, and Judy Hoffman. Model merging with svd to tie the knots. InProc. ICLR, 2025. 3

    2025

  42. [50]

    Cat merging: A training-free approach for resolving conflicts in model merging

    Wenju Sun, Qingyong Li, Yangliao Geng, and Boyang Li. Cat merging: A training-free approach for resolving conflicts in model merging. InProc. ICML, 2025. 2

    2025

  43. [51]

    Towards minimizing feature drift in model merging: Layer-wise task vector fusion for adaptive knowledge integration.arXiv preprint arXiv:2505.23859,

    Wenju Sun, Qingyong Li, Wen Wang, Yang Liu, Yangli-ao Geng, and Boyang Li. Towards minimizing feature drift in model merging: Layer-wise task vector fusion for adaptive knowledge integration.arXiv preprint arXiv:2505.23859,

  44. [52]

    Merging multi-task models via weight- ensembling mixture of experts

    Anke Tang, Li Shen, Yong Luo, Nan Yin, Lefei Zhang, and Dacheng Tao. Merging multi-task models via weight- ensembling mixture of experts. InProc. ICML, 2024. 3, 5, 6, 7, 8, 2, 4

    2024

  45. [53]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288, 2023. 2

    2023 arXiv

  46. [54]

    Glue: A multi-task benchmark and analysis platform for natural language under- standing

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language under- standing. InProc. ICLR, 2018. 5

    2018

  47. [55]

    Localizing task infor- mation for improved model merging and compression

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Localizing task infor- mation for improved model merging and compression. In Proc. ICML, 2024. 3, 5, 6, 7, 2, 4

    2024

  48. [56]

    Neural network acceptability judgments

    Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. InProc. ACL, 2019. 2, 5

    2019

  49. [57]

    A broad-coverage challenge corpus for sentence understanding through inference

    Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. InProc. ACL, 2018. 5

    2018

  50. [58]

    Huggingface’s transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv:1910.03771, 2019. 1, 2

    1910 arXiv

  51. [59]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing in- ference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Re- becca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Ko- rnblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing ...

    2022

  52. [60]

    Importance-based token merging for efficient image and video generation

    Haoyu Wu, Jingyi Xu, Hieu Le, and Dimitris Samaras. Importance-based token merging for efficient image and video generation. InProc. ICCV, 2025. 3

    2025

  53. [61]

    Sun database: Large-scale scene recog- nition from abbey to zoo

    Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recog- nition from abbey to zoo. InProc. CVPR, 2010. 5

    2010

  54. [62]

    Ties-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. InProc. NeurIPS, 2023. 1, 2, 3, 5, 6, 7, 8, 4

    2023

  55. [63]

    Calm: Consensus-aware localized merging for multi-task learning

    Kunda Yan, Min Zhang, Sen Cui, Qu Zikun, Bo Jiang, Feng Liu, and Changshui Zhang. Calm: Consensus-aware localized merging for multi-task learning. InProc. ICML, 2025. 3

    2025

  56. [64]

    Adamerging: Adap- tive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adap- tive model merging for multi-task learning. InProc. ICLR,

  57. [65]

    Continual model merg- ing without data: Dual projections for balancing stability and plasticity

    Enneng Yang, Anke Tang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, and Jie Zhang. Continual model merg- ing without data: Dual projections for balancing stability and plasticity. InProc. NeurIPS, 2025. 3

    2025

  58. [66]

    Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging

    Jinluan Yang, Dingnan Jin, Anke Tang, Li Shen, Didi Zhu, Zhengyu Chen, Ziyu Zhao, Daixin Wang, Qing Cui, Zhiqiang Zhang, et al. Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. InProc. NeurIPS, 2025. 1

    2025

  59. [67]

    Merg- ing vision transformers from different tasks and domains

    Peng Ye, Chenyu Huang, Mingzhu Shen, Tao Chen, Yongqi Huang, Yuning Zhang, and Wanli Ouyang. Merg- ing vision transformers from different tasks and domains. arXiv:2312.16240, 2023. 1

    2023 arXiv

  60. [68]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InProc. ICML, 2024. 2, 3, 5, 6, 7, 4

    2024

  61. [69]

    Robustmerge: Parameter-efficient model merging for mllms with direction robustness

    Fanhu Zeng, Haiyang Guo, Fei Zhu, Li Shen, and Hao Tang. Robustmerge: Parameter-efficient model merging for mllms with direction robustness. InProc. NeurIPS, 2025. 1

    2025

  62. [70]

    An overview of multi-task learn- ing.National Science Review, pages 30–43, 2018

    Yu Zhang and Qiang Yang. An overview of multi-task learn- ing.National Science Review, pages 30–43, 2018. 1

    2018

  63. [71]

    A survey on multi-task learning

    Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE transactions on knowledge and data engineering, pages 5586–5609, 2021. 1

    2021

  64. [72]

    Beyond training: Dynamic token merging for zero-shot video understanding

    Yiming Zhang, Zhuokai Zhao, Zhaorun Chen, Zenghui Ding, Xianjun Yang, and Yining Sun. Beyond training: Dynamic token merging for zero-shot video understanding. InProc. ICCV, 2025. 2

    2025

  65. [73]

    Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering

    Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, and Fei Wu. Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering. InProc. ICLR, 2025. 1

    2025

  66. [74]

    Free-merging: Fourier transform for efficient model merging

    Shenghe Zheng and Hongzhi Wang. Free-merging: Fourier transform for efficient model merging. InProc. CVPR, 2025. 3, 5, 6

    2025

  67. [75]

    Aim: Adaptive inference of multi-modal llms via token merging and pruning

    Yiwu Zhong, Zhuoming Liu, Yin Li, and Liwei Wang. Aim: Adaptive inference of multi-modal llms via token merging and pruning. InProc. ICCV, 2025. 3

    2025

  68. [76]

    Metagpt: Merging large language models using model exclusive task arithmetic

    Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. Metagpt: Merging large language models using model exclusive task arithmetic. InProc. EMNLP, 2024. 3

    2024

  69. [77]

    Hm3: Hierarchical multi-objective model merging for pretrained models

    Yu Zhou, Xingyu Wu, Jibin Wu, Liang Feng, and Kay Chen Tan. Hm3: Hierarchical multi-objective model merging for pretrained models. InProc. NeurIPS, 2025. 1

    2025

  70. [78]

    Remedy: Recipe merging dynam- ics in large vision-language models

    Didi Zhu, Yibing Song, Tao Shen, Ziyu Zhao, Jinluan Yang, Min Zhang, and Chao Wu. Remedy: Recipe merging dynam- ics in large vision-language models. InProc. ICLR, 2025. 3 Stay Unique, Stay Efficient: Preserving Model Personality in Multi-Task Merging Supplementary Material

    2025

  71. [79]

    The Complete Pipeline of DTS Here, we take the task vector as an example to illustrate the complete pipeline of DTS, which can also be applied to the difference vector. We first employ singular value decomposition on each task vector and retain only the top-r singular values a...

  72. [80]

    Baselines for seen tasks • Individual Modelsrefer to task-specific models before merging

    Baselines 7.1. Baselines for seen tasks • Individual Modelsrefer to task-specific models before merging. • Weight-Averagingperforms element-wise averaging of the weights across all fine-tuned models. • Fisher-Merging[ 36] leverages Fisher information matri- ces to estimate par...

  73. [81]

    Datasets For visual classification tasks, we employ classification accu- racy as the evaluation metric

    More Experimental Details 8.1. Datasets For visual classification tasks, we employ classification accu- racy as the evaluation metric. For natural language process- ing tasks, we follow the standard GLUE evaluation protocol: CoLA is assessed using Matthews correlation, STS-B u...

  74. [82]

    More Backbones In addition to the backbones evaluated in the main paper, we also assess the performance of various methods on ViT-B/16, ViT-L/14, and GPT-2 backbones

    More Experimental Results 9.1. More Backbones In addition to the backbones evaluated in the main paper, we also assess the performance of various methods on ViT-B/16, ViT-L/14, and GPT-2 backbones. As shown in Table 11, Ta- ble 7, and Tables 8, our method consistently outperfo...

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.