Pith. sign in

REVIEW 7 cited by

AdaMerging: Adaptive Model Merging for Multi-Task Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.02575 v2 pith:CWJJEWKK submitted 2023-10-04 cs.LG cs.CV

Enneng Yang , Zhenyi Wang , Li Shen , Shiwei Liu , Guibing Guo , Xingwei Wang , Dacheng Tao This is my paper
classification cs.LGcs.CV
keywords adamergingmodelmergingtasksdatamodelsarithmeticmulti-task
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multi-task learning (MTL) aims to empower a model to tackle multiple tasks simultaneously. A recent development known as task arithmetic has revealed that several models, each fine-tuned for distinct tasks, can be directly merged into a single model to execute MTL without necessitating a retraining process using the initial training data. Nevertheless, this direct addition of models often leads to a significant deterioration in the overall performance of the merged model. This decline occurs due to potential conflicts and intricate correlations among the multiple tasks. Consequently, the challenge emerges of how to merge pre-trained models more effectively without using their original training data. This paper introduces an innovative technique called Adaptive Model Merging (AdaMerging). This approach aims to autonomously learn the coefficients for model merging, either in a task-wise or layer-wise manner, without relying on the original training data. Specifically, our AdaMerging method operates as an automatic, unsupervised task arithmetic scheme. It leverages entropy minimization on unlabeled test samples from the multi-task setup as a surrogate objective function to iteratively refine the merging coefficients of the multiple models. Our experimental findings across eight tasks demonstrate the efficacy of the AdaMerging scheme we put forth. Compared to the current state-of-the-art task arithmetic merging scheme, AdaMerging showcases a remarkable 11\% improvement in performance. Notably, AdaMerging also exhibits superior generalization capabilities when applied to unseen downstream tasks. Furthermore, it displays a significantly enhanced robustness to data distribution shifts that may occur during the testing phase.

Discussion (0). Sign in to comment.

Forward citations

Cited by 7 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DivMerge: A divergence-based model merging method for multi-tasking

    cs.LG 2025-09 conditional novelty 6.0 of 10

    DivMerge learns task-arithmetic merging weights by minimizing Jensen-Shannon divergence between each specialist model and the merged model, improving multi-task performance and scalability.

  2. Interaction-Merged Motion Planning: Effectively Leveraging Diverse Motion Datasets for Robust Planning

    cs.RO 2025-07 conditional novelty 6.0 of 10

    A two-step model-merging method transfers interaction knowledge from multiple motion datasets to a target domain, outperforming ensembling and domain adaptation at the same inference cost.

  3. Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Graft merges two domain-specialized multimodal models by combining channel-wise gating, entropy-based global weighting, and an activation compatibility score to improve fusion without retraining.

  4. Harmonizing and Merging Source Models for CLIP-based Domain Generalization

    cs.CV 2025-06 conditional novelty 6.0 of 10

    HAM trains per-domain CLIP encoders, enriches them with confident cross-domain samples, aligns their update directions, and merges them with redundancy trimming, reaching 79.0% average accuracy on five DG benchmarks w...

  5. PSO-Merging: Merging Models Based on Particle Swarm Optimization

    cs.LG 2025-08 conditional novelty 5.0 of 10

    PSO-Merging applies particle swarm optimization over model weight space, seeded with original and sparsified experts, to build multitask models that outperform existing merging baselines on several language benchmarks.

  6. Exploring Sparse Adapters for Scalable Merging of Parameter Efficient Experts

    cs.LG 2025-07 conditional novelty 5.0 of 10

    Sparse adapters trained with max connection sensitivity outperform LoRA and full fine-tuning both alone and after merging 20 task experts, but still lag multitask training on unseen tasks.

  7. StatsMerging: Statistics-Guided Model Merging via Task-Specific Teacher Distillation

    cs.LG 2025-06 conditional novelty 5.0 of 10

    StatsMerging predicts per-layer merging coefficients from weight statistics and teacher pseudo-labels, achieving 94.5% average accuracy across eight vision tasks, 5.1 points above WEMoE.

Pith tools