REVIEW 15 cited by
Deep Model Fusion: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Deep Model Fusion: A Survey
read the original abstract
Deep model fusion/merging is an emerging technique that merges the parameters or predictions of multiple deep learning models into a single one. It combines the abilities of different models to make up for the biases and errors of a single model to achieve better performance. However, deep model fusion on large-scale deep learning models (e.g., LLMs and foundation models) faces several challenges, including high computational cost, high-dimensional parameter space, interference between different heterogeneous models, etc. Although model fusion has attracted widespread attention due to its potential to solve complex real-world tasks, there is still a lack of complete and detailed survey research on this technique. Accordingly, in order to understand the model fusion method better and promote its development, we present a comprehensive survey to summarize the recent progress. Specifically, we categorize existing deep model fusion methods as four-fold: (1) "Mode connectivity", which connects the solutions in weight space via a path of non-increasing loss, in order to obtain better initialization for model fusion; (2) "Alignment" matches units between neural networks to create better conditions for fusion; (3) "Weight average", a classical model fusion method, averages the weights of multiple models to obtain more accurate results closer to the optimal solution; (4) "Ensemble learning" combines the outputs of diverse models, which is a foundational technique for improving the accuracy and robustness of the final model. In addition, we analyze the challenges faced by deep model fusion and propose possible research directions for model fusion in the future. Our review is helpful in deeply understanding the correlation between different model fusion methods and practical application methods, which can enlighten the research in the field of deep model fusion.
Forward citations
Cited by 15 Pith papers
-
Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space
Model merging is cast as PoE inference with EBM experts, revealing Gaussian assumptions in prior work and proposing convergent Cauchy experts that improve empirical performance.
-
Good Agentic Friends Do Not Just Give Verbal Advice: They Can Update Your Weights
TFlow enables multi-agent LLMs to collaborate via transient low-rank LoRA perturbations derived from sender activations, yielding up to 8.5 accuracy gains and 83% token reduction versus text-based baselines on Qwen3-4...
-
UIPress: Bringing Optical Token Compression to UI-to-Code Generation
UIPress is the first encoder-side learned optical compression method for UI-to-Code that compresses visual tokens to 256, outperforming the uncompressed baseline by 7.5% CLIP score and the best inference-time baseline...
-
Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.
-
Personalized Image Aesthetic Assessment via Preference-rich Sample Mining and Cohort Merging
PRAC mines preference-rich images and merges LoRA adapters from aesthetically similar users to achieve state-of-the-art personalized aesthetic rating prediction.
-
PACT: Preserving Anchored Cores in Task-vectors for Model Merging
PACT preserves load-bearing wall dimensions from pre-trained weights inside task vectors to reduce conflicts and improve merged model performance.
-
DLLG: Dynamic Logit-Level Gating of LLM Experts
DLLG learns token-level fusion weights for LLM experts from sparse response supervision and outperforms routing, ensembling, and merging baselines on reasoning and code tasks.
-
Unlocking the Potential of Continual Model Merging: An ODE Perspective
Introduces ODE-M, an ODE-based merging method for continual model merging that follows low-loss connecting paths to mitigate catastrophic forgetting.
-
Unlocking the Potential of Continual Model Merging: An ODE Perspective
ODE-M traces low-loss connecting paths via time-dependent velocity fields and barrier constraints to improve controllability and reduce forgetting in continual model merging.
-
Unlocking the Potential of Continual Model Merging: An ODE Perspective
ODE-M formulates continual model merging as a barrier-aware ODE trajectory in parameter space, using first-order feedback and a utility-aware schedule to balance retained knowledge and new task performance.
-
Can Heterogeneous Language Models Be Fused?
HeteroFusion fuses heterogeneous LLMs via topology-based alignment and conflict-aware denoising, outperforming merging and ensemble baselines in cross-family and multi-source settings.
-
Efficient and Accurate Method for Separating Variant Components from Invariant Background and Component Model Fusion for Fast RFIC Design Space Exploration
RFIC design sweeps are accelerated about 37x by splitting the field solution into an invariant-background part (solved once) and a small variant-component part, using layer-wise translation symmetry to build the coupl...
-
Semantic-guided LoRA Parameters Generation
SG-LoRA generates LoRA parameters for unseen tasks from text descriptions alone, using semantic expert selection plus a conditional VAE, matching or exceeding oracle fine-tuning on retrieval benchmarks.
-
Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
MergeLock applies random invertible matrix transformations to Transformer attention weights, preserving the model's own output while forcing any merged model's accuracy down to near random.
-
Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
The paper introduces a new taxonomy for model merging methods and reviews their applications in LLMs, MLLMs, continual learning, multi-task learning, and other subfields while outlining open challenges.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.