REVIEW 25 cited by
Deep Model Fusion: A Survey
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Deep model fusion/merging is an emerging technique that merges the parameters or predictions of multiple deep learning models into a single one. It combines the abilities of different models to make up for the biases and errors of a single model to achieve better performance. However, deep model fusion on large-scale deep learning models (e.g., LLMs and foundation models) faces several challenges, including high computational cost, high-dimensional parameter space, interference between different heterogeneous models, etc. Although model fusion has attracted widespread attention due to its potential to solve complex real-world tasks, there is still a lack of complete and detailed survey research on this technique. Accordingly, in order to understand the model fusion method better and promote its development, we present a comprehensive survey to summarize the recent progress. Specifically, we categorize existing deep model fusion methods as four-fold: (1) "Mode connectivity", which connects the solutions in weight space via a path of non-increasing loss, in order to obtain better initialization for model fusion; (2) "Alignment" matches units between neural networks to create better conditions for fusion; (3) "Weight average", a classical model fusion method, averages the weights of multiple models to obtain more accurate results closer to the optimal solution; (4) "Ensemble learning" combines the outputs of diverse models, which is a foundational technique for improving the accuracy and robustness of the final model. In addition, we analyze the challenges faced by deep model fusion and propose possible research directions for model fusion in the future. Our review is helpful in deeply understanding the correlation between different model fusion methods and practical application methods, which can enlighten the research in the field of deep model fusion.
Forward citations
Cited by 25 Pith papers
-
Demystifying Mergeability: Interpretable Properties to Predict Model Merging Success
Model mergeability depends on merging method and partner tasks, with gradient alignment metrics emerging as the strongest predictors of normalized accuracy after merging.
-
Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs
RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.
-
Personalized Image Aesthetic Assessment via Preference-rich Sample Mining and Cohort Merging
PRAC mines preference-rich images and merges LoRA adapters from aesthetically similar users to achieve state-of-the-art personalized aesthetic rating prediction.
-
Forgetting of task-specific knowledge in model merging-based continual learning
Controlled vision experiments show that linear weight merging preserves shared knowledge but rapidly erases task-specific (unshared) knowledge, and merging sequentially trained models is safer than merging parallel-tr...
-
Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs
Graft merges two domain-specialized multimodal models by combining channel-wise gating, entropy-based global weighting, and an activation compatibility score to improve fusion without retraining.
-
Assembly of Experts: Linear-time construction of the Chimera LLM variants with emergent and adaptable behaviors
Weight interpolation between two DeepSeek models, restricted to routed expert tensors, yields a 671B Chimera model with near-R1 reasoning and roughly 40% lower output-token count.
-
Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning
First non-vacuous PAC-Bayes certificates for large vision and language models in the 100-example low-shot regime, obtained by reinterpreting model merging as a low-dimensional posterior.
-
Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning
Fine-tuning with sharpness-aware minimization reduces parameter interference in model merging, yielding better merged multi-task accuracy than standard fine-tuning, linearized fine-tuning, and linear-layer-only fine-tuning.
-
No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Uniformly rescaling singular values of summed task updates, plus adding task-specific residual directions, improves multi-task model merging.
-
Model Fusion via Neuron Transplantation
A new fusion method, Neuron Transplantation, concatenates ensemble members and prunes back down to a single model's size, outperforming individual models after fine-tuning.
-
Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent
DOGE merges fine-tuned models by optimizing a data-free loss-gap proxy with gradient steps projected orthogonal to a shared task subspace, improving average accuracy over previous methods.
-
Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learning
ACMap achieves exemplar-free class-incremental learning with constant inference time by averaging task-specific adapters and shifting previous prototypes with the current task's centroid drift.
-
Parameter-Efficient Interventions for Enhanced Model Merging
Applying ReFT-style interventions at every transformer block of a merged model improves multi-task accuracy beyond post-hoc single-layer repair, and slicing the representation keeps the parameter cost low.
-
Revisiting Weight Averaging for Model Merging
Centering task vectors around the weight average and keeping their top singular vectors yields a merged multi-task model that outperforms prior merging methods on vision and NLP benchmarks.
-
Automated item evaluation: Predicting item acceptance and rejection using LLM-generated critiques
A text-only classifier plus LLM-generated critiques predicts whether standardized test items will be permanently rejected with AUC 0.80, with much higher accuracy for math than ELA.
-
Efficient and Accurate Method for Separating Variant Components from Invariant Background and Component Model Fusion for Fast RFIC Design Space Exploration
RFIC design sweeps are accelerated about 37x by splitting the field solution into an invariant-background part (solved once) and a small variant-component part, using layer-wise translation symmetry to build the coupl...
-
Semantic-guided LoRA Parameters Generation
SG-LoRA generates LoRA parameters for unseen tasks from text descriptions alone, using semantic expert selection plus a conditional VAE, matching or exceeding oracle fine-tuning on retrieval benchmarks.
-
Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing
MergeLock applies random invertible matrix transformations to Transformer attention weights, preserving the model's own output while forcing any merged model's accuracy down to near random.
-
PSO-Merging: Merging Models Based on Particle Swarm Optimization
PSO-Merging applies particle swarm optimization over model weight space, seeded with original and sparsified experts, to build multitask models that outperform existing merging baselines on several language benchmarks.
-
ICM-Fusion: In-Context Meta-Optimized LoRA Fusion for Multi-Task Adaptation
ICM-Fusion uses a conditional VAE plus task-vector guidance to fuse multiple LoRA adapters into one model, reporting marginal average gains on vision and language benchmarks and larger gains in a few-shot long-tail setup.
-
FCOS: A Two-Stage Recoverable Model Pruning Framework for Automatic Modulation Recognition
FCOS combines channel clustering and layer collapse diagnosis to prune AMR models by over 95% with minimal accuracy loss.
-
SeWA: Selective Weight Average via Probabilistic Masking
SeWA adaptively selects a sparse set of checkpoints for weight averaging via learned probabilistic masks, claiming better generalization with fewer averaged points.
-
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion
Transformer attention layers admit continuous rotation symmetries, and aligning a source model's attention weights by the optimal rotation before weight averaging improves model fusion accuracy.
-
Learn Faster and Remember More: Balancing Exploration and Exploitation for Continual Test-time Adaptation
A mean-teacher framework with multi-level consistency regularization and complementary anchor replay improves continual test-time adaptation by balancing exploration and exploitation.
-
Rethinking Weight-Averaged Model-merging
Weight-averaged model merging is reinterpreted as template matching and implicit regularization, with systematic experiments showing logits ensembling generally outperforms weight averaging and ViTs degrade sharply un...
Discussion (0). Continue with ORCID to comment.