Task-Feature Specialization explains weight disentanglement in task arithmetic and leads to orthogonality, which OrthoReg enforces to enhance performance of model composition methods.
Multi-task model merging via adaptive weight disentanglement
6 Pith papers cite this work. Polarity classification is still indexing.
6
Pith papers citing it
citation-role summary
method 1
citation-polarity summary
roles
method 1polarities
use method 1representative citing papers
ReTeX predicts additive offsets to undo merging interference and uses an SVD subspace signature task identifier to recover over 95% of expert performance while improving generalization to unseen tasks.
PACT preserves load-bearing wall dimensions from pre-trained weights inside task vectors to reduce conflicts and improve merged model performance.
Merging breaks MoE routing via softmax sensitivity; HARC uses Hessian curvature for closed-form router calibration that improves merged model performance without retraining.
Memory Grafting improves language-model benchmarks by grafting offline hidden-state memory from a larger model into a recipient model using n-gram lookups and lightweight adapters, outperforming MoE and vanilla Engram baselines at 0.92B and 2.8B scales.
The paper introduces a new taxonomy for model merging methods and reviews their applications in LLMs, MLLMs, continual learning, multi-task learning, and other subfields while outlining open challenges.
citing papers explorer
-
Understanding and Enforcing Weight Disentanglement in Task Arithmetic
Task-Feature Specialization explains weight disentanglement in task arithmetic and leads to orthogonality, which OrthoReg enforces to enhance performance of model composition methods.
-
Learning to Recover Task Experts from a Multi-Task Merged Model
ReTeX predicts additive offsets to undo merging interference and uses an SVD subspace signature task identifier to recover over 95% of expert performance while improving generalization to unseen tasks.
-
PACT: Preserving Anchored Cores in Task-vectors for Model Merging
PACT preserves load-bearing wall dimensions from pre-trained weights inside task vectors to reduce conflicts and improve merged model performance.
-
When Model Merging Breaks Routing: Training-Free Calibration for MoE
Merging breaks MoE routing via softmax sensitivity; HARC uses Hessian curvature for closed-form router calibration that improves merged model performance without retraining.
-
Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory
Memory Grafting improves language-model benchmarks by grafting offline hidden-state memory from a larger model into a recipient model using n-gram lookups and lightweight adapters, outperforming MoE and vanilla Engram baselines at 0.92B and 2.8B scales.
-
Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunities
The paper introduces a new taxonomy for model merging methods and reviews their applications in LLMs, MLLMs, continual learning, multi-task learning, and other subfields while outlining open challenges.