REVIEW 15 cited by
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
read the original abstract
In this paper, we unveil that Language Models (LMs) can acquire new capabilities by assimilating parameters from homologous models without retraining or GPUs. We first introduce DARE to set most delta parameters (i.e., the disparity between fine-tuned and pre-trained parameters) to zeros without affecting the abilities of Supervised Fine-Tuning (SFT) LMs, which randomly Drops delta parameters with a ratio $p$ And REscales the remaining ones by $1 / (1 - p)$ to approximate the original embeddings. Then, we use DARE as a versatile plug-in to sparsify delta parameters of multiple SFT homologous models for mitigating parameter interference and merge them into a single model by parameter fusing. We experiment with encoder- and decoder-based LMs, showing that: (1) SFT delta parameter value ranges are typically small (within 0.002) with extreme redundancy, and DARE can effortlessly eliminate 90% or even 99% of them; (2) DARE can merge multiple task-specific LMs into one LM with diverse capabilities. Notably, this phenomenon is more pronounced in large-scale LMs, where the merged LM reveals the potential to surpass the performance of any source LM, providing a new discovery. We also utilize DARE to create a merged LM that ranks first among models with 7 billion parameters on the Open LLM Leaderboard.
Forward citations
Cited by 15 Pith papers
-
ResMerge: Residual-based Spectral Merging of Large Language Models
ResMerge improves merging of RL expert LLMs via a stable residual consensus backbone plus gated head correction, outperforming task-vector and spectral baselines in capability preservation.
-
Interpretability-Guided Layer Selection over Subspace Projection: SAEs as Stethoscopes, Not Scalpels, for Raw Task Vector Model Editing
SAEs used for layer selection with raw task vectors outperform subspace projection and raise math reasoning accuracy on Gemma-3-4B-IT.
-
CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
CapTrack shows post-training causes drift beyond facts, with instruction fine-tuning producing stronger behavioral changes than preference optimization across model families.
-
modelDNA: Calibrated Lineage Verification and Merge Decomposition from Sampled Weight Fingerprints
Sampled weight fingerprints recover LLM parentage with AUROC 1.0 and zero false positives, and recover published mergekit mixture weights without full downloads.
-
Spectral Rewiring for Exploration, Purification, and Model Merging
Subspace-Aligned Rewiring projects RL weight updates onto the base model’s SVD basis, retaining a compact rewiring matrix that preserves reasoning and improves exploration and multi-domain merging.
-
Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognition
Across four standard merging methods, refusal behavior from a large task vector overwrites fine-grained harm classification, leaving at most 12.9% accuracy.
-
Similarity-Guided Curriculum Fine-Tuning of LLMs for Neural Architecture Synthesis
MinHash-band curriculum fine-tuning of OlympicCoder-7B yields 60% peak success on high-similarity NAS code generation, yet merge drift and interface repair dominate later stages.
-
Spectral Unforgetting: Post-Hoc Recovery of Damaged Capabilities Without Retraining
DG-Hard uses Donoho-Gavish hard thresholding on the fine-tuning weight delta to separate task-aligned signal from noise-like residual, recovering damaged capabilities while preserving target-task gains.
-
Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts
BAR trains independent domain experts via separate mid-training, SFT, and RL pipelines then composes them with a MoE router to match monolithic retraining performance at lower cost and without catastrophic forgetting.
-
Weight Patching: Toward Source-Level Mechanistic Localization in LLMs
Weight Patching localizes capabilities to specific parameter modules in LLMs by replacing weights from a behavior-specialized model into a base model and validating recovery via a vector-anchor interface, revealing a ...
-
Can Heterogeneous Language Models Be Fused?
HeteroFusion fuses heterogeneous LLMs via topology-based alignment and conflict-aware denoising, outperforming merging and ensemble baselines in cross-family and multi-source settings.
-
Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective
After truncating or expanding checkpoints to a shared shape, small-ratio weight averaging slightly improves average benchmark scores over strong Qwen sources, but headline gains are inflated by per-task best-ratio selection.
-
Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting
A plug-and-play KL regularizer that masks the target token and renormalizes probabilities to improve the learning-forgetting trade-off in LoRA adaptation of LLMs.
-
Black-Box Optimization of Mixed Binary-Continuous Variables: Challenges and Opportunities in Evolutionary Model Merging
Data flow space model merging is formalized as a mixed binary-continuous black-box optimization problem, where a structured approach respecting variable dependencies achieves 6.7% higher accuracy and 51.4% smaller sea...
-
Sparse Orthogonal Parameters Tuning for Continual Learning
SoTU merges sparse orthogonal delta parameters learned across streaming tasks to fuse knowledge and mitigate forgetting in pre-trained model continual learning.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.