REVIEW 9 cited by
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
In this paper, we unveil that Language Models (LMs) can acquire new capabilities by assimilating parameters from homologous models without retraining or GPUs. We first introduce DARE to set most delta parameters (i.e., the disparity between fine-tuned and pre-trained parameters) to zeros without affecting the abilities of Supervised Fine-Tuning (SFT) LMs, which randomly Drops delta parameters with a ratio $p$ And REscales the remaining ones by $1 / (1 - p)$ to approximate the original embeddings. Then, we use DARE as a versatile plug-in to sparsify delta parameters of multiple SFT homologous models for mitigating parameter interference and merge them into a single model by parameter fusing. We experiment with encoder- and decoder-based LMs, showing that: (1) SFT delta parameter value ranges are typically small (within 0.002) with extreme redundancy, and DARE can effortlessly eliminate 90% or even 99% of them; (2) DARE can merge multiple task-specific LMs into one LM with diverse capabilities. Notably, this phenomenon is more pronounced in large-scale LMs, where the merged LM reveals the potential to surpass the performance of any source LM, providing a new discovery. We also utilize DARE to create a merged LM that ranks first among models with 7 billion parameters on the Open LLM Leaderboard.
Forward citations
Cited by 9 Pith papers
-
modelDNA: Calibrated Lineage Verification and Merge Decomposition from Sampled Weight Fingerprints
Sampled weight fingerprints recover LLM parentage with AUROC 1.0 and zero false positives, and recover published mergekit mixture weights without full downloads.
-
Spectral Rewiring for Exploration, Purification, and Model Merging
Subspace-Aligned Rewiring projects RL weight updates onto the base model’s SVD basis, retaining a compact rewiring matrix that preserves reasoning and improves exploration and multi-domain merging.
-
Asymmetric Collapse in Model Merging: When Refusal Over- writes Recognition
Across four standard merging methods, refusal behavior from a large task vector overwrites fine-grained harm classification, leaving at most 12.9% accuracy.
-
Similarity-Guided Curriculum Fine-Tuning of LLMs for Neural Architecture Synthesis
MinHash-band curriculum fine-tuning of OlympicCoder-7B yields 60% peak success on high-similarity NAS code generation, yet merge drift and interface repair dominate later stages.
-
Harnessing Optimization Dynamics for Curvature-Informed Model Merging
Optimization Trajectory Aware merging uses Adam second moments as a curvature proxy, first pruning task-vector edits with Fast Fisher Grafting, then reweighting survivors with a compressed curvature preconditioner.
-
DivMerge: A divergence-based model merging method for multi-tasking
DivMerge learns task-arithmetic merging weights by minimizing Jensen-Shannon divergence between each specialist model and the merged model, improving multi-task performance and scalability.
-
MiCoTA: Bridging the Learnability Gap with Intermediate CoT and Teacher Assistants
Training small language models on intermediate-length reasoning chains from a merged mid-sized teacher assistant improves their math reasoning scores over direct distillation from a large teacher.
-
Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective
After truncating or expanding checkpoints to a shared shape, small-ratio weight averaging slightly improves average benchmark scores over strong Qwen sources, but headline gains are inflated by per-task best-ratio selection.
-
Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
A comprehensive review that categorizes methods for shortening and adaptively triggering chain-of-thought reasoning in large language models.
Discussion (0). Sign in to comment.