Pith. sign in

REVIEW 28 cited by

Arcee's MergeKit: A Toolkit for Merging Large Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.13257 v3 pith:DWBHS4WL submitted 2024-03-20 cs.CL cs.AIcs.LG

Charles Goddard , Shamane Siriwardhana , Malikeh Ehghaghi , Luke Meyers , Vlad Karpukhin , Brian Benedict , Mark McQuade , Jacob Solawetz This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords modelmodelsmergekitmergingopen-sourcecheckpointscreationlanguage
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The rapid expansion of the open-source language model landscape presents an opportunity to merge the competencies of these model checkpoints by combining their parameters. Advances in transfer learning, the process of fine-tuning pretrained models for specific tasks, has resulted in the development of vast amounts of task-specific models, typically specialized in individual tasks and unable to utilize each other's strengths. Model merging facilitates the creation of multitask models without the need for additional training, offering a promising avenue for enhancing model performance and versatility. By preserving the intrinsic capabilities of the original models, model merging addresses complex challenges in AI - including the difficulties of catastrophic forgetting and multitask learning. To support this expanding area of research, we introduce MergeKit, a comprehensive, open-source library designed to facilitate the application of model merging strategies. MergeKit offers an extensible framework to efficiently merge models on any hardware, providing utility to researchers and practitioners. To date, thousands of models have been merged by the open-source community, leading to the creation of some of the worlds most powerful open-source model checkpoints, as assessed by the Open LLM Leaderboard. The library is accessible at https://github.com/arcee-ai/MergeKit.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 28 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. modelDNA: Calibrated Lineage Verification and Merge Decomposition from Sampled Weight Fingerprints

    cs.LG 2026-07 conditional novelty 6.5 of 10

    Sampled weight fingerprints recover LLM parentage with AUROC 1.0 and zero false positives, and recover published mergekit mixture weights without full downloads.

  2. Improving Ad-hoc Search Effectiveness for Conversational Information Retrieval via Model Merging

    cs.IR 2026-07 accept novelty 6.0 of 10

    Linear and spherical interpolation of ANCE and QRACDR parameters yields a single dense retriever that recovers ad-hoc effectiveness while retaining conversational skill and improving zero-shot generalization.

  3. Prompt Compression via Activation Aggregation

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A learned weighted sum of intermediate-layer activations compresses an instruction prompt into a single patch vector that, injected at an early layer, recovers task accuracy within ~2% of the full prompt.

  4. DivMerge: A divergence-based model merging method for multi-tasking

    cs.LG 2025-09 conditional novelty 6.0 of 10

    DivMerge learns task-arithmetic merging weights by minimizing Jensen-Shannon divergence between each specialist model and the merged model, improving multi-task performance and scalability.

  5. Lethe: Purifying Backdoored Large Language Models with Knowledge Dilution

    cs.CL 2025-08 conditional novelty 6.0 of 10

    LETHE uses parameter-level model merging plus prompt-level word definitions to dilute backdoor behavior in LLMs, cutting attack success to below 7% in most tested settings.

  6. Model Merging is Secretly Certifiable: Non-Vacuous Generalisation Bounds for Low-Shot Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    First non-vacuous PAC-Bayes certificates for large vision and language models in the 100-example low-shot regime, obtained by reinterpreting model merging as a low-dimensional posterior.

  7. EMoE: Training-Free Expert Disagreement for Uncertainty-Aware Text-to-Image Diffusion

    cs.AI 2025-05 conditional novelty 6.0 of 10

    Expert disagreement inside pretrained MoE diffusion models, measured as latent variance at the first denoising step, gives a training-free prompt uncertainty signal that correlates with text-image alignment across languages.

  8. DMM: Building a Versatile Image Generation Model via Distillation-Based Model Merging

    cs.CV 2025-04 conditional novelty 6.0 of 10

    DMM trains a single style-promptable diffusion model to reproduce the outputs of multiple teacher models, achieving a merged-model FIDt of 77.51 versus a reference of 74.91.

  9. Adapting Language-Specific LLMs to a Reasoning Model in One Day via Model Merging -- An Open Recipe

    cs.CL 2025-02 conditional novelty 6.0 of 10

    A Thai 70B model trained with an SFT-plus-DARE-merge recipe matches DeepSeek R1 on reasoning benchmarks while retaining most Thai language quality.

  10. Faster Machine Translation Ensembling with Reinforcement Learning and Competitive Correction

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A DQN-based candidate selection and a competitive correction block improve MT ensembling quality while reducing inference cost on English-Hindi and Hindi-English tasks.

  11. Merging Feed-Forward Sublayers for Compressed Transformers

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Merging aligned feed-forward sublayers into tied weights can remove over a third of a Transformer's feed-forward parameters with only small performance losses, after a short fine-tuning.

  12. Transformer-Squared: Self-adaptive LLMs

    cs.LG 2025-01 conditional novelty 6.0 of 10

    Transformer-Squared selectively rescales singular values of an LLM's weights with RL-trained expert vectors, then mixes these experts at inference to adapt to unseen tasks.

  13. Parameter-Efficient Interventions for Enhanced Model Merging

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Applying ReFT-style interventions at every transformer block of a merged model improves multi-task accuracy beyond post-hoc single-layer repair, and slicing the representation keeps the parameter cost low.

  14. Channel Merging: Preserving Specialization for Merged Experts

    cs.CL 2024-12 conditional novelty 6.0 of 10

    Channel Merging clusters similar channel parameters from fine-tuned LLMs and reconstructs the selected expert at inference, matching unmerged accuracy with about 53% of the ensemble's parameters.

  15. How to Merge Your Multimodal Models Over Time?

    cs.LG 2024-12 conditional novelty 6.0 of 10

    A systematic study of temporal model merging shows that initialization and deployment choices matter far more than the merging technique, with EMA-style weight interpolation as the best practice.

  16. AstroMLab 3: Achieving GPT-4o Level Performance in Astronomy with a Specialized 8B-Parameter Large Language Model

    astro-ph.IM 2024-11 conditional novelty 6.0 of 10

    A specialized 8B-parameter language model trained on astronomy literature and synthetic Q&A matches GPT-4o on the AstroMLab-1 astronomy benchmark at a fraction of the cost.

  17. Fishers for Free? Approximating the Fisher Information Matrix by Recycling the Squared Gradient Accumulator

    cs.LG 2025-07 conditional novelty 5.0 of 10

    The squared gradient accumulator from Adam can serve as a free, drop-in approximation of the empirical Fisher diagonal across model merging, pruning, task embedding, and continual learning, with comparable performance.

  18. Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Adaptive selection and dynamic weighted fusion of source LLMs reduces knowledge interference and improves target model accuracy compared to FuseLLM.

  19. PatientDx: Merging Large Language Models for Protecting Data-Privacy in Healthcare

    cs.CL 2025-04 reject novelty 5.0 of 10

    PatientDx merges a math-specialized LLM with a medical or instruct LLM via SLerp and reports mortality-prediction gains on MIMIC-IV, but the merging weight is tuned on the test set, undermining the claimed improvement.

  20. Speculate, then Collaborate: Fusing Knowledge of Language Models during Decoding

    cs.CL 2025-02 conditional novelty 5.0 of 10

    CoSD fuses two LLMs at decoding time: it keeps the draft model's tokens unless a confidence rule or a decision tree over token probabilities says the assistant model's token is better.

  21. MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs

    cs.CL 2025-02 conditional novelty 5.0 of 10

    Merging specialized LLMs into a MoE can be improved by replacing averaging with Dare/Ties merging and by using perplexity-based routing, while heterogeneous experts can be merged with projectors and a sequence-level router.

  22. Soup to go: mitigating forgetting during continual learning with model averaging

    cs.LG 2025-01 conditional novelty 5.0 of 10

    SFA reduces forgetting in continual learning by averaging the currently training model with a checkpoint from previous tasks during fine-tuning, matching data-buffer rehearsal without storing past data.

  23. Optimizing Large Language Models for Turkish: New Methodologies in Corpus Selection and Training

    cs.CL 2024-12 conditional novelty 5.0 of 10

    Fine-tuning Llama3-8B on a small-model-selected mix of translated and synthetic Turkish corpora improves few-shot benchmark scores and human preference ratings.

  24. CLUES: Collaborative High-Quality Data Selection for LLMs via Training Dynamics

    cs.CL 2025-07 conditional novelty 4.0 of 10

    A collaborative data-selection method that scores each private sample's influence on a public anchor set and filters by a global threshold before federated learning or model merging.

  25. The Aloe Family Recipe for Open and Specialized Healthcare LLMs

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Aloe Beta, a family of open-weights health LLMs built from Llama 3.1 and Qwen 2.5, matches or exceeds closed medical models on MCQA benchmarks while improving safety via DPO.

  26. Parameter-Efficient Checkpoint Merging via Metrics-Weighted Averaging

    cs.LG 2025-04 conditional novelty 4.0 of 10

    Merging LoRA checkpoints with weights based on inverse training loss, plus a rank-based penalty, beats uniform averaging on three fine-tuning tasks.

  27. BgGPT 1.0: Extending English-centric LLMs to other languages

    cs.CL 2024-12 conditional novelty 4.0 of 10

    Continually pretraining Gemma-2 on a curated Bulgarian corpus and merging with instruction-tuned models yields open Bulgarian-English models that beat larger open models on Bulgarian benchmarks.

  28. Neutralizing Backdoors through Information Conflicts for Large Language Models

    cs.CL 2024-11 conditional novelty 4.0 of 10

    A trigger-agnostic defense that merges a backdoored LLM with a clean-data LoRA model and adds contradictory prompt evidence, reducing attack success while keeping most clean-task accuracy.

Pith tools