Pith. sign in

REVIEW 6 cited by

Fusing finetuned models for better pretraining

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2204.03044 v1 pith:ACWHURWL submitted 2022-04-06 cs.CL cs.CVcs.LG

Fusing finetuned models for better pretraining

Leshem Choshen , Elad Venezian , Noam Slonim , Yoav Katz This is my paper
classification cs.CL cs.CVcs.LG
keywords fusingmodelsbetterintertrainingmodelpretrainedpretrainingapproach
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Pretrained models are the standard starting point for training. This approach consistently outperforms the use of a random initialization. However, pretraining is a costly endeavour that few can undertake. In this paper, we create better base models at hardly any cost, by fusing multiple existing fine tuned models into one. Specifically, we fuse by averaging the weights of these models. We show that the fused model results surpass the pretrained model ones. We also show that fusing is often better than intertraining. We find that fusing is less dependent on the target task. Furthermore, weight decay nullifies intertraining effects but not those of fusing.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Editing Models with Task Arithmetic

    cs.LG 2022-12 accept novelty 8.0

    Task vectors from weight differences allow arithmetic operations to edit pre-trained models, improving multiple tasks simultaneously and enabling analogical inference on unseen tasks.

  2. DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation

    cs.CV 2026-07 conditional novelty 6.0

    A hypernetwork generates per-column merging weights to combine source LoRA modules on CLIP, achieving state-of-the-art few-shot test-time domain adaptation.

  3. PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

    cs.CV 2026-04 unverdicted novelty 6.0

    PivotMerge merges heterogeneous multimodal pre-trained models via shared-space decomposition to filter conflicts and layer-wise weights based on alignment contributions, outperforming baselines on multimodal benchmarks.

  4. PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging

    cs.CV 2026-04 unverdicted novelty 6.0

    PivotMerge merges post-alignment cross-modal projectors from heterogeneous MLLM pre-training via shared-space filtering and layer-wise weights, beating prior merging baselines on multimodal benchmarks.

  5. Analytic Drift Resister for Non-Exemplar Continual Graph Learning

    cs.LG 2026-04 unverdicted novelty 6.0

    ADR achieves theoretically zero-forgetting class-incremental graph learning by combining backpropagation adaptation with ridge-regression-based layer-wise merging of GNN linear transformations.

  6. The Appeal and Reality of Recycling LoRAs with Adaptive Merging

    cs.LG 2026-02 conditional novelty 6.0

    Adaptive merging of recycled LoRAs gives little benefit over training a target-task LoRA, and randomly initialized LoRAs work as well as real ones once the target LoRA is in the pool.