Pith. sign in

REVIEW 8 cited by

Localizing Task Information for Improved Model Merging and Compression

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2405.07813 v1 pith:J5R52NFA submitted 2024-05-13 cs.LG cs.CV

Ke Wang , Nikolaos Dimitriadis , Guillermo Ortiz-Jimenez , François Fleuret , Pascal Frossard This is my paper
classification cs.LGcs.CV
keywords taskmergingmodelapproachesmulti-taskperformancetasksweights
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Model merging and task arithmetic have emerged as promising scalable approaches to merge multiple single-task checkpoints to one multi-task model, but their applicability is reduced by significant performance loss. Previous works have linked these drops to interference in the weight space and erasure of important task-specific features. Instead, in this work we show that the information required to solve each task is still preserved after merging as different tasks mostly use non-overlapping sets of weights. We propose TALL-masks, a method to identify these task supports given a collection of task vectors and show that one can retrieve >99% of the single task accuracy by applying our masks to the multi-task vector, effectively compressing the individual checkpoints. We study the statistics of intersections among constructed masks and reveal the existence of selfish and catastrophic weights, i.e., parameters that are important exclusively to one task and irrelevant to all tasks but detrimental to multi-task fusion. For this reason, we propose Consensus Merging, an algorithm that eliminates such weights and improves the general performance of existing model merging approaches. Our experiments in vision and NLP benchmarks with up to 20 tasks, show that Consensus Merging consistently improves existing approaches. Furthermore, our proposed compression scheme reduces storage from 57Gb to 8.2Gb while retaining 99.7% of original performance.

Discussion (0). Sign in to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Enough is as good as a feast: A Comprehensive Analysis of How Reinforcement Learning Mitigates Task Conflicts in LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    RL-trained LLMs keep most of their skills after weight merging, while SFT-trained LLMs drop about 19% on average, because RL keeps parameter updates smaller and more task-compatible.

  2. PACT: Preserving Anchored Cores in Task-vectors for Model Merging

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    PACT preserves load-bearing wall dimensions from pre-trained weights inside task vectors to reduce conflicts and improve merged model performance.

  3. Dynamic Model Merging Made Slim

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    DiDi-Merging achieves dynamic model merging performance matching or exceeding prior methods while using only 1.24x to 1.4x the parameters of a single fine-tuned model.

  4. DivMerge: A divergence-based model merging method for multi-tasking

    cs.LG 2025-09 conditional novelty 6.0 of 10

    DivMerge learns task-arithmetic merging weights by minimizing Jensen-Shannon divergence between each specialist model and the merged model, improving multi-task performance and scalability.

  5. Multiple quantum spin Hall states and topological current divider in Twisted Bilayer WSe$_2$

    cond-mat.mes-hall 2025-08 unverdicted novelty 6.0 of 10

    In twisted bilayer WSe2, the authors report emergent double and quadruple quantum spin Hall states whose helical channels sit at moire potential maxima, plus a gate-tunable transition and a five-terminal topological c...

  6. SEAT: Sparse Entity-Aware Tuning for Knowledge Adaptation while Preserving Epistemic Abstention

    cs.AI 2025-06 unverdicted novelty 6.0 of 10

    SEAT preserves epistemic abstention in LLMs during knowledge adaptation via sparse tuning and entity-perturbed KL regularization, yielding 18-101% better abstention on unknown queries while retaining near-perfect know...

  7. Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

    cs.AI 2026-07 conditional novelty 5.0 of 10

    After truncating or expanding checkpoints to a shared shape, small-ratio weight averaging slightly improves average benchmark scores over strong Qwen sources, but headline gains are inflated by per-task best-ratio selection.

  8. Robust Zero-Shot Generalization for Open-Vocabulary Action Recognition via Task Arithmetic

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    Merging task vectors extracted from fine-tuned OVAR models yields superior zero-shot generalization in out-of-distribution settings compared to the pre-trained base model.

Pith tools