Pith. sign in

REVIEW 5 major objections 6 minor 77 references

Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that pruning low-diversity task-vector layers and replacing them with pretrained weights improves out-of-distribution generalization of merged models while preserving in-domain accuracy.

desk verdict Useful plug-in for OOD-aware model merging with consistent H-score gains, but the ID-preservation claim is overstated and the theory is mostly borrowed. read the letter →

arxiv 2506.09093 v2 pith:FRSD7SFB submitted 2025-06-10 cs.LG cs.AI

Bingjie Zhang , Hongkang Li , Changlong Shi , Guowei Rong , He Zhao , Dongsheng Wang , Dandan Guo , Meng Wang This is my paper
classification cs.LGcs.AI
keywords modelmergingtaskvectorslayer-wisepruningout-of-distributiongeneralizationmulti-tasklearningsaliencyscoreCLIPvisiontransformersplug-and-play
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Model merging combines several fine-tuned models into one multi-task model, but existing methods are tuned for in-domain accuracy and often lose the pretrained model's out-of-distribution robustness. This paper claims that the lost robustness can be recovered by pruning task vectors layer-wise: layers whose updates are nearly the same across all tasks are treated as redundant and replaced with the original pretrained weights. The proposed LwPTV computes a per-layer saliency score from the deviation of each task's task vector from the cross-task mean, masks low-diversity layers, ORs the per-task masks so a layer is kept if any task needs it, and plugs the resulting mask into existing merging formulas. If correct, this gives a training-free, plug-and-play way to improve OOD generalization of Task Arithmetic, Ties-Merging, AdaMerging, and PCB-MERGING on CLIP and T5 models while largely preserving in-domain accuracy, as measured by the H-score, the harmonic mean of average ID and OOD accuracy.

What carries the argument

The central object is the layer-wise saliency score $s_l^k = \mathbb{E}[|\tau_l^k - \tfrac{1}{K}\sum_{j=1}^K \tau_l^j|]$, which measures how far one task's update at a given layer deviates from the average update across tasks. A high score is interpreted as evidence that the layer carries task-specific discriminative information; a low score means the layer is redundant, so the merged model keeps the pretrained weights there instead of the fine-tuned delta. The per-task masks are constructed by thresholding these scores, and the OR-aggregated shared mask ensures that a layer is removed only if all tasks agree to remove it. The theoretical motivation is Proposition 1, which the paper derives from a cited one-layer Transformer analysis and uses to argue that discriminative neurons vary more across tasks than non-discriminative neurons, making diversity a valid pruning criterion.

What would settle it

Compute each layer's saliency score on the eight benchmark tasks and also measure the drop in ID accuracy caused by forcing that layer to the pretrained weights; if any low-saliency layer shows a positive accuracy contribution, low cross-task diversity does not imply redundancy. The result would be strongest on a pair of tasks known to share a transferable visual feature, such as two datasets with similar textures, where pruning the shared low-diversity layer should preserve ID accuracy if the paper's claim is correct.

Watch

Extended reading notes

Core claim

The paper's central claim is that task-vector parameters that are consistently close to the cross-task average are redundant: they do not encode task-discriminative features, so substituting the pretrained weights in those layers improves OOD generalization without hurting in-domain performance. To realize this, LwPTV defines a layer-wise saliency score for task $k$ and layer $l$ as $s_l^k = \mathbb{E}[|\tau_l^k - \tfrac{1}{K}\sum_{j=1}^K \tau_l^j|]$, where $\tau_k = \theta_k - \theta_{\mathrm{pre}}$ is the task vector. Layers with scores below a per-task threshold are masked out, and a shared mask $\hat{m} = m^1 \lor \cdots \lor m^K$ is formed by OR-ing the per-task masks, so a layer is pruned in the merged model only when every task agrees it is redundant. The merged model is then $\theta_m = \theta_{\mathrm{pre}} + \hat{m} \odot \sum_k \lambda_k \tau_k$, with an additional coefficient scaling when combined with layer-wise AdaMerging. Across ViT-B/32, ViT-L/14, ViT-H/14, and T5-large, the paper reports H-score gains over Task Arithmetic, Ties, AdaMerging variants, and PCB-MERGING, with ID accuracy preserved or improved in most image experiments and some acknowledged ID loss for LW AdaMerging on ViT-B/32.

Load-bearing premise

The load-bearing premise is that a layer whose task-vector parameters are close to the average across tasks carries no task-specific information, so replacing that layer with the pretrained weights cannot hurt in-domain performance.

Editorial extensions

If this is right

  • On CLIP ViT-B/32, Task Arithmetic with LwPTV raises the H-score from 58.9 to 66.3 by improving average OOD accuracy from 51.3 to 60.9 while also raising average ID accuracy from 69.1 to 72.8.
  • On the same architecture, LwPTV combined with LW AdaMerging raises the H-score from 67.1 to 68.6, lifting average OOD accuracy from 57.7 to 61.7 at the cost of a 2.8-point drop in ID accuracy from 80.1 to 77.3.
  • On ViT-L/14, LwPTV improves H-scores for Task Arithmetic, Ties, TW AdaMerging, LW AdaMerging, and PCB-MERGING by between 0.7 and 2.1 points.
  • On T5-large, LwPTV raises the H-score of Task Arithmetic from 54.9 to 57.9, including a 3.0-point gain in average ID accuracy.
  • The method is training-free: mask computation costs $O(KL d_l)$ time, the ViT-B/32 merge adds under a second, and storage for Task Arithmetic drops from 432.77 MB to 226.75 MB on ViT-B/32.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit: the saliency criterion predicts that tasks with highly correlated fine-tuning updates will benefit most from pruning, so one could test this by merging deliberately related tasks and checking whether the OOD gain grows while the ID loss stays flat.
  • The theoretical support is proven for one-layer Transformers on binary classification; extending Proposition 1 to deep ViTs and multi-class tasks is an assumption, and a per-layer ablation of ID accuracy against saliency ordering would show whether the extension holds.
  • The NLP results in the appendix suggest the direction of the method may be domain-dependent: in language, where merged models already beat the pretrained model on OOD benchmarks, the paper switches to parameter-level pruning, implying that layer-wise pruning is a tool for rebalancing pretrained versus task-specific knowledge rather than a universal law.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes LwPTV (Layer-wise Pruning Task Vector), a plug-in preprocessing step for task-vector-based model merging. For each task, LwPTV computes a layer-wise saliency score (mean absolute deviation of the task-vector layer from the across-task mean), thresholds the scores to build a per-task layer mask, OR-aggregates the masks into a shared mask, and forms the merged model as θ_m = θ_pre + m_hat ⊙ Σ_k λ_k τ_k. The authors claim this improves out-of-domain (OOD) accuracy of existing merging methods while preserving in-domain (ID) accuracy, and support the claim with experiments on CLIP ViT-B/32, ViT-L/14, ViT-H/14 and T5-large, across Task Arithmetic, Ties-Merging, AdaMerging variants, and PCB-Merging.

Significance. If the empirical claim holds, LwPTV would be a simple, training-free, and broadly applicable plug-in that improves OOD generalization of task-vector merging, an underexplored axis in the model-merging literature. The paper's strengths are the breadth of the evaluation (multiple architectures, image and text tasks, seven merging baselines), the storage savings reported in Table 15, and the clear algorithmic presentation in Algorithm 1. The significance is tempered, however, by three issues: the headline ID-preservation claim is contradicted by the authors' own Table 2, the theoretical motivation does not bridge from neuron-level diversity to layer-level pruning, and the key hyperparameter η=0.7 is selected on the same OOD benchmarks used to report the gains. These issues do not invalidate the direction of the method but require substantial revision before the claims can be accepted.

major comments (5)
  1. [Abstract; §5.1, Table 2; §D.1] The central claim that LwPTV preserves ID performance is contradicted by the paper's own results. On ViT-B/32, LW AdaMerging drops from 80.1 to 77.3 ID average and LW AdaMerging++ drops from 81.1 to 77.1; on ViT-L/14, LW AdaMerging drops from 90.8 to 89.8. Section D.1 concedes that ID performance 'may be compromised' with certain merging techniques. The abstract and conclusion should be reworded to describe an ID/OOD trade-off rather than preservation, or the claims should be restricted to the methods and configurations where ID is actually preserved.
  2. [§4.1, Eq. (5), Proposition 1] Proposition 1, inherited from the one-layer binary-classification analysis in [38], establishes a neuron-level diversity gap between discriminative and non-discriminative neurons. The proposed method, however, prunes entire layers based on the scalar s_l^k, a layer-level average absolute deviation from the mean task vector. A layer can have a small mean deviation and still contain discriminative neurons, and a layer with large mean deviation can still contain redundant parameters; no argument bridges the two levels. The proof of Proposition 1 also contains inconsistent bounds: Eq. (10) states DV ≥ |S|·Ω(m^{-1}) while the following sentence says the diversity is 'in the order of |S|·Ω(m^{-1/2})'. This theoretical motivation therefore does not support the layer-wise redundancy assumption behind Eqs. (5)-(7).
  3. [§5, §5.2, Fig. 5] The pruning ratio η=0.7 is selected on the same OOD benchmarks that produce the headline results. Section 5.2 sweeps η in Fig. 5 and states that 'a pruning ratio η between 0.6 and 0.8 offers a relatively desired trade-off,' after which η=0.7 is used for all experiments. No validation split or selection rule is given. Because η directly controls how far the merged model moves toward the pretrained model, the reported OOD gains are partly fitted to the test distribution; the paper should report results for a prespecified η or a selection procedure that does not use OOD labels.
  4. [Table 2] The main results are reported without error bars or multiple seeds. Some claimed improvements are small (e.g., +0.7 H-score for LW AdaMerging++ on ViT-L/14 and +0.4 H-score for PCB on T5-large), so without variance estimates it is difficult to verify the consistency of the gains on which the H-score claims rest. The manuscript should report standard deviations across at least three seeds for the main tables, or at least for the smallest increments.
  5. [§B.16, Table 17] For the NLP experiments the method is changed from layer-wise pruning to parameter-level pruning ('for NLP tasks, we perform parameter pruning at the parameter level'), but no saliency score or mask construction for parameter-level pruning is defined. The T5-large results therefore do not evaluate the method introduced in Section 4, and the claimed generality across modalities is not substantiated by the reported experiments.
minor comments (6)
  1. [Eq. (5)] The summation index k is reused as the outer task index; the mean should be over a different index, e.g., Σ_j τ_l^j.
  2. [Eq. (6)] The threshold sorted(s_k)[⌊L·η⌋] is undefined for η=0 and for ties among saliency scores; the convention should be stated.
  3. [Table 2] Table 2 has formatting errors (e.g., '55.89.075.4' and '60.660.9+9.6'); the table needs to be regenerated so that all entries are legible.
  4. [Section 5] There are several typos: 'repsectively', 'Surgey', 'purning', and 'wether' should be corrected.
  5. [Table 1] LwPTV is marked 'No Test set', but when combined with AdaMerging variants the algorithm uses entropy minimization on unlabeled test data; the table should clarify that the 'No Test' property applies only to the standalone mask construction.
  6. [§B.1] The scaling rule λ̂_l^k = η·λ_l^k reuses η, the pruning ratio, as a coefficient-scaling factor without discussion; this conflation should be motivated or renamed.

Circularity Check

0 steps flagged · score 2.0 of 10

No definitional circularity: LwPTV's mask is computed from task vectors alone, and the cited theoretical support is external despite overlapping authors; minor caveats are η selection on the reported benchmark and the paper's own ID-drop concession.

full rationale

The derivation chain is not circular by construction. The saliency score in Eq. (5) is computed from task vectors τ_k and their layer-wise mean, with no OOD labels or OOD performance entering the mask equations (6)-(7); the merged model in Eq. (8) is a deterministic function of θ_pre, τ_k, λ_k, and the OR-aggregated mask. Proposition 1 is imported from [38], a prior theoretical paper co-authored by two of the present authors (Li and Wang), and its proof in Appendix A.1 is explicitly 'By Corollary 2 and Lemma 5 of [38]'; however, [38] is a theorem with its own stated assumptions (one-layer transformer, binary classification) that does not assume LwPTV's layer-level pruning claim, so this is legitimate self-citation/evidence rather than a definitional reduction. The unsupported step from neuron-level diversity to layer-level mean-absolute deviation is a correctness risk, not circularity. The pruning ratio η=0.7 is selected on the reported ID/OOD trade-off (Section 5.2), so part of the OOD gain is selection-fitted rather than a pure out-of-sample prediction; this is an evaluation caveat. Finally, Section D.1 admits 'it may compromise ID performance when employed with certain model merging techniques,' and Table 2 shows ID drops (e.g., LW AdaMerging 80.1→77.3 on ViT-B/32), which conflicts with the abstract's 'preserving the ability on ID tasks' claim, but inconsistency is a correctness issue, not circularity. No equation in the paper reduces to its own input by construction.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The paper's central cost is one free hyperparameter (eta=0.7) selected on the evaluation OOD benchmarks, plus an unproven transfer of a one-layer theory to deep models. No new entities are postulated.

free parameters (1)
  • layer pruning ratio eta = 0.7
    Selected from Fig. 5 trade-off curves on the same ViT-B/32 and ViT-L/14 OOD benchmarks later reported in Table 2; not chosen on a hold-out validation set. Table 1 claims 'No Test set' for LwPTV, but eta is effectively tuned on the test distribution.
assumptions (4)
  • ad hoc to paper Low cross-task diversity of a task-vector layer implies redundancy and safety of replacing it with pretrained weights.
    Core heuristic behind Eqs. (5)-(6); Proposition 1 supports it only for one-layer binary Transformers and depends on the self-cited result [38].
  • domain assumption The one-layer single-head Transformer analysis from [38] (Corollary 2, Lemma 5) transfers to deep ViT-B/L/H and T5 in multi-class settings.
    Proposition 1 is proved only in the simplified setting of Appendix A.1; no derivation is given for deep models, multiple layers, or multi-class tasks.
  • domain assumption The 13 OOD datasets (CIFAR10/100, ImageNet, etc.) are appropriate proxies for distribution shift in model merging.
    Standard OOD suites borrowed from [63, 74]; no independent validation protocol is used to select eta.
  • domain assumption All task models share the same architecture and initialization from the same pretrained checkpoint.
    Task vectors are only meaningful under shared initialization; the paper acknowledges this as a limitation in Section D.1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data." pith.science (2026). https://pith.science/paper/FRSD7SFB

@misc{pith2026250609093,
  author       = {Pith},
  title        = {Pith review of: Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FRSD7SFB}},
  note         = {Machine review of arXiv:2506.09093}
}
read the original abstract

Multi-task learning (MTL) concurrently trains a model on diverse task datasets to exploit common features, thereby improving overall performance across the tasks. Recent studies have dedicated efforts to merging multiple independent model parameters into a unified model for MTL, thus circumventing the need for training data and expanding the scope of applicable scenarios of MTL. However, current approaches to model merging predominantly concentrate on enhancing performance within in-domain (ID) datasets, often overlooking their efficacy on out-of-domain (OOD) datasets. In this work, we proposed LwPTV (Layer-wise Pruning Task Vector) by building a saliency score, measuring the redundancy of parameters in task vectors. Designed in this way ours can achieve mask vector for each task and thus perform layer-wise pruning on the task vectors, only keeping the pre-trained model parameters at the corresponding layer in merged model. Owing to its flexibility, our method can be seamlessly integrated with most of existing model merging methods to improve their performance on OOD tasks. Extensive experiments demonstrate that the application of our method results in substantial enhancements in OOD performance while preserving the ability on ID tasks.

Figures

Figures reproduced from arXiv: 2506.09093 by the authors.

Figure 1
Figure 1. ID and OOD performance of model merging methods on ViT-B/32; see ID and OOD tasks in Ex￾periments. Triangle and circle with same color mean baseline and baseline+ ours. In this work, we investigate the feasibility of pruning parameters of task vectors and only keeping the corresponding parameters in pre￾trained models to enhance the generalization performance on OOD data of merged mod￾els. The central challenge lies… view at source ↗
Figure 2
Figure 2. Illustration of our LwPTV framework: (a) Obtaining the Layer Mask; (b) Merging Task Vectors. Each row of the task vector block represents the layer-wise parameter. A white color denotes a value of 0, whereas a black color signifies a value of 1. The layer-wise salience score of each task vector is calculated, and a threshold operation is performed to generate a mask. The task vector is then layer-pruned through the … view at source ↗
Figure 5
Figure 5. Performance of Task Arithmetic+LwPTV with varying η. Effect of our components. We conduct ablation studies based on Task Arithmetic and LW AdaMerg￾ing with ViT-B/32. LwPTV includes three components: salience score, OR operation (∨), and coefficient scaling (scale) for LW AdaMerging, which learns the layer-wise merging weights. To systematically investigate the contribution of each individual component, we remove eac… view at source ↗
Figures from the paper (7 more)
Figure 7
Figure 7. Figure 7: T-SNE visualizations of (a) Task Arithmetic, and (b) Task Arithmetic w/ LwPTV on CIFAR10. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: The variations in λ l k of unpruned layers subsequent to the application of the LW AdaMerging and LW AdaMerging w/ LwPTV. B.2 Algorithm For Task Arithmetic, Ties, TW AdaMerging, TW AdaMerging++, LW AdaMerging, LW AdaMerg￾ing++, and PCB-MERGING, we present the algorithm…
Figure 9
Figure 9. Figure 9: The salience score of the task vectors with differ￾ent component on ViT-B/32. To investigate whether our method primarily prunes task vectors specific to certain struc￾tures, such as parameters in MLP, whose pre￾trained parameters inherently have stronger generalizatio…
Figure 10
Figure 10. Figure 10: Visualization of ℓ2 distance between merged model representations and pre-trained model representations. B.13 Storage overhead LwPTV can effectively diminish the storage requirements for task vectors for each merging method. As shown in Tab.15, we report the storage c…
Figure 11
Figure 11. Figure 11: T-SNE visualizations on STL10 comparing Task Arithmetic without and with LwPTV . [PITH_FULL_IMAGE:figures/full_fig_p026_11.png]
Figure 12
Figure 12. Figure 12: Visualizing of (a) salience score matrix, (b) mask vector for each task, and (c) final mask vector, all on VIT-L/14, where x-axis denotes the layer index, y-axis in (a-b) denotes task name. 50 1 00 1 50 200 250 300 3 50 SU. Ca. RE. Eu. SV. GT. MN. DT. T a s k 0. 0E+0 …
Figure 13
Figure 13. Figure 13: Visualizing of (a) salience score matrix, (b) mask vector for each task, and (c) final mask vector, all on VIT-H/14, where x-axis denotes the layer index, y-axis in (a-b) denotes task name. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

77 extracted references · 66 canonical work pages

  1. [38]

    When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers

    Hongkang Li, Yihua Zhang, Shuai Zhang, Pin-Yu Chen, Sijia Liu, and Meng Wang. When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers. InProceedings of the International Conference on Learning Representations (ICLR 2025), 2025

    work page 2025

  2. [1]

    Git re-basin: Merging models modulo permutation symmetries

    Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. InProceedings of the International Conference on Learning Representations (ICLR 2023), 2023

    work page 2023

  3. [2]

    Merging multi- task models via weight-ensembling mixture of experts

    Tang Anke, Shen Li, Luo Yong, Yin Nan, Zhang Lefei, and Tao Dacheng. Merging multi- task models via weight-ensembling mixture of experts. InProceedings of the International Conference on Machine Learning (ICML 2024), 2024

    work page 2024

  4. [3]

    Types of out-of-distribution texts and how to detect them

    Udit Arora, William Huang, and He He. Types of out-of-distribution texts and how to detect them. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), pages 10687–10701, 2021

    work page 2021

  5. [4]

    On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models.arXiv preprint arXiv:2108.07258, 2021

    arXiv 2021

  6. [5]

    Food-101–mining discriminative components with random forests

    Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101–mining discriminative components with random forests. InProceedings of the European Conference on Computer Vision (ECCV 2014), pages 446–461, 2014

    work page 2014

  7. [6]

    Swad: Domain generalization by seeking flat minima

    Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. InProceedings of the Conference on Neural Information Processing Systems (NeurIPS 2021), pages 22405–22418, 2021

    work page 2021

  8. [7]

    Fine-tuning deteriorates general textual out-of-distribution detection by distorting task-agnostic features

    Sishuo Chen, Wenkai Yang, Xiaohan Bi, and Xu Sun. Fine-tuning deteriorates general textual out-of-distribution detection by distorting task-agnostic features. InFindings of the Association for Computational Linguistics: EACL 2023, pages 564–579, 2023

    work page 2023

Show all 77 references
  1. [8]

    The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy

    Tianlong Chen, Zhenyu Zhang, Yu Cheng, Ahmed Awadallah, and Zhangyang Wang. The principle of diversity: Training stronger vision transformers calls for reducing all levels of redundancy. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR ...

    2022

  2. [9]

    Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017

    2017

  3. [10]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2014), pages 3606–3613, 2014

    2014

  4. [11]

    Deep learning for classical japanese literature.arXiv preprint arXiv:1812.01718, 2018

    Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature.arXiv preprint arXiv:1812.01718, 2018. 10

    2018 arXiv

  5. [12]

    An analysis of single-layer networks in unsuper- vised feature learning

    Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. InProceedings of the International conference on artificial intelligence and statistics (AISTATS 2011), pages 215–223, 2011

    2011

  6. [13]

    Emnist: Extending mnist to handwritten letters

    Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. InProceedings of the International Joint Conference on Neural Networks (IJCNN 2017), pages 2921–2926, 2017

    2017

  7. [14]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2009), pages 248–255, 2009

    2009

  8. [15]

    The mnist database of handwritten digit images for machine learning research [best of the web].IEEE signal processing magazine, 29(6):141–142, 2012

    Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web].IEEE signal processing magazine, 29(6):141–142, 2012

    2012

  9. [16]

    Merging by matching models in task parameter subspaces.Transactions on Machine Learning Research, 2024

    Tam Derek, Bansal Mohit, and Raffel Colin. Merging by matching models in task parameter subspaces.Transactions on Machine Learning Research, 2024

    2024

  10. [17]

    Qlora: Efficient fine- tuning of quantized llms

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient fine- tuning of quantized llms. InProceedings of the Conference on Neural Information Processing Systems (NeurIPS 2023), 2023

    2023

  11. [18]

    Revisiting large language model pruning using neuron semantic attribution.arXiv e-prints, pages arXiv–2503, 2025

    Yizhuo Ding, Xinwei Sun, Yanwei Fu, and Guosheng Hu. Revisiting large language model pruning using neuron semantic attribution.arXiv e-prints, pages arXiv–2503, 2025

    2025

  12. [19]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. InInternational Conf...

    2020

  13. [20]

    Parameter competition balancing for model merging

    Guodong Du, Junlin Lee, Jing Li, Runhua Jiang, Yifei Guo, Shuyang Yu, Hanting Liu, Sim Kuan Goh, Ho-Kin Tang, Daojing He, et al. Parameter competition balancing for model merging. InProceedings of the Conference on Neural Information Processing Systems (NeurIPS 2024), 2024

    2024

  14. [21]

    Memorization vs

    Aparna Elangovan, Jiayuan He, and Karin Verspoor. Memorization vs. generalization: Quanti- fying data leakage in nlp performance evaluation. InProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL 2021), ...

    2021

  15. [22]

    Efficiently identifying task groupings for multi-task learning

    Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. InProceedings of the Conference on Neural Information Processing Systems (NeurIPS 2021), pages 27503–27516, 2021

    2021

  16. [23]

    Challenges in representation learning: A report on three machine learning contests

    Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine learning contests. InProceedings of the Internation...

    2013

  17. [24]

    Stochastic weight averaging in parallel: Large-batch training that generalizes well

    Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. InProceedings of the International Conference on Learning Representations (ICLR 2020), 2020

    2020

  18. [25]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019

    2019

  19. [26]

    Multi-dimensional model compression of vision transformer

    Zejiang Hou and Sun-Yuan Kung. Multi-dimensional model compression of vision transformer. InProceedings of the IEEE International Conference on Multimedia and Expo (ICME 2022), pages 01–06, 2022. 11

    2022

  20. [27]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhad. Editing models with task arithmetic. InProceedings of the International Conference on Learning Representations (ICLR 2023), 2023

    2023

  21. [28]

    Model stock: All we need is just a few fine-tuned models

    Dong-Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All we need is just a few fine-tuned models. InProceedings of the European Conference on Computer Vision (ECCV 2024), pages 207–223, 2024

    2024

  22. [29]

    Jiarui Jiang, Wei Huang, Miao Zhang, Taiji Suzuki, and Liqiang Nie. Unveil benign overfitting for transformer in vision: Training dynamics, convergence, and generalization.Proceedings of the Conference on Neural Information Processing Systems (NeurIPS 2024), pages 135464– 135625, 2024

    2024

  23. [30]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. InProceedings of the International Conference on Learning Representations (ICLR 2023), 2023

    2023

  24. [31]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Languag...

    2019

  25. [32]

    3d object representations for fine- grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. InProceedings of the IEEE international conference on computer vision workshops (ICCVW 2013), pages 554–561, 2013

    2013

  26. [33]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009

    2009

  27. [34]

    Fine-tuning can distort pretrained features and underperform out-of-distribution

    Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. InProceedings of the International Conference on Learning Representations (ICLR 2022), 2022

    2022

  28. [35]

    Question and answer test-train overlap in open-domain question answering datasets

    Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. Question and answer test-train overlap in open-domain question answering datasets. InProceedings of the Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (EACL 2021), pages 10...

    2021

  29. [36]

    A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity

    Hongkang Li, Meng Wang, Sijia Liu, and Chen Pin-Yu. A theoretical understanding of shallow vision transformers: Learning, generalization, and sample complexity. InProceedings of the International Conference on Learning Representations (ICLR 2023), 2023

    2023

  30. [37]

    Hongkang Li, Meng Wang, Songtao Lu, Xiaodong Cui, and Pin-Yu Chen. How do nonlinear transformers learn and generalize in in-context learning? InProceedings of the International Conference on Learning Representations (ICLR 2024), pages 28734–28783, 2024

    2024

  31. [39]

    Clip-event: Connecting text and images with event structures

    Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. Clip-event: Connecting text and images with event structures. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 164...

    2022

  32. [40]

    Local to global: Learning dynamics and effect of initialization for transformers

    Ashok Vardhan Makkuva, Marco Bondaschi, Adway Girish, Alliot Nagle, Hyeji Kim, Michael Gastpar, and Chanakya Ekbote. Local to global: Learning dynamics and effect of initialization for transformers. InProceedings of the Conference on Neural Information Processing Systems (Neur...

    2024

  33. [41]

    Matena and Colin A

    Michael S. Matena and Colin A. Raffel. Merging models with fisher-weighted averaging. In Proceedings of the International Conference on Learning Representations (ICLR 2023), pages 17703–17716, 2022. 12

    2023

  34. [42]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. InNIPS workshop on deep learning and unsupervised feature learning, page 4, 2011

    2011

  35. [43]

    Saft: Towards out-of-distribution generalization in fine-tuning

    Bac Nguyen, Stefan Uhlich, Fabien Cardinaux, Lukas Mauch, Marzieh Edraki, and Aaron Courville. Saft: Towards out-of-distribution generalization in fine-tuning. InEuropean Confer- ence on Computer Vision, pages 138–154, 2024

    2024

  36. [44]

    Automated flower classification over a large number of classes

    Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. InProceedings of the Indian Conference on Computer Vision, Graphics & Image Processing (ICCVGIP 2008), pages 722–729, 2008

    2008

  37. [45]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2012), pages 3498–3505, 2012

    2012

  38. [46]

    What to pre-train on? efficient intermediate task selection

    Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. What to pre-train on? efficient intermediate task selection. InProceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP 2021), pages 10585–10605, 2021

    2021

  39. [47]

    What to pre-train on? efficient intermediate task selection.Empirical Methods in Natural Language Processing,Empirical Methods in Natural Language Processing, 2021

    Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. What to pre-train on? efficient intermediate task selection.Empirical Methods in Natural Language Processing,Empirical Methods in Natural Language Processing, 2021

    2021

  40. [48]

    Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R Bowman. Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work? In Proceed...

    2020

  41. [49]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  42. [50]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila an...

    2021

  43. [51]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InProceedings of the International Conference on Ma...

    2021

  44. [52]

    On the effect of dropping layers of pre-trained transformer models.Computer Speech and Language, 77, 2023

    Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. On the effect of dropping layers of pre-trained transformer models.Computer Speech and Language, 77, 2023

    2023

  45. [53]

    Movement pruning: Adaptive sparsity by fine-tuning

    Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. 2020

    2020

  46. [54]

    Multitask prompted training enables zero-shot task generalization

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. InProceedings of the International Conference on Lea...

    2022

  47. [55]

    Label sleuth: From unlabeled text to a classifier in a few hours

    Eyal Shnarch, Alon Halfon, Ariel Gera, Marina Danilevsky, Yannis Katsis, Leshem Choshen, Martin Santillan Cooper, Dina Epelboim, Zheng Zhang, Dakuo Wang, et al. Label sleuth: From unlabeled text to a classifier in a few hours. InProceedings of the Conference on Empirical Metho...

    2022

  48. [56]

    Lora vs full fine-tuning: An illusion of equivalence.arXiv e-prints, pages arXiv–2410, 2024

    Reece Shuttleworth, Jacob Andreas, Antonio Torralba, and Pratyusha Sharma. Lora vs full fine-tuning: An illusion of equivalence.arXiv e-prints, pages arXiv–2410, 2024. 13

    2024

  49. [57]

    Model fusion via optimal transport

    Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. InProceedings of the Conference on Neural Information Processing Systems (NeurIPS 2020), pages 22045–22055, 2020

    2020

  50. [58]

    Recursive deep models for semantic compositionality over a sentiment treebank

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProceedings of the Conference on Empirical Methods in Natural Language Processing ...

    2013

  51. [59]

    The german traffic sign recognition benchmark: a multi-class classification competition

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. InProceedings of the International Joint Conference on Neural Networks (IJCNN 2011), pages 1453–1460, 2011

    2011

  52. [60]

    Rotation equivariant cnns for digital pathology

    Bastiaan S.Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equivariant cnns for digital pathology. InProceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI 2018), pages 210– 218, 2018

    2018

  53. [61]

    Multi-task learning for dense prediction tasks: A survey.IEEE transactions on pattern analysis and machine intelligence, 44(7):3614–3633, 2021

    Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool. Multi-task learning for dense prediction tasks: A survey.IEEE transactions on pattern analysis and machine intelligence, 44(7):3614–3633, 2021

    2021

  54. [62]

    Lines: Post-training layer scaling prevents forgetting and enhances model merging.arXiv preprint arXiv:2410.17146, 2024

    Ke Wang, Nikolaos Dimitriadis, Alessandro Favero, Guillermo Ortiz-Jimenez, Francois Fleuret, and Pascal Frossard. Lines: Post-training layer scaling prevents forgetting and enhances model merging.arXiv preprint arXiv:2410.17146, 2024

    arXiv

  55. [63]

    Localizing task information for improved model merging and compression

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, François Fleuret, and Pascal Frossard. Localizing task information for improved model merging and compression. InProceedings of the International Conference on Machine Learning (ICML 2024), 2024

    2024

  56. [64]

    When to use multi-task learning vs intermediate fine-tuning for pre-trained encoder transfer learning

    Orion Weller, Kevin Seppi, and Matt Gardner. When to use multi-task learning vs intermediate fine-tuning for pre-trained encoder transfer learning. InProceedings of the Annual Meeting of the Association for Computational Linguistics (ACL 2022), 2022

    2022

  57. [65]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  58. [66]

    Robust fine-tuning of zero-shot models

    Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. InProceedings of the IEEE/CVF conference on computer vision ...

    2022

  59. [67]

    Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747, 2017

    Han Xiao, Kashif Rasul, and Roland V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.arXiv preprint arXiv:1708.07747, 2017

    2017 arXiv

  60. [68]

    Sun database: Exploring a large collection of scene categories.International Journal of Computer Vision, 119: 3–22, 2016

    Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories.International Journal of Computer Vision, 119: 3–22, 2016

    2016

  61. [69]

    Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interference when merging models. InProceedings of the Conference on Neural Information Processing Systems (NeurIPS 2023), 2023

    2023

  62. [70]

    Representation surgery for multi-task model merging

    Enneng Yang, Li Shen, Zhenyi Wang, Guibing Guo, Xiaojun Chen, Xingwei Wang, and Dacheng Tao. Representation surgery for multi-task model merging. InProceedings of the International Conference on Machine Learning (ICML 2024), 2024

    2024

  63. [71]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. InProceedings of the International Conference on Learning Representations (ICLR 2024), 2024. 14

    2024

  64. [72]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InProceedings of the International Conference on Machine Learning (ICML 2024), 2024

    2024

  65. [73]

    Uni3d: A unified baseline for multi-dataset 3d object detection

    Bo Zhang, Jiakang Yuan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Uni3d: A unified baseline for multi-dataset 3d object detection. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2023), pages 9253–9262, 2023

    2023

  66. [74]

    Learning to prompt for vision-language models.International Journal of Computer Vision, 130(9):2337–2348, 2022

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models.International Journal of Computer Vision, 130(9):2337–2348, 2022

    2022

  67. [75]

    Model tailor: Mitigating catastrophic forgetting in multi-modal large language models

    Didi Zhu, Zhongyisun Sun, Zexi Li, Tao Shen, Ke Yan, Shouhong Ding, Chao Wu, and Kun Kuang. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. InProceedings of the International Conference on Machine Learning (ICML 2024), 2024

    2024

  68. [76]

    Limitations

    Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning.Proceedings of the IEEE, 109: 43–76, 2020. 15 A Theoretical analyses A.1 Proof of proposition 1 We start by introducing the theoretic...

    2020

  69. [77]

    • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.