Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

Less is More: Efficient Model Merging with Binary Task Switch

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Binarizing task vectors matches full-precision merging while cutting storage to 1-3%.

desk verdict A genuinely useful binary task-vector merging method with real storage savings, but the paper's headline 'pulse-like' mechanistic claim is not yet supported because the key control is confounded with norm shrinkage. read the letter →

arxiv 2412.00054 v1 pith:QRTUPOXF submitted 2024-11-24 cs.LG

Biqing Qi , Fangyuan Li , Zhen Wang , Junqi Gao , Dong Li , Peng Ye , Bowen Zhou This is my paper · ORCID
classification cs.LG
keywords modelmergingtaskvectorsbinarizationparameterpruningmulti-tasklearningstorageefficiencydynamicpulseactivation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that task vectors, the parameter differences between a fine-tuned model and its pretrained base, are dominated by a small set of large-magnitude entries and that the rest are noise. It proposes discarding those small entries, binarizing the survivors to their signs, and rescaling by a single scalar, which it calls a task switch. The claim is that this binary approximation keeps or improves both single-task fine-tuning and multi-task merging accuracy, while cutting per-task storage to 1-3% of full precision. If true, multi-task deployment could store many specialized models as a few bits per parameter and combine them at inference without retraining.

What carries the argument

The load-bearing object is the pulse activation function $g_m(\tau)$, which keeps task-vector entries outside symmetric upper and lower quantile thresholds and zeros the rest. P-Discard applies this mask; Bin-Discard then replaces surviving entries with their signs $g_b(\tau)$ and rescales by the ratio of the $\ell^2$ norms of the masked full-precision vector and the masked sign vector. The product is a task switch with three parts: an activation mask $S_A$, a polarity sign $S_P$, and a scalar knob $\lambda$, combined dynamically at inference by Eq. 7 (or Eq. 8 for Auto-Switch). The mechanism works because the paper's experiments find that low-magnitude parameters are redundant noise whose removal reduces inter-task conflict.

What would settle it

Find one task vector where systematically discarding the smallest-magnitude entries, say at alpha = 0.5, produces a clear accuracy drop relative to the full task vector, or where the optimal alpha for a single layer is far outside the paper's global optimum. Alternatively, construct two task vectors whose low-magnitude entries are essential for one task, and show that binarizing both degrades merged accuracy below full-precision merging.

Watch

Extended reading notes

Core claim

The central discovery is that task vectors exhibit a pulse-like characteristic: parameters whose magnitude exceeds a per-task quantile threshold carry the useful signal, while parameters below that threshold actively hurt fine-tuning and merging performance. The paper demonstrates through controlled experiments that discarding the low-magnitude entries (P-Discard) improves average accuracy, and that replacing the surviving values with their signs times a scalar ratio (Bin-Discard) retains nearly all of that gain. From this it builds Task Switch (T-Switch), which stores each task as a binary activation mask, a binary sign vector, and one scalar knob, and Auto-Switch, which retrieves and combines these switches at inference from a small query set. The paper reports that T-Switch and Auto-Switch outperform prior merging methods on eight vision and eight language tasks while using 1-3% of the storage of full-precision task vectors.

Load-bearing premise

The load-bearing premise is that a single global discard ratio alpha, applied as symmetric per-task quantile thresholds, cleanly separates informative from noisy coordinates in every layer of every task vector; if the informative set is task-, layer-, or data-dependent, the switch discards the wrong parameters and the storage-accuracy trade-off collapses.

Editorial extensions

If this is right

  • If the pulse assumption holds, any model-merging pipeline can store each task as roughly 1-3% of its original parameter bits, consisting of a mask, signs, and one scalar per task.
  • Discarding small-magnitude entries can push merged accuracy above the average of the individually fine-tuned models, as observed in the vision experiments.
  • The same binarization applies to LoRA-style low-rank task vectors, suggesting that parameter-efficient fine-tuning results can be stored and merged this way too.
  • Auto-Switch shows that task selection can be performed without a learned router, using nearest-neighbor retrieval on a small label-free query set.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The pulse claim is about coordinate-wise magnitude, not task geometry; a testable extension is whether the optimal discard threshold varies by layer or task, and whether an adaptive per-layer alpha would outperform a single global ratio.
  • If the sign-and-scalar approximation preserves enough signal, it suggests that fine-tuning knowledge is largely directional: what matters is which parameters increase or decrease and by roughly how much, not the precise magnitudes.
  • A natural stress test is to merge tasks with heavily overlapping or adversarial task vectors, where the assumption that low-magnitude parameters are pure noise may break down.
  • The storage accounting (1-3%) assumes the mask and signs are stored efficiently; the paper reports example sizes but a reader should verify whether the mask is stored uncompressed or bit-packed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper claims that task vectors exhibit a pulse-like characteristic: only parameters whose magnitude exceeds a per-task quantile threshold contribute positively, and discarding the rest while binarizing the survivors to their signs times a scalar preserves or even improves fine-tuning and merging performance. This observation motivates T-Switch, a decomposition of each task vector into a binary activation mask, a binary sign vector, and a scalar scaling knob, and Auto-Switch, a training-free variant that weights these switches by nearest-neighbor retrieval on a small query set. The authors evaluate on eight vision tasks with CLIP ViT-B/32 and ViT-L/14, eight language tasks with RoBERTa, and LoRA fine-tuned settings, reporting state-of-the-art merging accuracy with storage of 1-3% of full-precision task vectors.

Significance. If the pulse-like property is real, it would provide a simple and powerful mechanistic account of task-vector redundancy and would make binary task-vector storage an attractive practical option for multi-task deployment. The paper's decomposition is conceptually clean, Auto-Switch is genuinely training-free, and the reported storage reductions are large and easy to verify from the given storage numbers. These are real strengths. However, the central mechanistic claim is not yet isolated from a norm-reduction confound, the main tables lack any uncertainty quantification, and T-Switch as defined in Eq. (7) does not actually combine task information, which undermines the framing as a merging method. The empirical results are suggestive but need additional controls and reframing before the paper's conclusions are supported.

major comments (4)
  1. [Section 3.2, Fig. 3, Table 1, Eq. (3)] The comparison between P-Discard and DARE is not norm-matched. P-Discard (Eq. 3) returns tau_i * g_m(tau_i) without rescaling, so its L2 norm shrinks as alpha grows, whereas the DARE control rescales the remaining entries by 1/(1-alpha) and approximately preserves the original task-vector norm. The observed advantage of 'Discard Low' over DARE, and the continued improvement as alpha increases, can therefore be explained by a shrinkage/regularization effect toward the pretrained weights rather than by the removal of low-magnitude coordinates. Bin-Discard (Eq. 5) inherits the same confound because it rescales to the P-Discarded norm. Please add norm-matched controls, such as random discarding with rescaling to exactly the P-Discarded norm and P-Discard rescaled back to the original norm, and report the per-alpha curves separately for magnitude selection and norm reduction before claiming the pulse-like mechanism.
  2. [Tables 2-4 and Table 6] All reported numbers appear to be single runs, with no error bars, confidence intervals, or significance tests. Claims such as T-Switch outperforming EMR-Merging by 2.24 points on ViT-B/32 (Table 2) and by 0.0415 on RoBERTa (Table 4) are central to the conclusions and could easily lie within fine-tuning run-to-run variation. Please report means and standard deviations over multiple fine-tune/merge seeds for the main comparisons, and use a statistical test where feasible.
  3. [Section 4.1 and Fig. 7] The main results fix the discard ratio at alpha=0.5 for all discarding methods, but this value is selected after sweeping alpha from 0.0 to 0.9 on the same evaluation benchmarks. The choice is therefore informed by the test set, and the reported improvements over baselines are not an independent evaluation at a pre-specified operating point. Please either select alpha on a held-out validation split or report the comparison to baselines across the full alpha range so that the sensitivity of the conclusions to this hyperparameter is visible.
  4. [Section 3.4, Eq. (7)] With U being a vector of all ones, Eq. (7) reduces to theta_i = theta + lambda_i * S_A^i * S_P^i, which is a binary decompression of a single task vector. No information from multiple tasks is combined, so T-Switch does not realize the multi-task merging objective in Eq. (1); given a task identity, it simply reconstructs an approximation of that task's fine-tuned model. The comparisons of T-Switch to static merging methods (Task-Arithmetic, TIES, DARE) and to dynamic merging methods in Tables 2-4 are therefore not comparisons of merging algorithms. Auto-Switch (Eq. 8) does combine switches across tasks; please reposition T-Switch as a per-task compression component or an ablation, or define a merging operation in Eq. (7) that actually combines task information.
minor comments (6)
  1. [Section 3.2] There is a typo in the text: 'obervation' should be 'observation'.
  2. [Section 3.2, Eqs. (2)-(3)] The relationship between gamma_u, gamma_l, and the discard ratio alpha is described only in prose; it would be clearer to define gamma_u and gamma_l directly as alpha-quantiles in Eq. (2), or to state the dependency explicitly.
  3. [Section 3.4] The notation f_ex used in the query-set construction is undefined; please specify which layer's feature outputs are used and how they are pooled.
  4. [Section 3.4] The text says the query set is built from 'directly averaged task vectors according to the scheme in equation 4', but Eq. (4) is a norm-weighted sum, not a simple average. Please clarify the exact construction.
  5. [Section 6, Table 5] The supplementary section title contains a typo: 'Exprimental Details' should be 'Experimental Details'.
  6. [General] No code or data release is mentioned, which hinders reproducibility of the reported merging results. Please provide code or at least detailed per-dataset fine-tuning and evaluation protocols.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the pulse-like claim is an empirical observation, the binary approximation is a deterministic normalization, and the method is benchmarked against external baselines.

full rationale

Walking the derivation chain: the pulse-like characteristic is introduced not as a theorem but as an observation from controlled experiments in Section 3.2 and Figures 3-5. The discard mask g_m is defined by quantile thresholds of the task vector itself, and the claim that low-magnitude parameters are redundant is tested by comparing Discard Low, Discard High, and DARE-Random; no equation defining the threshold presupposes the performance outcome. The Bin-Discard approximation in Eq. 5 is a deterministic L2-norm-preserving rescaling of a sign/mask decomposition, and the scaling factor lambda is computed from the task vector rather than fitted to downstream performance. T-Switch in Eq. 7 and Auto-Switch in Eq. 8 are constructive uses of that decomposition, not predictions that reduce to their inputs. The discard ratio alpha is set uniformly to 0.5 for all discard-based methods, and the ablation in Figure 7 explores performance across alpha; this is hyperparameter selection rather than a fitted parameter renamed as a prediction. The only co-authored citation is to EMR-merging [15], which is used as a baseline and experimental-setting reference, not as the justification for the pulse-like claim or the binary decomposition. The central results are evaluated against external benchmarks and compared with many non-self baselines. No step exhibits an equation equivalent to its input by construction, and no load-bearing premise is justified solely by a self-citation. The norm-matching objection to the causal interpretation is a correctness or experimental-design concern, not circularity.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The main free parameters are the discard ratio alpha and its induced quantile thresholds, plus the Auto-Switch query size and neighbor count. The axioms are domain assumptions about task-vector sparsity, sign-only representability, global thresholding, and KNN reliability; none are derived from first principles.

free parameters (4)
  • discard ratio alpha = 0.5 in main experiments; swept 0.1-0.9
    Global hyperparameter controlling the fraction of low-magnitude parameters removed. Chosen after ablation sweeps show best performance near 0.4-0.7 on the same benchmarks used for final evaluation.
  • quantile thresholds gamma_u, gamma_l = alpha-quantiles of positive and negative task-vector elements
    Defined by alpha and used in pulse activation equation (2). They determine which parameters are kept or discarded for each task.
  • Auto-Switch query size N = 100 samples per task
    Number of examples used to build the query set for Auto-Switch, listed in Table 5. Ablation shows more samples improve performance.
  • Auto-Switch neighbor count C = not reported in Table 5; ablated in Figure 7
    Number of nearest neighbors used to assign switch weights in Auto-Switch. Its default value is not clearly stated in the paper.
assumptions (4)
  • domain assumption Small-magnitude entries of task vectors are noise and can be discarded without hurting task performance.
    Section 3.2 states small fluctuations may be noise from labeling or outliers; this is the basis of pulse activation. Supported by controlled experiments on the chosen benchmarks, not by a proof or universal theory.
  • domain assumption Binarizing retained task-vector entries to their signs preserves the function of the task vector when scaled by the ratio of L2 norms (Eq. 5).
    Equation (5) replaces each nonzero entry by plus or minus one and rescales to preserve the norm of selected entries. No theoretical guarantee is given that sign-only coordinates plus a scalar preserve the directions needed for merging.
  • domain assumption A single global discard ratio alpha applies across all tasks, layers, and both vision and language models.
    P-Discard and T-Switch use one alpha per run; the paper does not investigate per-task or per-layer thresholds, so the method assumes uniformly redundant task vectors.
  • domain assumption KNN features from a directly averaged merged model can reliably identify the correct task for Auto-Switch.
    Auto-Switch builds query sets from features of the averaged model theta-bar; the choice of theta-bar and the feature metric is heuristic and validated only empirically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Less is More: Efficient Model Merging with Binary Task Switch." pith.science (2026). https://pith.science/paper/QRTUPOXF

@misc{pith2026241200054,
  author       = {Pith},
  title        = {Pith review of: Less is More: Efficient Model Merging with Binary Task Switch},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QRTUPOXF}},
  note         = {Machine review of arXiv:2412.00054}
}
read the original abstract

As an effective approach to equip models with multi-task capabilities without additional training, model merging has garnered significant attention. However, existing methods face challenges of redundant parameter conflicts and the excessive storage burden of parameters. In this work, through controlled experiments, we reveal that for task vectors, only those parameters with magnitudes above a certain threshold contribute positively to the task, exhibiting a pulse-like characteristic. We then attempt leveraging this characteristic to binarize the task vectors and reduce storage overhead. Further controlled experiments show that the binarized task vectors incur almost no decrease in fine-tuning and merging performance, and even exhibit stronger performance improvements as the proportion of redundant parameters increases. Based on these insights, we propose Task Switch (T-Switch), which decomposes task vectors into three components: 1) an activation switch instantiated by a binarized mask vector, 2) a polarity switch instantiated by a binarized sign vector, and 3) a scaling knob instantiated by a scalar coefficient. By storing task vectors in a binarized form, T-Switch alleviates parameter conflicts while ensuring efficient task parameter storage. Furthermore, to enable automated switch combination in T-Switch, we further introduce Auto-Switch, which enables training-free switch combination via retrieval from a small query set. Experiments indicate that our methods achieve significant performance improvements over existing baselines, requiring only 1-3% of the storage space of full-precision parameters.

Figures

Figures reproduced from arXiv: 2412.00054 by the authors.

Figure 1
Figure 1. Left: Challenges of model merging: conflicts in task [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our method: T-Switch and Auto-Switch. The left side illustrates the construction process of the task switch, where [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Left: comparison of the performance when discard [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Left: The accuracy difference between Bin-Discard [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Merging results (%) with discard ratios ranging from 0.0 to 0.9 across different model. Left: ViT-B/32, Right: RoBERTa. tasks. Baselines. We use the same baseline methods as in the merging experiments for vision models. However, since STS-B is a regression task while t…
Figure 7
Figure 7. Figure 7: Additional ablation results. Left: Merging results(%) [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A data-free LoRA merging framework that decouples weight magnitude from direction and orthogonalizes directions to reduce task interference, outperforming existing merging methods across vision, language and multimoda...

  2. FREE-Merging: Fourier Transform for Efficient Model Merging

    cs.CV 2024-11 conditional novelty 6.0 of 10

    High-pass filtering of task vectors in the Fourier domain reduces task interference in model merging, and lightweight rescaling experts recover lost performance.

Reference graph

Works this paper leans on

58 extracted references · 53 canonical work pages · cited by 2 Pith papers

  1. [1]

    Ainsworth, Jonathan Hayase, and Siddhartha S

    Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permuta- tion symmetries. In ICLR, 2023. 2

    work page 2023

  2. [2]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. 1

    arXiv 2023

  3. [3]

    Xnor-net++: Im- proved binary neural networks

    Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Im- proved binary neural networks. In BMVC, 2019. 3

    work page 2019

  4. [4]

    Cer, Mona T

    Daniel M. Cer, Mona T. Diab, Eneko Agirre, I ˜nigo Lopez- Gazpio, and Lucia Specia. Semeval-2017 task 1: Seman- tic textual similarity - multilingual and cross-lingual focused evaluation. CoRR, 2017. 7

    work page 2017

  5. [5]

    Mmdetection: Open mmlab detection tool- box and benchmark

    Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection tool- box and benchmark. arXiv preprint arXiv:1906.07155, 2019. 1

    arXiv 1906

  6. [6]

    Remote sens- ing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sens- ing image scene classification: Benchmark and state of the art. Proc. IEEE, 2017. 4, 6

    work page 2017

  7. [7]

    Describing textures in the wild

    Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014 , pages 3606–3613. IEEE Computer Society,

    work page 2014

  8. [8]

    Binaryconnect: Training deep neural networks with binary weights during propagations

    Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurIPS, 2015. 3

    work page 2015

Show all 58 references
  1. [9]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. Automatically con- structing a corpus of sentential paraphrases. In IWP, 2005. 7

    2005

  2. [10]

    1-bit wavenet: compressing a generative neural network in speech recognition with two binarized methods

    Sicheng Gao, Runqi Wang, Liuyang Jiang, and Baochang Zhang. 1-bit wavenet: compressing a generative neural network in speech recognition with two binarized methods. In Conference on Industrial Electronics and Applications (ICIEA), pages 2043–2047, 2021. 4

    2021

  3. [11]

    The third PASCAL recognizing textual entailment challenge

    Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In PASCAL Workshop, 2007. 7

    2007

  4. [12]

    Differen- tiable soft quantization: Bridging full-precision and low-bit neural networks

    Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differen- tiable soft quantization: Bridging full-precision and low-bit neural networks. In ICCV, 2019. 3

    2019

  5. [13]

    Gradient reweighting: Towards imbalanced class-incremental learning

    Jiangpeng He. Gradient reweighting: Towards imbalanced class-incremental learning. In CVPR. IEEE, 2024. 3

    2024

  6. [14]

    Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens., 2019. 4, 6

    2019

  7. [15]

    Emr-merging: Tuning-free high- performance model merging

    Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. Emr-merging: Tuning-free high- performance model merging. CoRR, 2024. 1, 2, 6, 7

    2024

  8. [16]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco T ´ulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In ICLR, 2023. 1, 3, 4, 6

    2023

  9. [17]

    CLAP4CLIP: contin- ual learning with probabilistic finetuning for vision-language models

    Saurav Jha, Dong Gong, and Lina Yao. CLAP4CLIP: contin- ual learning with probabilistic finetuning for vision-language models. CoRR, 2024. 3

    2024

  10. [18]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In ICLR, 2023. 1, 3, 6

    2023

  11. [19]

    REPAIR: renormalizing permuted activations for interpolation repair

    Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: renormalizing permuted activations for interpolation repair. In ICLR. OpenRe- view.net, 2023. 2

    2023

  12. [20]

    Warping the space: Weight space rotation for class- incremental few-shot learning

    Do-Yeon Kim, Dong-Jun Han, Jun Seo, and Jaekyun Moon. Warping the space: Weight space rotation for class- incremental few-shot learning. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. 3

    2023

  13. [21]

    3d object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV, 2013. 4, 6

    2013

  14. [22]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 3

    2009

  15. [23]

    Y . LeCun. The mnist database of handwritten digits. In http://yann. lecun. com/exdb/mnist/, 1998. 4, 6

    1998

  16. [24]

    Fully quantized network for object detec- tion

    Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detec- tion. In CVPR, 2019. 3

    2019

  17. [25]

    Roberta: A robustly optimized BERT pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettle- moyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 2019. 7

    2019

  18. [26]

    Bi-real net: Enhancing the perfor- mance of 1-bit cnns with improved representational capabil- ity and advanced training algorithm

    Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the perfor- mance of 1-bit cnns with improved representational capabil- ity and advanced training algorithm. In ECCV, 2018. 3

    2018

  19. [27]

    Twin-merging: Dynamic integration of modular expertise in model merging

    Zhenyi Lu, Chenghao Fan, Wei Wei, Xiaoye Qu, Dangyang Chen, and Yu Cheng. Twin-merging: Dynamic integration of modular expertise in model merging. CoRR, 2024. 1, 2, 4, 6, 7

    2024

  20. [28]

    A&b bnn: Add&bit- operation-only hardware-friendly binary neural network

    Ruichen Ma, Guanchao Qiao, Yian Liu, Liwei Meng, Ning Ning, Yang Liu, and Shaogang Hu. A&b bnn: Add&bit- operation-only hardware-friendly binary neural network. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition (CVPR) , pages 5704–5713,

  21. [29]

    Bagdanov

    Simone Magistri, Tomaso Trinci, Albin Soutif-Cormerais, Joost van de Weijer, and Andrew D. Bagdanov. Elastic fea- ture consolidation for cold start exemplar-free incremental learning. In ICLR, 2024. 3

    2024

  22. [30]

    Merging models with fisher-weighted averaging

    Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. In NeurIPS, 2022. 1, 2, 3, 6

    2022

  23. [31]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bis- sacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop, 2011. 4, 6

    2011

  24. [32]

    Binary neural networks for speech recognition

    Yanmin Qian and Xu Xiang. Binary neural networks for speech recognition. Frontiers Inf. Technol. Electron. Eng. , 20(5):701–715, 2019. 4

    2019

  25. [33]

    Guanchao Qiao, Shaogang Hu, Tupei Chen, L. M. Rong, Ning Ning, Qi Yu, and Y . Liu. STBNN: hardware-friendly spatio-temporal binary neural network with high pattern recognition accuracy. Neurocomputing, 409:351–360, 2020. 4

    2020

  26. [34]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 4, 6

    2021

  27. [35]

    Squad: 100, 000+ questions for machine com- prehension of text

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100, 000+ questions for machine com- prehension of text. In EMNLP, 2016. 7

    2016

  28. [36]

    Xnor-net: Imagenet classification using bi- nary convolutional neural networks

    Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using bi- nary convolutional neural networks. In ECCV, 2016. 3

    2016

  29. [37]

    Dandekar S

    N. Dandekar S. Iyer, K. Csernai, and et al. First quora dataset release: Question pairs. data. quora. com. 2017. 7

    2017

  30. [38]

    Manning, Andrew Y

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y . Ng, and Christopher Potts. Recursive deep models for semantic compositional- ity over a sentiment treebank. In EMNLP, 2013. 7

    2013

  31. [39]

    The german traffic sign recognition bench- mark: A multi-class classification competition

    Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition bench- mark: A multi-class classification competition. In IJCNN,

  32. [40]

    Fast object detection based on binary deep convolution neural networks

    Siyang Sun, Yingjie Yin, Xingang Wang, De Xu, Wenqi Wu, and Qingyi Gu. Fast object detection based on binary deep convolution neural networks. CAAI Trans. Intell. Technol., 3 (4):191–197, 2018. 3

    2018

  33. [41]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  34. [42]

    Correa, Carlos Hinojosa, and Henry Arguello

    Edwin Vargas, Claudia V . Correa, Carlos Hinojosa, and Henry Arguello. Biper: Binary neural networks using a peri- odic function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5684–5693, 2024. 3

    2024

  35. [43]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language under- standing. In ICLR, 2019. 7

    2019

  36. [44]

    Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments.Trans. Assoc. Com- put. Linguistics, 7:625–641, 2019. 7

    2019

  37. [45]

    Adina Williams, Nikita Nangia, and Samuel R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, 2018. 7

    2018

  38. [46]

    Hug- gingface’s transformers: State-of-the-art natural language processing

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chau- mond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´emi Louf, Morgan Funtowicz, and Jamie Brew. Hug- gingface’s transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771, 2019. 1

    1910 arXiv

  39. [47]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy with...

    2022

  40. [48]

    Binary deep neural networks for speech recognition

    Xu Xiang, Yanmin Qian, and Kai Yu. Binary deep neural networks for speech recognition. In ISCA, 2017. 4

    2017

  41. [49]

    Ehinger, Aude Oliva, and Antonio Torralba

    Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010. 4, 6

    2010

  42. [50]

    Raf- fel, and Mohit Bansal

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raf- fel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In NeurIPS, 2023. 2, 3, 4, 6

    2023

  43. [51]

    Adamerging: Adap- tive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adap- tive model merging for multi-task learning. In ICLR, 2024. 3, 6

    2024

  44. [52]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024. 2, 3, 4, 6

    2024

  45. [53]

    Lq-nets: Learned quantization for highly accurate and compact deep neural networks

    Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In ECCV, 2018. 3

    2018

  46. [54]

    Preventing zero-shot transfer degradation in continual learning of vision-language models

    Zangwei Zheng, Mingyuan Ma, Kai Wang, Ziheng Qin, Xi- angyu Yue, and Yang You. Preventing zero-shot transfer degradation in continual learning of vision-language models. In ICCV, 2023. 3

    2023

  47. [55]

    Dorefa-net: Training low bitwidth convo- lutional neural networks with low bitwidth gradients

    Shuchang Zhou, Zekun Ni, Xinyu Zhou, He Wen, Yuxin Wu, and Yuheng Zou. Dorefa-net: Training low bitwidth convo- lutional neural networks with low bitwidth gradients. CoRR,

  48. [57]

    Exprimental Details To provide a comprehensive overview of the experimental setup, we list the hyperparameter settings for our method and all baseline methods in Table 5. Methods α N Scaling Coef LR Epochs Task-Arithmetic – – 0.3 – – Ties-Merging 0.5 – 0.3 – – DARE 0.5 – – – –...

  49. [58]

    To evaluate the effectiveness of our method in merging larger mod- els, we conducted experiments on eight visual tasks using the ViT-L/14 model

    Additional Results Merging results on the ViT-L/14 model. To evaluate the effectiveness of our method in merging larger mod- els, we conducted experiments on eight visual tasks using the ViT-L/14 model. Table 6 shows the combined perfor- mance of our method and various baselin...

  50. [2016]

    3 Less is More: Efficient Model Merging with Binary Task Switch Supplementary Material

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.