STAR: Spectral Truncation and Rescale for Model Merging

Ching-Yun Ko; I-Hsin Chung; Mi-Yen Yeh; Pin-Yu Chen; Tejaswini Pedapati; Yu-Ang Lee

REVIEW 3 major objections 5 minor 35 references

STAR: Spectral Truncation and Rescale for Model Merging

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read STAR claims data-free merging of up to 20 fine-tuned models by truncating small singular components of task vectors and rescaling to restore nuclear norm, outperforming baselines by 4.2% on 12-model Flan-T5 merges.

desk verdict Clean, data-free merging trick with fair baselines, but the appendix bound only covers truncation without the rescale step, and the empirical support is thinner than the prose suggests. read the letter →

arxiv 2502.10339 v1 pith:EXPRVP4P submitted 2025-02-14 cs.CL cs.AIcs.LG

Yu-Ang Lee , Ching-Yun Ko , Tejaswini Pedapati , I-Hsin Chung , Mi-Yen Yeh , Pin-Yu Chen This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords modelmergingtaskvectorsspectraltruncationsingularvaluedecompositionnuclearnormrescalinglow-rankpriormulti-tasklanguagemodelsdata-free

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

STAR is a data-free recipe for merging many fine-tuned versions of the same pretrained model into one multi-task model. The paper argues that the performance collapse seen as the number of merged models grows comes partly from noisy and redundant directions in the task vectors, and that those directions can be identified in the spectral domain. For each weight matrix of each task vector, the method keeps only the singular components whose cumulative energy reaches a threshold, rescales the survivors to restore the matrix's original nuclear norm, and then averages the cleaned task vectors. The authors report that this slows the decay of merged-model performance across model sizes and architectures, and that merging 12 Flan-T5 models outperforms the compared baselines by 4.2%, with successful merges of up to 20 Mistral-7B adapters.

What carries the argument

The machinery is per-layer singular value decomposition of task vectors. Each task vector is $\theta_{\mathrm{ft}} - \theta_{\mathrm{pre}}$; STAR reshapes its entries back into weight matrices, decomposes each matrix as $\sum_i \sigma_i u_i v_i^\top$, and keeps only the components needed so that the kept singular values contain at least $\eta\%$ of the total nuclear norm (default $\eta = 40$). It then multiplies each kept $\sigma_k$ by $\|\sigma\|_1 / \|\sigma_{1:r}\|_1$, reconstructs the cleaned matrix, and averages the cleaned task vectors across tasks. The rescale step is what makes truncation safe: without it the task vector shrinks and the merged model drifts back toward the pretrained model; with it the matrix keeps its original size in the nuclear norm while shedding the small components. This combination gives STAR an automatic, layer-specific rank choice and a data-free pipeline.

What would settle it

Take a fine-tuned model, keep only the small singular components of its task vector (discard the top directions), merge that residual into the pretrained model, and measure task accuracy: if accuracy stays high, task knowledge is not concentrated in the large singular directions and STAR's premise is wrong. A cheaper check is to compare STAR with a variant that truncates the same number of singular components chosen at random; if random truncation performs equally well, the benefit is not specifically spectral.

Watch

Extended reading notes

Core claim

The paper's central claim is that the ceiling on how many models can be merged is set less by the number of tasks than by how much conflicting, low-signal content each task vector carries. STAR's hypothesis is that task-specific knowledge lives in the top singular directions of each layer's weight update, while the small singular directions are mostly noise or redundant fine details, so removing them before averaging reduces cross-task interference without sacrificing task performance. The method makes this precise with a bound: for a point on task A's data manifold, the interference contributed by task B's matrix satisfies $\|Bx\| \le r_B \beta \sqrt{r_A}$, and truncating $B$ to a lower rank shrinks that upper bound. After truncation, STAR rescales the kept singular values by a common factor so the matrix's nuclear norm matches the original, preventing the cleaned task vector from shrinking relative to the pretrained model. On this basis the paper claims that STAR outperforms TIES, TALL-masks, and MetaGPT in one-shot use, degrades more gracefully as the model count grows to 12 and 20, and needs no tuning of its threshold across combinations.

Load-bearing premise

The load-bearing premise is that each task's useful knowledge sits in the large singular directions of its weight update, so discarding the small singular components removes noise rather than skill.

Editorial extensions

If this is right

Merging 20 LoRA-tuned adapters into one static Mistral-7B model keeps averaged normalized performance above the pretrained baseline, so multi-task assembly no longer has to stop at a handful of models.
Because the same $\eta = 40$ works across Flan-T5-base, Flan-T5-large, and Mistral-7B-Instruct, deployment can use the default threshold instead of a per-task hyperparameter search.
The optimal threshold decreases as more models are merged, implying that stronger truncation is exactly what is needed when conflicts are most severe.
The method operates on task vectors before the averaging step, so it can be composed with other merging rules that also start from task vectors.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the low-rank prior is right, the singular-value spectrum of a task vector becomes a cheap, data-free predictor of how safely that model can be merged; steep spectra should merge cleanly, flat spectra should not.
The same spectral truncation and nuclear-norm rescale recipe is architecture-agnostic, so it could plausibly transfer to vision or multimodal adapters, or to fusion rules other than simple averaging; the paper tests only NLP models and averaging.
A stress test the paper does not run: fine-tune a model so that a rare but essential capability is encoded only in small singular components, then check whether STAR removes precisely that capability. If it does, the method's advantage would disappear exactly on long-tail skills.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: STAR is a simple, data-free way to merge many task vectors, and empirically it holds up better than TIES, TALL-masks, and MetaGPT as the model count grows. The core idea—SVD-truncate each task matrix and then rescale to preserve nuclear norm—is new relative to the cited work, and the ablation in Table 1 shows the rescale matters, especially at low ranks. They also did the right comparison against the standard baselines, including DARE variants, and released code.

The weak spot is the theory, not the experiments. The Sec. A.1 bound is meant to motivate why truncation reduces interference: for two task matrices A and B, ||Bx|| ≤ r_B β√r_A, and truncating B to rank r lowers that bound by (r_B−r)β√r_A. But the actual STAR algorithm always rescales retained singular values by c = ||σ||_1 / ||σ_{1:r}||_1 > 1. After rescaling, the bound becomes r·(c·max|σ^i_B α_j|)·√r_A, which can be larger than the original if r·c > r_B. The paper never states or checks that condition. So the proof as written applies to a different transform than the one being evaluated. It's a motivation gap, not a fatal flaw, but it should be fixed in revision—either re-derive the bound for the rescaled matrix or drop the formal claim and keep the low-rank intuition.

The experimental section is thinner than the prose suggests. The main curves come from 5 sampled task combinations per model count on Flan-T5 and 3 on Mistral, with no error bars. That's enough to see a trend but not enough to be confident about the 4.2% claim. Excluding CoLA is defensible, but the reason given ('tends to output the same answer') deserves a footnote. The hyperparameter-robustness claim is mostly supported—std dev in Fig 6 is small—but Fig 5 shows the optimal η shifts downward as model count grows, so 'robust' should be qualified.

The method itself is sound as an empirical contribution. The math is straightforward, the code is out there, and the comparisons are fair. I'd send this to a serious referee, with instructions to push on the theory gap and ask for error bars or a sensitivity analysis over task combinations.

Worth your reading-group time if they like discussing the gap between theoretical motivation and actual algorithm.

Referee Report

3 major / 5 minor

Summary. The paper proposes STAR (Spectral Truncation and Rescale), a data-free model merging method that applies per-weight-matrix singular value decomposition (SVD) to task vectors, truncates small singular-value components to reduce rank, rescales retained singular values to restore the matrix nuclear norm, and finally averages the processed task vectors. The authors claim that STAR outperforms TIES, TALL-masks, and MetaGPT as the number of merged models grows, with a 4.2% improvement over baselines when merging 12 models on Flan-T5, robustness to the single hyperparameter η, and the ability to merge up to 20 models on Mistral-7B. The manuscript includes a theoretical bound on cross-task interference (App. A.1), ablations of the rescaling step, hyperparameter sensitivity experiments, and comparisons with DARE as a plug-in.

Significance. If the empirical claims hold, STAR would be a practically valuable contribution to data-free model merging: it requires no inference on training data, has a single hyperparameter with demonstrated robustness, and is clearly specified with public code. The strongest aspect is the consistent empirical trend across three model families (Flan-T5-base, Flan-T5-large, Mistral-7B-Instruct) showing that STAR's performance decay is slower than that of prior methods as the number of merged models increases. The paper also provides a formal, albeit incomplete, motivation for spectral truncation. The significance is tempered by the limited number of sampled task combinations and by the gap between the theoretical bound and the actual STAR algorithm, both of which need to be addressed for the claims to be fully credible.

major comments (3)

[App. A.1 and Sec. 3.2] The conflict-reduction bound ∥Bx∥ ≤ r_B β√r_A is derived for a truncated B without rescaling, but Algorithm 1 always rescales retained singular values by c = ∥σ∥_1/∥σ_{1:r}∥_1 > 1 (Sec. 3.2). After rescaling, the upper bound becomes r · c · max_{i≤r}|σ_B^i α_j| · √r_A, which is not necessarily smaller than the original r_B β√r_A; the reduction holds only under a spectral condition that is neither stated nor verified in the paper. Since this bound is the paper's stated theoretical rationale for why STAR mitigates merging conflicts, the motivation is incomplete and should be corrected to a bound for the full STAR transform or explicitly reframed as a heuristic.
[Sec. 4.2 and Fig. 4] The central empirical claims, including the abstract's 4.2% improvement and the ability to merge up to 20 models, rest on averaged results over only 5 sampled task combinations for Flan-T5 and 3 for Mistral-7B, with no error bars or confidence intervals on the averaged curves. For a claim about robustness across model sizes and numbers of models, this is a thin statistical basis; the authors should report variance and ideally use more random task combinations to substantiate the comparisons.
[Sec. 1 and Fig. 4] The introduction states that STAR can merge up to 20 models while achieving positive performance gains compared to the pretrained model, but the pretrained-model baseline is never plotted in Fig. 4 or Fig. 1, nor are the exact values reported. Without this baseline, the reader cannot verify whether the merged model remains above the pretrained model, which is the paper's own criterion for the purpose of model merging (Sec. 4.1). Please add the pretrained baseline to the figures or state the numerical values.

minor comments (5)

[Abstract and Sec. 5] There are typos in the abstract and conclusion: 'hyperparamater' should be 'hyperparameter' and 'principaled' should be 'principled'.
[Sec. 3.1] The low-rank prior ('small singular values often correlate with noise or fine details') is stated as fact; since it is the key assumption underlying the method, it should be presented more cautiously as a working assumption and discussed in the limitations.
[Alg. 1 and Sec. 3.3] Algorithm 1 calls the subroutine 'rank_keep(σ, η, p)' but the variable p is never defined; the rank-determination rule in Sec. 3.3 involves only σ and η, so the pseudocode should be corrected.
[Sec. 4.1 and Sec. A.6] The main text says the total number of task vectors for Flan-T5-base/large is 13, but Sec. A.6 indicates Flan-T5-base has 12 (7 from FusionBench plus 5 fine-tuned) and Flan-T5-large has 13 (7 plus 6); this inconsistency should be corrected.
[Sec. 3.2 and Table 2] In the DARE comparison (Table 2), DARE's drop rate p is selected as the best-performing value over a grid (p*), while STAR uses a fixed η=40; this asymmetry should be disclosed in the main text or in the table caption to avoid overstating STAR's advantage.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: STAR's ranking, truncation, and rescaling are defined independently of the evaluated outputs, and no self-citation chain is load-bearing.

full rationale

STAR is a data-free manipulation of task vectors: SVD, adaptive rank selection by cumulative singular-value energy threshold eta, nuclear-norm-preserving rescaling, and averaging. The rank criterion r=argmin_k(... >= eta%) and rescale sigma'_k=||sigma||_1/||sigma_{1:r}||_1 * sigma_k are definitional algorithm steps, not quantities fitted to the benchmark tasks; eta=40 is fixed a priori and the sensitivity analysis spans eta in {10,...,70}. The conflict-reduction argument in Sec. A.1 is an independent inequality about truncation, and while it is not extended to the rescaled transform (so the rescaling step's theoretical motivation is under-supported), that is a rigor gap, not a circular reduction. The paper's positive-gains and 4.2% claims are measured against external benchmarks (GLUE, PIQA, etc.) and baselines, not derived from the values being predicted. There are no load-bearing self-citations: references to task arithmetic, TIES, MetaGPT, DARE, and FusionBench are external prior work, and the authors' own fine-tuned models are data, not premises. The stated limitation (PEFT-only evaluation) is an honest scope restriction. No circular step can be exhibited.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central method introduces one hyperparameter, the energy retention ratio eta, fixed at 40. The methodological core relies on a low-rank prior for task vectors and on nuclear-norm restoration as a compensation scheme; both are assumptions from the broader signal-processing and task-arithmetic literature rather than derived from first principles within the paper. No new entities are invented.

free parameters (1)

rank retention ratio eta = 40
Energy threshold for truncation in Algorithm 1; fixed at 40 for the one-shot results. Sensitivity analysis (Fig 6) shows modest variation, but Fig 5 shows the optimal eta decreases as the number of merged models increases, so the default is not universally optimal.

assumptions (6)

domain assumption Task vectors delta = theta_ft - theta_pre encode the task-specific update (Ilharco et al., 2022).
Sec 2.1 defines task vectors and treats them as additive updates; STAR builds on this.
domain assumption Small singular values of task-vector weight matrices correspond to noise or fine details and can be removed without hurting task performance.
Sec 3.1 invokes a low-rank prior from compressed sensing and denoising; this is the key heuristic for truncation.
domain assumption Nuclear norm is a proper measure of matrix size for the merging context; restoring it compensates for truncation.
Sec 3.2 proposes rescaling singular values so that the sum of singular values is unchanged, citing nuclear norm as a convex relaxation of rank.
domain assumption Simple averaging of truncated-and-rescaled task vectors is a valid merging operation.
Sec 3.3 applies simple averaging after reconstruction; this assumes the cleaned task vectors are directly addable.
domain assumption The Cauchy-Schwarz based bound in Sec A.1 is a meaningful proxy for actual merging conflict.
The bound uses an upper bound on cross-task output norm, but no formal connection to task performance is shown.
standard math SVD exists for all weight matrices considered.
SVD is a standard matrix factorization; no issue.

how reviews work

0 comments

Cite this review

Pith. "Pith review of STAR: Spectral Truncation and Rescale for Model Merging." pith.science (2026). https://pith.science/paper/EXPRVP4P

@misc{pith2026250210339,
  author       = {Pith},
  title        = {Pith review of: STAR: Spectral Truncation and Rescale for Model Merging},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EXPRVP4P}},
  note         = {Machine review of arXiv:2502.10339}
}

abstract

Model merging is an efficient way of obtaining a multi-task model from several pretrained models without further fine-tuning, and it has gained attention in various domains, including natural language processing (NLP). Despite the efficiency, a key challenge in model merging is the seemingly inevitable decrease in task performance as the number of models increases. In this paper, we propose $\mathbf{S}$pectral $\mathbf{T}$runcation $\mathbf{A}$nd $\mathbf{R}$escale (STAR) that aims at mitigating ``merging conflicts'' by truncating small components in the respective spectral spaces, which is followed by an automatic parameter rescaling scheme to retain the nuclear norm of the original matrix. STAR requires no additional inference on original training data and is robust to hyperparamater choice. We demonstrate the effectiveness of STAR through extensive model merging cases on diverse NLP tasks. Specifically, STAR works robustly across varying model sizes, and can outperform baselines by 4.2$\%$ when merging 12 models on Flan-T5. Our code is publicly available at https://github.com/IBM/STAR.

Figures

Figures reproduced from arXiv: 2502.10339 by the authors.

**Figure 2.** An overview of the STAR workflow. When merging two task vectors, δ1 and δ2, (1) STAR transforms both task vectors into their spectral spaces with their singular vectors being the orthogonal basis using singular value decomposition (SVD) (singular values are represented by the length of the arrows), (2) STAR removes redundant dimensions by truncating singular vectors with small singular values, (3) STAR restores the … view at source ↗

**Figure 3.** An example of the automatic rank determi [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Model merging results on Flan-T5-large and Mistral-7B-Instruct. For all numbers of models merged, [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: The mean and standard deviation of the opti [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: The average model merging results on Flan [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 7.** Figure 7: The model merging results on Flan-T5-base [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 7 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...
[2]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[3]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[4]

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

2020
[5]

Rickard Br \"u el-Gabrielsson, Jiacheng Zhu, Onkar Bhardwaj, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, and Justin Solomon. 2024. Compress then serve: Serving thousands of lora adapters with little overhead. arXiv preprint arXiv:2407.00066

arXiv 2024
[6]

Jian-Feng Cai, Emmanuel J Cand \`e s, and Zuowei Shen. 2010. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4):1956--1982

work page 2010
[7]

Emmanuel Candes and Benjamin Recht. 2012. Exact matrix completion via convex optimization. Communications of the ACM, 55(6):111--119

work page 2012
[8]

Emmanuel J Candes and Yaniv Plan. 2010. Matrix completion with noise. Proceedings of the IEEE, 98(6):925--936

work page 2010

Show all 35 references

[9]

Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055

2017 arXiv
[10]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1--53

2024
[11]

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL

2019
[12]

Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. 2007. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 16(8):2080--2095

2007
[13]

Nico Daheim, Thomas M \"o llenhoff, Edoardo Maria Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. 2023. Model merging by uncertainty-based gradient matching. arXiv preprint arXiv:2310.12808

2023 arXiv
[14]

MohammadReza Davari and Eugene Belilovsky. 2023. Model breadcrumbs: Scaling multi-task model merging with sparse masks. arXiv preprint arXiv:2312.06795

2023 arXiv
[15]

Fidel A Guerrero Pena, Heitor R Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. 2022. Re-basin via implicit sinkhorn differentiation. in 2023 ieee. In CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20237--20246

2022
[16]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

2021 arXiv
[17]

Chenyu Huang, Peng Ye, Tao Chen, Tong He, Xiangyu Yue, and Wanli Ouyang. 2024. Emr-merging: Tuning-free high-performance model merging. arXiv preprint arXiv:2405.17461

2024 arXiv
[18]

Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089

2022 arXiv
[19]

Moritz Imfeld, Jacopo Graldi, Marco Giordano, Thomas Hofmann, Sotiris Anagnostidis, and Sidak Pal Singh. 2023. Transformer fusion with optimal transport. arXiv preprint arXiv:2310.05719

2023 arXiv
[20]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

2023 arXiv
[21]

Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. arXiv:1910.11473v2

2020 arXiv
[22]

Maas, Raymond E

Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. http://www.aclweb.org/anthology/P11-1015 Learning word vectors for sentiment analysis . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguist...

2011
[23]

P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65

2014
[24]

Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703--17716

2022
[25]

Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2024. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36

2024
[26]

Derek Tam, Mohit Bansal, and Colin Raffel. 2024. Merging by matching models in task parameter subspaces. Transactions on Machine Learning Research

2024
[27]

Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Do, and Dacheng Tao. 2024. Fusionbench: A comprehensive benchmark of deep model fusion. arXiv preprint arXiv:2406.03280

2024
[28]

Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, Fran c ois Fleuret, and Pascal Frossard. 2024. https://openreview.net/forum?id=DWT9uiGjxT Localizing task information for improved model merging and compression . In Forty-first International Conference on Machine Learning

2024
[29]

Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. 2024. Ties-merging: Resolving interference when merging models. Advances in Neural Information Processing Systems, 36

2024
[30]

Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024. Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666

2024 arXiv
[31]

Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. 2023. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575

2023 arXiv
[32]

Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning

2024
[33]

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

2019
[34]

Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28

2015
[35]

Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. 2024. Metagpt: Merging large language models using model exclusive task arithmetic. arXiv preprint arXiv:2406.11385

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

[2] [2]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[3] [3]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[4] [4]

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence

2020

[5] [5]

Rickard Br \"u el-Gabrielsson, Jiacheng Zhu, Onkar Bhardwaj, Leshem Choshen, Kristjan Greenewald, Mikhail Yurochkin, and Justin Solomon. 2024. Compress then serve: Serving thousands of lora adapters with little overhead. arXiv preprint arXiv:2407.00066

arXiv 2024

[6] [6]

Jian-Feng Cai, Emmanuel J Cand \`e s, and Zuowei Shen. 2010. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4):1956--1982

work page 2010

[7] [7]

Emmanuel Candes and Benjamin Recht. 2012. Exact matrix completion via convex optimization. Communications of the ACM, 55(6):111--119

work page 2012

[8] [8]

Emmanuel J Candes and Yaniv Plan. 2010. Matrix completion with noise. Proceedings of the IEEE, 98(6):925--936

work page 2010