Pith. sign in

REVIEW 2 major objections 6 minor 6 cited by

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces

T0 review · 2 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Merged multitask models get closer to single-task accuracy by flattening the singular spectrum of their summed weight updates.

desk verdict A genuinely simple and effective merging recipe, but the SAR-based explanation for why it works is confounded and should not be the sales pitch. read the letter →

arxiv 2502.04959 v3 pith:VJVXNYRZ submitted 2025-02-07 cs.LG

Daniel Marczak , Simone Magistri , Sebastian Cygert , Bart{l}omiej Twardowski , Andrew D. Bagdanov , Joost van de Weijer This is my paper · ORCID
classification cs.LG MSC 68T0715A18
keywords modelmergingtaskarithmeticsingularvaluedecompositionsubspacealignmentisotropicmulti-tasklearningLoRAspectrumflattening
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to explain why merging several fine-tuned models into one multi-task model still loses accuracy, and to close that gap without retraining. It shows that a task's performance in the merged model tracks how well the task's weight-update matrix aligns with the merged weight-update matrix, measured by a new quantity, the Subspace Alignment Ratio. On that evidence it proposes two merging recipes: Iso-C, which replaces the skewed singular-value spectrum of the summed task updates with a flat one, and Iso-CTS, which additionally keeps a few singular directions from each task. On 8, 14 and 20 vision tasks, and on LoRA-tuned vision models and T5 language models, the recipes outperform existing merging methods and shrink the gap toward the per-task models.

What carries the argument

The load-bearing objects are the task matrix $\Delta_t = \theta_t - \theta_0$ (the weight update from fine-tuning), its singular value decomposition, and the Subspace Alignment Ratio $\text{SAR}(\Delta_t, \Delta_M; k_M)$, which projects $\Delta_t$ onto the top $k_M$ left-singular vectors of the merged matrix and reports the fraction of $\Delta_t$'s magnitude retained. The rank $k_M$ is chosen automatically as the smallest rank whose reconstruction error falls below 5 percent of the total energy, so a flatter singular spectrum automatically yields a larger effective subspace. Iso-C replaces the skewed spectrum of the summed matrix $\Delta_{\text{TA}}$ with a single average singular value $\bar{\sigma}$, reconstructing $\bar{\sigma} U V^\top$; Iso-CTS concatenates top-$k$ common directions with per-task directions taken from the residual $\Delta_t - U_{1:k}U_{1:k}^\top\Delta_t$, whitens the combined basis, and scales the result by the average of the selected singular values. The same machinery serves as an analysis tool: the paper uses SAR to identify which tasks are underrepresented and to predict which ones will improve.

What would settle it

Merge 8 or 20 tasks with Iso-CTS, then run a control that keeps the same rank and the same isotropic scaling but replaces the task-specific singular vectors with randomly chosen orthonormal vectors orthogonal to the common subspace; if the random-direction control matches Iso-CTS accuracy, the content of the task-specific directions is not what matters and the alignment explanation is in doubt. A second check: if a task's dominant directions are orthogonal to the common subspace, adding them should specifically rescue that task, whereas adding directions from a different task would not.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that the singular-value structure of the summed task matrix, not just pairwise cosine similarity between task vectors, governs how well a merged model preserves each task. The authors define the Subspace Alignment Ratio as the fraction of a task matrix's total magnitude captured by projection onto the dominant left-singular subspace of the merged matrix, and report a Pearson correlation of 0.94 between the average of this ratio over layers and normalized accuracy improvement. They then show that flattening the singular values of the Task Arithmetic sum to their mean, leaving the singular vectors untouched and rescaling by a single coefficient $\alpha$, raises the alignment and with it accuracy; adding per-task directions that lie orthogonal to the common subspace, after whitening the combined basis, extends the gains to larger task sets. The resulting Iso-CTS method reaches state-of-the-art merged accuracy on vision benchmarks at three model scales, on LoRA-adapted models, and on T5 language tasks, with no additional training.

Load-bearing premise

The paper assumes that the Subspace Alignment Ratio, computed with an automatically chosen effective rank, is a causal driver of merged-model accuracy, so that any operation that raises alignment, such as flattening the spectrum, is the right cure; the support is a strong correlation and interpolation experiments, not a proof that alignment itself causes the gains.

Editorial extensions

If this is right

  • Iso-C and Iso-CTS require no retraining and no data beyond a validation set, so they apply to any family of models fine-tuned from a shared checkpoint.
  • As the number of tasks grows, retaining task-specific directions matters more: Iso-CTS beats Iso-C by up to 2.8 percent absolute accuracy when going from 8 to 20 tasks.
  • The recipe transfers to low-rank (LoRA) adapters by first reconstructing $\Delta W_t = B_t A_t$, and outperforms LoRA-specific merging methods.
  • Spectrum flattening does not help individual single-task models (it hurts them by 3.3 percent), so the gain comes from counteracting interference in summed task matrices, not from improving the task matrices themselves.
  • On T5 language models, both Iso variants beat prior merging methods, showing the effect is not vision-specific.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If alignment is truly causal, merging quality could be optimized directly by choosing the merged subspace that maximizes total alignment across tasks, rather than by flattening the spectrum and adding residuals; that is a tractable, testable objective.
  • The result reframes post-hoc merging: spectral skewness of the summed updates, not only parameter sign conflicts, drives interference, so sign-trimming methods and spectrum flattening address different mechanisms and may be complementary.
  • Because the method only needs the final weight matrices, it should apply to other architectures and modalities, such as CNNs or diffusion models, as long as their weight updates are matrices; the paper tests transformer vision and text backbones only.
  • A natural extension is to replace the fixed $\epsilon=0.05$ rank rule and the uniform mean scaling with per-layer or per-task weights derived from validation accuracy, which could preserve the alignment mechanism while beating the uniform isotropic choice.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes two training-free model merging methods: Iso-C, which flattens the singular value spectrum of the Task Arithmetic sum matrix, and Iso-CTS, which augments the common subspace with per-task directions before isotropic scaling. The central explanatory claim is that the Subspace Alignment Ratio (SAR) between task-specific and merged matrices strongly correlates with performance, and that isotropic merging improves both SAR and accuracy. The methods are evaluated on vision benchmarks (8/14/20 tasks with ViT-B/32, ViT-B/16, ViT-L/14), LoRA fine-tuned vision models, and T5 language models, reporting state-of-the-art results against Task Arithmetic, TIES, Consensus TA, TSV-M, and other baselines.

Significance. If the results hold, the paper offers a simple, training-free merging technique that substantially reduces the gap to single-task models, and it demonstrates effectiveness across vision, LoRA, and language settings. The approach is computationally cheap for Iso-C (one SVD per layer), reproducible code is provided, and the ablation study in Section 5.4 plus the appendix derivations give useful insight into spectral properties of merged matrices. However, the explanatory mechanism is currently not supported as stated: the measured SAR increase from Task Arithmetic to Iso-C is largely a consequence of the metric's dependence on the effective rank k_M, not an independent improvement in directional alignment. The empirical accuracy gains are real and consistently measured, but the causal story in the abstract and Section 3.3 needs either stronger evidence or a reframed claim.

major comments (2)
  1. [Section 3.3, Appendix A.2, Eqs. (5)-(8), (16)] The claim that Iso-C 'enhances alignment' and that this drives the performance gain is not supported by the SAR metric because the metric is not invariant to the spectral flattening. Iso-C replaces the singular values of Δ_TA by their mean (Eq. 7-8) but leaves U and V unchanged; therefore, for any fixed k, the top-k projection subspaces of Δ_TA and Δ_Iso-C are identical. The inequality in Eq. (16) follows solely from k_Iso-C > k_TA, which is a direct consequence of the threshold in Eq. (6). Consequently, the SAR increases in Fig. 4b and Fig. 5b measure the growth of k_M, not any improvement in the overlap between the task matrix and the merged matrix's span. To rescue the causal explanation, please report SAR with a fixed k (e.g., k = k_TA for both methods) or use a metric that does not depend on singular value magnitudes, such as principal angles or fixed-rank projections; alternatively, reframe the contribution as demonstrating that spectrum flattening itself improves merging, without attributing the gain to increased subspace alignment.
  2. [Section 4.2, Algorithm 2] The definition s = (r-k)/T requires this quantity to be an integer because U* and V* are constructed by concatenating k common and T·s task-specific columns to form matrices in R^{m×r} and R^{n×r}. For the default setting k/r = 0.8, a typical square layer of ViT-B/16 has r = 768, and for T = 20 this yields s = 7.68, which is not an integer. The paper does not specify how to handle non-divisible (r-k)/T (e.g., floor/ceil, dropping leftover directions, or adjusting k). This affects the exact definition of the method used to produce Table 1 and must be clarified for reproducibility.
minor comments (6)
  1. [Tables 1 and 2] No variance or number of seeds is reported. Although the merging itself is deterministic given the checkpoints, the fine-tuning and hyperparameter selection involve randomness; please report seeds or variance, especially for close comparisons such as Iso-C vs TSV-M on ViT-B/32 with 14 tasks.
  2. [Figure 3a] The Pearson correlation ρ = 0.94 is computed on only 8 points; please provide a p-value or confidence interval to assess the strength of the correlation.
  3. [Section 4.2, after Eq. (10)] The sentence 'both the left- and right-singular vectors of Δ_t ... are orthogonal to the subspace spanned by the common directions' is not correct for right-singular vectors; the projection in Eq. (10) only ensures orthogonality of the left singular vectors to U^{1:k}. The subsequent whitening handles the right side, but the text should be corrected.
  4. [Section 5.3, Table 3] In both NLP settings Iso-C slightly outperforms Iso-CTS (75.6 vs 75.2 and 83.3 vs 82.8); the sentence 'Iso-CTS achieves very similar results to Iso-C suggesting that the common space captures all the directions necessary' should be adjusted to reflect that Iso-C is numerically better.
  5. [Related work, Table 1] The baseline is introduced as 'Task Singular Vectors (TSV)' in Section 2 but compared as TSV-M in Table 1; please define TSV-M explicitly when it first appears in a table.
  6. [Eq. (6)] The choice of ε = 0.05 in the effective-rank criterion is not discussed; a sensitivity analysis of the SAR metric to this threshold would strengthen the claim that the metric is robust.

Circularity Check

2 steps flagged · score 6.0 of 10

The reported alignment improvement of Iso-C over Task Arithmetic is by construction: spectrum flattening leaves the singular vectors unchanged and inflates the metric's adaptive rank k_M, so the SAR increase used as the causal mechanism is definitional; the accuracy gains themselves are independently measured.

  1. self definitional [Section 4.1 (Eqs. 7-8), Section 3.3 (Eqs. 5-6), Appendix A.2 (Eq. 16)]
    "Flattening the spectrum of ∆TA (Figure 4a), yields ∆Iso-C with effective rank kIso-C > kTA (as discussed in Appendix A.1). This flattening modifies only the singular values of TA, leaving the singular vectors unchanged. Therefore, the original subspace T is contained within the larger subspace spanned by the top singular vectors of ∆Iso-C... Thus, by construction, we have T ⊂ I. ... The increase in SAR is due to a higher number of dominant components kIso-c in ∆Iso-c (see Equation (6)), derived from the singular vectors of ∆TA."

    Iso-C replaces the singular values of ∆TA by their average (Eqs. 7-8) while keeping U and V identical to those of Task Arithmetic. The SAR metric (Eq. 5) projects onto the top k_M left-singular vectors, with k_M selected by the 5% reconstruction criterion (Eq. 6). Because the flattened spectrum has more components above that threshold, k_Iso-C > k_TA, and Appendix A.2 Eq. 16 proves the SAR increase by the containment T ⊂ I. Thus the paper's own equations make the 'enhanced alignment' of Iso-C a mathematical consequence of the metric, not an empirical discovery. For any fixed k, the top-k subspaces of ∆TA and ∆Iso-C coincide exactly, since the singular vectors are unchanged. The accuracy gains are independent, but the alignment increase used as the explanatory mechanism is definitional.

  2. self definitional [Section 5.4, Figure 4 and Eq. 15]
    "Interpolating from ∆TA (β = 0) towards ∆Iso-C (β = 1) makes the spectrum of singular values of ∆M more uniform and increases the number of preserved components kM (Eq. (6)) denoted by dashed lines. (b) This results in an increased alignment between each task-specific model and merged model measured by SARavg. (c) As alignment increases, the performance also improves as predicted based on the strong correlation between these two properties investigated in Section 3.3."

    The interpolation in Eq. 15 changes only singular values; U and V remain those of ∆TA. The caption explicitly attributes the SAR increase to the growth of k_M under Eq. 6. Since SAR at any fixed k is identical for TA and Iso-C, the monotone SAR curve in Figure 4b is the same by-construction artifact as in Appendix A.2. Claiming that Figure 4 confirms the Section 3.3 correlation therefore double-counts a metric that was defined to move with the spectrum. The performance curve in Figure 4c is real evidence, but the stated mechanism ('as alignment increases, performance improves') is not independently supported by this experiment.

full rationale

The paper's accuracy comparisons are self-contained and measured on held-out data: Iso-C and Iso-CTS beat strong baselines across vision and language benchmarks, and no target accuracy is fitted into the singular-value construction. The circularity is confined to the paper's explanatory claim. The central metric SAR (Eq. 5) uses an adaptive rank k_M (Eq. 6), and Iso-C's only change is to flatten the spectrum while keeping U and V fixed (Eqs. 7-8). Hence the claimed alignment gain over Task Arithmetic is guaranteed by Appendix A.2 Eq. 16: it is a property of the metric, not new geometric alignment. The interpolation experiment in Section 5.4 inherits the same confound. Because the paper uses this alignment increase as the mechanism behind the accuracy gain, part of the central claim reduces by construction. There are no load-bearing self-citations or imported uniqueness theorems; the whitening step cites external work. The honest finding is partial circularity: the method's usefulness is empirically real, but the stated alignment-based explanation is not independently supported.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on SVD of task matrices, a validation-set scalar alpha, and a k/r split chosen from benchmark accuracy. No new physical entities are introduced; SAR is a measurement definition, not a postulated entity. The main unproven axiom is the causal role of SAR in merging success.

free parameters (3)
  • alpha scaling factor for merged task matrices = varies per setting, e.g. 1.30 to 1.90 in Table 5
    Global multiplier applied to the merged task matrix (Eqs. 9 and 14); selected on a held-out validation set for each method, backbone, and task count.
  • common-subspace fraction k/r for Iso-CTS = 0.8
    Default split between common and task-specific subspaces; chosen from the 20-task accuracy curve (Figure 6) and then fixed across all experiments. Robustness within [0.5, 0.9] is shown, but the default is benchmark-selected.
  • epsilon threshold in SAR (Eq. 6) = 0.05
    Defines the effective rank kM by approximation error; an arbitrary tolerance that affects all SAR values and the reported correlation.
assumptions (5)
  • domain assumption Task Arithmetic additivity: the merged model is the pretrained weights plus a scaled sum of task matrices (Eq. 3)
    Inherited from Ilharco et al. (2023); all proposed methods operate within this additive framework.
  • standard math SVD and projection properties: Frobenius norm, projection onto top-k singular vectors, whitening via SVD
    Used in Eqs. 5, 6, 11 and Appendix A; standard linear algebra.
  • domain assumption Singular vectors of task matrices carry task-relevant semantic structure; the dominant common subspace from Task Arithmetic is a valid basis for merging
    Motivated by prior SVD fine-tuning work (MiLORA, SVFT) and by the paper's own SAR analysis; not proven.
  • ad hoc to paper SAR with epsilon=0.05 is a faithful proxy for multi-task interference, so increasing SAR by construction is a valid route to better accuracy
    Introduced in Section 3.3 and Appendix A.2; supported only by correlation and interpolation, not derived.
  • ad hoc to paper Equal task-specific budget s=(r-k)/T is sufficient to capture underrepresented task directions
    Algorithm 2 divides the residual subspace equally among tasks; no complexity weighting; the paper acknowledges weak and non-common tasks may remain underrepresented in Appendix A.4.

how reviews work

0 comments
Cite this review

Pith. "Pith review of No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces." pith.science (2026). https://pith.science/paper/VJVXNYRZ

@misc{pith2026250204959,
  author       = {Pith},
  title        = {Pith review of: No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJVXNYRZ}},
  note         = {Machine review of arXiv:2502.04959}
}
read the original abstract

Model merging integrates the weights of multiple task-specific models into a single multi-task model. Despite recent interest in the problem, a significant performance gap between the combined and single-task models remains. In this paper, we investigate the key characteristics of task matrices -- weight update matrices applied to a pre-trained model -- that enable effective merging. We show that alignment between singular components of task-specific and merged matrices strongly correlates with performance improvement over the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task matrices, enhances alignment, and reduces the performance gap. Additionally, we incorporate both common and task-specific subspaces to further improve alignment and performance. Our proposed approach achieves state-of-the-art performance on vision and language tasks across various sets of tasks and model scales. This work advances the understanding of model merging dynamics, offering an effective methodology to merge models without requiring additional training. Code is available at https://github.com/danielm1405/iso-merging .

Figures

Figures reproduced from arXiv: 2502.04959 by the authors.

Figure 1
Figure 1. Spectrum of singular values for a single layer weight update matrix obtained by merging using Task Arithmetic (top) compared to our approaches: Iso-C (middle) and Iso-CTS (bottom). Task Arithmetic sums the task-specific matrices, which result in a spectrum with a few dominant components. Iso-C instead replaces this spectrum with a uniform one, which results in significant performance improvement. Iso-CTS en￾hances t… view at source ↗
Figure 2
Figure 2. (a) Tasks vectors are typically close to orthogonal to each other. (b) Models with very different normalized accuracy improvements (NAI) exhibit very close cosine similarities, and the correlation between cosine similarity and NAI is low. of the model θt and the pre-trained model θ0 for layer ℓ: ∆ (ℓ) t = θ (ℓ) t − θ (ℓ) 0 . (2) In the rest of the paper, the ℓ superscript is omitted when not relevant to the discussi… view at source ↗
Figure 3
Figure 3. (a) NAI strongly correlates with SARavg (Pearson correlation coefficient ρTA = 0.94). (b) Note the groups of highly aligned tasks such as {MNIST, SVHN, GTSRB} and {EuroSAT, RESISC45}. By comparing (b) and (a), the mutually aligned datasets exhibit higher alignment with the merged model and consequently achieve good performance. On the other hand, tasks with low mutual alignment, such as DTD, Cars, and SUN397, are le… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: (a) Interpolating from ∆TA (β = 0) towards ∆Iso-C (β = 1) makes the spectrum of singular values of ∆M more uniform and increases the number of preserved components kM (Eq. (6)) denoted by dashed lines. (b) This results in an increased alignment between each task-specif…
Figure 5
Figure 5. Figure 5: (a) The directions associated with the least significant singular values of ∆TA have a minor contribution to the performance of Iso-C model. (b) Task-specific directions introduced in Iso-CTS improve the Average Subspace Alignment Ratio (SARavg) between task-specific m…
Figure 6
Figure 6. Figure 6: Iso-CTS is robust to the selected size of the common subspace as any value leads to improvement over Iso-C. These results are for the 20-task scenario. for i > k. Note that the leftmost k directions are the ones associated with the highest singular values of ∆TA. We pl…
Figure 7
Figure 7. Figure 7: Mean L1 distance between the final embeddings of task-specific models and the merged one for Iso-C and TA. We used ViT-B/16 model. D.2. Interference quantification In this Section, we experimentally show that merging interference (defined in Appendix A.3) is lower when…
Figure 8
Figure 8. Figure 8: Performance of Iso-CTS with and without isotropic scaling (Eq.13). Isotropic scaling is a crucial component of Iso-CTS. Results for merging 20 tasks with ViT-B/16. crucial component of Iso-CTS as long as common subspace exists. When only task-specific subspaces are in …
Figure 9
Figure 9. Figure 9: Normalized Accuracy Improvement (NAI) vs. Average Subspace Alignment Ratio (SARavg) for ViT-L/14. E.2. Visualization of per layer Subspace Alignment Ratio In Figure 3a and Figure 4b in the main paper, we presented SARavg – Subspace Alignment Ratio averaged across all t…
Figure 10
Figure 10. Figure 10 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: Validation Accuracy while scaling task matrices with α coefficient (Eq. (3) applied for a single task). We observe a performance gap between the accuracy of original and modified models for the optimal values of α (denoted by square). 21 [PITH_FULL_IMAGE:figures/full…
Figure 12
Figure 12. Figure 12: Visualization of singular value spectra of different task matrices for different types of layers in ViT/B-16. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: Per layer Subspace Alignment Ratio between fine-tuned and merged weight matrices for ViT-B/16. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning

    cs.MM 2026-04 conditional novelty 6.5 of 10

    Localized Muon-style spectral orthogonalization of primary and constraint momentum subspaces (SIFT) reduces objective conflicts and improves constrained LLM steering across four tasks.

  2. DA-MergeLoRA: Hypernetwork-Based LoRA Merging for Few-Shot Test-Time Domain Adaptation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A hypernetwork generates per-column merging weights to combine source LoRA modules on CLIP, achieving state-of-the-art few-shot test-time domain adaptation.

  3. Composable Cross-prompt Essay Scoring by Merging Models

    cs.CL 2025-05 conditional novelty 6.0 of 10

    Merging LoRA adapters with Bayesian-optimized weights guided by a prior-encoded information maximization objective enables source-free cross-prompt essay scoring.

  4. Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmetic

    cs.LG 2025-02 conditional novelty 6.0 of 10

    Task Vector Bases compresses T task vectors into M softmax-mixed basis vectors that preserve task arithmetic operations, with empirical gains over PCA and random selection.

  5. Model Unmerging: Making Your Models Unmergeable for Secure Model Sharing

    cs.LG 2025-09 conditional novelty 5.0 of 10

    MergeLock applies random invertible matrix transformations to Transformer attention weights, preserving the model's own output while forcing any merged model's accuracy down to near random.

  6. Tensorized Clustered LoRA Merging for Multi-Task Interference

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    Clustering training data by embedding similarity and jointly CP-decomposing LoRA adapters cuts multi-task merging interference: +1.4% on Phi-3 and +2.3% on Mistral-7B over SVD baselines.

Reference graph

Works this paper leans on

13 extracted references · 7 canonical work pages · cited by 6 Pith papers

  1. [7]

    We used ViT-B/16 model

    Mean L1 distance between the final embeddings of task-specific models and the merged one for Iso-C and TA. We used ViT-B/16 model. D.2. Interference quantification In this Section, we experimentally show that merging interference (defined in Appendix A.3) is lower when merging is performed with Iso-C than with TA. Following Yang et al. (2024), we measure ...

    work page 2024

  2. [8]

    and Ramakrishna, M

    Vasudevan, V . and Ramakrishna, M. A hierarchical singular value decomposition algorithm for low rank matrices. arXiv preprint arXiv: 1710.02812,

    arXiv

  3. [12]

    We empirically assess the reduced interference of Iso-C compared to Task Arithmetic and analyze the impact of the scaling factor α on our approaches

    for evaluation. We empirically assess the reduced interference of Iso-C compared to Task Arithmetic and analyze the impact of the scaling factor α on our approaches. Finally, we present an ablation study showing what happens when spectrum flattening is applied to each task model individually. D.1. Additional vision baselines In this Section we provide res...

    work page 2021

  4. [13]

    These methods were originally evaluated on checkpoints from Task Arithmetic (Ilharco et al.,

    and CART (Lee et al., 2025). These methods were originally evaluated on checkpoints from Task Arithmetic (Ilharco et al.,

    work page 2025

  5. [1998]

    Adarank: Adaptive rank pruning for enhanced model merging

    Lee, C., Choi, J., Lee, C., Kim, D., and Hong, S. Adarank: Adaptive rank pruning for enhanced model merging. arXiv preprint arXiv: 2503.22178,

    arXiv

  6. [2009]

    Lecun, Y ., Bottou, L., Bengio, Y ., and Haffner, P

    URL https://www.cs.toronto.edu/ ˜kriz/learning-features-2009-TR.pdf . Lecun, Y ., Bottou, L., Bengio, Y ., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE,

    work page 2009

  7. [2011]

    and Zisserman, A

    Nilsback, M.-E. and Zisserman, A. Automated Flower Classification over a Large Number of Classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing,

    work page 2008

  8. [2014]

    Deep Learning for Classical Japanese Literature

    Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K., and Ha, D. Deep Learning for Classical Japanese Literature. arXiv preprint arXiv: 1607.06450,

    arXiv

Show all 13 references
  1. [2018]

    Milora: Harnessing minor singular compo- nents for parameter-efficient LLM finetuning

    Wang, H., Xiao, Z., Li, Y ., Wang, S., Chen, G., and Chen, Y . Milora: Harnessing minor singular compo- nents for parameter-efficient LLM finetuning. CoRR, abs/2406.09044, 2024a. Wang, K., Dimitriadis, N., Ortiz-Jim ´enez, G., Fleuret, F., and Frossard, P. Localizing task info...

    arXiv

  2. [2021]

    Deep model fusion: A survey

    Li, W., Peng, Y ., Zhang, M., Ding, L., Hu, H., and Shen, L. Deep model fusion: A survey. arXiv preprint arXiv: 2309.15698,

    arXiv

  3. [2022]

    12 No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces A. Theoretical properties of Iso-C In this Appendix, we discuss the theoretical properties of Iso-C by explicitly showing the connection between spectral skewness and the increased subspace...

    2025

  4. [2023]

    K., Ghosh, J., Dimakis, A., Choi, E., Bojchevski, A., and Sanghavi, S

    Lingam, V ., Tejaswi, A., Vavre, A., Shetty, A., Gudur, G. K., Ghosh, J., Dimakis, A., Choi, E., Bojchevski, A., and Sanghavi, S. SVFT: parameter-efficient fine-tuning with singular vectors. CoRR, abs/2405.19597,

    arXiv

  5. [2025]

    1 from Gargiulo et al

    performs: – T SVDs per layer on each task matrix (line 1, Alg. 1 from Gargiulo et al. (2025)): O(T Ln3) – Two additional SVDs per layer (lines 10-11, Alg.1 from Gargiulo et al. (2025)): O(2Ln3) Yielding the total complexity: O(TSV) = O(T Ln3 + 2Ln3) = O((T + 2)Ln3) = O(T Ln3) ...

    2025

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.