{"id":"9e862456-e89e-4638-8c08-d506baeba2c3","arxiv_id":"2502.04959","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Uniformly rescaling singular values of summed task updates, plus adding task-specific residual directions, improves multi-task model merging.","lead":"This paper introduces a way to combine several fine-tuned AI models into one model without retraining, by smoothing out the differences between their weight updates. The method makes the merged model nearly as accurate as the individual specialists, and it tops prior merging methods on vision and language benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Iso-C's reported SAR increase is inherited from the adaptive effective rank k_M, not from new alignment: with a fixed k, Iso-C and Task Arithmetic have identical top-k subspaces, so the correlation claimed in §3.3 cannot support the causal story.","rationale":"The paper is a strong empirical contribution: Iso-C/Iso-CTS deliver consistent gains across backbones and modalities, with code and a clean SVD-based procedure. My concern targets the explanatory claim, not the method. Appendix A.2 already concedes that the SAR increase is partly definitional; the stronger observation is that the increase is entirely definitional for Iso-C, because the singular vectors are unchanged and only k_M changes. This makes the ρ=0.94 correlation and the β-interpolation evidence confounded by the rank-selection rule. A fixed-k recomputation would settle the issue in less than an hour on the released code. If the fixed-k result is flat, the paper needs revision but the method may still be accepted on empirical grounds; hence I keep the reader's CONDITIONAL verdict rather than escalating to REJECT. No variance or error-bar issue changes this assessment.","tokens_in":24059,"tokens_out":11166,"duration_ms":115586,"concrete_test":"Recompute Fig. 3a and Fig. 4b/c on ViT-B/16 with a fixed projection rank k for both TA and Iso-C (e.g., k = k_TA from Eq. 6, and k = 50 as a robustness check). Since Iso-C does not change singular vectors, SAR_fixed(Iso-C) will equal SAR_fixed(TA) for every task up to numerical precision. If NAI still improves from TA to Iso-C while SAR_fixed is flat, the reported alignment increase is not the driver. For Iso-CTS, additionally compare SAR with k fixed against the method's default adaptive k_M; if task-specific directions improve SAR even at fixed k, that part of the explanation survives. The paper should either report these rank-normalized numbers or explicitly reframe the contribution as an empirical merging method without the causal alignment claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Iso-C replaces the singular values of Δ_TA by their mean (Eq. 7-8), leaving U and V unchanged. The SAR metric (Eq. 5) uses k_M selected by the 5% reconstruction criterion (Eq. 6). As the spectrum is flattened, k_M grows until the projection subspace covers almost all of R^m; for a flat spectrum k_M ≈ 0.9975r, so SAR approaches 1 for any task matrix Δ_t. Appendix A.2 (Eq. 16) proves that this SAR increase is by construction, because the top-k subspace of TA is contained in the larger top-k_M subspace of Iso-C. More sharply: for any fixed k, the top-k subspaces of Δ_TA and Δ_Iso-C are identical, since the singular vectors are unchanged. Thus SAR at fixed k is exactly equal for TA and Iso-C on every task, and the entire reported alignment gain is an artifact of letting k_M vary with the spectrum. The interpolation experiment in §5.4 (Fig. 4) is subject to the same confound: as β grows, k_M grows, so SAR rises even though the singular vectors are fixed. Consequently, the central explanatory claim—that increased subspace alignment drives the accuracy gain—is not supported by the presented metric. The accuracy improvements themselves are real and independently measured, but they cannot be attributed to the measured alignment increase.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two training-free model merging methods: Iso-C, which flattens the singular value spectrum of the Task Arithmetic sum matrix, and Iso-CTS, which augments the common subspace with per-task directions before isotropic scaling. The central explanatory claim is that the Subspace Alignment Ratio (SAR) between task-specific and merged matrices strongly correlates with performance, and that isotropic merging improves both SAR and accuracy. The methods are evaluated on vision benchmarks (8/14/20 tasks with ViT-B/32, ViT-B/16, ViT-L/14), LoRA fine-tuned vision models, and T5 language models, reporting state-of-the-art results against Task Arithmetic, TIES, Consensus TA, TSV-M, and other baselines.","tokens_in":24410,"tokens_out":7616,"duration_ms":69751,"significance":"If the results hold, the paper offers a simple, training-free merging technique that substantially reduces the gap to single-task models, and it demonstrates effectiveness across vision, LoRA, and language settings. The approach is computationally cheap for Iso-C (one SVD per layer), reproducible code is provided, and the ablation study in Section 5.4 plus the appendix derivations give useful insight into spectral properties of merged matrices. However, the explanatory mechanism is currently not supported as stated: the measured SAR increase from Task Arithmetic to Iso-C is largely a consequence of the metric's dependence on the effective rank k_M, not an independent improvement in directional alignment. The empirical accuracy gains are real and consistently measured, but the causal story in the abstract and Section 3.3 needs either stronger evidence or a reframed claim.","major_comments":[{"comment":"The claim that Iso-C 'enhances alignment' and that this drives the performance gain is not supported by the SAR metric because the metric is not invariant to the spectral flattening. Iso-C replaces the singular values of Δ_TA by their mean (Eq. 7-8) but leaves U and V unchanged; therefore, for any fixed k, the top-k projection subspaces of Δ_TA and Δ_Iso-C are identical. The inequality in Eq. (16) follows solely from k_Iso-C > k_TA, which is a direct consequence of the threshold in Eq. (6). Consequently, the SAR increases in Fig. 4b and Fig. 5b measure the growth of k_M, not any improvement in the overlap between the task matrix and the merged matrix's span. To rescue the causal explanation, please report SAR with a fixed k (e.g., k = k_TA for both methods) or use a metric that does not depend on singular value magnitudes, such as principal angles or fixed-rank projections; alternatively, reframe the contribution as demonstrating that spectrum flattening itself improves merging, without attributing the gain to increased subspace alignment.","section":"Section 3.3, Appendix A.2, Eqs. (5)-(8), (16)"},{"comment":"The definition s = (r-k)/T requires this quantity to be an integer because U* and V* are constructed by concatenating k common and T·s task-specific columns to form matrices in R^{m×r} and R^{n×r}. For the default setting k/r = 0.8, a typical square layer of ViT-B/16 has r = 768, and for T = 20 this yields s = 7.68, which is not an integer. The paper does not specify how to handle non-divisible (r-k)/T (e.g., floor/ceil, dropping leftover directions, or adjusting k). This affects the exact definition of the method used to produce Table 1 and must be clarified for reproducibility.","section":"Section 4.2, Algorithm 2"}],"minor_comments":[{"comment":"No variance or number of seeds is reported. Although the merging itself is deterministic given the checkpoints, the fine-tuning and hyperparameter selection involve randomness; please report seeds or variance, especially for close comparisons such as Iso-C vs TSV-M on ViT-B/32 with 14 tasks.","section":"Tables 1 and 2"},{"comment":"The Pearson correlation ρ = 0.94 is computed on only 8 points; please provide a p-value or confidence interval to assess the strength of the correlation.","section":"Figure 3a"},{"comment":"The sentence 'both the left- and right-singular vectors of Δ_t ... are orthogonal to the subspace spanned by the common directions' is not correct for right-singular vectors; the projection in Eq. (10) only ensures orthogonality of the left singular vectors to U^{1:k}. The subsequent whitening handles the right side, but the text should be corrected.","section":"Section 4.2, after Eq. (10)"},{"comment":"In both NLP settings Iso-C slightly outperforms Iso-CTS (75.6 vs 75.2 and 83.3 vs 82.8); the sentence 'Iso-CTS achieves very similar results to Iso-C suggesting that the common space captures all the directions necessary' should be adjusted to reflect that Iso-C is numerically better.","section":"Section 5.3, Table 3"},{"comment":"The baseline is introduced as 'Task Singular Vectors (TSV)' in Section 2 but compared as TSV-M in Table 1; please define TSV-M explicitly when it first appears in a table.","section":"Related work, Table 1"},{"comment":"The choice of ε = 0.05 in the effective-rank criterion is not discussed; a sensitivity analysis of the SAR metric to this threshold would strengthen the claim that the metric is robust.","section":"Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The empirical results are strong and the method is clearly presented, but the central explanatory narrative in the abstract and Section 3.3 is currently not supported because the SAR gain from Task Arithmetic to Iso-C is definitional. The authors have been transparent in Appendix A.2 about the construction, which is a positive sign, but the claims need to be reframed or supplemented with a fixed-rank alignment measure. The integer-divisibility issue in Algorithm 2 is a concrete reproducibility problem that should be straightforward to fix. If the authors address these points, the paper could be acceptable; there is no concern about novelty or scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The empirical core is real: replacing the singular values of the summed task matrix by their mean, then optionally adding task-specific directions, gives state-of-the-art merging results across three ViT scales, LoRA models, and T5. That is a useful, training-free recipe, and the benchmark coverage is genuinely broad. The paper also does a few things convincingly well: the ablation in Figure 5a shows that the low singular directions of the task-arithmetic matrix contribute little, and Appendix D.5 shows that flattening individual task matrices hurts, which strongly suggests the gain comes from the merging operation itself, not from per-task improvement. The method is simple enough to be adopted quickly, and code is provided.\n\nThe soft spot is the explanatory story. The stress-test note is correct, and the paper half-admits it in Appendix A.2: the SAR increase for Iso-C is by construction, because the effective rank k_M grows as the spectrum flattens while the singular vectors stay identical to Task Arithmetic. For any fixed k, the top-k subspaces of Δ_TA and Δ_Iso-C are exactly the same, so the SAR metric at fixed k is unchanged. That means the correlation plotted in Section 3.3 and the monotone SAR increase in Figure 4b are mostly a re-statement of \"we keep more components of the projection subspace,\" not evidence that alignment causes the accuracy gain. The accuracy improvements themselves are independently measured and robust; they just cannot be attributed to the measured SAR increase. The paper should either fix k when comparing SAR, or drop the causal language and present isotropic scaling as a heuristic that works, with the SAR metric as descriptive only.\n\nOther issues are minor by comparison: no error bars or repeated runs anywhere, and the hyperparameters (alpha, k/r) are selected on the same benchmarks, though Figure 6 shows the method is reasonably robust to k/r. The LoRA results are strong but computed on reconstructed rank-16 matrices; still, they are honest apples-to-apples comparisons.\n\nWho should read this: practitioners who want a cheap, effective merging method, and researchers interested in why merging works — they should be wary of the metric-based argument. The paper deserves a serious referee, but the authors should be pushed to reframe the SAR claim or provide a fixed-rank version of the analysis. I would cite the empirical finding, not the metric.","headline":"A genuinely simple and effective merging recipe, but the SAR-based explanation for why it works is confounded and should not be the sales pitch.","tokens_in":24907,"tokens_out":2159,"would_cite":true,"duration_ms":23334,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","15A18"],"pacs":[],"model":"deepseek-v4-flash","headline":"Merged multitask models get closer to single-task accuracy by flattening the singular spectrum of their summed weight updates.","keywords":["model merging","task arithmetic","singular value decomposition","subspace alignment","isotropic merging","multi-task learning","LoRA","spectrum flattening"],"falsifier":"Merge 8 or 20 tasks with Iso-CTS, then run a control that keeps the same rank and the same isotropic scaling but replaces the task-specific singular vectors with randomly chosen orthonormal vectors orthogonal to the common subspace; if the random-direction control matches Iso-CTS accuracy, the content of the task-specific directions is not what matters and the alignment explanation is in doubt. A second check: if a task's dominant directions are orthogonal to the common subspace, adding them should specifically rescue that task, whereas adding directions from a different task would not.","tokens_in":23887,"feed_emoji":"🧩","tokens_out":6274,"duration_ms":55903,"temperature":0.7,"pith_summary":"This paper tries to explain why merging several fine-tuned models into one multi-task model still loses accuracy, and to close that gap without retraining. It shows that a task's performance in the merged model tracks how well the task's weight-update matrix aligns with the merged weight-update matrix, measured by a new quantity, the Subspace Alignment Ratio. On that evidence it proposes two merging recipes: Iso-C, which replaces the skewed singular-value spectrum of the summed task updates with a flat one, and Iso-CTS, which additionally keeps a few singular directions from each task. On 8, 14 and 20 vision tasks, and on LoRA-tuned vision models and T5 language models, the recipes outperform existing merging methods and shrink the gap toward the per-task models.","feed_headline":"Flattening singular spectra narrows the model-merging gap","feed_subtitle":"Subspace alignment predicts merged accuracy; flattening the spectrum exploits it to beat prior merging on vision and language.","key_machinery":"The load-bearing objects are the task matrix $\\Delta_t = \\theta_t - \\theta_0$ (the weight update from fine-tuning), its singular value decomposition, and the Subspace Alignment Ratio $\\text{SAR}(\\Delta_t, \\Delta_M; k_M)$, which projects $\\Delta_t$ onto the top $k_M$ left-singular vectors of the merged matrix and reports the fraction of $\\Delta_t$'s magnitude retained. The rank $k_M$ is chosen automatically as the smallest rank whose reconstruction error falls below 5 percent of the total energy, so a flatter singular spectrum automatically yields a larger effective subspace. Iso-C replaces the skewed spectrum of the summed matrix $\\Delta_{\\text{TA}}$ with a single average singular value $\\bar{\\sigma}$, reconstructing $\\bar{\\sigma} U V^\\top$; Iso-CTS concatenates top-$k$ common directions with per-task directions taken from the residual $\\Delta_t - U_{1:k}U_{1:k}^\\top\\Delta_t$, whitens the combined basis, and scales the result by the average of the selected singular values. The same machinery serves as an analysis tool: the paper uses SAR to identify which tasks are underrepresented and to predict which ones will improve.","core_discovery":"On the paper's own terms, the central discovery is that the singular-value structure of the summed task matrix, not just pairwise cosine similarity between task vectors, governs how well a merged model preserves each task. The authors define the Subspace Alignment Ratio as the fraction of a task matrix's total magnitude captured by projection onto the dominant left-singular subspace of the merged matrix, and report a Pearson correlation of 0.94 between the average of this ratio over layers and normalized accuracy improvement. They then show that flattening the singular values of the Task Arithmetic sum to their mean, leaving the singular vectors untouched and rescaling by a single coefficient $\\alpha$, raises the alignment and with it accuracy; adding per-task directions that lie orthogonal to the common subspace, after whitening the combined basis, extends the gains to larger task sets. The resulting Iso-CTS method reaches state-of-the-art merged accuracy on vision benchmarks at three model scales, on LoRA-adapted models, and on T5 language tasks, with no additional training.","pith_inferences":["If alignment is truly causal, merging quality could be optimized directly by choosing the merged subspace that maximizes total alignment across tasks, rather than by flattening the spectrum and adding residuals; that is a tractable, testable objective.","The result reframes post-hoc merging: spectral skewness of the summed updates, not only parameter sign conflicts, drives interference, so sign-trimming methods and spectrum flattening address different mechanisms and may be complementary.","Because the method only needs the final weight matrices, it should apply to other architectures and modalities, such as CNNs or diffusion models, as long as their weight updates are matrices; the paper tests transformer vision and text backbones only.","A natural extension is to replace the fixed $\\epsilon=0.05$ rank rule and the uniform mean scaling with per-layer or per-task weights derived from validation accuracy, which could preserve the alignment mechanism while beating the uniform isotropic choice."],"forward_implications":["Iso-C and Iso-CTS require no retraining and no data beyond a validation set, so they apply to any family of models fine-tuned from a shared checkpoint.","As the number of tasks grows, retaining task-specific directions matters more: Iso-CTS beats Iso-C by up to 2.8 percent absolute accuracy when going from 8 to 20 tasks.","The recipe transfers to low-rank (LoRA) adapters by first reconstructing $\\Delta W_t = B_t A_t$, and outperforms LoRA-specific merging methods.","Spectrum flattening does not help individual single-task models (it hurts them by 3.3 percent), so the gain comes from counteracting interference in summed task matrices, not from improving the task matrices themselves.","On T5 language models, both Iso variants beat prior merging methods, showing the effect is not vision-specific."],"supporting_citations":[{"why":"Defines task vectors and Task Arithmetic merging, the starting point and main baseline that the isotropic methods build on and compare against.","marker":"(Ilharco et al., 2023)"},{"why":"Introduces TIES merging, a sign-conflict baseline that Iso-C and Iso-CTS outperform on vision and language benchmarks.","marker":"(Yadav et al., 2023)"},{"why":"Consensus Merging supplies the 8, 14 and 20-task vision benchmarks and the fine-tuned CLIP checkpoints used in the main experiments.","marker":"(Wang et al., 2024b)"},{"why":"Task Singular Vectors is the closest SVD-based merging method, providing the whitening step and the strongest vision competitor.","marker":"(Gargiulo et al., 2025)"},{"why":"KnOTS provides the LoRA merging evaluation protocol and checkpoints used in the low-rank experiments.","marker":"(Stoica et al., 2025)"},{"why":"MaTS defines the NLP fine-tuning setup and T0 task mixtures used for the T5 language-model experiments.","marker":"(Tam et al., 2023)"},{"why":"CLIP is the pre-trained vision backbone on which the vision merging experiments are built.","marker":"(Radford et al., 2021)"},{"why":"LoRA provides the low-rank adaptation formalism whose reconstructed task matrices are merged in the LoRA experiments.","marker":"(Hu et al., 2021)"}],"fun_headline_variants":["Flatten spectra, align subspaces, merge better","Subspace alignment ratio predicts model merging gains","Iso-CTS: common plus task-specific subspaces for merging","Flatten spectra to reduce model-merging gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that the Subspace Alignment Ratio, computed with an automatically chosen effective rank, is a causal driver of merged-model accuracy, so that any operation that raises alignment, such as flattening the spectrum, is the right cure; the support is a strong correlation and interpolation experiments, not a proof that alignment itself causes the gains.","fun_headline_variants_meta":{"raw":{"variants":["Flatten spectra, align subspaces, merge better","Subspace alignment ratio predicts model merging gains","Iso-CTS: common plus task-specific subspaces for merging","Flatten spectra to reduce model-merging gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00099,"raw_usage":{"total_tokens":4186,"prompt_tokens":925,"completion_tokens":3261,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":3198}},"tokens_in":541,"tokens_out":3261,"duration_ms":22606,"temperature":1.0,"reasoning_tokens":3198,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T20:49:40.522301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Merge 8 or 20 tasks with Iso-CTS, then run a control that keeps the same rank and the same isotropic scaling but replaces the task-specific singular vectors with randomly chosen orthonormal vectors orthogonal to the common subspace; if the random-direction control matches Iso-CTS accuracy, the content of the task-specific directions is not what matters and the alignment explanation is in doubt. A second check: if a task's dominant directions are orthogonal to the common subspace, adding them should specifically rescue that task, whereas adding directions from a different task would not.","supporting_citations":[],"review_version":1}