{"id":"01f8c2c0-4ded-4a39-866a-f92584840e4d","arxiv_id":"2506.09093","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LwPTV prunes low-saliency layers of task vectors before merging, improving OOD accuracy of merged models while roughly maintaining ID accuracy.","lead":"This paper introduces LwPTV, a plug-in that prunes low-saliency layers of task vectors before merging fine-tuned models, pushing the merged model closer to the pretrained weights. A generalist should read it because it offers a simple, training-free way to improve out-of-domain robustness of multi-task model merging, but its gains depend on a hyperparameter chosen on the test sets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core redundancy assumption behind Eqs. (5)-(7) is unproven at the layer level, and the paper's own Table 2 shows ID drops that contradict the abstract's 'preserving ID' claim.","rationale":"The reader's weakest assumption and mine coincide: low cross-task diversity in a layer is treated as evidence of redundancy, but the theoretical support (Proposition 1, inherited from [38]) concerns individual neurons in a simplified model, not whole layers in deep multi-class ViTs. The paper's own results substantiate the risk: on ViT-B/32, applying LwPTV to LW AdaMerging and LW AdaMerging++ reduces ID Average by 2.8 and 4.0 points respectively, which conflicts with the abstract's 'preserving the ability on ID tasks.' The method may still be a useful plug-in, and the reported H-score improvements are consistent across architectures, so the appropriate verdict remains conditional. My proposed random-layer control directly tests whether the saliency criterion, rather than the general act of moving toward pretrained weights, is responsible for the favorable ID/OOD trade-off. Secondary concerns about selecting eta=0.7 on OOD benchmarks and the absence of error bars remain worth addressing in revision but are not the single load-bearing issue.","tokens_in":34268,"tokens_out":12154,"duration_ms":152306,"concrete_test":"On ViT-B/32, run a control in which each task's layer mask is drawn uniformly at random with the same per-task retention count as LwPTV, then OR-aggregated exactly as in Eq. (7), repeated over 10 seeds. Compare ID Average, OOD Average, and H-score against Table 2. If random layer pruning achieves comparable OOD gains with similar or better ID retention, the saliency score is not carrying the claimed ID/OOD trade-off; if random pruning loses substantially more ID accuracy, the redundancy assumption gains direct empirical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LwPTV's central operation zeroes entire layers of all task vectors when the layer-level saliency score in Eq. (5) is low across tasks (Eqs. 6-7), replacing those layers with pretrained weights. Proposition 1 only shows that, in a one-layer binary-classification transformer analyzed in [38], individual discriminative neurons have higher across-task diversity than non-discriminative neurons. It does not imply that a layer with a small mean absolute deviation from the mean task vector contains no discriminative neurons, nor that a uniformly shared layer update is redundant for ID accuracy. The step from neuron-level diversity to layer-level pruning is an unproven leap for deep multi-class ViTs. This is not merely a theoretical gap: Table 2 reports ID Average dropping from 80.1 to 77.3 for LW AdaMerging and from 81.1 to 77.1 for LW AdaMerging++ on ViT-B/32, and Section D.1 concedes that ID performance can be compromised. Without the redundancy assumption, LwPTV's guaranteed effect is to move the merged model toward the pretrained model, which can raise OOD zero-shot accuracy while reducing ID accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LwPTV (Layer-wise Pruning Task Vector), a plug-in preprocessing step for task-vector-based model merging. For each task, LwPTV computes a layer-wise saliency score (mean absolute deviation of the task-vector layer from the across-task mean), thresholds the scores to build a per-task layer mask, OR-aggregates the masks into a shared mask, and forms the merged model as θ_m = θ_pre + m_hat ⊙ Σ_k λ_k τ_k. The authors claim this improves out-of-domain (OOD) accuracy of existing merging methods while preserving in-domain (ID) accuracy, and support the claim with experiments on CLIP ViT-B/32, ViT-L/14, ViT-H/14 and T5-large, across Task Arithmetic, Ties-Merging, AdaMerging variants, and PCB-Merging.","tokens_in":34469,"tokens_out":6087,"duration_ms":66879,"significance":"If the empirical claim holds, LwPTV would be a simple, training-free, and broadly applicable plug-in that improves OOD generalization of task-vector merging, an underexplored axis in the model-merging literature. The paper's strengths are the breadth of the evaluation (multiple architectures, image and text tasks, seven merging baselines), the storage savings reported in Table 15, and the clear algorithmic presentation in Algorithm 1. The significance is tempered, however, by three issues: the headline ID-preservation claim is contradicted by the authors' own Table 2, the theoretical motivation does not bridge from neuron-level diversity to layer-level pruning, and the key hyperparameter η=0.7 is selected on the same OOD benchmarks used to report the gains. These issues do not invalidate the direction of the method but require substantial revision before the claims can be accepted.","major_comments":[{"comment":"The central claim that LwPTV preserves ID performance is contradicted by the paper's own results. On ViT-B/32, LW AdaMerging drops from 80.1 to 77.3 ID average and LW AdaMerging++ drops from 81.1 to 77.1; on ViT-L/14, LW AdaMerging drops from 90.8 to 89.8. Section D.1 concedes that ID performance 'may be compromised' with certain merging techniques. The abstract and conclusion should be reworded to describe an ID/OOD trade-off rather than preservation, or the claims should be restricted to the methods and configurations where ID is actually preserved.","section":"Abstract; §5.1, Table 2; §D.1"},{"comment":"Proposition 1, inherited from the one-layer binary-classification analysis in [38], establishes a neuron-level diversity gap between discriminative and non-discriminative neurons. The proposed method, however, prunes entire layers based on the scalar s_l^k, a layer-level average absolute deviation from the mean task vector. A layer can have a small mean deviation and still contain discriminative neurons, and a layer with large mean deviation can still contain redundant parameters; no argument bridges the two levels. The proof of Proposition 1 also contains inconsistent bounds: Eq. (10) states DV ≥ |S|·Ω(m^{-1}) while the following sentence says the diversity is 'in the order of |S|·Ω(m^{-1/2})'. This theoretical motivation therefore does not support the layer-wise redundancy assumption behind Eqs. (5)-(7).","section":"§4.1, Eq. (5), Proposition 1"},{"comment":"The pruning ratio η=0.7 is selected on the same OOD benchmarks that produce the headline results. Section 5.2 sweeps η in Fig. 5 and states that 'a pruning ratio η between 0.6 and 0.8 offers a relatively desired trade-off,' after which η=0.7 is used for all experiments. No validation split or selection rule is given. Because η directly controls how far the merged model moves toward the pretrained model, the reported OOD gains are partly fitted to the test distribution; the paper should report results for a prespecified η or a selection procedure that does not use OOD labels.","section":"§5, §5.2, Fig. 5"},{"comment":"The main results are reported without error bars or multiple seeds. Some claimed improvements are small (e.g., +0.7 H-score for LW AdaMerging++ on ViT-L/14 and +0.4 H-score for PCB on T5-large), so without variance estimates it is difficult to verify the consistency of the gains on which the H-score claims rest. The manuscript should report standard deviations across at least three seeds for the main tables, or at least for the smallest increments.","section":"Table 2"},{"comment":"For the NLP experiments the method is changed from layer-wise pruning to parameter-level pruning ('for NLP tasks, we perform parameter pruning at the parameter level'), but no saliency score or mask construction for parameter-level pruning is defined. The T5-large results therefore do not evaluate the method introduced in Section 4, and the claimed generality across modalities is not substantiated by the reported experiments.","section":"§B.16, Table 17"}],"minor_comments":[{"comment":"The summation index k is reused as the outer task index; the mean should be over a different index, e.g., Σ_j τ_l^j.","section":"Eq. (5)"},{"comment":"The threshold sorted(s_k)[⌊L·η⌋] is undefined for η=0 and for ties among saliency scores; the convention should be stated.","section":"Eq. (6)"},{"comment":"Table 2 has formatting errors (e.g., '55.89.075.4' and '60.660.9+9.6'); the table needs to be regenerated so that all entries are legible.","section":"Table 2"},{"comment":"There are several typos: 'repsectively', 'Surgey', 'purning', and 'wether' should be corrected.","section":"Section 5"},{"comment":"LwPTV is marked 'No Test set', but when combined with AdaMerging variants the algorithm uses entropy minimization on unlabeled test data; the table should clarify that the 'No Test' property applies only to the standalone mask construction.","section":"Table 1"},{"comment":"The scaling rule λ̂_l^k = η·λ_l^k reuses η, the pruning ratio, as a coefficient-scaling factor without discussion; this conflation should be motivated or renamed.","section":"§B.1"}],"recommendation":"major_revision","confidential_remarks":"The motivating theoretical result is taken almost verbatim from [38], which shares two co-authors with the present paper (Hongkang Li and Meng Wang). Proposition 1 is essentially a restatement of Corollary 2 and Lemma 5 of [38] in the same one-layer binary setting, so the theoretical contribution is thin. This is not by itself a reason to reject, but the editor may want to verify the novelty disclosure and ask the authors to state the dependence explicitly. Also, the checklist states that code has been released, but the arXiv submission does not include a code repository; please confirm the code availability for any camera-ready version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: LwPTV is a plug-in layer-wise masking method for task-vector merging that consistently raises H-scores across ViT-B/32, ViT-L/14, ViT-H/14, and T5-large. It is a genuine but incremental contribution: DARE, Ties, MWP, LiNeS all prune or scale task vectors, but the layer-wise cross-task diversity score plus OR-aggregated shared mask is a new criterion and it is cleanly described.\n\nThe empirical work is the strong part. Eight ID datasets, thirteen OOD datasets, three CLIP scales, T5 NLP, ablations, comparison against DARE/MWP/random mask/WiSE-FT/Model Stock/LiNeS, storage and time overhead. The H-score gains are consistent across nearly every baseline and architecture. I believe the direction is real.\n\nThe soft spots are in proportion. First, the abstract claims 'preserving the ability on ID tasks,' but Table 2 shows LW AdaMerging ID dropping from 80.1 to 77.3 and LW AdaMerging++ from 81.1 to 77.1 on ViT-B/32. The paper's own limitation section D.1 concedes this. That is an overclaim, not a fatal flaw, but it should be fixed.\n\nSecond, the theoretical motivation is thinner than it looks. Proposition 1 is essentially a restatement of a neuron-level result from reference [38], a paper with overlapping authors, and the leap from neuron diversity to layer-level pruning is not proven. The stress-test concern is valid: a layer with low average deviation could still contain discriminative neurons, and pruning it may remove shared features that matter for ID. That said, the theory is motivation, not the load-bearing part; the ablations show the saliency score outperforms magnitude and random masking, so the criterion does more than just nudge the model toward the pretrained weights.\n\nThird, the main hyperparameter eta=0.7 is selected on the same OOD benchmarks that produce the headline numbers, and no error bars are reported. This makes the precise gains uncertain. The consistency across many settings mitigates this, but it is still a real weakness.\n\nMy recommendation: yes, send to peer review. The method is useful, the experiments are extensive, and the issues are addressable: soften the abstract, report error bars, and either justify eta via validation or show sensitivity across a range. The theory section could be trimmed or reframed as intuition. A serious referee would be able to work with this.","headline":"Useful plug-in for OOD-aware model merging with consistent H-score gains, but the ID-preservation claim is overstated and the theory is mostly borrowed.","tokens_in":35043,"tokens_out":4107,"would_cite":true,"duration_ms":43357,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that pruning low-diversity task-vector layers and replacing them with pretrained weights improves out-of-distribution generalization of merged models while preserving in-domain accuracy.","keywords":["model merging","task vectors","layer-wise pruning","out-of-distribution generalization","multi-task learning","saliency score","CLIP vision transformers","plug-and-play merging"],"falsifier":"Compute each layer's saliency score on the eight benchmark tasks and also measure the drop in ID accuracy caused by forcing that layer to the pretrained weights; if any low-saliency layer shows a positive accuracy contribution, low cross-task diversity does not imply redundancy. The result would be strongest on a pair of tasks known to share a transferable visual feature, such as two datasets with similar textures, where pruning the shared low-diversity layer should preserve ID accuracy if the paper's claim is correct.","tokens_in":34037,"feed_emoji":"✂️","tokens_out":8293,"duration_ms":90796,"temperature":0.7,"pith_summary":"Model merging combines several fine-tuned models into one multi-task model, but existing methods are tuned for in-domain accuracy and often lose the pretrained model's out-of-distribution robustness. This paper claims that the lost robustness can be recovered by pruning task vectors layer-wise: layers whose updates are nearly the same across all tasks are treated as redundant and replaced with the original pretrained weights. The proposed LwPTV computes a per-layer saliency score from the deviation of each task's task vector from the cross-task mean, masks low-diversity layers, ORs the per-task masks so a layer is kept if any task needs it, and plugs the resulting mask into existing merging formulas. If correct, this gives a training-free, plug-and-play way to improve OOD generalization of Task Arithmetic, Ties-Merging, AdaMerging, and PCB-MERGING on CLIP and T5 models while largely preserving in-domain accuracy, as measured by the H-score, the harmonic mean of average ID and OOD accuracy.","feed_headline":"Pruning low-diversity task-vector layers lifts OOD accuracy of merged models","feed_subtitle":"Pruning low-diversity task-vector layers lifts OOD accuracy without hurting in-domain tasks.","key_machinery":"The central object is the layer-wise saliency score $s_l^k = \\mathbb{E}[|\\tau_l^k - \\tfrac{1}{K}\\sum_{j=1}^K \\tau_l^j|]$, which measures how far one task's update at a given layer deviates from the average update across tasks. A high score is interpreted as evidence that the layer carries task-specific discriminative information; a low score means the layer is redundant, so the merged model keeps the pretrained weights there instead of the fine-tuned delta. The per-task masks are constructed by thresholding these scores, and the OR-aggregated shared mask ensures that a layer is removed only if all tasks agree to remove it. The theoretical motivation is Proposition 1, which the paper derives from a cited one-layer Transformer analysis and uses to argue that discriminative neurons vary more across tasks than non-discriminative neurons, making diversity a valid pruning criterion.","core_discovery":"The paper's central claim is that task-vector parameters that are consistently close to the cross-task average are redundant: they do not encode task-discriminative features, so substituting the pretrained weights in those layers improves OOD generalization without hurting in-domain performance. To realize this, LwPTV defines a layer-wise saliency score for task $k$ and layer $l$ as $s_l^k = \\mathbb{E}[|\\tau_l^k - \\tfrac{1}{K}\\sum_{j=1}^K \\tau_l^j|]$, where $\\tau_k = \\theta_k - \\theta_{\\mathrm{pre}}$ is the task vector. Layers with scores below a per-task threshold are masked out, and a shared mask $\\hat{m} = m^1 \\lor \\cdots \\lor m^K$ is formed by OR-ing the per-task masks, so a layer is pruned in the merged model only when every task agrees it is redundant. The merged model is then $\\theta_m = \\theta_{\\mathrm{pre}} + \\hat{m} \\odot \\sum_k \\lambda_k \\tau_k$, with an additional coefficient scaling when combined with layer-wise AdaMerging. Across ViT-B/32, ViT-L/14, ViT-H/14, and T5-large, the paper reports H-score gains over Task Arithmetic, Ties, AdaMerging variants, and PCB-MERGING, with ID accuracy preserved or improved in most image experiments and some acknowledged ID loss for LW AdaMerging on ViT-B/32.","pith_inferences":["An extension the paper leaves implicit: the saliency criterion predicts that tasks with highly correlated fine-tuning updates will benefit most from pruning, so one could test this by merging deliberately related tasks and checking whether the OOD gain grows while the ID loss stays flat.","The theoretical support is proven for one-layer Transformers on binary classification; extending Proposition 1 to deep ViTs and multi-class tasks is an assumption, and a per-layer ablation of ID accuracy against saliency ordering would show whether the extension holds.","The NLP results in the appendix suggest the direction of the method may be domain-dependent: in language, where merged models already beat the pretrained model on OOD benchmarks, the paper switches to parameter-level pruning, implying that layer-wise pruning is a tool for rebalancing pretrained versus task-specific knowledge rather than a universal law."],"forward_implications":["On CLIP ViT-B/32, Task Arithmetic with LwPTV raises the H-score from 58.9 to 66.3 by improving average OOD accuracy from 51.3 to 60.9 while also raising average ID accuracy from 69.1 to 72.8.","On the same architecture, LwPTV combined with LW AdaMerging raises the H-score from 67.1 to 68.6, lifting average OOD accuracy from 57.7 to 61.7 at the cost of a 2.8-point drop in ID accuracy from 80.1 to 77.3.","On ViT-L/14, LwPTV improves H-scores for Task Arithmetic, Ties, TW AdaMerging, LW AdaMerging, and PCB-MERGING by between 0.7 and 2.1 points.","On T5-large, LwPTV raises the H-score of Task Arithmetic from 54.9 to 57.9, including a 3.0-point gain in average ID accuracy.","The method is training-free: mask computation costs $O(KL d_l)$ time, the ViT-B/32 merge adds under a second, and storage for Task Arithmetic drops from 432.77 MB to 226.75 MB on ViT-B/32."],"supporting_citations":[{"why":"Supplies the task-vector generalization analysis of one-layer Transformers that motivates the diversity-based pruning criterion and Proposition 1.","marker":"[38]"},{"why":"Defines task vectors and the Task Arithmetic merging formula that LwPTV plugs into and improves.","marker":"[27]"},{"why":"Provides the Ties-Merging baseline with its own task-vector trimming, which LwPTV is combined with and compared against.","marker":"[69]"},{"why":"Provides the AdaMerging baselines with layer-wise coefficients, motivating the coefficient-scaling component when masks are applied.","marker":"[71]"},{"why":"Provides the PCB-MERGING baseline that balances parameter competition and is one of the methods LwPTV is shown to improve.","marker":"[20]"},{"why":"Provides the WiSE-FT baseline for improving OOD robustness of fine-tuned models, used for comparison.","marker":"[66]"},{"why":"Provides the LiNeS baseline that scales fine-tuned updates layer-wise and is used as an OOD-focused comparison.","marker":"[62]"},{"why":"Provides the DARE baseline that drops and rescales task-vector entries, the pruning competitor LwPTV outperforms.","marker":"[72]"}],"fun_headline_variants":["Prune low-diversity task-vector layers to boost OOD merging","LwPTV masks redundant task-vector layers for OOD gains","Mask common task-vector layers to improve OOD model merges","Layer-wise task-vector pruning enhances OOD generalization","Drop redundant task-vector layers, keep ID, lift OOD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a layer whose task-vector parameters are close to the average across tasks carries no task-specific information, so replacing that layer with the pretrained weights cannot hurt in-domain performance.","fun_headline_variants_meta":{"raw":{"variants":["Prune low-diversity task-vector layers to boost OOD merging","LwPTV masks redundant task-vector layers for OOD gains","Mask common task-vector layers to improve OOD model merges","Layer-wise task-vector pruning enhances OOD generalization","Drop redundant task-vector layers, keep ID, lift OOD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000278,"raw_usage":{"total_tokens":1714,"prompt_tokens":1065,"completion_tokens":649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":559}},"tokens_in":681,"tokens_out":649,"duration_ms":7206,"temperature":1.0,"reasoning_tokens":559,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:03:35.474140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute each layer's saliency score on the eight benchmark tasks and also measure the drop in ID accuracy caused by forcing that layer to the pretrained weights; if any low-saliency layer shows a positive accuracy contribution, low cross-task diversity does not imply redundancy. The result would be strongest on a pair of tasks known to share a transferable visual feature, such as two datasets with similar textures, where pruning the shared low-diversity layer should preserve ID accuracy if the paper's claim is correct.","supporting_citations":[{"cited_title":"When is task vector provably effective for model editing? a generalization analysis of nonlinear transformers","cited_arxiv_id":null,"evidence_quote":"Supplies the task-vector generalization analysis of one-layer Transformers that motivates the diversity-based pruning criterion and Proposition 1."},{"cited_title":"Editing models with task arithmetic","cited_arxiv_id":null,"evidence_quote":"Defines task vectors and the Task Arithmetic merging formula that LwPTV plugs into and improves."},{"cited_title":"Resolving interference when merging models","cited_arxiv_id":null,"evidence_quote":"Provides the Ties-Merging baseline with its own task-vector trimming, which LwPTV is combined with and compared against."},{"cited_title":"Adamerging: Adaptive model merging for multi-task learning","cited_arxiv_id":null,"evidence_quote":"Provides the AdaMerging baselines with layer-wise coefficients, motivating the coefficient-scaling component when masks are applied."},{"cited_title":"Parameter competition balancing for model merging","cited_arxiv_id":null,"evidence_quote":"Provides the PCB-MERGING baseline that balances parameter competition and is one of the methods LwPTV is shown to improve."},{"cited_title":"Robust fine-tuning of zero-shot models","cited_arxiv_id":null,"evidence_quote":"Provides the WiSE-FT baseline for improving OOD robustness of fine-tuned models, used for comparison."},{"cited_title":"LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging","cited_arxiv_id":"2410.17146","evidence_quote":"Provides the LiNeS baseline that scales fine-tuned updates layer-wise and is used as an OOD-focused comparison."},{"cited_title":"Language models are super mario: Absorbing abilities from homologous models as a free lunch","cited_arxiv_id":null,"evidence_quote":"Provides the DARE baseline that drops and rescales task-vector entries, the pruning competitor LwPTV outperforms."}],"review_version":1}