{"id":"c9589b4c-f404-4079-aa7a-aa243de28e5c","arxiv_id":"2505.18713","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Neural Parameter Search (NPS) prunes fine-tuned models by evolutionary reweighting of magnitude-based task vector subspaces, improving transfer, fusion, and compression.","lead":"This paper introduces a pruning method for fine-tuned models that reweights magnitude-based subspaces of the task vector using an evolutionary search, then combines the pruned model with the pre-trained model for transfer, fusion, and compression. It reports consistent but modest gains across vision, NLP, and multimodal benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NPS gains may reflect its larger validation-set search budget (CMA-ES on subspace weights and fusion coefficients) rather than the subspace-reweighting mechanism itself; matched-budget controls are needed to support the central claim.","rationale":"The reader's verdict is CONDITIONAL, citing the method's reliance on calibration-set tuning and lack of error bars. My concern is more specific: the paper does not establish that NPS's mechanism—rather than its larger search budget—produces the gains. NPS uses CMA-ES to optimize per-subspace weights and fusion coefficients on the validation set, while baselines receive only a small grid search. This is a confound that could explain all the reported improvements. It is not an internal inconsistency but a threat to the attribution of the central claim. The proposed control experiment (giving baselines the same CMA-ES budget) would settle it. The 4.3% T5-base discrepancy is also worth flagging, but it is a reporting error rather than a challenge to the method's validity. Since the concern is addressable and the paper otherwise presents consistent (if small) gains, the CONDITIONAL verdict remains appropriate; no change in verdict is needed, but the conditions should include matched-budget baselines and error bars.","tokens_in":23240,"tokens_out":7554,"duration_ms":65348,"concrete_test":"Run a matched-budget comparison on the ViT-B/32 8-task fusion setting (Section 4.4): use CMA-ES to optimize TIES's hyperparameters and/or its per-parameter mask (or DARE's drop ratio and rescale) on the same calibration set used for NPS, with the same number of generations and validation evaluations; report mean and standard deviation across 5 seeds. If optimized baselines match or beat NPS's 76.5 average, the claimed advantage is attributable to search budget rather than the NPS subspace-reweighting mechanism. Additionally, recompute the T5-base '4.3% performance gain' claimed in the introduction from Table 2; if the maximum gain over any baseline is +2.1 percentage points, the introduction overstates the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that NPS's magnitude-subspace reweighting improves pruning for transfer, fusion, and compression. However, NPS's gains may be an artifact of its larger validation-set search budget rather than the proposed mechanism. In §3.2, CMA-ES optimizes per-subspace weights w_m against calibration-set accuracy, with 20–50 generations (Table 13), and in Eq. 7 the fusion coefficients λ_i are also evolved. Baselines (TIES, DARE, Task Arithmetic) are only grid-searched over r and λ (Table 12) and do not receive a comparable black-box optimizer. Because the search directly maximizes validation accuracy of the pruned/merged model, NPS has more capacity to fit the validation set. The reported gains of +0.9 to +3.0 percentage points over TIES (Table 2) could therefore reflect validation overfitting or search budget rather than the specific scalar-reweighting hypothesis. The paper does not include a control where baselines are optimized with the same evolutionary budget, nor does it report error bars (except one five-seed emotion experiment), so the small margins are hard to interpret. This confound is load-bearing: if it lands, the claimed superiority of NPS over prior pruning and merging methods is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Neural Parameter Search (NPS), a pruning method for fine-tuned models built on task vectors. NPS splits a task vector into M subspaces by parameter magnitude, uses CMA-ES to reweight these subspaces against a calibration set, and then applies magnitude pruning to the reweighted vector. The resulting pruned models are used for knowledge transfer, model fusion, and compression. Experiments cover LLaVA-based multimodal transfer, T5/LLaMA/ViT/RoBERTa merging, PEFT adapter merging, and checkpoint compression; the paper reports consistent gains over baselines such as DARE, TIES, and Model Tailor, including a +2.1/+1.6 gain on T5-base/T5-large fusion and improved storage-accuracy trade-offs.","tokens_in":23516,"tokens_out":3803,"duration_ms":30985,"significance":"If the reported gains hold, NPS offers a simple, gradient-free way to improve pruned fine-tuned models for transfer, fusion, and compression, with only a few scalar weights per magnitude stratum. The paper's strengths include a broad set of benchmarks across NLP, vision, and multimodal settings, public code, and a Limitations section that honestly acknowledges the need for validation data and the computational cost of the search. However, the central claim that NPS's subspace-reweighting mechanism is responsible for the gains is not yet established, because the search budget given to NPS is far larger than that given to the baselines and because most tables lack error bars; the reported margins are often only 1–3 points, so the confounds are load-bearing.","major_comments":[{"comment":"The reported gains may partly reflect a larger validation-set search budget rather than the proposed subspace-reweighting mechanism. In §3.2 and §B.3, NPS runs 20–50 generations of CMA-ES on the per-subspace weights, and in Eq. (7) the fusion coefficients λ_i are also evolved; the baselines in Table 12 are only grid-searched over r and λ. Because the optimization objective is the calibration-set accuracy of the final pruned/merged model, NPS has substantially more capacity to fit the validation set. The paper does not report a control in which TIES, DARE, or Task Arithmetic are given the same number of validation-set evaluations (or in which NPS is restricted to the baseline grid budget). I would like to see such matched-budget experiments, or at least an analysis that varies the CMA-ES generations for NPS and shows that the gains persist with a comparable budget; without this, the claimed superiority over the baselines is not established.","section":"§3.2, Eq. (7), Table 13"},{"comment":"The mask formula in Eq. (4) as printed, `md = 1 if τd ≥ sorted(τ)[r × d]`, appears to contain a typo: the threshold `[r × d]` depends on the index d, which is not a meaningful sparsity threshold. It should presumably be `[r × D]`, where D is the total number of parameters, or an equivalent quantile expression. Please correct the equation and clarify whether the threshold is computed over the original task vector or the reweighted task vector from Eq. (2).","section":"Eq. (4)"},{"comment":"With the exception of the five-seed emotion experiment reported in §4.4, most tables report a single run and no variance information. Given that NPS's gains over the best baseline are modest (+0.9 to +3.0 percentage points in Table 2, and similar in Tables 1 and 3), the results could be within run-to-run noise. The authors should either provide multiple seeds with standard deviations for the main fusion, transfer, and compression comparisons, or explain why variance is negligible for these deterministic setups. Without this, the small margins are hard to interpret as robust improvements.","section":"Tables 1–3, §4.5"}],"minor_comments":[{"comment":"The abstract and §3.2 describe the method as searching 'within low-rank subspaces.' However, the actual construction partitions parameters by magnitude into strata and reweights them; this is not a low-rank decomposition. Please reword the description to avoid implying a low-rank structure that is not used (e.g., 'magnitude-based subspaces' or 'weight strata').","section":"Abstract and §3.2"},{"comment":"The notation in Eqs. (2)–(5) is ambiguous because τ is reused for both the original task vector and the reweighted task vector. The text says 'we reallocated weights ... to obtain a new task vector: τ = ...' and then uses τ again in Eqs. (4)–(5). Please use a distinct symbol, such as τ_w, for the reweighted vector and state explicitly which vector enters the mask computation.","section":"§3.2, Eq. (2)–(5)"},{"comment":"The first row in both blocks of Table 1 is labeled 'Zero-shot-' with a trailing hyphen; this appears to be a typo for 'Zero-shot'. Please correct the label.","section":"Table 1"},{"comment":"Minor naming inconsistencies: the method is referred to as both 'Talls Mask' and 'TALL-Mask' in different places (e.g., Appendix A, Section 4.2, Table 3). Please standardize the name.","section":"Appendix A/B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well within the scope of the journal and the experiments are extensive, but the central comparison is confounded by the unequal search budget given to NPS versus the baselines. The public code should make the requested matched-budget controls feasible. I would ask the authors to add those controls and error bars before acceptance; the 'low-rank' wording is a smaller but important correctness fix. There is no indication of a citation or novelty problem from the referee's perspective."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before anything else: it combines CMA-ES with magnitude-stratified reweighting of task vectors, and it shows consistent, modest gains across vision, NLP, and multimodal benchmarks. That combination is genuinely new, and the experiments are broad enough that the method is probably useful in practice even if the stated mechanism is not fully proven. But the central claim—that magnitude-based subspaces are what make the reweighting work—is under-supported: the baselines are grid-searched while NPS gets a black-box optimizer over subspace weights and, in fusion, over the interpolation coefficients too. That is a real confound, and the stress-test concern lands.\n\nWhat's good: the paper is honest about requiring calibration data and reports search costs in the appendix (roughly 2–6 hours even for LLMs). The ablation on number of subspaces, calibration volume, and sparsity ratio is exactly what a reader wants. The storage/accuracy Pareto results for compression are the strongest part of the paper, and the reported gains of +0.9 to +3.0 points over TIES are consistent across settings. Code is promised; I can't verify it from the preprint.\n\nSoft spots, in order of importance. First, the matched-budget control is missing: let TIES or DARE use the same CMA-ES budget over their own per-layer or global scaling coefficients, or better, run CMA-ES directly on per-parameter mask scores, and see whether the magnitude-subspace structure still earns its keep. Without that, the improved test accuracy may reflect more search capacity rather than the proposed mechanism. Second, error bars are nearly absent; only the emotion experiment reports multiple seeds, and several margins are within noise. Third, Eq. 4 has a typo (r*d should be r*D), and calling the magnitude partition 'low-rank subspaces' is wrong—it's a coordinate partition, not a low-rank decomposition. Minor. Fourth, the 'training-free' label in Section 4.2 is misleading; the method does require a search over validation data, though the Limitations section is appropriately candid about the overhead.\n\nFor a reader: this is for people doing model merging, task-vector pruning, or efficient deployment of multiple fine-tuned checkpoints. It is not a breakthrough that changes how transfer learning is done, but it is a solid practical contribution that deserves serious referee time. I would send it to review, but I would ask the authors to add the matched-budget control and at least a few error bars before acceptance. My own verdict: the practical method is likely useful even if the subspace-specific mechanism turns out to be less special than claimed. Recommend: engage, but make the authors do the control experiment.","headline":"A practical, useful pruning/merging trick with broad experiments, but the core mechanistic claim needs matched-budget controls before I'd believe the subspace story.","tokens_in":24055,"tokens_out":2396,"would_cite":true,"duration_ms":23576,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reweight task-vector subspaces before pruning fine-tuned models","keywords":["task vector","model pruning","model merging","catastrophic forgetting","knowledge transfer","model compression","evolutionary search","CMA-ES"],"falsifier":"Take a fine-tuned model where the largest-magnitude task-vector entries are deliberately corrupted or irrelevant while small-magnitude entries carry the task signal; if NPS still prunes away the small entries and loses accuracy, then the magnitude-reweighting premise fails. A concrete version is to swap in a random perturbation on the top 10% of task-vector entries, run the NPS search on a calibration set, and then check whether the pruned model recovers the original accuracy.","tokens_in":1580,"feed_emoji":"✂️","tokens_out":7118,"duration_ms":95974,"temperature":0.7,"pith_summary":"This paper argues that fine-tuned models can be slimmed more effectively by first reweighting the parameter groups of their task vector, the difference from the pretrained weights, rather than pruning the raw fine-tuned weights. The proposed method, Neural Parameter Search (NPS), splits the task vector into magnitude-ranked subspaces, uses an evolutionary search to assign each subspace a scalar weight on a calibration set, and then prunes by magnitude. If this works, practitioners get a training-free way to reduce catastrophic forgetting when interpolating with the pretrained model, to merge multiple models without retraining, and to compress checkpoints while keeping near-original accuracy. The paper reports consistent gains across vision, NLP, and multimodal benchmarks, including a 4.3% improvement on T5-base fusion and about 40% better compression efficiency on vision tasks.","feed_headline":"Search subspace weights to prune fine-tuned models","feed_subtitle":"A calibration set guides evolutionary search; better fusion, compression, and transfer follow.","key_machinery":"The central object is the reweighted task vector $\\tau = \\sum_{m=1}^{M} w_m \\cdot q_m$, where $q_m$ are magnitude-ranked subspaces of the difference between fine-tuned and pretrained weights. The scalar weights are searched by CMA-ES on a calibration set, and a final magnitude mask selects the surviving parameters. This machinery turns pruning into a low-dimensional search problem rather than a per-parameter choice, and the same pruned task vector plugs into interpolation, merging, and compression.","core_discovery":"On the paper's own terms, the central discovery is that the parameters of a task vector are not equally important, and that importance can be captured by learning a small number of scalar weights over magnitude-sorted subspaces. Starting from $\\tau = \\theta_{\\mathrm{finetuned}} - \\theta_{\\mathrm{pretrained}}$, the method splits $\\tau$ into $M$ subspaces $q_m$ by magnitude, searches weights $\\{w_m\\}$ with CMA-ES against validation accuracy on a calibration set, and only then applies a magnitude mask to obtain the pruned model $\\hat{\\theta}_{ft} = \\theta_{\\mathrm{pre}} + m \\odot \\tau$. The paper claims this reweighted-and-then-pruned task vector preserves performance at far higher sparsity than baselines such as TIES and DARE, and that the same pruned representation yields better interpolation for forgetting mitigation, better merging across tasks and modalities, and lower storage cost.","pith_inferences":["If magnitude-ranked scalar reweighting generalizes, it suggests a cheap universal preprocessing step for any task-vector-based operation, since it only needs a handful of calibration examples.","Because CMA-ES is population-based, the search cost grows with the number of subspaces, so an adaptive partition could remove the need to tune that hyperparameter.","The reliance on a calibration set means NPS is not fully data-free, and in low-data regimes the search could overfit to the calibration split, so a natural test is whether performance holds on held-out tasks.","A testable extension is per-layer or per-block weighting instead of a single scalar per global magnitude stratum, which may further improve the pruning-versus-accuracy trade-off."],"forward_implications":["At sparsity ratios where TIES and DARE degrade, NPS claims to preserve accuracy; for example, ViT-B/32 keeps near-original accuracy at a sparsity ratio of 0.04.","Merging NPS-pruned task vectors outperforms the baselines in all six reported settings, including fully fine-tuned NLP models, (IA)3 PEFT adapters, LLMs, vision models, and emotion-domain models.","Knowledge transfer on the LLaVA multimodal benchmark can be improved by interpolating the NPS-pruned model with the pretrained model, preserving performance at 10% sparsity while raising average metrics over prior forgetting-mitigation methods.","Knowledge compression can store only the pretrained weights, pruned task vectors, and binary masks, reducing storage below the TALL Mask + TIES baseline while maintaining roughly 99-100% normalized accuracy.","The whole search is gradient-free and completes in a few hours even for large language models, so it needs no backpropagation through the model."],"supporting_citations":[{"why":"Supplies the task vector definition and the Task Arithmetic baseline that the whole NPS construction builds on.","marker":"(Ilharco et al., 2023a)"},{"why":"TIES-Merging is the main fusion baseline and also provides the experimental setups for T5 and (IA)3 merging that NPS compares against.","marker":"(Yadav et al., 2024)"},{"why":"DARE is the primary pruning baseline for task vectors, and NPS is compared against its drop-and-rescale strategy.","marker":"(Yu et al., 2023a)"},{"why":"Model Tailor is the forgetting-mitigation baseline in the LLaVA knowledge transfer experiments, and its H-score metric is used for evaluation.","marker":"(Zhu et al., 2024)"},{"why":"TALL-masks and Consensus Merging are the key compression and merging baselines that NPS claims to beat in storage and accuracy.","marker":"(Wang et al., 2024)"},{"why":"CMA-ES is the evolutionary algorithm actually used to search the subspace weights in NPS.","marker":"(Hansen and Ostermeier, 1996)"}],"fun_headline_variants":["Subspace-weight search slims fine-tuned models","Task-vector pruning with CMA-ES boosts transfer","Slim fine-tuned models via subspace weight search","Prune task vectors: smarter, smaller, transferable","Search task-vector subspaces for leaner models"],"cache_read_input_tokens":26240,"weakest_assumption_plain":"The load-bearing premise is that a handful of scalar weights, one per magnitude-ranked chunk of the task vector, can be found on a calibration set that tells the method which parameters are actually important, and that pruning by magnitude after that reweighting keeps the important ones.","fun_headline_variants_meta":{"raw":{"variants":["Subspace-weight search slims fine-tuned models","Task-vector pruning with CMA-ES boosts transfer","Slim fine-tuned models via subspace weight search","Prune task vectors: smarter, smaller, transferable","Search task-vector subspaces for leaner models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1356,"prompt_tokens":968,"completion_tokens":388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":311}},"tokens_in":584,"tokens_out":388,"duration_ms":3259,"temperature":1.0,"reasoning_tokens":311,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:26:04.373589+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fine-tuned model where the largest-magnitude task-vector entries are deliberately corrupted or irrelevant while small-magnitude entries carry the task signal; if NPS still prunes away the small entries and loses accuracy, then the magnitude-reweighting premise fails. A concrete version is to swap in a random perturbation on the top 10% of task-vector entries, run the NPS search on a calibration set, and then check whether the pruned model recovers the original accuracy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TIES-Merging is the main fusion baseline and also provides the experimental setups for T5 and (IA)3 merging that NPS compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Model Tailor is the forgetting-mitigation baseline in the LLaVA knowledge transfer experiments, and its H-score metric is used for evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"CMA-ES is the evolutionary algorithm actually used to search the subspace weights in NPS."}],"review_version":1}