{"id":"7aa28dbb-489d-48d7-ac71-62215b6184d6","arxiv_id":"2412.15467","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"NP Merge learns a separate interpolation weight for each parameter when merging two aligned networks, improving on several baselines but not on ensembles.","lead":"NP Merge is a new way to combine two trained neural networks: instead of averaging every weight with the same coefficient, it learns a separate blending weight for each parameter. The authors report gains over several merging baselines, though their own tables show it does not beat simple ensembles.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Empirical support for 'outperforms past methods' omits the closest per-parameter merging baselines (Fisher-weighted averaging [13], function-space aggregation [18]); without them, the central claim is not established.","rationale":"The reader's formal weakest_assumption concerns the (0,1) convex-hull constraint, an interesting limitation but not the decisive issue here: the paper's own fine-tuning comparisons show NP Merge often matching or beating unconstrained fine-tuning, so the restricted search space is not clearly hurting performance in the tested regimes. The more load-bearing problem is that the closest prior per-parameter merging methods are cited in related work but never compared against. Because NP Merge's contribution is precisely per-parameter interpolation, a fair evaluation must include Fisher-weighted averaging and function-space aggregation, which also assign parameter-specific weights. Without these baselines, the empirical section cannot distinguish whether gains come from per-parameter flexibility or simply from post-alignment optimization on training data. The reader's rationale does mention the missing parameter-wise baselines, but the stated weakest_assumption is different, hence 'partial' agreement. The recommended verdict stays CONDITIONAL: the paper presents a plausible method and useful experiments, but the central 'outperforms past methods' claim requires the missing comparisons before acceptance as stated.","tokens_in":14223,"tokens_out":4005,"duration_ms":34829,"concrete_test":"Add Fisher-weighted averaging (Matena & Raffel, 2022) and function-space aggregation (Dhawan et al., 2024) to Tables I-III, using the same Permute and Weight Matching alignments, the same training data for computing Fisher/function-space weights, and the same evaluation protocol. If NP Merge does not exceed these per-parameter baselines by more than one standard deviation on all three tables, the headline claim should be weakened to a qualified improvement over uniform merging and fine-tuning.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that NP Merge 'outperform[s] past methods.' But Section IV-A lists only uniform-α aggregation baselines: base-model averaging, ensembles, direct averaging, Permute, Weight Matching, CCA Merge, and fine-tuning. Absent are Fisher-weighted averaging (Matena & Raffel, 2022) and function-space aggregation (Dhawan et al., 2024), the two methods the authors themselves identify in Section II-C as proposing per-parameter weighted merging—the same category as NP Merge. NP Merge optimizes per-parameter α with 10 epochs over the training data; the comparison baselines use a single scalar α and no data-driven optimization. This conflates the effect of per-parameter interpolation with the effect of using labeled data after alignment. If Fisher-weighted averaging or function-space aggregation were run on the same aligned models with the same data, they might match or beat NP Merge, in which case the abstract's 'outperforming past methods' is unsupported. This is load-bearing because the paper's stated novelty is exactly per-parameter weighting, and its headline empirical claim is relative to past methods. The prose claim in Section IV-C that NP Merge 'can outperform all the baseline accuracies, including the ensemble accuracy' is also contradicted by Tables I-III, but that is an overstatement separate from the comparison gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NP Merge, a method that learns a per-parameter interpolation coefficient for each weight when merging two neural networks that have already been aligned. The coefficients are constrained to (0,1) via a sigmoid reparameterization and optimized by gradient descent on labeled data, with batch-norm statistics reset after merging. The authors evaluate NP Merge on VGG11 and ResNet20 models trained on CIFAR-10/100 under same-data and non-uniform class splits, on ResNet18 on ImageNet-200, and in a many-model setting via iterative pairwise merging. They compare against uniform averaging, permutation-based and CCA alignment, model ensembles, and fine-tuning. The central claims are that NP Merge outperforms past merging methods and can outperform ensembles, but the reported tables do not support the stronger ensemble claims.","tokens_in":14483,"tokens_out":7232,"duration_ms":63723,"significance":"If the empirical claims were fully supported, NP Merge would be a useful and simple addition to the model-merging toolkit: it is architecture-agnostic, compatible with existing alignment methods, and the data-size robustness analysis and iterative multi-model extension are practically relevant. However, the evaluation omits the closest per-parameter merging baselines (Fisher-weighted averaging and function-space aggregation), and the manuscript's prose claims about outperforming ensembles are contradicted by its own tables. The underlying optimization procedure appears sound, but the central empirical claim needs substantial revision before the result can be considered established.","major_comments":[{"comment":"The sentence in Section IV-C that 'results in Tables II and III show that NP Merge can outperform all the baseline accuracies, including the ensemble accuracy' is directly contradicted by the numbers in those tables. In Table II (80%-20%) the ensemble reaches 77.84% while NP-P reaches 73.13% and NP-WM 72.94%; in the Dirichlet column the ensemble is 73.77% while both NP variants are 73.45%; in Table III the ensemble Top-1 accuracy is 75.96% while NP-P is 62.21% and NP-WM is 58.55%. The conclusion in Section V that NP Merge 'consistently outperforms traditional merging methods and ensemble techniques' is also false for Table I, where the ensemble achieves 89.65% versus 88.38% (NP-P) and 88.49% (NP-WM) on CIFAR-10, and 73.51% versus 62.88% (NP-P) and 61.68% (NP-WM) on CIFAR-100. These overstatements are load-bearing because the abstract's central claim is 'outperforming past methods'; the manuscript must either weaken the claims to match the data or justify why the ensemble comparison is not the relevant one.","section":"Section IV-C, Tables II and III"},{"comment":"The baseline list in Section IV-A omits Fisher-weighted averaging [13] and function-space aggregation [18], the two methods that the paper itself identifies in Section II-C as the closest prior work on per-parameter weighted merging. Because the abstract claims NP Merge 'outperform[s] past methods' and the stated novelty is per-parameter weighting, a fair comparison should include these baselines under the same alignment procedure (Permute or Weight Matching) and the same optimization data. Without those numbers, the central claim that learned per-parameter weights improve over existing per-parameter merging methods is not established.","section":"Section IV-A and Section II-C"},{"comment":"The sigmoid parameterization constrains every alpha to (0,1), so the merged model is restricted to the per-parameter convex hull of the two aligned models. The paper presents this boundedness as a regularization advantage, but it also means that any optimal merge requiring extrapolation outside the segment is unreachable. This assumption is load-bearing for the method's design and is not tested. Please report results with unconstrained alpha (or a comparison to fine-tuning in settings where the best weights may lie outside the hull) to characterize when the bounded search space helps and when it limits performance.","section":"Section III-D"},{"comment":"The comparison with fine-tuning is not fully specified. NP Merge uses a fixed 10 epochs, a fixed learning rate, and the sigmoid-induced regularization, while fine-tuning is said to use 'optimal hyper-parameters' without reporting them. Because one of the paper's stated contributions is that NP Merge is more stable than fine-tuning with little data, the fine-tuning protocol (epochs, learning rate schedule, early stopping, and data budget) needs to be reported so the comparison is budget-matched and reproducible.","section":"Section IV-D"}],"minor_comments":[{"comment":"The affiliation contains a typo: 'Insitute' should be 'Institute'.","section":"Author affiliation"},{"comment":"The notation 'alpha = S_{i=1} alpha_i tensor' is unclear; use a concatenation or Cartesian-product notation for the per-layer alpha tensors.","section":"Section III-B"},{"comment":"Figure 1 reports trends without error bars or standard deviations, although the tables report them; the 'stable accuracy' claim is hard to evaluate without variability information.","section":"Figure 1"},{"comment":"There is a small inconsistency between Table II and Table IV for the Weight Matching training-data accuracy (60.86% in Table II versus 60.20% in Table IV after batch-norm reset); please reconcile or explain.","section":"Tables II and IV"},{"comment":"The manuscript alternates between 'NP Merge' and 'NP-Merge'; please choose one consistent spelling.","section":"Throughout"},{"comment":"The learning rate is reported for CIFAR-10 and CIFAR-100 but not for the ImageNet-200 experiment; please clarify the optimization hyperparameters used there.","section":"Section IV-A"},{"comment":"The memory and computational overhead discussion is qualitative; a quantitative measurement (e.g., wall-clock time and memory usage relative to baselines) would strengthen the trade-off discussion.","section":"Section IV-F"}],"recommendation":"major_revision","confidential_remarks":"The paper's scope is a good fit for a model-merging venue and the method is simple enough to be of interest. However, the discrepancy between the prose claims and the paper's own tables is severe, and the missing closest baselines directly affect the abstract's central claim. Please require the revision to address the Fisher-weighted and function-space baselines and to correct the ensemble overclaims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"NP Merge is a straightforward idea: after aligning two models, learn a per-parameter interpolation weight by gradient descent on training data. That is worth something. It is not a new capability, but it is a clean way to get more out of existing alignments, and the paper shows it consistently beats uniform averaging and standard alignment baselines across several settings. The extension to multiple models via pairwise merging is sensible, and the data-efficiency experiments—showing NP Merge degrades more gracefully than fine-tuning when only a few examples per class are available—are a genuinely useful addition.\n\nThe problems are two, and they are real. First, the paper never compares against the closest per-parameter baselines it cites: Fisher-weighted averaging (Matena & Raffel) and function-space aggregation (Dhawan et al.). Those methods also assign per-parameter weights, so NP Merge is an alternative to them, not a new category. Without those numbers, the abstract's claim that it \"outperforms past methods\" simply is not established. Second, the prose goes further than the tables. Section IV-C and the conclusion say NP Merge beats ensemble accuracy, but in every reported table the ensemble is better—often by a lot (Table III: 75.96 vs 62.21 top-1). That is not a minor wording issue; it is a false claim that should be removed.\n\nThere is also a smaller point about the sigmoid constraint pinning alpha to (0,1). It is a deliberate regularizer, and the comparison to fine-tuning helps. But the method cannot escape the convex hull of the two aligned models. One sentence acknowledging that limitation would be appropriate.\n\nOverall, the paper is a useful incremental contribution. It is not a revolution, but the method is clean, the experimental coverage is reasonable, and the data-efficiency analysis is a plus. The missing baselines and the ensemble overclaim are correctable. I would send it to a serious referee and ask for a revision that adds the two missing comparisons and tones down the claims. For a reading group, it would be a good case study in how easy it is to overstate when you omit the nearest competitors.","headline":"A promising per-parameter merging method that needs a fair comparison against the closest prior work before its claims hold.","tokens_in":15000,"tokens_out":3057,"would_cite":true,"duration_ms":26025,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes NP Merge, a model-merging technique that learns a distinct interpolation weight for every parameter of two aligned neural networks via gradient-based optimization, and claims this per-parameter flexibility produces…","keywords":["model merging","parameter interpolation","gradient-based optimization","linear mode connectivity","model alignment","federated learning","ensemble methods","non-uniform merging"],"falsifier":"Construct a pair of models where the optimal merged weight for at least some parameters is known to lie outside the interval between the two base weights (for example, a case where the best combined weight has larger magnitude than either base weight, which could be created by deliberately scaling one model's weights). If NP Merge's accuracy on that pair is worse than unconstrained fine-tuning or even uniform averaging, that would show the convex-hull restriction is a real limitation. A simpler check is to compare NP Merge against uniform averaging on a task where no alignment is applied; if NP Merge fails to beat the trivial average in that regime, the method's advantage disappears.","tokens_in":14034,"feed_emoji":"🧩","tokens_out":5340,"duration_ms":45142,"temperature":0.7,"pith_summary":"The paper introduces NP Merge, a method for combining two or more neural networks by learning a separate interpolation coefficient for each weight, rather than using one scalar for the whole model. After the models are aligned, the coefficients are optimized by gradient descent on labeled data, with a sigmoid keeping each coefficient between 0 and 1. The paper argues that this per-parameter flexibility better exploits linear mode connectivity, especially when models were trained on different data distributions, and reports accuracy gains over uniform averaging, standard fine-tuning, and prior merging methods in several settings. The central empirical claim is that NP Merge consistently outperforms past merging methods; this matters because model merging offers the memory and inference savings of a single model plus some of the accuracy gains of ensembling.","feed_headline":"Learning a weight for every parameter beats uniform model merging","feed_subtitle":"Per-parameter interpolation improves merged-model accuracy and holds up when data is scarce.","key_machinery":"The central object is the per-parameter interpolation tensor $\\alpha_i$, which assigns one scalar in $(0,1)$ to every weight in every layer, combined with the elementwise blend $W_i = \\alpha_i \\odot W^A_i + (1-\\alpha_i) \\odot W^{B'}_i$. Each coefficient is parameterized as $\\alpha_i = \\sigma(\\alpha^{\\text{pre}}_i)$, so the learned values stay strictly between 0 and 1 and the merged model remains inside the line segment connecting the two aligned models. This bounded formulation acts as a regularizer that pulls the solution back toward the equal average ($\\alpha = 0.5$), and the alignment of the two models is held fixed while the coefficients are optimized by gradient descent.","core_discovery":"The paper's central claim is that not all parameters of a neural network are equally important when merging, and that assigning a learned, per-parameter weight to each weight in the interpolation between two aligned models yields merged models that are more accurate than the uniform average. Concretely, after aligning two models into the same mode-connectivity basin, the merged weights are given by $W_i = \\alpha_i \\odot W^A_i + (1-\\alpha_i) \\odot W^{B'}_i$, where each scalar $\\alpha_i$ is optimized by backpropagating through the merged network on labeled data. The search is constrained to the convex hull of the two models' weight vectors because each $\\alpha_i$ is produced by a sigmoid, so the coefficients lie strictly between 0 and 1. The paper demonstrates this on CIFAR-10, CIFAR-100, and ImageNet-200 with VGG and ResNet architectures, using Permute or Weight Matching for alignment, and reports that NP Merge remains more stable than fine-tuning when the optimization dataset is as small as one example per class.","pith_inferences":["The learned coefficients may implicitly approximate a Fisher-information-style weighting, so a testable prediction is that the optimized $\\alpha_i$ values correlate with the diagonal entries of each model's Fisher information matrix.","Because the sigmoid biases coefficients toward 0.5, NP Merge behaves like a constrained fine-tuning that begins at the uniform average; relaxing the $(0,1)$ bound in a controlled way might unlock further gains, at the cost of robustness.","NP Merge's success on disjoint and imbalanced data splits suggests it could be applied to task-vector arithmetic, learning per-parameter scaling of task vectors rather than the scalar coefficients used in current task arithmetic.","The current method requires labeled optimization data; a natural extension is to optimize the coefficients unsupervised, for example by matching activation statistics between the merged and base models."],"forward_implications":["NP Merge can be applied on top of any existing alignment method by replacing the uniform scalar with learned per-parameter coefficients, so existing alignment research can be directly upgraded.","With limited optimization data, NP Merge degrades more gracefully than fine-tuning, making it better suited to federated or privacy-restricted settings where labeled validation data is scarce.","Repeated pairwise merging extends NP Merge to 4, 8, or more models with stable accuracy in balanced settings, providing a practical route to merging many decentralized models.","The learned per-parameter coefficients are interpretable as the contribution of each weight from each base model, which could be used to diagnose which layers or units carry the most task-specific information."],"supporting_citations":[{"why":"Supplies the Permute alignment method, which is used as the activation-based alignment prior for the NP-P variant.","marker":"[5]"},{"why":"Supplies the Weight Matching alignment method, which is used as the weight-based alignment prior for the NP-WM variant.","marker":"[6]"},{"why":"Provides the batch-norm reset procedure used after merging to avoid variance collapse.","marker":"[8]"},{"why":"CCA Merge is the state-of-the-art alignment baseline that NP Merge is compared against in the non-uniform data settings.","marker":"[17]"},{"why":"Fisher-weighted averaging is a prior non-uniform merging approach that NP Merge extends conceptually by learning weights without computing Fisher information.","marker":"[13]"},{"why":"Function space aggregation is the related per-parameter weighting method in federated settings that motivates the comparison between data-dependent merging approaches.","marker":"[18]"}],"fun_headline_variants":["Per-parameter weights beat uniform merging for model fusion","Optimize each weight to merge models better than averaging","NP Merge learns an alpha for every parameter","Uniform model merging? Try per-weight interpolation","Learned per-parameter coefficients improve merged models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the best merged model can be expressed as a per-parameter convex combination of the two aligned models, with every coefficient between 0 and 1; if the optimal merge requires extrapolating outside that segment, NP Merge cannot reach it.","fun_headline_variants_meta":{"raw":{"variants":["Per-parameter weights beat uniform merging for model fusion","Optimize each weight to merge models better than averaging","NP Merge learns an alpha for every parameter","Uniform model merging? Try per-weight interpolation","Learned per-parameter coefficients improve merged models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3075,"prompt_tokens":906,"completion_tokens":2169,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":2098}},"tokens_in":522,"tokens_out":2169,"duration_ms":13829,"temperature":1.0,"reasoning_tokens":2098,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:24:25.527794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a pair of models where the optimal merged weight for at least some parameters is known to lie outside the interval between the two base weights (for example, a case where the best combined weight has larger magnitude than either base weight, which could be created by deliberately scaling one model's weights). If NP Merge's accuracy on that pair is worse than unconstrained fine-tuning or even uniform averaging, that would show the convex-hull restriction is a real limitation. A simpler check is to compare NP Merge against uniform averaging on a task where no alignment is applied; if NP Merge fails to beat the trivial average in that regime, the method's advantage disappears.","supporting_citations":[{"cited_title":"Optimizing mode connectivity via neuron alignment,","cited_arxiv_id":null,"evidence_quote":"Supplies the Permute alignment method, which is used as the activation-based alignment prior for the NP-P variant."},{"cited_title":"Git re-basin: Merging models modulo permutation symmetries,","cited_arxiv_id":null,"evidence_quote":"Supplies the Weight Matching alignment method, which is used as the weight-based alignment prior for the NP-WM variant."},{"cited_title":"REPAIR: REnormalizing permuted activations for interpolation repair,","cited_arxiv_id":null,"evidence_quote":"Provides the batch-norm reset procedure used after merging to avoid variance collapse."},{"cited_title":"Harmony in diversity: Merging neural networks with canonical correlation analysis,","cited_arxiv_id":null,"evidence_quote":"CCA Merge is the state-of-the-art alignment baseline that NP Merge is compared against in the non-uniform data settings."},{"cited_title":"Merging models with fisher-weighted averaging,","cited_arxiv_id":null,"evidence_quote":"Fisher-weighted averaging is a prior non-uniform merging approach that NP Merge extends conceptually by learning weights without computing Fisher information."},{"cited_title":"Leveraging function space aggregation for federated learning at scale,","cited_arxiv_id":null,"evidence_quote":"Function space aggregation is the related per-parameter weighting method in federated settings that motivates the comparison between data-dependent merging approaches."}],"review_version":1}