{"id":"447ae032-172e-4f19-baa0-7b52962b13dd","arxiv_id":"2505.16148","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"NAN sets merging coefficients inversely proportional to each model's parameter norm and claims a least-squares justification, but the derivation yields a different formula and performance gains are inconsistent.","lead":"The paper proposes NAN, a training-free rule for model merging that weights each model by the inverse of its parameter norm. The claimed least-squares justification does not actually produce the rule, and the experiments show mixed, sometimes negative, results.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (10) is not a consequence of the §3 derivation: the stated proxy n ∝ 1/Var(W) with Var(W) ∝ ||W||_F^2 yields inverse-squared weights, and the switch to inverse first power is ungrounded.","rationale":"The reader's verdict identifies the same weakest assumption: the bridge between the least-squares derivation and the implemented formula is an empirical law plus a hand-chosen norm exponent. My stress-test focuses on the internal inconsistency within that bridge. Eq. (9) is a legitimate least-squares result under the stated isotropy assumption, but Eq. (10) does not follow from it. If Var(W) ∝ ||W||_F^2, then inverse variance gives inverse squared norm, not inverse norm. The paper's justification for dropping the square is numerical stability, which is an engineering heuristic, not a theoretical step. Therefore the central claim that NAN is 'theoretically grounded' is unsupported. The empirical claim of consistent improvement is also contradicted by Table 1: for example, TA+NAN drops SVHN accuracy from 80.2 to 76.2 relative to Task Arithmetic, and Ties+NAN drops MNIST from 98.3 to 97.8 relative to Ties-Merging. These are not large margins, but they directly contradict the word 'consistently' in the abstract and Section 4. The lack of code or error bars further weakens the empirical case, though the theoretical gap is sufficient on its own. I agree with the rejection verdict: the paper's core contribution, a training-free plugin with a principled derivation, does not hold. The re-derivation test would settle the matter: if the authors can produce a valid derivation of the inverse first power from the least-squares objective, the rejection should be reconsidered; otherwise the formula is another heuristic coefficient rule.","tokens_in":8522,"tokens_out":3798,"duration_ms":29435,"concrete_test":"Independently re-derive Eq. (10) from §3.1–3.3 without changing the norm exponent. Combine Eq. (9) with n ∝ 1/Var(W) and the paper's own Var(W) ∝ ||W||_F^2: the sample-size-weighted average becomes α_i = (1/||W_i||_F^2) / Σ_j (1/||W_j||_F^2). If no additional argument produces the inverse first power, Eq. (10) is not a consequence of the least-squares framework. As a supplementary empirical check, test the proxy on the ViT-B/32 models of Table 1: compute the actual training-set sizes n_i and the Frobenius norms of the fine-tuned weight matrices, then check whether n_i correlates with 1/||W_i||_F or with 1/||W_i||_F^2; the correct relation should be visible if the proxy is to carry the derivation.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim that NAN's coefficients are theoretically grounded fails at the step connecting Eq. (9) to Eq. (10). Eq. (9) says optimal merging weights are sample-size weighted: α_i = n_i / Σ_j n_j. To make this training-free, §3.3 invokes the empirical relation n ∝ 1/Var(W), citing Si et al. (2025) and Du et al. (2025), and then, assuming zero-mean weights, states Var(W) ∝ ||W||_F^2. Taken together, these two relations imply n_i ∝ 1/||W_i||_F^2, hence α_i = (1/||W_i||_F^2) / Σ_j (1/||W_j||_F^2). The paper instead proposes Eq. (10) with 1/||W_i||_F, justified only by a comment about numerical stability and large scaling disparities. That is a change of functional form, not a derivation. The stated empirical law itself is also load-bearing: it is cited primarily to the authors' own prior work and to Du et al., and no evidence is given that it holds for the fine-tuned ViT-B/32, ViT-L/14, or LLaMA-2 checkpoints used in the experiments, where parameter variance is influenced by initialization scale, learning rate, regularization, and task difficulty in addition to sample size. Because Eq. (10) does not follow from the least-squares framework, NAN reduces to a heuristic coefficient rule, undermining the paper's central claim of a theoretically grounded method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Norm-Aware Merging (NAN), a training-free coefficient estimator for model merging. Starting from a least-squares formulation for two tasks, the authors derive that under isotropic normalized features the optimal merged model is approximately a sample-size-weighted average of individually trained models (Eq. (9)). Because sample sizes are unknown in practice, they invoke an empirical relation n ∝ 1/Var(W), approximate Var(W) by the squared Frobenius norm, and then define coefficients α_i = (1/||W_i||_F) / Σ_j (1/||W_j||_F) (Eq. (10)). They plug these coefficients into Task Arithmetic and Ties-Merging and evaluate on eight image classification tasks, language tasks with LLaMA-2, and VLM tasks with LLaVA, reporting average improvements.","tokens_in":8868,"tokens_out":5273,"duration_ms":43387,"significance":"If the derivation were valid, NAN would be a valuable zero-cost plugin for task-vector merging. The paper has strengths: the method is simple, training-free, and tested across three domains (ViT-B/32, ViT-L/14, LLaMA-2-13B, LLaVA-13B), and the writing is concise. The experimental comparisons cover representative baselines. However, the central theoretical claim is not supported by the derivation: Eq. (10) is a heuristic re-scaling of the implied inverse-squared norm rule. The empirical improvements are also modest and occasionally negative. Significance is therefore limited; at best the paper offers a plausible heuristic with partial empirical support. No code is provided, which further limits reproducibility.","major_comments":[{"comment":"Eq. (10) does not follow from Eqs. (8)–(9). The derivation in §3.2 yields sample-size weights α_i = n_i / Σ_j n_j; combining this with the stated proxy n ∝ 1/Var(W) and Var(W) ∝ ||W||_F^2 gives α_i ∝ 1/||W_i||_F^2, not the 1/||W_i||_F used in Eq. (10). The intervening sentence about numerical stability changes the functional form rather than deriving it, so the central claim that NAN's coefficients are theoretically grounded is unsupported.","section":"§3.3, Eq. (10)"},{"comment":"The empirical law n ∝ 1/Var(W) is load-bearing and is cited primarily to the authors' own Si et al. (2025) and to Du et al. (2025), with no demonstration that it holds for the ViT, LLaMA-2, or LLaVA fine-tuned checkpoints used in the experiments. Since parameter variance also depends on initialization scale, learning rate, regularization, and task difficulty, this proxy is not established for the models at hand; without it, the bridge from sample-size weights to norm-based weights collapses.","section":"§3.3, paragraph beginning 'Recent findings suggest'"},{"comment":"The linear least-squares solution is used as a proxy for fine-tuned deep-network weights, but no argument is given that the closed-form solution of Eq. (2) approximates the SGD-trained checkpoints being merged. The matrix-valued coefficients in Eq. (8) are then reduced to scalar sample-size weights using an isotropy assumption that is plausible for normalized features but not verified for the learned representations of the tested models.","section":"§3.1–§3.2"},{"comment":"The claim that NAN 'consistently improves' the baselines is too strong. Several entries show degradation, e.g., TA+NAN on ViT-B/32 SVHN drops from 80.2 to 76.2, and Ties+NAN on ViT-L/14 is below plain Ties on SUN397, Cars, and RESISC45. Without standard deviations or significance tests, the evidence supports only 'sometimes improves, sometimes slightly hurts,' which weakens the empirical claim as stated.","section":"§4, Tables 1–3"}],"minor_comments":[{"comment":"The sentence 'Extensive experiments on show that NAN consistently improves...' is incomplete; the missing specification should be supplied.","section":"Abstract"},{"comment":"AdaMerging is attributed to Yadav et al. (2023), but the AdaMerging paper is Yang et al. (2023); additionally, AdaMerging++ and Layer-wise AdaMerging appear in Table 1 but are not described in the baselines paragraph.","section":"§4, Baselines"},{"comment":"The global scaling factor m/2 is introduced with no derivation or ablation; since it scales all coefficients equally it cannot affect relative coefficients, so its role should be stated explicitly.","section":"§3.3"},{"comment":"Clarify whether W_i denotes full model weights or task vectors; the text says 'model's weights W' but the method is applied both directly to raw weights and as a reweighting step in task-vector merging.","section":"Eq. (10)"}],"recommendation":"reject","confidential_remarks":"The theoretical gap at Eq. (10) is the main obstacle; the empirical evidence alone is not strong enough to carry the paper. The authors might resubmit a version that frames NAN as a heuristic with a broader empirical study, but that would be a substantial reframing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one genuinely new thing here is the coefficient rule alpha_i = 1/||W_i||_F, normalized, as a training-free plugin for task-vector merging. That exact formula doesn't appear in the earlier merging literature, and the plug-and-play framing is nice. The authors also run a reasonably broad evaluation across ViT, LLaMA, and LLaVA backbones, which is more than many merging papers do.\n\nThe problem is that the theoretical grounding doesn't actually produce the formula. The least-squares derivation in Sections 3.1 and 3.2 is correct for linear models: with normalized isotropic features, the optimal merged model is approximately the sample-size-weighted average. Fine. Then Section 3.3 invokes an empirical law n ∝ 1/Var(W), cites the authors' own prior work and Du et al., assumes zero-mean weights, and gets Var(W) ∝ ||W||_F^2. Combined, that gives alpha_i ∝ 1/||W_i||_F^2, not 1/||W_i||_F. The switch to first power is justified only by a remark about numerical stability and scaling disparities. That is a change of functional form, not a derivation. So the central claim—that NAN is theory-derived—does not hold up. The method is a heuristic with a least-squares preamble.\n\nThe empirical claim is also oversold. The tables show several datasets where NAN hurts accuracy (e.g., SVHN and MNIST for TA+NAN in Table 1; BBH for TA+NAN in Table 2; AI2D in Table 3). Averages improve by 1–3 points, but there are no error bars and no code, so it's hard to tell whether those gains are noise. \"Consistently improves\" is not what the tables say.\n\nNone of this makes the paper worthless. A simple training-free coefficient rule that helps on average could be a useful baseline even if its justification is heuristic. But the paper presents itself as theoretically grounded, and that claim is not supported. The load-bearing proxy law is also cited mostly to the authors' own prior work, which is a circularity worth noting.\n\nI'd send it to peer review rather than desk reject, because the idea is timely and the experiments are broad, but the referee should insist that the authors either fix the derivation or reframe NAN honestly as a heuristic. As it stands, the version-of-record claim needs revision.","headline":"A simple training-free norm heuristic for merging coefficients, wrapped in a derivation that doesn't actually produce the formula.","tokens_in":9395,"tokens_out":2803,"would_cite":false,"duration_ms":20937,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Optimal merging weights track task-specific information, estimated by inverse parameter norm without training.","keywords":["model merging","training-free","task vector","coefficient estimation","Frobenius norm","least squares","multi-task learning","parameter variance"],"falsifier":"Take a set of fine-tuned checkpoints with publicly known training sample sizes and check whether the inverse Frobenius norm ranking matches the sample-size ranking across tasks; if a model trained on considerably more data has a larger norm and therefore gets a smaller NAN weight, the proxy's core relation fails for that case. A second, cheaper check is the ablation of $\\|W\\|_F$ versus $\\|W\\|_F^2$: if the unsquared choice is load-bearing, swapping it should visibly change merged accuracy.","tokens_in":8308,"feed_emoji":"🧩","tokens_out":4807,"duration_ms":34351,"temperature":0.7,"pith_summary":"The paper tries to establish that when merging independently fine-tuned models, the best mixing coefficient for each model is proportional to how much task-specific information that model carries, and that this quantity can be read off the model weights alone. It derives from a least-squares view of merging that the ideal weights are sample-size-weighted, then connects sample size to the inverse variance of the trained parameters. That motivates a training-free plug-in, NAN, which sets each model's coefficient to the inverse of its Frobenius norm, normalized across models. If the derivation and proxy hold, NAN would give a zero-cost way to improve any task-vector merging method, including in settings where training data and sample sizes are unavailable.","feed_headline":"Inverse norm sets merge weights, no training needed","feed_subtitle":"A plug-in that lifts task-vector merging scores on vision, language, and multimodal models using only the weights themselves.","key_machinery":"The argument is carried by a chain of substitutions. First, the closed-form least-squares solution for two tasks writes the merged model as a weighted combination $W^* = \\Omega_1 W_1^* + \\Omega_2 W_2^*$, where the matrix weights are relative information contents $(A_1+A_2)^{-1} A_i$. Second, normalized features make $A_i \\approx n_i I$, turning the weights into sample-size fractions. Third, the empirical law $n \\propto 1/\\mathrm{Var}(W)$ lets the paper substitute variance for unknown sample size, and zero-centered weights make $\\mathrm{Var}(W) \\propto \\|W\\|_F^2$. The final step replaces the squared norm by the unsquared Frobenius norm, yielding the inverse-norm NAN coefficients; a global factor $m/2$ rescales the merged weights when many models are combined.","core_discovery":"On the paper's own terms, the central claim is that the optimal merging coefficients should scale with the amount of task-specific information encoded in each model, and that the inverse parameter norm is a faithful, computable proxy for that information. Starting from two least-squares problems, the paper shows the jointly optimal solution is a weighted average of the individually optimal solutions, with matrix weights proportional to the unnormalized input covariances; under normalized, roughly isotropic features this becomes a sample-size-weighted average. Because sample sizes are unknown for open models, the paper invokes the empirical regularity that parameter variance decreases with training data volume, assumes zero-centered updates so variance is tracked by the squared Frobenius norm, and then deliberately uses the unsquared norm for numerical stability. The resulting NAN coefficient for model $i$ is $\\alpha_i = (1/\\|W_i\\|_F) / \\sum_j (1/\\|W_j\\|_F)$, applied either to raw weights or as a reweighting of an existing merging strategy's output.","pith_inferences":["The derivation's sample-size step assumes normalized, isotropic features, so the cleanest test of the paper's logic is comparing NAN's inverse-norm weights against true sample-size weights on models where those sizes are published; a mismatch there would point to the proxy, not the least-squares core, as the fragile link.","Because the paper uses the unsquared norm while its variance argument naturally gives the squared norm, a direct ablation switching between $\\|W\\|_F$ and $\\|W\\|_F^2$ would isolate whether the stability motivation or the variance proxy is doing the empirical work.","The $m/2$ global rescaling is a practical band-aid for coefficient shrinkage with many models; one could test whether per-layer or per-block norm ratios behave differently from the global norm, which would decide whether a single scalar per model is the right granularity.","If the inverse-norm rule is genuinely capturing task-specific information, it should also predict which pairs of fine-tuned models will merge well before evaluation, making it useful as a cheap screening tool for model selection."],"forward_implications":["Plugging NAN into Task Arithmetic or Ties-Merging raises average accuracy across the eight vision tasks, the two-LLM language setting, and the two-VLM setting, with the largest gains on GSM8K and GeoQA.","NAN is training-free and requires only the checkpoints themselves, so it can be applied when data, Fisher information, or sample sizes are unavailable.","Because the coefficient rule is a one-line formula, it can be composed with any merging strategy that produces a weighted combination of task vectors or raw weights.","The same inverse-norm rule works across backbones (ViT-B/32, ViT-L/14, LLaMA2-13B, LLaVA-v1.5-13B) and across vision, language, and multimodal tasks."],"supporting_citations":[{"why":"Supplies the empirical finding that parameter variance decreases with training data volume and that weights are approximately zero-centered; this is the bridge from sample size to the norm proxy.","marker":"Si et al., 2025"},{"why":"Corroborates the same variance-data-volume relation and zero-centered weight assumption used to justify the Frobenius-norm proxy.","marker":"Du et al., 2025"},{"why":"Cited for the observation that weight variability is lower for models trained on more data, part of the empirical basis for n proportional to 1/Var(W).","marker":"Fort et al., 2019"},{"why":"Cited alongside Fort et al. for the variance-data-volume relation that connects the least-squares derivation to a computable coefficient.","marker":"Izmailov et al., 2018"},{"why":"Defines task-vector arithmetic, the base merging strategy that NAN reweights in the vision, language, and VLM experiments.","marker":"Ilharco et al., 2022"},{"why":"Defines Ties-Merging, the other base strategy NAN is plugged into, and its coefficient heuristic is what NAN replaces.","marker":"Yadav et al., 2023"}],"fun_headline_variants":["Inverse norm sets merge weights, zero training","Training-free merge coefficients from parameter norms","Norm-based weight scaling improves model merging","NAN: no-training merge via inverse parameter norm","Use inverse norm to pick merge coefficients"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands on the empirical relation that more training data makes a model's parameters lower-variance, and on the extra step of using the plain Frobenius norm instead of its square as the proxy for that variance.","fun_headline_variants_meta":{"raw":{"variants":["Inverse norm sets merge weights, zero training","Training-free merge coefficients from parameter norms","Norm-based weight scaling improves model merging","NAN: no-training merge via inverse parameter norm","Use inverse norm to pick merge coefficients"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1498,"prompt_tokens":870,"completion_tokens":628,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":486,"completion_tokens_details":{"reasoning_tokens":562}},"tokens_in":486,"tokens_out":628,"duration_ms":5904,"temperature":1.0,"reasoning_tokens":562,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:05:31.533698+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of fine-tuned checkpoints with publicly known training sample sizes and check whether the inverse Frobenius norm ranking matches the sample-size ranking across tasks; if a model trained on considerably more data has a larger norm and therefore gets a smaller NAN weight, the proxy's core relation fails for that case. A second, cheaper check is the ablation of $\\|W\\|_F$ versus $\\|W\\|_F^2$: if the unsquared choice is load-bearing, swapping it should visibly change merged accuracy.","supporting_citations":[],"review_version":1}