{"id":"dd22ccfa-ed33-45c3-ad86-9251f13d11a7","arxiv_id":"2506.04567","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"StatsMerging predicts per-layer merging coefficients from weight statistics and teacher pseudo-labels, achieving 94.5% average accuracy across eight vision tasks, 5.1 points above WEMoE.","lead":"This paper proposes StatsMerging, a method that merges several fine-tuned image classifiers into one model by predicting blending coefficients from weight statistics. It uses singular values from SVD and teacher-model pseudo-labels, and reports higher average accuracy than prior merging methods on eight vision tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on the ill-specified StatsMerging++ variant (94.5%), while the base LW StatsMerging in Table 2 scores 84.5%, below WEMoE (89.4%); the extra validation budget is never disclosed.","rationale":"The reader's weakest assumption correctly identifies that the method relies on unlabeled validation data and pseudo-labels, but the more specific and load-bearing problem is that the advertised SOTA result is produced only by StatsMerging++—a variant whose only stated difference is 'more validation data'—and no quantitative description of that data is given. This matters because the base StatsMerging, for which the full algorithmic description is provided, does not outperform WEMoE in the layer-wise setting (84.5% vs. 89.4%). Thus the central claim of 5.1% improvement is not a claim about the method as specified; it is a claim about an unspecified experimental configuration. The concern is not merely about representativeness of the validation data; it is about whether the comparison is fair at all, since data-free baselines are being compared against an approach whose performance may depend on access to a large pseudo-labeled set drawn from the same distribution as the training data. A precise disclosure of the validation split and a replication with a capped validation budget would settle the issue. The paper otherwise has a plausible mechanism, an internally consistent ablation, and a reasonable base method, but the headline result cannot be evaluated until this ambiguity is resolved.","tokens_in":18530,"tokens_out":6537,"duration_ms":72810,"concrete_test":"Request the exact per-task validation set construction for StatsMerging++: number of images per task, source split (train/test), and any overlap with the test set. Then rerun the eight-task merging experiment with StatsMerging++ using a validation budget capped at the same 'small portion' used for base StatsMerging. If the average accuracy drops from 94.5% toward the reported 84.5% (or below WEMoE's 89.4%), the headline SOTA claim is an artifact of the undisclosed larger validation set rather than of the statistics-guided merging mechanism.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The abstract and conclusion claim StatsMerging outperforms WEMoE by 5.1% (94.5% vs. 89.4%), but Table 2 shows the actual proposed method, LW StatsMerging, achieves only 84.5%, which is 4.9 points below WEMoE. The 94.5% number is produced exclusively by 'LW StatsMerging++', described in Appendix A only as 'an extended version of StatsMerging trained on more validation data'. No sample counts, split provenance, or overlap checks are provided. Several datasets (MNIST, SVHN, GTSRB, Stanford Cars) have no official validation split, so it is unclear whether the 'validation set' is carved from the training set or includes test images. If 'more validation data' means a large fraction of the training split used with teacher pseudo-labels, the method gains a substantial data advantage over data-free WEMoE and over AdaMerging's test-time entropy minimization, while still claiming 'without test samples' and 'without ground truth labels'. The per-task gains over individual models (SUN397: 75.3 to 92.4, Cars: 77.7 to 95.4) are anomalously large for a fixed coefficient merging scheme and could be explained by access to a large, distribution-matched pseudo-labeled validation set. Because the central empirical claim is defined by an undisclosed data budget, the result is not reproducible and the comparison is not interpretable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes StatsMerging, a model merging method that predicts per-task (or per-layer) merging coefficients with a small MLP, StatsMergeLearner, whose input is a vector of weight statistics (mean, variance, norm, and top-3 singular values). The learner is trained by distilling pseudo-labels from the task-specific fine-tuned models themselves on validation images, avoiding human annotations. The authors report results on eight ViT-B/32 classification tasks (Table 2), generalization to unseen tasks (Table 3), and a heterogeneous-architecture variant where ViT teachers are distilled into a ResNet50 before merging (Table 4). The headline claim is 94.5% average accuracy for LW StatsMerging++, 5.1% above WEMoE.","tokens_in":18916,"tokens_out":6848,"duration_ms":69210,"significance":"The proposed direction of using weight-distribution statistics to predict merging coefficients and training the predictor by teacher distillation is interesting and, if properly validated, would be a useful addition to the model-merging literature. The paper ships code and includes ablations (Tables 5 and 6) that support the incremental value of the statistical features, and the unseen-task generalization experiments in Table 3 are a relevant extension beyond the standard benchmark. The main concerns are empirical reproducibility and the fairness of the comparison: the state-of-the-art claim rests on StatsMerging++, whose extra validation data are never quantified, and the base method in Table 2 is 4.9 points below WEMoE. These issues must be resolved before the central claim can be assessed.","major_comments":[{"comment":"The abstract, Section 4.2, and Section 5 claim that StatsMerging outperforms WEMoE by 5.1% (94.5% vs. 89.4%), but Table 2 reports the method actually described in Section 3 and Algorithm 1, LW StatsMerging, at 84.5%, which is 4.9 points below WEMoE. The 94.5% figure comes from LW StatsMerging++, which Appendix A defines only as \"an extended version of StatsMerging trained on more validation data.\" No sample counts, split provenance, or overlap checks are given for this variant, and the paper does not state its training budget. Because the central SOTA claim is defined by an undisclosed data quantity, the result is not reproducible and the comparison to data-free methods such as WEMoE is not interpretable. Please report the exact validation sets and sizes for both StatsMerging and StatsMerging++, and either promote the base method as the headline or justify the extra data as a fair comparison.","section":"Abstract, Section 4.2, Table 2, Appendix A"},{"comment":"There is an inconsistency in the merging formula. Section 3.1 defines the merged weights as theta_m = sum_k lambda_k theta_k, and Algorithm 1 line 19 writes theta^l_merged = sum_i lambda_k theta_k, but Figure 1(d) and its caption show theta_MTL = theta_pre + lambda_A T_A + lambda_B T_B with T_k = theta_k - theta_pre. If the experimental implementation uses the task-vector form, the text and Algorithm 1 are wrong; if it uses the raw-weight form, the diagram is misleading. The two formulas produce different merged models for the same lambda, so this ambiguity affects every reported number. Please state the exact merging equation used in the code and align Algorithm 1, Figure 1, and Section 3.1.","section":"Section 3.1, Algorithm 1, Figure 1"},{"comment":"Section 3.3 states that all SML training samples are collected from the validation set only, but the paper never defines a validation split per dataset. Of the eight datasets, MNIST, SVHN, GTSRB, and Stanford Cars do not have official validation splits in the cited versions, and Appendix A does not say whether the validation set was carved from training data or includes test images. If any test images were used to train SML, the reported accuracies, especially the large per-task gains in Table 2 (e.g., SUN397 75.3 to 92.4 and Cars 77.7 to 95.4), would be inflated. Please provide per-dataset validation construction details and an overlap check with the test sets; the claim \"without test samples\" in the abstract needs this evidence.","section":"Section 3.3, Appendix A, Table 2"},{"comment":"The coefficients lambda are learned on validation samples drawn from the same task distributions on which the merged model is tested, and the pseudo-labels come from the very models being merged. This makes lambda a fitted quantity rather than a prediction from weight statistics alone, so the comparison with data-free baselines such as WEMoE and EMR-MERGING is not apples-to-apples. I do not view this as a tautology, but it is a correctness-risk concern: the paper should report a concrete control, e.g., training SML on a held-out subset of validation data and testing on the remaining validation tasks, or limiting the validation budget to the same size used by AdaMerging's test-time adaptation, to show that the gains are not primarily an artifact of fitting to the target distributions.","section":"Section 3.3, Table 2"},{"comment":"The heterogeneous-architecture experiment uses the distillation loss in Eq. (8), L = alpha L_CE(y, y_hat) + (1-alpha) T^2 L_KL(...), which requires ground-truth labels y. This contradicts the abstract's claim that StatsMerging works \"without requiring ground truth labels\" for the heterogeneous setting. In addition, Table 4 compares only against three simple baselines on three tasks; no WEMoE, AdaMerging, or other recent merging method is evaluated in this setting, so the \"first heterogeneous architectural merging method\" claim is not supported by the experiments reported. Please clarify the label requirement for this extension and add heterogeneous-architecture comparisons with the baselines used in Table 2.","section":"Section 4.2, Eq. (8), Table 4"},{"comment":"No error bars or multiple-seed results are reported. Given that several differences are small (e.g., Table 3: AdaMerging++ 68.7 vs. StatsMerging 70.8 in the first group; Table 2: TW StatsMerging 76.4 vs. TW AdaMerging++ 73.7), the reader cannot determine whether the gains are statistically significant. Please report means and standard deviations over at least three seeds for the main merging and generalization tables.","section":"Tables 2 and 3"}],"minor_comments":[{"comment":"Table 2 is referred to in the text as \"Table 4.2,\" and the Appendix robustness tables are numbered Table 5 and Table 6, which collide with the main-text Tables 5 and 6. Please renumber all tables consistently.","section":"Section 4.2 and Appendix B"},{"comment":"The sentence \"Notably, the Equation 3 above is task-wise\" appears to refer to Eq. (2); Eq. (3) is introduced later. Please fix the cross-reference.","section":"Section 3.2"},{"comment":"Appendix B.2 contains an orphaned paragraph beginning \"Identifies 'regularization samples'...\" that appears to be leftover draft text; it should be removed or fully integrated into the discussion.","section":"Appendix B.2"},{"comment":"The caption of Figure 3 does not explain how coefficients are normalized or which layer indices correspond to MHSA, MLP, and LayerNorm blocks; please add this information to make the coefficient analysis interpretable.","section":"Figure 3"},{"comment":"The relationship between Eq. (6), Eq. (7), and Eq. (10) is unclear: Eq. (6) uses ground-truth labels but Section 3.3 says no labels are used. Please clarify which loss is actually minimized in each experiment.","section":"Equations (6), (7), and (10)"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is worth considering, but the main empirical claim currently rests on an underspecified variant with an undisclosed validation budget. The authors should be asked to supply exact data splits, validation sizes, and code-level clarification of the merging formula before the SOTA claim can be evaluated. I also note that the related-work discussion cites several very recent preprints; a tighter focus on the direct baselines would help the reader."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline number is the first thing to check: the 94.5% average accuracy that drives the abstract and conclusion belongs to StatsMerging++, a variant described only as “trained on more validation data.” The actual proposed method, LW StatsMerging, gets 84.5% in Table 2, which is 4.9 points below WEMoE (89.4%). Read the main table before trusting the abstract.\n\nThe core idea has merit. Using weight statistics (mean, variance, magnitude, and SVD singular values) as input to a lightweight MLP that predicts merging coefficients is a plausible design, and the teacher-distillation training scheme is a sensible way to avoid manual labels. The ablation in Table 6 is internally consistent: each statistic contributes, and SVD adds 3.0–3.2% accuracy. That gives some confidence the feature design isn't arbitrary.\n\nThe problems are real and load-bearing. Most importantly, the central SOTA claim is not reproducible: StatsMerging++'s validation data budget is never quantified, and several datasets (MNIST, SVHN, GTSRB, Stanford Cars) have no official validation split, so it's unclear whether the method is carving into training data. That could give it a large, hidden data advantage over data-free WEMoE and explain the anomalously large per-task gains on SUN397 and Cars. Second, Algorithm 1's merge formula (θ_merged = Σ λ_k θ_k) doesn't match the task-vector diagram (θ_pre + Σ λ_k T_k); the paper never clarifies which is actually used. Third, the “first heterogeneous architectural merging” claim is an overreach: the authors distill ViT teachers into a ResNet student and then merge homogeneous models, and they cite Adamms (Du et al., 2025) without discussing it. There's also a stray sentence in Appendix B.2 about “regularization samples” that reads like leftover from another draft.\n\nWho should read this? Practitioners curious about learned coefficient merging might find the feature idea worth testing, but as written, the results cannot be trusted and the method's actual advantage over WEMoE is unestablished. I would still send it to peer review, because the approach is plausible and the reported gap, if real, matters; but the referee should demand exact splits, seeds, and a full specification of StatsMerging++ before any acceptance.","headline":"The headline 94.5% comes from an ill-specified StatsMerging++ variant; the base StatsMerging scores 84.5%, below WEMoE's 89.4%, and the paper overclaims both first-ness and heterogeneous merging.","tokens_in":19380,"tokens_out":2973,"would_cite":false,"duration_ms":32709,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"StatsMerging claims that a small neural network can merge eight task-specific ViT-B/32 models into one that reaches 94.5% average accuracy, using only weight statistics and teacher-distilled pseudo-labels, without ground-truth labels or…","keywords":["model merging","weight statistics","singular value decomposition","task-specific teacher distillation","multi-task learning","vision transformer","task vectors","knowledge distillation"],"falsifier":"Run the same eight-task merging experiment but train StatsMergeLearner using pseudo-labels generated by deliberately broken teachers, for example models with shuffled or randomly reinitialized final layers, on the same validation images; if merged accuracy remains near 94.5%, teacher quality is not what carries the result, and the claimed mechanism is not the active one. A second check is to hold out one task entirely and use the learner trained on the remaining seven to predict its coefficients; a large accuracy collapse on the held-out task would falsify the generalization claim.","tokens_in":18362,"feed_emoji":"🧩","tokens_out":7854,"duration_ms":66850,"temperature":0.7,"pith_summary":"StatsMerging claims that the right signal for merging several fine-tuned vision models is the distribution of their weights, not the raw weights or hand-tuned coefficients. A lightweight MLP, StatsMergeLearner, takes four statistics of each task model's weights — mean, variance, magnitude, and the leading singular values from SVD — and predicts per-layer merging coefficients. The learner is trained by task-specific teacher distillation: each task model labels unlabeled validation images, and the merged model is trained to match those pseudo-labels, so no ground-truth annotations or test samples enter the loop. On eight ViT-B/32 classification tasks, the layer-wise extension StatsMerging++ achieves 94.5% average accuracy, beating WEMoE (89.4%) by 5.1%, and the paper claims this is the first merging method to handle heterogeneous architectures, by distilling different backbones into a single target architecture before merging.","feed_headline":"Merging 8 vision models hits 94.5% via weight stats","feed_subtitle":"A lightweight learner blends models by reading each task model's weight statistics — no labels or test data required.","key_machinery":"The load-bearing object is StatsMergeLearner, a compact two-layer MLP that predicts merging coefficients $\\lambda$ from a weight-statistics vector. The vector is $[\\mu, \\sigma^2, m, \\sigma'_r]$: mean, variance, L2 magnitude, and top-$r$ singular values from SVD of the weight matrices; the singular values are the distinctive component, acting as a proxy for task importance and weight distribution, and the ablation attributes +3.0% (same architecture) and +3.2% (different architectures) average accuracy to them. Training relies on task-specific teacher distillation: each fine-tuned task model is a teacher that pseudo-labels unlabeled validation images, and StatsMergeLearner is fit so the merged model matches those one-hot labels under cross-entropy. For heterogeneous architectures, the machinery adds a preprocessing step in which every task model is distilled into one selected target backbone before statistics are extracted and coefficients are predicted.","core_discovery":"The paper's central claim is that weight-distribution statistics, and especially singular values, are a sufficient guide for predicting how much each task model should contribute to a merged model. For each task $k$, StatsMerging builds a statistics vector $S_k = \\mathrm{stats}(\\theta_k) = [\\mu, \\sigma^2, m, \\sigma'_r]$, where $\\mu$ and $\\sigma^2$ are the mean and variance of the weights, $m$ is the weight magnitude, and $\\sigma'_r$ are the top $r$ singular values from SVD of the weight matrices. A two-layer MLP, StatsMergeLearner, maps $S_k$ to merging coefficients $\\lambda_k$ at task or layer granularity, and the merged weights are assembled in the task-vector form $\\theta_{\\mathrm{merged}} = \\theta_{\\mathrm{pre}} + \\sum_k \\lambda_k (\\theta_k - \\theta_{\\mathrm{pre}})$. The learner is trained by task-specific teacher distillation: each task-specific model pseudo-labels its own unlabeled validation images, and the merged model is optimized with cross-entropy against those pseudo-labels, so no ground-truth annotations or test samples are needed. On eight ViT-B/32 tasks the layer-wise StatsMerging++ reaches 94.5% average accuracy versus 89.4% for WEMoE, and the paper claims the same machinery works for heterogeneous architectures by first distilling all backbones into one target architecture.","pith_inferences":["Editorial inference: if singular values really encode task importance, the same statistics-to-coefficient map might transfer to a ninth unseen task without retraining StatsMergeLearner; the paper tests generalization to two unseen tasks but does not test this zero-shot coefficient transfer.","Editorial inference: the paper explicitly limits itself to vision classification, so applying the same weight-statistics recipe to detection, restoration, or language models is an open question, since their weight distributions and teacher reliability differ.","Editorial inference: a natural stress test would be to train StatsMergeLearner on a few tasks and apply it to many more; the sparse, layer-recurring coefficient patterns the paper visualizes suggest the learner may be capturing architecture structure (attention blocks, MLP layers, norms) more than task identity, which would imply broad transfer.","Editorial inference: the teacher-distillation setup could be tightened by ablating teacher quality, for example by corrupting a fraction of pseudo-labels, to measure how much of the 94.5% depends on teacher accuracy versus the statistics features themselves."],"forward_implications":["A single merged ViT-B/32 can replace eight separately fine-tuned models for SUN397, Cars, RESISC45, EuroSAT, SVHN, GTSRB, MNIST, and DTD, holding 94.5% average accuracy versus 90.5% for the individually trained models.","Merging no longer needs ground-truth labels or test samples, only unlabeled validation images and the existing task teachers, which removes the main supervision bottleneck in learning-based merging.","Because coefficients are inferred per layer from weight statistics, the predictor can be applied at test time to new task models without retraining the learner, provided the same statistics are available.","Heterogeneous architectures can be merged by first distilling all task models into a shared backbone; the paper reports 81.3% average accuracy across ResNet50 and ViT-B/32 models, beating Task Arithmetic's 73.7%.","Robustness to image corruption improves: under impulse noise, StatsMerging gains 6.3 percentage points over AdaMerging on a four-task suite."],"supporting_citations":[{"why":"Supplies the AdaMerging protocol, evaluation setup, and the AdaMerging baselines that StatsMerging is compared against and outperforms.","marker":"Yang et al., 2023"},{"why":"WEMoE is the state-of-the-art baseline (89.4% average accuracy) that the paper's 94.5% result is framed against.","marker":"Tang et al., 2024"},{"why":"Defines task vectors and task arithmetic, the combination structure that StatsMerging's coefficients plug into.","marker":"Ilharco et al., 2023"},{"why":"Ties-Merging supplies the high-magnitude parameter intuition that motivates including weight magnitude in the statistics vector, and is a baseline.","marker":"Yadav et al., 2023a"},{"why":"Provides the CLIP ViT-B/32 pre-trained backbone from which all task-specific models are fine-tuned.","marker":"Radford et al., 2021"},{"why":"Provides the knowledge-distillation framework that task-specific teacher distillation builds on.","marker":"Hinton et al., 2015"}],"fun_headline_variants":["Weight stats guide model merging without labels","Singular values proxy task importance for model merging","Teacher distillation enables label-free model merging","StatsMerging blends vision models via weight distributions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that unlabeled validation images for each task are available and that the task-specific teachers' pseudo-labels on those images are accurate enough to train the coefficient predictor; the paper itself restricts its claims to vision classification tasks, so if the validation images are unrepresentative, or the teachers are unreliable, the predicted coefficients and the reported gains will not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Weight stats guide model merging without labels","Singular values proxy task importance for model merging","Teacher distillation enables label-free model merging","StatsMerging blends vision models via weight distributions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1468,"prompt_tokens":1084,"completion_tokens":384,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":700,"completion_tokens_details":{"reasoning_tokens":328}},"tokens_in":700,"tokens_out":384,"duration_ms":14256,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:39:50.709977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same eight-task merging experiment but train StatsMergeLearner using pseudo-labels generated by deliberately broken teachers, for example models with shuffled or randomly reinitialized final layers, on the same validation images; if merged accuracy remains near 94.5%, teacher quality is not what carries the result, and the claimed mechanism is not the active one. A second check is to hold out one task entirely and use the learner trained on the remaining seven to predict its coefficients; a large accuracy collapse on the held-out task would falsify the generalization claim.","supporting_citations":[],"review_version":1}