{"id":"2124cf2d-0e2f-4889-9b90-6a753e65f078","arxiv_id":"2608.01184","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SAFE-Merge masks risk-prone parameter updates and recovers lost task information with a constrained low-rank correction, achieving the best H-score in data-free continual model merging benchmarks.","lead":"This paper presents a method for continually merging fine-tuned AI models without access to the original training data, while trying to preserve the general knowledge of the pretrained model. The method, SAFE-Merge, masks risky parameter updates and adds a low-rank correction, and the authors report higher balanced H-score performance than prior merging methods on vision and language benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported H-score gains may reflect per-benchmark tuning of ρ and λ on the evaluation tasks themselves, not a data-free advantage","rationale":"The reader identified the risk-proxy assumption (Eq. 3) as the weakest assumption. I agree that is a substantive concern, and it is well articulated. However, I believe the more load-bearing issue for the paper's central empirical claim is the hyperparameter selection procedure. The paper explicitly uses the benchmark trade-off curves to set ρ and λ, without a separate validation set or nested protocol. This directly affects the reported H-scores and the fairness of comparisons with baselines, and it undermines the 'data-free' framing. The risk proxy, while theoretically under-supported, has direct empirical validation in Fig. 1 (positive vs. negative selection), which gives it some credibility. The hyperparameter tuning has no such direct support and is, in fact, an acknowledged post-hoc choice. A concrete check—fixing defaults or using held-out task orders—would settle whether the claimed advantage is real. The reader's verdict of CONDITIONAL already encompasses this concern, so I recommend UNCHANGED.","tokens_in":25057,"tokens_out":9056,"duration_ms":94521,"concrete_test":"Re-run Tables 1, 2, and S.5 with a single fixed configuration (e.g., ρ=0.5, λ=0.8) across all backbones and stream lengths, or select ρ/λ on two held-out task orders and evaluate on the remaining eight. Compare SAFE-Merge's H-score with the baselines under the same protocol. If SAFE-Merge does not achieve the best H-score in a majority of settings, the headline claim is not supported as a data-free method.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim 'consistently achieves the best H-score' is not yet supported as a data-free statement because the key hyperparameters (keep ratio ρ, plasticity weight λ) are selected using the evaluation tasks themselves. In Fig. 3, ACC and Gen. are plotted against ρ and λ for the exact 8/14/20-task streams and the ImageNet/IN-R/ObjectNet probes that later appear in Tables 1 and S.5. The appendix A.6 then states the knees at ρ=0.3–0.5 'motivate stronger sparsification for longer sequences', indicating the final ρ values were read off these benchmark curves. No held-out validation split or nested protocol is described; the 10 task orders are all used for the reported means. Thus the reported gains over baselines (which follow their original, not per-benchmark, hyperparameters) may be an artifact of oracle tuning rather than a property of the merging algorithm. This is especially problematic because the method is advertised as data-free: a deployer without access to these exact benchmarks cannot reproduce the reported configuration. The risk-proxy assumption in Eq. (3) is also a concern, but it has at least the direct positive/negative support of Fig. 1; the tuning issue has no such support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"SAFE-Merge proposes a data-free continual model merging method that first masks incoming task-vector updates according to a per-parameter risk score, then learns a masked low-rank recovery term, and finally fuses the result into the running backbone. The risk score contrasts energy along right singular vectors of the pretrained weights (a proxy for general-knowledge-sensitive directions) with energy along task-specific directions. The paper reports that SAFE-Merge consistently achieves the best harmonic mean of merged-task accuracy and held-out general performance across CLIP vision streams of 8, 14, and 20 tasks, three CLIP backbones, ten task orders, and a Flan-T5 GLUE benchmark, while adding no inference-time parameters. An ablation study and rank-sensitivity analysis support the design choices, and a positive-versus-negative selection experiment (Fig. 1) gives direct evidence that the risk score orders parameter updates by their damage to held-out general accuracy.","tokens_in":25387,"tokens_out":5014,"duration_ms":52195,"significance":"If the central claims hold, SAFE-Merge is a worthwhile step for data-free continual merging: it explicitly targets preservation of pretrained general knowledge rather than only downstream-task interference, and it does so with a simple parameter-space procedure that leaves masked parameters untouched and adds no inference cost. The empirical coverage is unusually thorough for this area: three backbones, three stream lengths, ten task orders, ablations, rank sensitivity, a language benchmark, and an overhead table. The positive/negative selection experiment in Fig. 1 is a particularly good falsifiable check of the risk-scoring mechanism. However, the paper's central claim of a \"data-free\" advantage is weakened by the use of the evaluation benchmarks themselves to select key hyperparameters, and the principal risk-scoring proxy is validated on only one backbone. Neither issue is fatal, but both need to be addressed before the claims can be accepted at face value.","major_comments":[{"comment":"The keep ratio ρ and plasticity weight λ are selected using the evaluation benchmarks themselves. Fig. 3(a) plots ACC/Gen. on the exact 8/14/20-task streams with the exact IN/IN-R/ObjNet probes for the ρ values later used in Tables 1 and S.5, and Fig. 3(b) does the same for λ; A.6 states the knees “motivate stronger sparsification for longer sequences,” so the final ρ values are read off these curves. No held-out split or nested protocol is described. Since the baselines use their original, not per-benchmark, hyperparameters, the reported “data-free” gains and the claim of consistently best H-score may be an artifact of oracle tuning. Please add a validation protocol (e.g., tune on one held-out task set and evaluate on another, or fix ρ/λ a priori) and show the gains survive.","section":"§Experiments/Implementation Details, Fig. 3, Appendix A.6"},{"comment":"The risk score replaces the unavailable general-data covariance C_g with Q_0 Q_0^T, the top right singular subspace of the pretrained weight. Appendix A.1 explicitly calls this a proxy, not an exact identity, and Fig. 1 validates it on ViT-B/32 only. The guarantee that low-risk masks protect general knowledge depends on this correspondence holding for the deployment distribution. Please provide the same positive/negative validation for ViT-B/16 and ViT-L/14 (and ideally the NLP backbone), or an explicit comparison with a data-derived covariance on a held-out general set; otherwise the central risk-ordering claim is under-supported.","section":"§Motivation, Eq. (3), Appendix A.1"},{"comment":"The theoretical justification of the recovery objective imports Corollary 1 of Qiu et al. (2026) without proof and without checking the misalignment term ζ or the data-dependent scales σ_1(X). The paper correctly states that the result only motivates the objective, but the main-text presentation nevertheless frames the recovery as solving the preservation problem. The gap between an unmeasured upper bound and the actual loss should be stated more prominently in the main text, or the corollary should be proved in the appendix under the specific SAFE-Merge variables.","section":"§Appendix A.2, Eq. (S.3)–(S.5)"}],"minor_comments":[{"comment":"The Naive accumulation and Risk-aware mask-only rows report ±0.0 even though results are said to be averaged over ten task orders; task order should affect these quantities. Please clarify whether these rows use a single order or whether the zeros are a typographical artifact.","section":"Table 3"},{"comment":"The phrase “without task-specific tuning” is contradicted by ρ varying with stream length and backbone (0.5/0.3 for ViT-B/32, 0.7/0.5 for ViT-B/16 and L/14). Rephrase to say ranks are shared while ρ is adjusted by protocol.","section":"§Experiments/Implementation Details"},{"comment":"No code or checkpoint release is indicated. For reproducibility, please release the merge script, the exact GF/FusionBench checkpoint identifiers, and the task permutations used (Table S.1 is helpful but not machine-readable).","section":"§Experiments, reproducibility"},{"comment":"The terms “positive selection” and “negative selection” are defined only in the running text; add a one-line definition in the caption so the figure stands alone.","section":"Fig. 1 caption"}],"recommendation":"major_revision","confidential_remarks":"The overlap with the same group's NUFILT paper (Qiu et al. 2026) is significant: shared first author, shared task-vector/subspace machinery, and identical rank choices. The novelty here is the risk-aware mask and the masked recovery; that is enough for a distinct contribution if the evaluation is made fair. The hyperparameter-selection issue is the main risk to the paper's central claim. If the authors can show the H-score advantage survives under an honest selection protocol (or at least report both tuned and fixed-default configurations), the paper could be acceptable after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: SAFE-Merge is a genuine step in continual model merging — it explicitly targets preservation of pretrained general knowledge rather than only task interference, and it has the experiments to back that up. The risk-aware mask that contrasts pretrained singular directions against task-unique directions, plus the masked low-rank recovery constrained to safe entries, is a clean idea and not present in the cited literature. On three CLIP backbones, three stream lengths, ten task orders, and a GLUE benchmark, it reports the best H-score, and the ablations show the two components complement each other: masking alone helps a lot, recovery alone is unstable at length, together they're solid. I believe the 3–8 point H-score gains over NUFILT are real as reported.\n\nThe soft spots are two. First, the keep ratio ρ is chosen per backbone and stream length by looking at the ACC/Gen trade-off curves on the exact evaluation tasks (Fig. 3, Appendix A.6). The appendix says the knees at ρ=0.3–0.5 'motivate stronger sparsification for longer sequences' — that reads like the test tasks were used to pick the final configuration. There is no held-out validation or nested protocol described. For a method advertised as data-free, that's a load-bearing gap: a deployer without access to these exact benchmarks wouldn't know how to set ρ. This doesn't destroy the contribution, because the risk-score validation in Fig. 1 (positive vs negative selection) holds over a wide range of ρ, but it weakens the specific 'consistently best H-score' claim.\n\nSecond, the theoretical justification for the recovery objective leans on Corollary 1 from Qiu et al. (2026), a first-author-overlapping prior paper, and the central risk proxy — using pretrained weight singular vectors to approximate general-data covariance — is admitted to be a proxy, validated only on one backbone and three probes. That is a reasonable engineering bet, but it's not a theorem, and the paper would be stronger with a second backbone or a synthetic distribution check.\n\nMinor: no code is released, so reproducing the exact numbers requires significant effort. That matters less for the idea and more for the headline claims.\n\nBottom line: this deserves a serious referee. The method is clearly specified, the evaluation is above the field's average, and the authors are honest about several limitations. I'd send it out and ask the reviewers to press on the hyperparameter protocol and ask for a minimal code release. If those come back clean, this is a solid paper that will be cited.","headline":"A well-evidenced continual-merging method that deserves review, but its headline advantage may depend on hyperparameters chosen on the evaluation tasks themselves.","tokens_in":25841,"tokens_out":3737,"would_cite":true,"duration_ms":33515,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SAFE-Merge protects a merged model's general knowledge by masking risky parameter updates, then recovers lost task skill with a low-rank correction.","keywords":["continual model merging","data-free merging","general knowledge preservation","risk-aware masking","low-rank recovery","task vectors","H-score","CLIP"],"falsifier":"On a backbone where the pretrained weights' singular directions are known not to align with the general data covariance (e.g., a CLIP text tower applied to a vision stream, or a randomly permuted weight matrix), run the positive-versus-negative selection experiment from the paper's Figure 1: if retaining low-risk entries no longer keeps held-out accuracy near the pretrained level while high-risk selection does no worse, the proxy fails.","tokens_in":24921,"feed_emoji":"🛡️","tokens_out":7487,"duration_ms":70327,"temperature":0.7,"pith_summary":"This paper takes on a blind spot in data-free continual model merging: existing methods protect previously merged tasks from interference but treat the pretrained backbone as a neutral container, so general knowledge silently erodes. SAFE-Merge claims that this erosion can be prevented without any task data by deciding, parameter by parameter, which updates are safe. It scores each entry of an incoming task vector by contrasting its energy along directions of the pretrained weights (risky) against directions unique to the new task (valuable), masks out the high-risk entries, and then recovers most of the lost task information with a low-rank correction confined to the kept entries. On CLIP vision streams of 8, 14, and 20 tasks and on five GLUE tasks with Flan-T5, it reports the best harmonic mean of merged-task accuracy and held-out general performance, with the gap over baselines growing as sequences lengthen. The payoff, if right, is that a deployment system can keep folding in specialized checkpoints without sacrificing the model's broader competence.","feed_headline":"Masking risky updates preserves general knowledge in model merging","feed_subtitle":"SAFE-Merge scores each parameter change and recovers lost task skill—no task data, no extra inference cost.","key_machinery":"The method has two coupled parts. (1) Risk-aware sparse masking: for each linear layer, compute $Q_0^{(l)}$ from the top-$r_p$ right singular vectors of the pretrained weight and $Q_{\\mathrm{new}}^{(l)}$ from the incoming task vector's right singular vectors after removing its overlap with $Q_0^{(l)}$; the per-entry score $R^{(l)}_{ij} = (\\tau_t^{(l)} Q_0^{(l)}{Q_0^{(l)}}^\\top)^2_{ij} - (\\tau_t^{(l)} Q_{\\mathrm{new}}^{(l)}{Q_{\\mathrm{new}}^{(l)}}^\\top)^2_{ij}$ separates overwrite risk from task utility, and the mask $M_t$ keeps the lowest-scoring $\\rho$ fraction. (2) Masked low-rank recovery: a low-rank factor pair $(B_t^{(l)}, A_t^{(l)})$ is optimized, its product masked by $M_t$, so no rej","core_discovery":"The central claim is that parameter-wise overwrite risk can be read off parameters alone. Treating the pretrained weight matrix as a generalized update, its top-$r_p$ right singular vectors $V_0^{(l)}$ approximate the dominant input-activation subspace of the general training distribution; the risk score $R^{(l)}_{ij}$ in Eq. (6) then quantifies, for each entry of the incoming task vector $\\tau_t^{(l)}$, whether its projected energy falls mostly on those general directions or on the orthogonalized task-unique directions $Q_{\\mathrm{new}}^{(l)}$. Retaining only the low-risk entries (the mask $M_t$) and adding a low-rank recovery $M_t \\odot (\\tau_t + B_t A_t)$ optimized against old-task and ne","pith_inferences":["If the singular-direction proxy transfers beyond CLIP and Flan-T5, the same score could act as a general safety filter for deciding which user-adapted checkpoints are safe to aggregate or share.","The keep ratio $\\rho$ is scheduled by backbone and stream length; a natural extension is to make it adaptive per layer or per incoming task using the separation between the risk-score distributions.","The mask-plus-recovery split suggests a general design: spend a hard safety budget on a small, trusted set of coordinates and use a constrained correction to squeeze task value out of that same set — an idea that could apply to other plasticity-stability settings.","A testable consequence: apply the risk score to partially fine-tuned or LoRA-adapted models; if the weight-to-covariance correspondence holds there, the method extends beyond full fine-tuning."],"forward_implications":["If correct, data-free continual merging can protect held-out generalization without any calibration data or replay, since the risk score is computed purely from pretrained, merged, and incoming parameters.","The safety-rather-than-interference framing means the benefits should grow as more tasks accumulate, and indeed the paper reports the largest H-score gains on the 20-task stream.","The fused update adds no inference-time parameters or latency, so the preservation comes for free at deployment.","The positive/negative selection experiment offers a reusable diagnostic: the risk score can rank any task vector's entries by how much they threaten general knowledge."],"supporting_citations":[{"why":"Introduces task-vector arithmetic, the representation of fine-tuned models as displacements from a shared pretrained initialization that SAFE-Merge operates on.","marker":"(Ilharco et al. 2023)"},{"why":"Supplies the parameter-derived singular-direction proxy for data subspaces and the upper-bound result (Corollary 1) used to justify the data-free recovery objective.","marker":"(Qiu et al. 2026)"},{"why":"Ties-Merging, the sign/magnitude sparsification baseline whose treatment of all parameter updates alike motivates risk-aware selection.","marker":"(Yadav et al. 2023)"},{"why":"The CLIP vision-language backbones whose pretrained general knowledge is the object of preservation in all vision experiments.","marker":"(Radford et al. 2021)"},{"why":"FusionBench supplies the public fine-tuned CLIP checkpoints and the 20-dataset vision suite used for the merge streams.","marker":"(Tang et al. 2024a)"},{"why":"GLUE provides the five merged NLP tasks and the three held-out probes (MNLI, QNLI, RTE) of general linguistic knowledge.","marker":"(Wang et al. 2019)"},{"why":"OPCM, the orthogonal-projection baseline whose rigid removal of overlapping directions the method contrasts with.","marker":"(Tang et al. 2025)"},{"why":"ImageNet serves as a held-out probe of general visual knowledge in the risk-score validation and final evaluations.","marker":"(Deng et al. 2009)"},{"why":"ImageNet-R serves as a held-out robustness probe of general visual knowledge.","marker":"(Hendrycks et al. 2021)"},{"why":"ObjectNet serves as a held-out bias-controlled probe of general visual knowledge.","marker":"(Barbu et al. 2019)"}],"fun_headline_variants":["Mask risky updates, keep general knowledge","Data-free merging that shields pretrained knowledge","Risk-scored masks keep general knowledge intact","SAFE-Merge: protect base knowledge while merging"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the top right singular vectors of the pretrained weight matrix capture the same directions that carry large energy in the general training distribution's input covariance; if that correspondence breaks on a different backbone or under distribution shift, the mask will protect the wrong parameters and the general-knowledge guarantee is lost.","fun_headline_variants_meta":{"raw":{"variants":["Mask risky updates, keep general knowledge","Data-free merging that shields pretrained knowledge","Risk-scored masks keep general knowledge intact","SAFE-Merge: protect base knowledge while merging"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000473,"raw_usage":{"total_tokens":2196,"prompt_tokens":759,"completion_tokens":1437,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":1381}},"tokens_in":503,"tokens_out":1437,"duration_ms":9807,"temperature":1.0,"reasoning_tokens":1381,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:26:35.646962+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a backbone where the pretrained weights' singular directions are known not to align with the general data covariance (e.g., a CLIP text tower applied to a vision stream, or a randomly permuted weight matrix), run the positive-versus-negative selection experiment from the paper's Figure 1: if retaining low-risk entries no longer keeps held-out accuracy near the pretrained level while high-risk selection does no worse, the proxy fails.","supporting_citations":[],"review_version":1}