{"id":"f743c866-eb1c-4356-b8bf-9ba5d9d9c5c8","arxiv_id":"2602.04509","paper_version":7,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A data-free importance score (Jacobian norm × weight × activation) computed on synthetic prompts, used to freeze the most output-critical parameters during fine-tuning, mitigates catastrophic forgetting in MLLMs better than prior methods.","lead":"Model-Dowser is a fine-tuning recipe that freezes the parameters of a multimodal AI model that are most important for its pretrained skills — identified by probing the model with synthetic text and measuring output sensitivity — so that learning a new task does not wipe out old knowledge. It is memory-light, needs no original training data, and beats prior anti-forgetting methods when fine-tuning reaches deep layers of the language model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The mask's theoretical justification hinges on substituting |W| for the true update ΔW; with Adam-style updates this substitution is unvalidated, so the claimed principled importance score may not explain the reported retention.","rationale":"The paper's empirical core is coherent: the arithmetic in the tables checks out, the method is memory-light, and the depth-sweep diagnosis is a useful contribution. The random-selection baselines with significance tests and the Appendix G stability analysis are genuine supporting evidence. However, the central theoretical claim—that the mask preserves pretrained generalization because it freezes the parameters that would dominate the Corollary 3.2 bound—depends entirely on equating the unknown perturbation ΔW with the known weight magnitude |W|. In modern optimizers this equation has no a priori justification; if the proposed test shows little correlation between |W| and |ΔW|, then the paper's explanation of its own success collapses to an untested heuristic. That would not overturn the empirical H-score improvements, but it would remove the claimed 'principled' basis and change the paper's contribution from a theoretically motivated method to an empirical heuristic. This is exactly the weakest assumption identified by the reader, and it remains the most load-bearing point because it sits between the theorem and the algorithm. The separate SPIDER-depth inconsistency and single-seed reporting are real but secondary; they affect strength of comparison, not the internal logic of the method. Therefore the appropriate verdict remains CONDITIONAL, pending the correlation test and the requested clarifications.","tokens_in":29528,"tokens_out":9666,"duration_ms":114984,"concrete_test":"Run a full fine-tuning run on NVILA-Lite-2B COCO-Caption using the paper's exact setup (10k samples, 5 epochs, AdamW, lr=2e-5) but without any mask, and record the actual per-parameter ΔW. Then compute (a) Spearman correlation between |W_{ij}| and |ΔW_{ij}|, and (b) Spearman correlation between the proposed S̄_{ij} and the true per-parameter contribution (1/N)Σ_n ||J^{(l)}_{i,n}||_2 · |ΔW_{ij}| · |h^{(l-1)}_{j,n}|, using the same synthetic probes. If either correlation is weak (e.g., < 0.1), the mask selected by Eq. (3) is not selecting the output-shift-dominant parameters, and the theoretical justification in Corollary 3.2 is not load-bearing for the empirical result. A complementary check: compare the top-90% mask from S̄ with the top-90% mask from the true contribution; chance-level overlap would indicate the theory does not explain the freezing strategy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is Eq. (3): S^{(l)}_{ij} = ||J^{(l)}_i||_2 · |W^{(l)}_{ij}| · |h^{(l-1)}_j|. Theorem 3.1 and Corollary 3.2 bound output shift in terms of the true perturbation |ΔW_{ij}|, but the mask (Eq. 7) ranks parameters by |W_{ij}|. Appendix B explicitly concedes that this is only a 'proxy' for ΔW. Nothing in the paper establishes that Adam/AdamW updates scale with current weight magnitude; in fact, Adam's update is roughly lr·sign(g)/sqrt(v), which is nearly independent of |W|. If the rank correlation between |W| and |ΔW| is low, then the top-90% parameters frozen by Eq. (3) are not the dominant terms in the Corollary 3.2 bound, and the theoretical justification does not explain the reported retention. This is not a disagreement with external consensus; it is an unsupported inference inside the paper's own derivation. Appendix G validates the Hutchinson/text-probe estimator against real-data scores, but it does not validate the ΔW→|W| substitution, which is the actual bridge from the bound to the mask.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Model-Dowser, a sparse fine-tuning method for multimodal large language models. Before adaptation, it computes a parameter importance score S̄_ij = (1/N) Σ_n ‖J^{(l)}_{i,n}‖₂ |W^{(l)}_{ij}| |h^{(l-1)}_{j,n}|, where the Jacobian is estimated with Hutchinson probing on synthetic text-only prompts, and then freezes the top-ρ parameters during downstream fine-tuning. The authors claim that this preserves pretrained generalization better than full fine-tuning, post-merging methods (Grafting, DARE, ModelTailor), and the sparse baseline SPIDER across LLaVA-1.5-7B and NVILA-Lite-2B on captioning, classification, and VQA benchmarks. The paper includes depth sweeps, mask-ratio sweeps, random-mask comparisons with significance tests, a memory-complexity analysis, and a continual-learning extension.","tokens_in":29922,"tokens_out":11626,"duration_ms":123675,"significance":"The empirical work is broad and internally consistent: across two architectures, four downstream tasks, six upstream benchmarks, and multiple fine-tuning depths, Model-Dowser reports the best H-score in almost every condition, with large margins over Tailor and SPIDER in deep fine-tuning regimes. The random-mask baselines with significance tests provide credible evidence that the importance ranking itself carries signal. If the two central gaps—the ΔW→|W| substitution and the text-only probing distribution—can be closed by direct validation, Model-Dowser would be a useful and scalable addition to the forgetting-mitigation toolbox. In its current form, however, the theoretical framing overstates what is proven, and the headline H-scores on captioning tasks are computed from a unit-inconsistent metric. The method is not circular in the sense that upstream evaluation benchmarks do not enter the score computation, and the reported arithmetic is reproducible.","major_comments":[{"comment":"The bridge from the bound to the mask is the substitution ΔW_{ij} → |W_{ij}|. Theorem 3.1 and Corollary 3.2 bound the output shift in terms of the true perturbation |ΔW|, but Eq. (7) freezes parameters ranked by |W|. Appendix B explicitly calls this a 'conservative first-order surrogate,' yet no evidence is given that fine-tuning updates scale with current weight magnitude. For Adam/AdamW the per-step update is adaptive and roughly proportional to sign(g)/√v, not to |W|. If |W| and |ΔW| are weakly correlated, freezing the |W|-largest parameters need not suppress the dominant terms in Corollary 3.2. At minimum, the authors should measure the rank correlation between |W| and the actual |ΔW| observed after fine-tuning, or compare masks built from the two quantities; otherwise the claim that the score 'explains why preserving high-score parameters helps' is unsupported.","section":"§3.1, Eq. (3), Appendix B"},{"comment":"The probing distribution is text-only: the experiments state 'synthetic text samples N=64,' and Eq. (6) generates prompts from random token seeds. Downstream fine-tuning, however, is multimodal and the forgetting being mitigated is driven by image-conditioned inputs. The score never exercises the vision encoder or cross-attention pathway, so those parameters are effectively unscored. Appendix G validates the Hutchinson/text-probe estimator against 'real data samples,' but it does not establish that text-only probes activate the same functional structure as the multimodal inputs used during downstream training. The authors should either compare masks computed with image-conditioned probes, or give an explicit argument for why language-decoder activations alone are sufficient.","section":"§3.2, Eq. (6), §4.1, Appendix G"},{"comment":"For image captioning, A_down is CIDEr, which is unbounded and can exceed 100 (e.g., 135.5 in Table 1), while A_up is an accuracy in [0,100]. The reported H-score is the harmonic mean of these two quantities, and Avg is their arithmetic mean. This conflates different units and makes the captioning H-scores/AVG not interpretable as accuracies. For example, with A_up≈62.7 and A_down≈135.5, the harmonic mean is ≈85.7 solely because CIDEr is large. The headline captioning results should be recomputed with a normalized A_down (e.g., CIDEr normalized to [0,100]) or reported separately for upstream and downstream without a single combined score.","section":"§4.1, Eq. (9), Tables 1–2, 5"},{"comment":"There is no ablation isolating the three factors of the importance score. The paper's central claim is that jointly combining output sensitivity ‖J_i‖, weight magnitude |W_{ij}|, and input activity |h_j| is what makes the mask effective. The comparisons with SPIDER, Tailor, etc., are against whole methods that differ in multiple design choices, so they do not identify the contribution of the proposed score. Please include masks based on |W| alone, ‖J‖·|W|, ‖J‖·|h|, |W|·|h|, and the full product; this would directly test the 'jointly considering' claim and the role of each term.","section":"§3.1–3.2, §4.2"}],"minor_comments":[{"comment":"The Hutchinson estimator in Eq. (4) yields ‖J_i‖², while the score uses ‖J_i‖ and the main text also mentions E|ξ^T J_i| as an L1 surrogate. Please state explicitly how the square root is taken in practice and which estimator is actually used in the reported results.","section":"Eq. (4) vs Eqs. (3)/(5), Appendix D"},{"comment":"The significance tests use t-tests over only three random seeds. Please state the normality assumption or provide more seeds, and make the one-sided/two-sided choice consistent between Tables 4 and 6.","section":"Tables 4 and 6"},{"comment":"The row for Grafting lists ρ=100% but '# params' = 143M/438M, the same as methods with ρ=10%. Please clarify what '# params' counts and why Grafting's count differs from Full-FT's if it fine-tunes 100% of the last-20-layer parameters.","section":"Table 3"},{"comment":"Several typos: 'COCO-Cpation' in Appendix F, 'NVLILA' in the Table 6 caption, and 'upstream performance average of v' in Appendix H.2. The duplicated citation for Xu & Zhang (2024a/b) should also be merged.","section":"Appendix F, Table 6 caption, Appendix H.2"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a strong, consistent empirical trend and the arithmetic checks out, so I am not recommending rejection. However, the theoretical contribution is currently not supported by the experiments: the ΔW→|W| substitution is unvalidated, the text-only probing is not shown to cover the visual pathway, and the captioning H-scores are unit-inconsistent. These are addressable with additional experiments or a substantially tempered set of claims, which is why I recommend major revision rather than reject."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe two things you should know: first, the central empirical claim survives reading. A static, data-free importance mask computed before fine-tuning preserves upstream generalization substantially better than existing baselines when you fine-tune deep into an MLLM, and the arithmetic across the tables is consistent. Second, the theoretical story is weaker than the authors claim, and the weak link is exactly the substitution of |W| for the true update ΔW in Eq. (3).\n\nThe depth-sweep diagnosis is the genuine contribution. Prior work mostly evaluated shallow fine-tuning; showing that post-merging baselines collapse at larger depths, and that a sparse mask keeps working there, is useful and could change what practitioners do when adapting billion-parameter models. The method is memory-light and conceptually clean, and the appendix validation of the Hutchinson estimator against real-sample scores is a nice, reproducible check. The continual-learning experiment is brief but consistent with the main story.\n\nThe soft spots are real but mostly do not break the empirical claim. The stress-test concern is correct: the bound in Corollary 3.2 is about the true perturbation |ΔW|, while the mask ranks by |W|. Appendix B concedes this is a proxy, but the concession is buried and the main text later calls the score \"principled\" and says freezing high-score parameters preserves the dominant contributors. For Adam, ΔW is roughly lr·sign(g)/sqrt(v), which is nearly independent of |W|; nothing in the paper shows the rank correlation between |W| and |ΔW| is high. So the theory as written is motivational, not explanatory. That is a legitimate referee point, but it is separable from the empirical result: the mask could work for other reasons, and the comparison against random selection at multiple mask ratios shows the importance ranking matters.\n\nTwo more genuinely soft spots. The probing distribution is text-only synthetic prompts; the visual pathway is never scored, yet the forgetting being mitigated is driven by multimodal inputs. The appendix's real-data comparison helps, but the paper should clarify what \"real data\" means and justify why text-only probing suffices. There is also an internal contradiction: the text says SPIDER cannot be trained on LLaVA-1.5-7B when L>20 due to memory, but Figure 1(c) shows SPIDER at depth 32. The main tables have no error bars or significance testing against SPIDER, only against random selection. None of this is fatal, but it needs fixing.\n\nWho is this for? Anyone adapting MLLMs who wants a cheap, data-free way to protect pretrained knowledge. It deserves a serious referee. I'd send it out, with requests for code, multi-seed results against the strongest baseline, a clarification of the probing distribution, and either validation or a clear downgrade of the ΔW proxy.\n\nBest,\n[Your name]","headline":"The depth-sweep empirical result is solid and genuinely useful; the theoretical justification is a heuristic that the authors should either validate or clearly label as such.","tokens_in":30405,"tokens_out":2810,"would_cite":true,"duration_ms":34845,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A pre-fine-tuning mask built from Jacobian sensitivity, weight magnitude, and input activation preserves MLLM generalization during deep fine-tuning, outperforming prior methods.","keywords":["catastrophic forgetting","multimodal large language models","sparse fine-tuning","parameter importance","Jacobian sensitivity","data-free probing","stochastic trace estimator","model merging"],"falsifier":"Compute the Model-Dowser mask on a small MLLM, run standard Adam fine-tuning, and measure the rank correlation between |W| and actual per-parameter update magnitudes ΔW; if the correlation is near zero, or if a mask built from true ΔW performs no better than the |W|-based mask, the proxy bridging theory to algorithm is not doing the work.","tokens_in":29389,"feed_emoji":"🧠","tokens_out":8903,"duration_ms":74318,"temperature":0.7,"pith_summary":"Model-Dowser is a sparse fine-tuning method for multimodal large language models (MLLMs) that aims to prevent catastrophic forgetting when models are adapted to downstream tasks. It computes a per-parameter importance score before fine-tuning—weight magnitude times input activation times output sensitivity estimated via synthetic probes—and freezes the most important parameters while updating the rest. The paper proves a first-order bound showing that these scores aggregate to an upper bound on the output shift caused by parameter updates, and shows empirically that the method maintains pretrained zero-shot performance even when fine-tuning extends to earlier decoder layers, outperforming prior methods on LLaVA and NVILA across captioning, classification, and VQA tasks. The approach is data-free, needs no extra training memory, and scales to multi-billion-parameter models.","feed_headline":"One precomputed mask preserves MLLM knowledge during deep fine-tuning","feed_subtitle":"Freezing the 10% most output-sensitive weights keeps zero-shot knowledge while adapting to new tasks, at no extra memory.","key_machinery":"The load-bearing object is the parameter-wise importance score S_ij = ‖J_i‖_2 |W_ij| |h_j|, a first-order sensitivity measure combining output Jacobian norm, weight magnitude, and input activation. It is estimated without data via stochastic trace estimation on synthetic text prompts, and converted into a static binary mask before fine-tuning. The argument that freezing high-S parameters controls output drift rests on the first-order Taylor bound ‖Δf‖_2 ⪅ Σ_ij ‖J_i‖_2 |ΔW_ij| |h_j| (Corollary 3.2).","core_discovery":"The paper shows that catastrophic forgetting in MLLMs can be mitigated by preserving the parameters whose perturbation would most shift the network's outputs, measured by the score S_ij = ‖J_i‖_2 · |W_ij| · |h_j|, where J_i is the Jacobian column relating the output to the i-th pre-activation, |W_ij| is the weight magnitude, and |h_j| is the input activation. The score is estimated data-free by probing the frozen model with synthetic prompts and stochastic trace estimation with random sign vectors, then converted into a per-layer binary mask that freezes the top (1−ρ) fraction of high-importance weights. The authors prove (Corollary 3.2) that the total output shift under multi-weight perturb","pith_inferences":["The same importance score could be repurposed for other data-free operations on frozen models—pruning, quantization sensitivity analysis, or identifying safety-critical neurons—since it ranks parameters by output functional impact without needing task data.","The paper's text-only synthetic probes may under-sample the visual pathway of an MLLM; the method could be extended by generating synthetic image-text prompts to score the vision encoder and connector blocks, which are currently frozen.","If the |W| proxy for ΔW holds across optimizers, the mask could be computed once and reused across multiple downstream tasks; an empirical check of rank correlation between |W| and actual update magnitudes would settle this.","The continual-learning result (backward transfer of −2.5 vs −7.8 for a strong baseline) suggests the static mask could serve as a simple backbone for sequential task adaptation, though only a five-task benchmark is evaluated."],"forward_implications":["Sparse fine-tuning with a static pre-computed mask can be as effective as adaptive methods like SPIDER while using only O(|P|) memory.","Deep fine-tuning of earlier decoder layers, which previously caused severe forgetting, becomes stable when only low-importance parameters are updated.","The method works across architectures (LLaVA-1.5-7B, NVILA-Lite-2B) and task types (captioning, classification, VQA), with H-score gains of several points over state-of-the-art baselines.","Because the mask is computed once before training, the method scales to multi-billion-parameter models and remains data-free."],"fun_headline_variants":["Freeze the right weights: data-free probing stops MLLM forgetting","Selective weight freezing beats catastrophic forgetting in MLLMs","Model-Dowser: data-free importance masks curb forgetting in MLLMs","Keep the sensitive weights: a data-free fix for MLLM forgetting","Data-free importance score preserves MLLM knowledge during fine-tuning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a parameter's actual fine-tuning update is proportional to its current weight magnitude, so freezing the parameters with the largest |W| is the same as freezing those whose perturbation would shift the output most.","fun_headline_variants_meta":{"raw":{"variants":["Freeze the right weights: data-free probing stops MLLM forgetting","Selective weight freezing beats catastrophic forgetting in MLLMs","Model-Dowser: data-free importance masks curb forgetting in MLLMs","Keep the sensitive weights: a data-free fix for MLLM forgetting","Data-free importance score preserves MLLM knowledge during fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001144,"raw_usage":{"total_tokens":4586,"prompt_tokens":748,"completion_tokens":3838,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":3746}},"tokens_in":492,"tokens_out":3838,"duration_ms":23204,"temperature":1.0,"reasoning_tokens":3746,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T04:37:39.442489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the Model-Dowser mask on a small MLLM, run standard Adam fine-tuning, and measure the rank correlation between |W| and actual per-parameter update magnitudes ΔW; if the correlation is near zero, or if a mask built from true ΔW performs no better than the |W|-based mask, the proxy bridging theory to algorithm is not doing the work.","supporting_citations":[],"review_version":3}