Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

A pre-fine-tuning mask built from Jacobian sensitivity, weight magnitude, and input activation preserves MLLM generalization during deep fine-tuning, outperforming prior methods.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:37 UTC pith:UXVIOBJL

load-bearing objection The depth-sweep empirical result is solid and genuinely useful; the theoretical justification is a heuristic that the authors should either validate or clearly label as such. the 4 major comments →

arxiv 2602.04509 v7 pith:UXVIOBJL submitted 2026-02-04 cs.CL

Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models

Hyeontaek Hwang , Nguyen Dinh Son , Daeyoung Kim This is my paper
classification cs.CL
keywords catastrophic forgettingmultimodal large language modelssparse fine-tuningparameter importanceJacobian sensitivitydata-free probingstochastic trace estimatormodel merging
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Model-Dowser is a sparse fine-tuning method for multimodal large language models (MLLMs) that aims to prevent catastrophic forgetting when models are adapted to downstream tasks. It computes a per-parameter importance score before fine-tuning—weight magnitude times input activation times output sensitivity estimated via synthetic probes—and freezes the most important parameters while updating the rest. The paper proves a first-order bound showing that these scores aggregate to an upper bound on the output shift caused by parameter updates, and shows empirically that the method maintains pretrained zero-shot performance even when fine-tuning extends to earlier decoder layers, outperforming prior methods on LLaVA and NVILA across captioning, classification, and VQA tasks. The approach is data-free, needs no extra training memory, and scales to multi-billion-parameter models.

Core claim

The paper shows that catastrophic forgetting in MLLMs can be mitigated by preserving the parameters whose perturbation would most shift the network's outputs, measured by the score S_ij = ‖J_i‖_2 · |W_ij| · |h_j|, where J_i is the Jacobian column relating the output to the i-th pre-activation, |W_ij| is the weight magnitude, and |h_j| is the input activation. The score is estimated data-free by probing the frozen model with synthetic prompts and stochastic trace estimation with random sign vectors, then converted into a per-layer binary mask that freezes the top (1−ρ) fraction of high-importance weights. The authors prove (Corollary 3.2) that the total output shift under multi-weight perturb

What carries the argument

The load-bearing object is the parameter-wise importance score S_ij = ‖J_i‖_2 |W_ij| |h_j|, a first-order sensitivity measure combining output Jacobian norm, weight magnitude, and input activation. It is estimated without data via stochastic trace estimation on synthetic text prompts, and converted into a static binary mask before fine-tuning. The argument that freezing high-S parameters controls output drift rests on the first-order Taylor bound ‖Δf‖_2 ⪅ Σ_ij ‖J_i‖_2 |ΔW_ij| |h_j| (Corollary 3.2).

Load-bearing premise

The load-bearing premise is that a parameter's actual fine-tuning update is proportional to its current weight magnitude, so freezing the parameters with the largest |W| is the same as freezing those whose perturbation would shift the output most.

What would settle it

Compute the Model-Dowser mask on a small MLLM, run standard Adam fine-tuning, and measure the rank correlation between |W| and actual per-parameter update magnitudes ΔW; if the correlation is near zero, or if a mask built from true ΔW performs no better than the |W|-based mask, the proxy bridging theory to algorithm is not doing the work.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Sparse fine-tuning with a static pre-computed mask can be as effective as adaptive methods like SPIDER while using only O(|P|) memory.
  • Deep fine-tuning of earlier decoder layers, which previously caused severe forgetting, becomes stable when only low-importance parameters are updated.
  • The method works across architectures (LLaVA-1.5-7B, NVILA-Lite-2B) and task types (captioning, classification, VQA), with H-score gains of several points over state-of-the-art baselines.
  • Because the mask is computed once before training, the method scales to multi-billion-parameter models and remains data-free.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same importance score could be repurposed for other data-free operations on frozen models—pruning, quantization sensitivity analysis, or identifying safety-critical neurons—since it ranks parameters by output functional impact without needing task data.
  • The paper's text-only synthetic probes may under-sample the visual pathway of an MLLM; the method could be extended by generating synthetic image-text prompts to score the vision encoder and connector blocks, which are currently frozen.
  • If the |W| proxy for ΔW holds across optimizers, the mask could be computed once and reused across multiple downstream tasks; an empirical check of rank correlation between |W| and actual update magnitudes would settle this.
  • The continual-learning result (backward transfer of −2.5 vs −7.8 for a strong baseline) suggests the static mask could serve as a simple backbone for sequential task adaptation, though only a five-task benchmark is evaluated.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes Model-Dowser, a sparse fine-tuning method for multimodal large language models. Before adaptation, it computes a parameter importance score S̄_ij = (1/N) Σ_n ‖J^{(l)}_{i,n}‖₂ |W^{(l)}_{ij}| |h^{(l-1)}_{j,n}|, where the Jacobian is estimated with Hutchinson probing on synthetic text-only prompts, and then freezes the top-ρ parameters during downstream fine-tuning. The authors claim that this preserves pretrained generalization better than full fine-tuning, post-merging methods (Grafting, DARE, ModelTailor), and the sparse baseline SPIDER across LLaVA-1.5-7B and NVILA-Lite-2B on captioning, classification, and VQA benchmarks. The paper includes depth sweeps, mask-ratio sweeps, random-mask comparisons with significance tests, a memory-complexity analysis, and a continual-learning extension.

Significance. The empirical work is broad and internally consistent: across two architectures, four downstream tasks, six upstream benchmarks, and multiple fine-tuning depths, Model-Dowser reports the best H-score in almost every condition, with large margins over Tailor and SPIDER in deep fine-tuning regimes. The random-mask baselines with significance tests provide credible evidence that the importance ranking itself carries signal. If the two central gaps—the ΔW→|W| substitution and the text-only probing distribution—can be closed by direct validation, Model-Dowser would be a useful and scalable addition to the forgetting-mitigation toolbox. In its current form, however, the theoretical framing overstates what is proven, and the headline H-scores on captioning tasks are computed from a unit-inconsistent metric. The method is not circular in the sense that upstream evaluation benchmarks do not enter the score computation, and the reported arithmetic is reproducible.

major comments (4)
  1. [§3.1, Eq. (3), Appendix B] The bridge from the bound to the mask is the substitution ΔW_{ij} → |W_{ij}|. Theorem 3.1 and Corollary 3.2 bound the output shift in terms of the true perturbation |ΔW|, but Eq. (7) freezes parameters ranked by |W|. Appendix B explicitly calls this a 'conservative first-order surrogate,' yet no evidence is given that fine-tuning updates scale with current weight magnitude. For Adam/AdamW the per-step update is adaptive and roughly proportional to sign(g)/√v, not to |W|. If |W| and |ΔW| are weakly correlated, freezing the |W|-largest parameters need not suppress the dominant terms in Corollary 3.2. At minimum, the authors should measure the rank correlation between |W| and the actual |ΔW| observed after fine-tuning, or compare masks built from the two quantities; otherwise the claim that the score 'explains why preserving high-score parameters helps' is unsupported.
  2. [§3.2, Eq. (6), §4.1, Appendix G] The probing distribution is text-only: the experiments state 'synthetic text samples N=64,' and Eq. (6) generates prompts from random token seeds. Downstream fine-tuning, however, is multimodal and the forgetting being mitigated is driven by image-conditioned inputs. The score never exercises the vision encoder or cross-attention pathway, so those parameters are effectively unscored. Appendix G validates the Hutchinson/text-probe estimator against 'real data samples,' but it does not establish that text-only probes activate the same functional structure as the multimodal inputs used during downstream training. The authors should either compare masks computed with image-conditioned probes, or give an explicit argument for why language-decoder activations alone are sufficient.
  3. [§4.1, Eq. (9), Tables 1–2, 5] For image captioning, A_down is CIDEr, which is unbounded and can exceed 100 (e.g., 135.5 in Table 1), while A_up is an accuracy in [0,100]. The reported H-score is the harmonic mean of these two quantities, and Avg is their arithmetic mean. This conflates different units and makes the captioning H-scores/AVG not interpretable as accuracies. For example, with A_up≈62.7 and A_down≈135.5, the harmonic mean is ≈85.7 solely because CIDEr is large. The headline captioning results should be recomputed with a normalized A_down (e.g., CIDEr normalized to [0,100]) or reported separately for upstream and downstream without a single combined score.
  4. [§3.1–3.2, §4.2] There is no ablation isolating the three factors of the importance score. The paper's central claim is that jointly combining output sensitivity ‖J_i‖, weight magnitude |W_{ij}|, and input activity |h_j| is what makes the mask effective. The comparisons with SPIDER, Tailor, etc., are against whole methods that differ in multiple design choices, so they do not identify the contribution of the proposed score. Please include masks based on |W| alone, ‖J‖·|W|, ‖J‖·|h|, |W|·|h|, and the full product; this would directly test the 'jointly considering' claim and the role of each term.
minor comments (4)
  1. [Eq. (4) vs Eqs. (3)/(5), Appendix D] The Hutchinson estimator in Eq. (4) yields ‖J_i‖², while the score uses ‖J_i‖ and the main text also mentions E|ξ^T J_i| as an L1 surrogate. Please state explicitly how the square root is taken in practice and which estimator is actually used in the reported results.
  2. [Tables 4 and 6] The significance tests use t-tests over only three random seeds. Please state the normality assumption or provide more seeds, and make the one-sided/two-sided choice consistent between Tables 4 and 6.
  3. [Table 3] The row for Grafting lists ρ=100% but '# params' = 143M/438M, the same as methods with ρ=10%. Please clarify what '# params' counts and why Grafting's count differs from Full-FT's if it fine-tunes 100% of the last-20-layer parameters.
  4. [Appendix F, Table 6 caption, Appendix H.2] Several typos: 'COCO-Cpation' in Appendix F, 'NVLILA' in the Table 6 caption, and 'upstream performance average of v' in Appendix H.2. The duplicated citation for Xu & Zhang (2024a/b) should also be merged.

Circularity Check

0 steps flagged

No significant circularity; the importance score is self-contained and benchmark evaluation is held out, with an acknowledged non-circular proxy gap.

full rationale

The derivation chain is not circular. The importance score in Eq. (3) is defined as ||J_i||·|W_ij|·|h_j|, and Eq. (7) turns it into a binary mask; Eq. (8) applies the mask to gradients. The theoretical bound in Cor. 3.2 is proved for true perturbations |ΔW|, and the substitution of |W| for |ΔW| is explicitly flagged in Appendix B as a 'conservative first-order surrogate' — an approximation, not an identity. The mask is computed entirely from the pretrained model and synthetic inputs; no parameter of the method is fit to the evaluation benchmarks (TextVQA, OKVQA, MMBench, GQA, etc.), and the reported H-scores are measured on held-out upstream tasks. The random-selection comparisons (Tables 4, 6, 7) provide a non-circular control: if the score were merely a renamed input, random masks would match its performance. Self-citations are absent from the load-bearing argument; the Hutchinson estimator and the Khintchine/Haagerup bounds are external results. The acknowledged ΔW→|W| proxy is a validity gap that belongs in a correctness risk assessment, not in a circularity score.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The method contributes no fitted constants to the evaluation target: the score is computed pre-adaptation on synthetic data and evaluated on held-out benchmarks. Its dependence on upstream machinery is mostly standard math (Taylor, Hutchinson, Khintchine) plus two ad hoc premises: the |ΔW|→|W| proxy and the text-only synthetic probing distribution. The free parameters ρ, N, R, L are chosen by hand or by saturation, not fitted to the results.

free parameters (5)
  • mask update ratio ρ = 0.1 (main); swept 0.01-1.0
    Fraction of per-layer weights updated; the core knob of the method. The paper shows robustness across ρ, but the headline tables all use ρ=0.1.
  • Monte Carlo probing samples N = 64
    Number of synthetic prompts; chosen by saturation in Appendix G (Hamming distance/Spearman plateau). Affects the stability of the score.
  • Rademacher vectors R = 8
    Number of Hutchinson projection vectors per sample; chosen by saturation in Appendix G. Affects variance of the Jacobian estimate.
  • Fine-tune depth L = 20 (main tables); swept 4-32
    Number of decoder layers updated. The paper's main claim is stated at L=20; the depth sweep is an experimental axis, not fitted to the target.
  • Training hyperparameters (lr, epochs, batch) = 2e-5, 5 epochs, batch 128
    Standard settings shared with baselines; not fitted to the target but required to reproduce the reported numbers.
axioms (6)
  • domain assumption First-order Taylor expansion of the network output in the pre-activations is accurate at the scale of fine-tuning weight updates (remainder term is negligible).
    Theorem 3.1 and Corollary 3.2 drop O(‖Δz‖²) without quantifying it; for large models and multi-step fine-tuning, the approximation is unverified.
  • ad hoc to paper The actual fine-tuning perturbation |ΔW_ij| is proportional to the current weight magnitude |W_ij|.
    Eq. (3) and Appendix B substitute |W_ij| for ΔW_ij ('we interpret this as a proxy'). This is the load-bearing bridge between the Taylor bound and the mask; it is not derived and is questionable under adaptive optimizers.
  • ad hoc to paper Synthetic text-only prompts generated from the MLLM itself activate the same functional structure as real multimodal inputs.
    Sec. 3.2 synthesizes prompts from random token seeds (Eq. 6); no synthetic images are generated. The visual pathway is never probed, yet the forgetting being mitigated occurs under multimodal inputs. Appendix G validates against an unidentified 'real data' score.
  • domain assumption Preserving the model output on synthetic probes implies preserving accuracy on upstream benchmarks.
    The bound in Cor. 3.2 bounds output shift; the leap from 'small output shift on probing distribution' to 'retained zero-shot accuracy on TextVQA/MMBench/GQA' is never formalized.
  • domain assumption Per-layer percentile ranking (top-ρ within each layer, Eq. 7) yields the same protection as global ranking.
    The theoretical bound is global over all parameters; the implementation ranks within each layer, which guarantees 10% updates everywhere but can freeze low-sensitivity parameters in some layers while updating high-sensitivity ones in others. The paper does not discuss this mismatch.
  • standard math Hutchinson-style estimator identity: E_ξ[(ξ^T J_i)^2] = ‖J_i‖²₂ for Rademacher ξ (Eq. 4).
    Correct as stated and used; a standard stochastic norm estimator. Also Khintchine's inequality (Theorem D.1) with Haagerup constant 1/√2 is used correctly.
invented entities (2)
  • Functional importance score S̄_ij = ‖J_i‖·|W_ij|·|h_j| independent evidence
    purpose: Postulated measure of parameter importance for freezing during fine-tuning.
    The score has an external falsifiable handle: it is tested against real-data rankings (App. G) and against random masking across ρ (Tables 4, 6, 7), where it predicts retention differences. It is not circular, but its theoretical status is heuristic.
  • Synthetic probe distribution (random-seed model generations) independent evidence
    purpose: Data-free proxy for the input distribution used to estimate Jacobians and activations.
    Appendix G provides evidence (Spearman ≈ 0.89 vs real-data scores) that the synthetic distribution approximates real activations, though the 'real data' used for anchoring is never identified. The text-only construction leaves the visual modality unprobed.

pith-pipeline@v1.3.0-alltime-deepseek · 29037 in / 26591 out tokens · 277100 ms · 2026-08-03T04:37:39.442489+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/UXVIOBJL

@misc{pith2026260204509,
  author       = {Pith},
  title        = {Pith review of: Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UXVIOBJL}},
  note         = {Machine review of arXiv:2602.04509}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Fine-tuning Multimodal Large Language Models (MLLMs) on task-specific data is an effective way to improve performance on downstream applications. However, such adaptation often leads to a degradation in generalization on pretrained tasks, a phenomenon known as Catastrophic Forgetting. Existing methods that aim to mitigate this issue either become ineffective when fine-tuning deeper layers of the language decoder or scale poorly with increasing model size. To address these limitations, we propose Model-Dowser, a novel sparse fine-tuning approach for MLLMs. Model-Dowser measures a principled importance score for each model parameter with respect to pretrained generalization (prior to downstream adaptation) by jointly considering weight magnitudes, input activations, and output sensitivities. During fine-tuning, Model-Dowser selectively preserves high-importance parameters and updates the remaining. Comprehensive experiments on two representative MLLMs, LLaVA and NVILA, demonstrate that Model-Dowser effectively mitigates catastrophic forgetting and consistently outperforms prior methods, while remaining resource-efficient and scalable to multi-billion-parameter models.

Figures

Figures reproduced from arXiv: 2602.04509 by Daeyoung Kim, Hyeontaek Hwang, Nguyen Dinh Son.

Figure 1
Figure 1. Figure 1: Performance comparison of catastrophic forgetting mitigation methods on LLaVA-1.5 (7B) and NVILA-Lite (2B) fine-tuned on ImageNet-R. (a)-(b) Radar charts illustrating the balance between downstream adaptation and upstream capabili￾ties. (c)-(d) H-score1 stability across varying fine-tuning depths. Model-Dowser (red line) consistently achieves robust performance compared to previous works. MLLMs can be attr… view at source ↗
Figure 2
Figure 2. Figure 2: Overall Architecture of Model-Dowser. The proposed method consists of three main steps. 1. Probing (Section 3.2): samples Jacobian matrix and input activation with synthetic data samples on every layer (l). 2. Compute Score (Section 3.2): generate parameter-wise importance score with Jacobian matrix, weight magnitude, and activation. 3. Sparse Finetune (Section 3.3): update the least important ρ% of parame… view at source ↗
Figure 3
Figure 3. Figure 3: Performance comparison across fine-tuning depths on COCO and ImageNet-R. Results show the average accuracy across all tasks for an update ratio of ρ = 0.1 and various merging methods. The x-axis denotes the number of layers fine-tuned, counted incrementally from the final output layer toward the initial input layer. Increased Vulnerability of Early Decoder Layers to Catastrophic Forgetting. Layer-wise anal… view at source ↗
Figure 4
Figure 4. Figure 4: Performance comparison across various mask ratios (ρ) on COCO-Caption and ImageNet-R using (a-b) NVILA-Lite-2b, and (c-d) LLaVA-1.5-7B. Results show the upstream and down￾stream performance (Aup, Avg, H-score) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Radar chart on diverse benchmarks on LLaVA-1.5-7B and NVILA-Lite when finetuning all layers. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance comparison across fine-tuning depths on Flickr30k and IconQA. Results show the average accuracy across all tasks for an update ratio of ρ = 0.1 and various merging methods. The x-axis denotes the number of layers fine-tuned, counted incrementally from the final output layer toward the initial input layer. 0.01 0.05 0.1 0.25 0.5 0.75 0.9 1.0 Mask Ratio (ρ) 55 60 65 70 75 (a) Flickr30k LLaVA 0.01… view at source ↗
Figure 7
Figure 7. Figure 7: Performance comparison across various mask ratios (ρ) on Flickr30k and IconQA using LLaVA-1.5-7B. Results show the upstream and downstream performance. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: demonstrates the stability and robustness of our approach compared to random selection. There is a significant disparity between the two methods; as the number of probing samples (N) increases, our estimated importance score closely aligns with the score rankings calculated from real data samples. This indicates that our data-free proxy effectively captures the model’s true structural sensitivity. Furtherm… view at source ↗

Review history (3 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

    cs.CL 2026-06 unverdicted novelty 5.0

    The survey formalizes MLLM perception as a unified vision-language capability and traces its evolution via a new five-stage taxonomy while outlining future challenges.

Reference graph

Works this paper leans on

5 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [1]

    URL https: //aclanthology.org/2025.acl-long.666/

    doi: 10.18653/v1/2025.acl-long.666. URL https: //aclanthology.org/2025.acl-long.666/. Guo, H., Zeng, F., Zhu, F., Liu, W., Wang, D.-H., Xu, J., Zhang, X.-Y ., and Liu, C.-L. Federated continual instruc- tion tuning.arXiv preprint arXiv:2503.12897, 2025b. Haagerup, U. The best constants in the khintchine inequality. Studia Mathematica, 70(3):231–283, 1981....

    Pith/arXiv arXiv 2025

  2. [243]

    emnlp-main.243/

    URL https://aclanthology.org/2021. emnlp-main.243/. Li, X. L. and Liang, P. Prefix-tuning: Optimizing continu- ous prompts for generation. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.),Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing ...

    doi:10.18653/v1/2021.acl-long 2021

  3. [353]

    acl-long.353/

    URL https://aclanthology.org/2021. acl-long.353/. Li, Y ., Luo, F., Tan, C., Wang, M., Huang, S., Li, S., and Bai, J. Parameter-efficient sparsity for large language models fine-tuning.arXiv preprint arXiv:2205.11005, 2022. Lin, T.-Y ., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Doll´ar, P., and Zitnick, C. L. Microsoft coco: Common obj...

    Pith/arXiv arXiv 2021

  4. [598]

    findings-acl.598/

    URL https://aclanthology.org/2024. findings-acl.598/. Zhou, X., He, J., Ke, Y ., Zhu, G., Guti´errez-Basulto, V ., and Pan, J. An empirical study on parameter-efficient fine- tuning for multimodal large language models. InFindings of the Association for Computational Linguistics: ACL 2024, pp. 10057–10084, 2024b. Zhu, D., Sun, Z., Li, Z., Shen, T., Yan, K...

    2024

  5. [2024]

    URL https://proceedings.mlr.press/ v235/zhu24l.html. 12 Appendix of Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in MLLMs In this appendix, we provide a more detailed discussion, proof, and result from the main text. •Appendix A: An extended disscussion on the related work from Section 2. •Appendix B: Proof of Theorem 3.1...

    2013