{"id":"7fb2f0da-7f8b-4499-8b35-9e29cc395ae3","arxiv_id":"2607.16442","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Cross-modal unlearning transfer in vision-language models is asymmetric, architecture-dependent, and shallow under typographic attacks; influence-guided block selection reduces the measured gap.","lead":"This paper studies whether removing harmful knowledge from a vision-language model in one input channel (text or image) also suppresses it in the other. It finds the transfer is uneven, that typographic attacks recover much of the forgotten content, and proposes a block-selection method that improves cross-modal transfer.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CrossInf's reported near-zero cross-modal/typographic ASR may reflect test-set selection: top-k is tuned on the same 72 text probes used to measure success (Table 3), and D_cross appears to be drawn from that pool; no held-out or random-block control separates influence scores from hyperparameter lu","rationale":"The reader's weakest assumption correctly identifies the most load-bearing concern: the evaluation protocol for CrossInf allows test-set information to enter through both the influence-scoring probe set and the top-k selection, and the primary success metric is one the paper itself shows underreports unsafe post-unlearning behavior. My reading of the manuscript supports this: Table 3 explicitly selects k by minimizing ASR on the Exp 1 cross-modal probes, and §6.2 uses those selected configurations to report the headline reductions. The random-block control is absent, so even the influence mechanism's causal contribution is not isolated. These are fixable methodological issues, not evidence of bad faith; the paper is unusually transparent about the LG-4 recall problem and provides examples of the target-string failure mode. The diagnostic contributions — bidirectional cross-modal transfer asymmetry, architecture-dependent fusion effects, and typographic-attack recovery of unlearned behaviors — rest on comparisons that are less affected by this concern and remain plausible. For that reason I do not recommend changing the reader's CONDITIONAL verdict: the paper should be accepted only with additional held-out validation, a random-block baseline, and metric-robust reporting of the CrossInf mitigation claims.","tokens_in":27239,"tokens_out":4162,"duration_ms":40649,"concrete_test":"Construct a held-out text probe set disjoint from the 72 Exp 1 probes and from the 50-sample D_cross pool, using new phrasings and related-but-not-identical concepts. Split it into a development set and a test set: choose top-k only on the development set, then evaluate once on the held-out test set. Also add a random-block control: for each architecture, run CrossInf with the same k values but with randomly selected blocks instead of influence-selected blocks, repeated over at least 3 seeds. If the held-out ASR no longer shows the 13.9%→1.4% drop, or if random-block selection matches influence-selected performance, the reported CrossInf benefit is substantially attributable to test-set selection rather than to the influence mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mitigation claim — that influence-guided block selection reduces visual-to-text LG-4 ASR from 13.9% to 1.4% for LLaVA and typographic ASR to 0% across architectures — is underdetermined by the reported experiments. In §6.2 the per-model top-k is selected as the value minimizing ASR on the Exp 1 cross-modal text probes reported in Table 3. Those same 72 probes constitute the evaluation set for the headline visual-to-text numbers (§5.1.2, §5.2). This is direct selection on the test set: with k∈{10,30,50,100}, the reported 'best' configuration is the one that makes the headline number smallest. Moreover, §4.2 describes D_cross as a small cross-modal probe set used to compute influence scores; §6.2 says the overhead measurement used 50 cross-modal samples, suggesting D_cross is drawn from the same 72-probe pool that later measures success. Even if the influence scores themselves are not updated on these samples, their use in block selection plus the test-set-driven k choice means the pipeline has access to evaluation-set information. A second, compounding issue is that the 'near zero' claim is carried by LlamaGuard-4, which the paper's own Table 5 shows underreports unsafe outputs after unlearning: LG-4 recall is 51% overall and its false-negative rate rises to 74% on post-unlearning outputs. The target-string ASR after CrossInf remains 43.1% for LLaVA, which the paper attributes to degenerate non-canonical responses, but without human validation of the specific CrossInf outputs the 'near-zero ASR' claim is fragile. These concerns do not undermine the diagnostic findings of asymmetric, architecture-dependent transfer or shallow unlearning under typographic attacks; they specifically affect the CrossInf mitigation claims.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports a bidirectional measurement of cross-modal unlearning transfer in three VLMs (LLaVA-1.5, InstructBLIP, IDEFICS) with different fusion mechanisms. It finds that text-to-visual unlearning transfer is strong (cross-modal ASR below 5% across models), while visual-to-text transfer is weaker and architecture-dependent (text-probe ASR of 13.9% for LLaVA vs. 3% for InstructBLIP after visual unlearning). It further shows that typographic attacks re-encode harmful text as images and recover 58–69% of unlearned behaviors. To close the transfer gap, it proposes CrossInf, an influence-guided block-selection method that concentrates the unlearning loss on the transformer blocks most influential for cross-modal generalization, reporting reductions in LlamaGuard-4 ASR from 13.9% to 1.4% for LLaVA in the visual-to-text direction and to 0% under typographic attacks across all three architectures, while preserving utility. Human evaluation with three annotators (Fleiss κ=0.77) is used to validate the automated metrics, and CKA analysis is used to relate transfer to representational change.","tokens_in":27645,"tokens_out":5443,"duration_ms":51266,"significance":"If the empirical claims hold, this is a useful systematic study. The diagnostic contributions — asymmetry of cross-modal transfer, the role of the fusion architecture, and the fragility of gradient-ascent unlearning under typographic attacks — are novel in scope and directly relevant to VLM safety. The paper also transparently releases code, data probes, and configurations (Appendix F), and it reports both target-string and LlamaGuard metrics alongside a human evaluation, which is good practice. The proposed CrossInf mitigation is potentially valuable, but as presented its quantitative claims are substantially weakened by selection-on-the-evaluation-set issues, the absence of a random-block control, and the reliance on LlamaGuard-4, which the paper itself shows underreports post-unlearning unsafe outputs. The diagnostic findings may survive revision; the mitigation claims are not yet established.","major_comments":[{"comment":"The per-model top-k is selected on the same Exp 1 cross-modal text probes that constitute the headline evaluation set. Table 3 reports Cross-modal ASR for k∈{10,30,50,100} and §6.2 then cites the k=50 (LLaVA) and k=30 (InstructBLIP/IDEFICS) numbers as the CrossInf result. Since the same 72-probe set is used both to pick k and to report the final 13.9%→1.4% reduction, this is selection on the evaluation set. The improvement is therefore inflated. A held-out probe set (or nested selection) is required to support the claim, or the paper should explicitly frame the result as a training-set result rather than a deployment prediction.","section":"§6.2, Table 3"},{"comment":"The provenance of D_cross is not stated precisely. §4.2 describes D_cross as a 'small cross-modal probe set' used only to compute influence scores; §6.2 says the overhead measurement used '50 cross-modal samples (matching our experimental setup)'. The only cross-modal text probe set described in §5.2 is the 72-item custom set used for Exp 1 evaluation. If D_cross is a subset of that 72-item pool, the influence scores have access to the evaluation distribution, and the block selection is not independent of the test set. Please state explicitly where D_cross is drawn from; if it is the same pool, the 'no multimodal unlearning data' claim in the introduction and abstract is misleading, because a labeled probe set from the target modality was available for model selection.","section":"§4.2 and §6.2"},{"comment":"The near-zero ASR claims for CrossInf rest on LlamaGuard-4, which the paper's own Table 5 shows has 51% recall overall and a false-negative rate of 74% on post-unlearning outputs. The target-string ASR after CrossInf is still 43.1% for LLaVA in Exp 1 and 47.1% under typographic attacks. The paper attributes the TS/LG gap to degenerate non-canonical responses (Appendix D), but the human evaluation in §6.5 is not reported for the CrossInf outputs or the typographic-attack outputs specifically, so it does not validate that all TS positives are benign. Without human labels on a stratified sample of the CrossInf TS-positive responses, the claim that the TS metric is a blind spot rather than residual unsafe behavior is unsupported.","section":"§6.3, Table 5"},{"comment":"No control condition separates the contribution of the influence scores from the effect of freezing most blocks. The reported comparison is vanilla unlearning (k=100) versus influence-selected top-k; there is no random-block selection with the same k and the same training budget. The observed improvement could come from the k sweep or from the implicit regularization of updating fewer blocks. A random-block or magnitude-based baseline is necessary to attribute the improvements to Eq. (4). Additionally, the block-level influence score in Eq. (4) is an ad hoc construction (sum of absolute entries of IF_b) whose validity as an estimator of causal contribution is simply assumed; some validation of the score (e.g., correlation with held-out transfer improvement) would be needed.","section":"§4.2, Eq. (4)"},{"comment":"The evaluation is small-N, single-run, with no uncertainty quantification. Exp 1 cross-modal evaluation uses 72 text probes; Exp 3 uses 90 typographic images. The reported differences are on the order of a handful of items (13.9% is about 10/72; 1.4% is about 1/72; 0% is 0/90). No seeds, multiple runs, or confidence intervals are reported. The architecture-dependence claims (e.g., LLaVA 13.9% vs InstructBLIP 2.8%) may be sensitive to a few responses. At minimum, per-item results, bootstrap CIs, or multiple unlearning runs with different random seeds should be provided for the headline numbers.","section":"§5.1.2, §5.2, Tables 8–10"}],"minor_comments":[{"comment":"Notation is inconsistent: 'LlamaGuard-4' and 'Llama-Guard-4' are used interchangeably; 'CrossInfapplies' and similar missing spaces occur after macro names; 'we did a study over the selection of k' should be rephrased. A careful copyedit is needed.","section":"Throughout"},{"comment":"The figure captions say 'Bars = target-string ASR; diamonds = LlamaGuard ASR,' but the legends list 'Baseline (LG) Post (LG) Baseline (TS) Post (TS)' with no explicit mapping to marker shapes. The current presentation makes it difficult to tell which series is which; please align the legend order and marker shapes, and define them in the caption.","section":"Figures 7 and 8"},{"comment":"The header row 'Probe set Pre-Unlearning Vanilla (k=100) k=10 k=30 k=50' is confusing because 'Probe set' is not actually a column that varies; use row grouping labels for each model block. Also define the 'TS/LG' slash notation in the caption.","section":"Table 3"},{"comment":"The intervention names are written differently in the table ('Vision only', 'Fusion only', 'LLM only', 'Fusion + LLM') and in the text ('vision-only', 'fusion-only', 'LLM-only', 'fusion+LLM'). Please standardize the notation for consistency.","section":"Table 10 and §6.4"}],"recommendation":"major_revision","confidential_remarks":"The diagnostic core of the paper is interesting and likely salvageable, but the mitigation claims for CrossInf are currently supported by evaluation-set selection and by a classifier with a known high false-negative rate. In revision, the authors should either make the held-out nature of the probes and D_cross explicit and repeat the experiments with a genuinely disjoint evaluation set, or downgrade the abstract/introduction claims accordingly. I would ask for the revised manuscript to be re-reviewed rather than accept on the current evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Best read this for the diagnosis, not the cure. The first contribution—a bidirectional measurement of cross-modal unlearning transfer across three fusion architectures—is genuinely new and mostly convincing. Text unlearning transfers strongly to vision; visual unlearning transfers weakly and architecture-dependently (InstructBLIP's Q-Former saturates, LLaVA's MLP leaves a 40-point gap). The typographic-attack results (58–69% recovery) support the shallow-unlearning claim, and the intervention ablation cleanly shows the LLM is the critical component. The authors are also transparent about LlamaGuard-4's blind spots and include a human eval with decent agreement.\n\nThe soft spots are in the CrossInf mitigation. The headline improvement—13.9% to 1.4% visual-to-text ASR and 'near-zero' typographic ASR—is underdetermined. The per-model top-k is chosen from Table 3 as the value minimizing ASR on the same 72 text probes that constitute the evaluation set. D_cross also appears to be drawn from that pool (50 samples vs 72 probes). That is selection on the test set: even if the influence scores aren't updated on these samples, block selection plus k-tuning use evaluation-set information. There is no random-block baseline, so we can't tell whether the influence scores are doing anything beyond 'unlearn on fewer blocks'. And the near-zero numbers are carried by LlamaGuard-4, which the paper's own Table 5 shows has a 74% false-negative rate on post-unlearning outputs; target-string ASR actually rises for LLaVA (40.3 to 43.1%) and was not human-validated specifically for CrossInf outputs. Single-run, small-N measurements without error bars also make the differences less solid than they look.\n\nThese concerns don't undercut the diagnostic findings, which rely on vanilla unlearning and don't depend on k or D_cross. But they mean the paper's central mitigation claim is not yet supported.\n\nWho this is for: people working on safety unlearning in multimodal models will value the measurement study and the architecture-dependence result. The CrossInf idea is worth pursuing, but the evidence needs a major redraft: held-out probes, a random-block control, and reporting that doesn't lean solely on an unreliable classifier.\n\nRecommendation: send to peer review. A serious referee could help the authors fix the evaluation; desk-rejecting would lose a worthwhile diagnostic contribution. But the acceptance should be conditional on the mitigation experiments being redone.","headline":"Cross-modal unlearning diagnosis is worth reading; the CrossInf mitigation numbers as reported are not to be trusted because of test-set selection.","tokens_in":28163,"tokens_out":3803,"would_cite":true,"duration_ms":33532,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unlearning in a vision-language model is asymmetric and incomplete: text unlearning transfers to images, but image unlearning leaves text vulnerabilities that typographic attacks recover.","keywords":["machine unlearning","vision-language models","cross-modal transfer","typographic attacks","influence functions","asymmetric unlearning","shallow unlearning","model safety"],"falsifier":"Take a new VLM from one of the three families, run CrossInf with top-k chosen on one held-out set of cross-modal probes, then evaluate on a disjoint set of text probes for the same 12 concepts and on typographic images that were never used in scoring or selection. If the visual-to-text LlamaGuard ASR does not stay near 1.4% (or the typographic ASR at 0%), the headline mitigation is an artifact of selection on the evaluation set.","tokens_in":27107,"feed_emoji":"🛡️","tokens_out":6288,"duration_ms":53416,"temperature":0.7,"pith_summary":"This paper asks whether safety unlearning applied in one input modality of a vision-language model—text or image—also suppresses semantically equivalent harmful inputs in the other modality. It finds the transfer is real but asymmetric and incomplete: text unlearning sharply reduces harmful responses to images, while image unlearning leaves harmful text responses partly alive (13.9% LlamaGuard attack rate on LLaVA, 9.7% on IDEFICS, 2.8% on InstructBLIP). It also finds the unlearning is shallow: rendering the forbidden text as an image via typographic attacks recovers 58–69% of the supposedly erased behaviors across all three architectures. To close the gap, the paper proposes CrossInf, which uses block-level influence scores to identify the transformer blocks most responsible for cross-modal coupling and applies unlearning only there; this cuts the LLaVA visual-to-text attack rate to 1.4% and typographic recovery to near zero across all three models, without needing multimodal unlearning data. The stakes are practical: single-modality unlearning is the current default, and these results say it gives a false sense of safety for multimodal deployments.","feed_headline":"Unlearning one modality leaves the other vulnerable","feed_subtitle":"Text-to-image transfer is strong, but image-to-text leaves a 13.9% attack rate and typographic images recover most erased knowledge.","key_machinery":"The central object is a block-level influence function over LoRA adapters: for each (layer, adapted-module) block, the paper computes IF_b = -G_cross (regularized Fisher)^{-1} G_f^T between the same-modality forget set and a small cross-modal probe set (DataInf's 2-D gradient features per LoRA matrix), aggregates the absolute entries into a scalar score, and applies the three-term unlearning loss (gradient ascent, refusal, KL-to-reference) only to the top-k% blocks, freezing the rest. The probe set is never used as an unlearning target; it only identifies which blocks couple the modalities.","core_discovery":"Gradient-ascent unlearning in one modality propagates unevenly through the shared backbone: text-to-visual transfer is near-complete (cross-modal attack success below 0.5% on LlamaGuard-4), while visual-to-text transfer leaves residual risk and is strongly mediated by fusion design. The Q-Former bottleneck of InstructBLIP transfers almost fully; the MLP projector of LLaVA leaves a large gap. Typographic attacks that re-encode forbidden text as pixels recover 58–69% of unlearned behaviors, showing the knowledge was suppressed, not erased. The central claim is that CrossInf, which selects top-k transformer blocks by cross-modal influence, closes most of the gap: visual-to-text LlamaGuard ASR d","pith_inferences":["Inference: The numbers for CrossInf's headline reductions and the per-model top-k choice both come from the same 72-item text-probe set, so the 13.9%→1.4% and 58–69%→0% results are likely optimistic relative to a deployment attack distribution the method never saw.","Inference: The asymmetry suggests a cheap pre-deployment audit: measure cross-modal ASR in the direction opposite to the unlearning modality; if the gap is large, either the fusion architecture cannot support transfer or the unlearning update is too shallow.","Inference: If the block-selection mechanism is right, the same influence scores could predict whether a given VLM will transfer any safety intervention, and could be used to choose fusion architectures that are inherently more unlearning-friendly.","Inference: A testable extension would run CrossInf's influence scoring on one probe set and evaluate on a disjoint held-out set of probes and new typographic images; if the gains persist, the method is a genuine transfer enhancer rather than a selection artifact."],"forward_implications":["Deployers who unlearn in one modality should assume the other modality is still partially vulnerable and test both directions plus typographic re-encoding.","The LLM blocks are the necessary and sufficient intervention point for cross-modal transfer; vision encoders and fusion modules can be left untouched.","Typographic attacks are a cheap, no-optimization bypass for gradient-ascent unlearning, so unlearning evaluations should include them as a standard stress test.","A small cross-modal probe set can substitute for large multimodal unlearning datasets when the unlearning is targeted to high-influence blocks.","Architecture choice matters: bottlenecks (Q-Former) saturate and transfer on their own; distributed cross-attention (IDEFICS) needs targeted selection; loose MLP projection (LLaVA) can only be partially rescued."],"fun_headline_variants":["One modality forgets, the other remembers","Vision-language unlearning: one side is shallow","Typographic attacks undo unlearning in VLMs","Cross-modal unlearning gap cut by CrossInf","Unlearning text doesn't unlearn pixels"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The cross-modal probe set used to score blocks and to choose the target k is the same set used to measure success, so the reported mitigation benefit assumes those 72 text probes and the 90 typographic images are representative of the attacks that will actually be deployed.","fun_headline_variants_meta":{"raw":{"variants":["One modality forgets, the other remembers","Vision-language unlearning: one side is shallow","Typographic attacks undo unlearning in VLMs","Cross-modal unlearning gap cut by CrossInf","Unlearning text doesn't unlearn pixels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000283,"raw_usage":{"total_tokens":1555,"prompt_tokens":840,"completion_tokens":715,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":646}},"tokens_in":584,"tokens_out":715,"duration_ms":27018,"temperature":1.0,"reasoning_tokens":646,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T20:56:27.166656+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a new VLM from one of the three families, run CrossInf with top-k chosen on one held-out set of cross-modal probes, then evaluate on a disjoint set of text probes for the same 12 concepts and on typographic images that were never used in scoring or selection. If the visual-to-text LlamaGuard ASR does not stay near 1.4% (or the typographic ASR at 0%), the headline mitigation is an artifact of selection on the evaluation set.","supporting_citations":[],"review_version":1}