{"id":"7c3f510a-e5c1-4081-aed6-2e69666bbf22","arxiv_id":"2511.20196","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An MLLM unlearning method and benchmark that aim to erase targeted private facts while preserving image understanding.","lead":"This paper presents a benchmark and a method for making multimodal AI models forget specific private facts without losing the ability to understand images. Its method, SMFA, preserves visual understanding better than existing unlearning approaches while reducing targeted answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SMFA's forget-set Fact Scores remain near 5/10, so the claimed precise targeted erasure is not actually demonstrated, undercutting the central 'benign forgetting' claim.","rationale":"The reader's weakest_assumption focused on the reliability of the few-shot retaining anchor, which is a methodological generalization concern. While that is real, the single most load-bearing issue is more direct: the paper's own Table 1 shows that SMFA's forget-set Fact Scores remain near the midpoint (4.7–5.3), so the model still leaks substantial target knowledge on average. This contradicts the abstract's and Section 5.2's claims of 'precise removal' and 'targeted knowledge erasure.' If the target knowledge is not reliably erased, the method fails the primary objective of benign forgetting, regardless of how well it preserves image understanding. The reader did mention this overstatement as point (3) in the rationale, but not as the weakest assumption, hence partial agreement. The concrete leakage audit would settle whether the high average Fact Score is due to a few high-scoring outliers or widespread partial leakage. If widespread, the paper's central claim is unsupported and the verdict should remain conditional only if the authors substantially revise their claims and provide this analysis; if the audit shows near-complete erasure, the original concern would dissolve. Therefore, I keep the reader's CONDITIONAL verdict unchanged, conditioned on this evidence.","tokens_in":18572,"tokens_out":6850,"duration_ms":80321,"concrete_test":"Run a leakage audit on the released checkpoints: after applying SMFA (k=5, 5% split), generate outputs for all forget-set image-memory queries. Then (1) have Qwen-Plus score each output with Fact Score and report the fraction with Fact Score==0 and with Fact Score>=8; (2) perform a string match for the gold attribute (e.g., city, name, salary) in each output. If more than 5% of queries leak the exact attribute or score >=8, the targeted-erasure claim is falsified. This directly tests whether 'benign forgetting'—complete removal of target knowledge—actually occurs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that SMFA achieves precise and controllable unlearning, i.e., erases targeted sensitive knowledge while preserving general image understanding (Abstract; Sec. 5.2). Table 1 directly undercuts this: for every model and forget ratio, SMFA's forget-set image-memory Fact Score is only 4.7–5.3 on a 0–10 scale, compared with ~7.5 for the fine-tuned original and ~1.4 for the untrained base. An average near 5 means many outputs still contain substantial correct information, not clean refusals. The case-study examples with Fact Score 0 are not representative. Because the primary objective is rigorous removal of privacy-sensitive content, this partial erasure means the headline claim that SMFA 'successfully achieves targeted knowledge erasure without compromising foundational visual capabilities' is not supported by the paper's own measurements. The masking mechanism may reduce leakage and preserve retention, but it does not yield benign forgetting as defined. This is a direct empirical inconsistency, not just a concern about benchmark external validity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces S-MLLMUn Bench, a benchmark for multimodal LLM unlearning that jointly evaluates forgetting of privacy-sensitive image-text memories and preservation of general image understanding. It uses 1,000 synthetic profiles with character and ophthalmic images. The authors propose SMFA: first fine-tune the model with refusal labels on the forget set to obtain a Memory Forgetting Adapter (MFA); then train a retaining anchor on a few-shot retain set and mask entries of the forget update that directionally conflict with the anchor and dominate it in magnitude (Eqs. 6–10). Experiments on LLaVA-OneVision-7B and Qwen2.5-VL-7B, against four baselines, show SMFA preserves image understanding and retain-set memory much better than baselines, while reducing forget-set Fact Scores from ~7.5 to ~4.7–5.3. The paper claims this is 'precise and controllable unlearning' and 'benign forgetting.'","tokens_in":18758,"tokens_out":7991,"duration_ms":86174,"significance":"The benchmark is a timely and useful attempt to quantify the forgetting-retention trade-off for MLLMs, and the sculpting idea is simple and plausible. Providing code/data is a plus. However, the forget-set Fact Scores are far above the base-model floor, so the central claim of targeted knowledge erasure is not supported by the paper's own measurements. The benchmark may still be valuable, but the method as presented does not yet substantiate 'benign forgetting' as rigorously defined.","major_comments":[{"comment":"Forget-set Fact Scores are 4.7–5.3 for image memory across models/ratios, compared to the base model's ~1.2–1.5 and the original's ~7.5. A mean of ~5 implies many responses still contain correct facts, not clean refusals. Thus the abstract's 'targeted knowledge erasure' and Sec. 5.2's 'effectively erases targeted knowledge' are not supported. Report the score distribution (e.g., percentage of Fact Score 0) and use the base model as the reference floor.","section":"Table 1, all SMFA rows"},{"comment":"No error bars, multiple seeds, or significance tests are reported. The forget-set ratios are small and the few-shot retain set is equal in size, so the trade-off results could be unstable. Run at least three seeds and report mean±std, or justify determinism, before claiming superiority over baselines.","section":"Sec. 5.1, Table 1"},{"comment":"The retaining anchor is trained on a few-shot retain subset equal to the forget set. The paper relies on the MLLM's 'strong generalization capability' to propagate this limited signal, but no analysis shows how the choice of few-shot items affects the mask. Add an ablation varying the few-shot retain set size/random seed to demonstrate robustness of the anchor.","section":"Sec. 3.3, Eqs. (6)–(10)"},{"comment":"The masking rule is close to TIES-Merging (trim and sign-conflict resolution), which is cited but not compared. Without a TIES-style baseline applied to the MFA and retaining anchor, the specific contribution of the 'sculpting' is unclear. Add such a comparison.","section":"Sec. 2, Eq. (7)"}],"minor_comments":[{"comment":"Header reads 'LLaV A-OneVision'; should be 'LLaVA-OneVision'. Also the table is dense; consider splitting by model.","section":"Table 1"},{"comment":"Define 'F Score' and 'M Score' in the caption (Fact Score and Meaningful Score).","section":"Fig. 6"},{"comment":"Use 'Meaningful Score' consistently; the metric is named 'Meaningful Score' in Sec. 4.3 but appears in lower-case in some places.","section":"Sec. 5.2"},{"comment":"State explicitly that the evaluation retain set is the complete retain set; the main text only says 'retain set.'","section":"App. A.1"},{"comment":"Provide the exact prompt used for the Qwen-Plus Fact Score and Meaningful Score in the appendix to ensure reproducibility.","section":"Sec. 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's own numbers contradict its headline claim. The benchmark and method are a reasonable start, but the authors should either achieve near-base forget-set Fact Scores or reframe the contribution as a trade-off study rather than 'benign forgetting.' I recommend major revision with a request for stronger empirical evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: SMFA is a genuinely useful method and S-MLLMUn Bench is a step forward for MLLM unlearning, but the paper overclaims \"benign forgetting.\" Its own numbers show forget-set Fact Scores around 4.7–5.3 out of 10 across all settings, which means a lot of correct information still leaks through. That is not clean refusal, and it undercuts the phrase \"precise and controllable unlearning.\"\n\nWhat's new: the benchmark is the first I've seen that jointly measures forgetting efficacy and preservation of image understanding as first-class metrics. That's a genuinely useful artifact for a field that has mostly cared about erasure alone. The method is a sensible combination of IDK-style refusal fine-tuning with TIES-inspired weight masking; the retaining anchor idea is simple and plausible, and the ablation shows both mask criteria matter. The experiments cover two 7B MLLMs, three forget ratios, and four baselines, and SMFA does win the tradeoff on its own benchmark while keeping meaningful scores high. The case studies show it refusing on some forget queries while retaining image understanding.\n\nWhere it's soft: (1) The central claim is stronger than the data. Forget-set Fact Scores stay near 5/10, so \"benign forgetting\" as the paper defines it — precise removal of targeted knowledge — is not actually demonstrated. The stress-test note is right about this. (2) No error bars, no multiple seeds, and the metrics rely on a Qwen-Plus judge with no spot-checked agreement or calibration. (3) The hyperparameter k=5 is hand-picked and directly controls the reported tradeoff; Fig. 4 shows sensitivity, so a principled selection or a small sensitivity table would help. (4) There is no direct comparison against TIES-Merging or task arithmetic, which is odd given the method is essentially a masked weight merge. That baseline would contextualize the contribution and is cheap to run.\n\nThe method is not circular: forget-set behavior is optimized directly and evaluated on paraphrased queries, and retention is a side effect tested against baselines. The reader's worry about the few-shot anchor being unrepresentative is plausible but not fatal — the strong generalization assumption is reasonable for 7B MLLMs, and the retain-set numbers mostly hold up.\n\nWho it's for: anyone working on MLLM unlearning, privacy compliance, or model editing. It deserves a serious referee. I'd send it out, but I'd push for variance reporting, a direct TIES-Merging baseline, and a more measured statement of what level of erasure counts as \"benign.\" With those revisions, it becomes a solid, citable contribution.","headline":"SMFA and S-MLLMUn Bench are a real step forward for MLLM unlearning, but the paper overclaims 'benign forgetting' — its own forget-set Fact Scores sit near 5/10, so erasure is partial, not precise.","tokens_in":19299,"tokens_out":2101,"would_cite":true,"duration_ms":24184,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Selective MLLM unlearning can erase private image-text memories without sacrificing general image understanding, and the paper's SMFA method plus S-MLLMUn Bench demonstrate how to measure and achieve that balance.","keywords":["machine unlearning","multimodal large language models","selective forgetting","image understanding","parameter masking","right to be forgotten","refusal tuning","benchmark"],"falsifier":"Train SMFA with a retain anchor built only from character-image questions, then measure ophthalmic image understanding on the forget set. If the few-shot anchor fails to generalize to the ophthalmic modality, that score should drop visibly after unlearning, showing the anchor does not protect unrelated capabilities. Alternatively, resample the few-shot retain set several times and check whether retain-set image understanding varies sharply across runs; high variance would falsify the anchor's reliability.","tokens_in":18413,"feed_emoji":"🔒","tokens_out":7236,"duration_ms":76937,"temperature":0.7,"pith_summary":"The paper targets a known failure of multimodal unlearning: methods that scrub private facts from an MLLM often damage its general ability to look at images and answer basic questions. It argues that the damage comes from over-generalization of the forgetting update, and that this can be filtered in weight space. It proposes SMFA, which fine-tunes the model on refusal labels to create a memory-forgetting adapter, then masks out adapter entries that conflict with a retaining anchor trained on a small set of knowledge to preserve. To make the trade-off measurable, it introduces S-MLLMUn Bench, a benchmark with 1,000 synthetic profiles pairing private image-memory questions with image-understanding questions. On two MLLMs, the paper reports that existing methods trade away image understanding, while SMFA forgets targeted knowledge and keeps understanding close to the original model.","feed_headline":"Masked weight updates forget private data and preserve vision","feed_subtitle":"A few retained examples guide targeted multimodal unlearning that erases sensitive facts while keeping image understanding intact.","key_machinery":"The load-bearing object is the Sculpted Memory Forgetting Adapter (SMFA), built from two weight updates. The Memory Forgetting Adapter ΔW_f comes from refusal-label fine-tuning on the forget set and encodes the forgetting effect. The Retaining Anchor ΔW_a comes from fine-tuning on a few-shot subset of the retain set and encodes the direction that preserves unrelated knowledge. The masking rule kills an entry of ΔW_f when the anchor and forgetting updates have opposite signs (directional conflict) and the forgetting entry exceeds |ρ·ΔW_a| (relative magnitude), with ρ = ||ΔW_f||_F / ||ΔW_a||_F making the two updates comparable. Merging the masked adapter back into the base model leaves the non","core_discovery":"The central claim is that benign forgetting—removing the targeted privacy-sensitive multimodal memory while preserving unrelated knowledge and foundational image understanding—is achievable by sculpting the forgetting update rather than by stronger forgetting. SMFA first obtains a Memory Forgetting Adapter ΔW_f by fine-tuning on forget-set questions with refusal labels, then builds a Retaining Anchor ΔW_a from a few-shot retain set. The mask removes entries of ΔW_f only when the two updates point in opposite directions and the forget entry is large relative to the anchor entry after scale normalization. The surviving update is merged into the base model. On S-MLLMUn Bench, SMFA drives Fact S","pith_inferences":["The paper tests SMFA on synthetic profiles; a natural extension is to apply the same anchor-mask rule to naturally occurring private images such as faces, documents, and medical photos, where the anchor would need to generalize across less contrived visual variation.","A testable consequence of the anchor's 'strong generalization' assumption is that deliberately narrowing the few-shot retain set—for example, omitting an entire visual modality like the ophthalmic images—should make that modality vulnerable to unlearning; the paper's design could measure this directly.","The same weight-sculpting principle could plausibly apply to knowledge editing in MLLMs, where a factual correction is an update that should be confined rather than broadcast; the paper does not claim this, but the mechanism is structurally identical.","If benign forgetting becomes a standard objective, safety post-training could shift from broad refusal behavior toward identity-specific forgetting—models could describe a face generically while refusing to name or link it to private attributes."],"forward_implications":["If SMFA is right, MLLMs can be granted a practical right to be forgotten: targeted privacy erasure without retraining and without losing the vision-language skills that make the model useful.","The few-shot retain set (matched in size to the forget set, 5–15% of the data) means unlearning can proceed even when the full training corpus is unavailable.","S-MLLMUn Bench's joint evaluation of forgetting, retention, and image understanding gives the field a shared protocol for judging whether unlearning is benign rather than merely destructive.","The k hyperparameter gives practitioners a dial: raising k strengthens forgetting on the forget set while retain-set scores stay largely stable until k becomes very large.","The sign-and-magnitude masking rule connects unlearning to model merging, suggesting that conflicting updates can be reconciled in weight space without gradient ascent or neuron pruning."],"fun_headline_variants":["Masked updates forget private data, keep vision intact","Selective unlearning preserves image understanding","Forget sensitive facts, keep visual skills","Benign forgetting: erase targeted memories, preserve vision","SMFA unlearns privacy, preserves MLLM capabilities"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The method depends on the claim that a few-shot retain set, matched in size to the forget set, produces a retaining anchor whose weight direction represents all unrelated knowledge and general visual understanding well enough that masking every conflicting dominant entry cannot delete anything important.","fun_headline_variants_meta":{"raw":{"variants":["Masked updates forget private data, keep vision intact","Selective unlearning preserves image understanding","Forget sensitive facts, keep visual skills","Benign forgetting: erase targeted memories, preserve vision","SMFA unlearns privacy, preserves MLLM capabilities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000133,"raw_usage":{"total_tokens":974,"prompt_tokens":749,"completion_tokens":225,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":152}},"tokens_in":493,"tokens_out":225,"duration_ms":2963,"temperature":1.0,"reasoning_tokens":152,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T20:18:22.442841+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SMFA with a retain anchor built only from character-image questions, then measure ophthalmic image understanding on the forget set. If the few-shot anchor fails to generalize to the ophthalmic modality, that score should drop visibly after unlearning, showing the anchor does not protect unrelated capabilities. Alternatively, resample the few-shot retain set several times and check whether retain-set image understanding varies sharply across runs; high variance would falsify the anchor's reliability.","supporting_citations":[],"review_version":1}