{"id":"070dc58b-8c04-47dd-ab43-0941dcc0e4f5","arxiv_id":"2607.18615","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A bilevel meta-training method aligns text-only backbone unlearning with full vision-language model forgetting, improving forget-retain trade-offs across two VLMs and two meme datasets.","lead":"Researchers propose Stochastic Meta-Unlearning, a two-stage training method that tunes the language part of a vision-language model so a few text-only unlearning steps still erase a target concept once the model is put back together. The approach is aimed at making \"machine unlearning\" reliable for multimodal systems rather than just pure text models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Cluster-based forget/retain split in Appendix C.2 is unvalidated; SMU's transfer may reflect distributional robustness to separable clusters rather than multimodal concept unlearning.","rationale":"The reader's weakest assumption highlighted the cluster-based split and the label-accuracy metric. I agree that the cluster split is the most method-specific load-bearing point because it defines the forgetting target and all transfer tasks; if the clusters are not semantically coherent, the method's success on these benchmarks is not evidence of multimodal concept unlearning. I do not select the label-accuracy concern as primary because it affects essentially all unlearning evaluations and is explicitly acknowledged in the Limitations (Appendix D), whereas the cluster split is a preprocessing choice unique to this paper that underpins the task definition. The proposed random-split ablation would directly test whether the clusters' coherence is necessary for SMU's reported behavior. Since the reader already issued a CONDITIONAL verdict, this concern reinforces the need for additional validation but does not change the direction or magnitude of the verdict, so 'UNCHANGED' is appropriate.","tokens_in":15307,"tokens_out":5142,"duration_ms":53778,"concrete_test":"Run the full SMU meta-train/meta-test protocol with forget/retain sets formed by a random split of the training set at the same proportion as the clustered split, holding all other hyperparameters and evaluation sets fixed. If SMU's forget-retain trade-off advantage over the strongest baseline per metric is largely preserved under the random split, the cluster semantics are not load-bearing. If the advantage collapses, the claimed transfer is an artifact of separable clusters and fails for non-coherent targets. Additionally, compute the adjusted Rand index between the cluster assignments and ground-truth labels (e.g., hateful/not-hateful for Hateful Memes, harm levels for Covid Memes) to check whether the clusters correspond to a coherent concept.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SMU enables few text-side unlearning steps to remove a target from the recomposed VLM rests on the definition of the 'target'. Appendix C.2 constructs forget/retain sets by KMeans (k=10, seed 42) on sentence-transformer text embeddings of meme text, with no validation that the resulting clusters correspond to a semantically coherent concept. All transfer experiments (RQ2/RQ3) switch between these clusters, so the 'new forgetting targets' are simply other clusters in the same embedding space. If the clusters are trivially separable (e.g., by topic keywords or style), a meta-learned initialization could learn to suppress cluster-identifying text patterns, which would transfer to any cluster from the same distribution but does not constitute unlearning of a real-world concept. The reported 10.52-point Forget reduction and 20.10/17.01-point utility gains could then be specific to the cluster geometry rather than to a general multimodal unlearning capability. Since the target definition is entirely determined by this unvalidated split, the strongest claim that SMU 'removes the target from the recomposed VLM' is not yet established. This concern is method-specific and load-bearing because it affects the interpretation of both the main results and the transfer experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses machine unlearning in vision-language models (VLMs). It observes that applying text-only unlearning to the language backbone of a VLM does not guarantee that the target is forgotten once the backbone is recomposed with the frozen visual components; image-conditioned queries can still elicit the target. To overcome this, the authors propose Stochastic Meta-Unlearning (SMU), a bilevel meta-learning framework. In the inner loop, a few text-side unlearning steps are applied to the language backbone; in the outer loop, the updated backbone is recomposed with the frozen VLM and the outer loss is evaluated on multimodal forget/retain batches. The outer loss is backpropagated through the inner unlearning steps to update the initialization. Experiments on Qwen2.5-VL and InternVL3 with Hateful Memes and COVID Memes report that SMU achieves a better forget-retain trade-off than three direct VLM-side baselines (GradDiff, NPO, RMU), and that the learned initialization transfers to new forgetting targets and to different meta-test unlearning operators. The paper also includes explicit limitations about computation, frozen visual components, and lack of formal guarantees.","tokens_in":15593,"tokens_out":6591,"duration_ms":74271,"significance":"If the reported results hold, the paper makes a useful conceptual contribution: it shows that language-backbone-only unlearning can be made robust under multimodal recomposition by meta-learning the initialization with VLM-level feedback. This is an interesting and non-obvious direction, and the bilevel formulation is well motivated by the component-system mismatch the authors identify. The paper also explicitly states limitations and does not overclaim formal guarantees. However, the empirical support is currently incomplete in several load-bearing places: the experimental design relies on an unvalidated cluster-based target split, no variance or multiple-seed results are reported, the sampling distribution used in stochastic meta-training is not specified, and no quantitative comparison is made to the text-side-only unlearning baseline that motivates the method. These issues do not invalidate the idea, but they must be addressed before the central empirical claims can be accepted.","major_comments":[{"comment":"No error bars, standard deviations, or multiple-seed results are reported for any of the quantitative claims. The headline numbers (10.52-point Forget reduction, 20.10/17.01-point utility improvements) are point estimates from a single run. Since the differences between methods are sometimes large but the table has no uncertainty information, I cannot tell whether the reported ordering is stable. Please report means and variances over at least three seeds, or justify why single-run evaluation is appropriate.","section":"§5, Table 1"},{"comment":"The cluster-based forget/retain split is load-bearing but unvalidated. KMeans (k=10, seed 42) is applied to MiniLM text embeddings of meme text, and the forget set is 'selected clusters'. The paper claims these form 'semantically coherent' targets, but no evidence is given: no cluster-size table, no label distribution per cluster, no human evaluation or concept-name inspection. Since RQ2/RQ3 switch between these clusters and call them 'new forgetting targets', the transfer claim could reflect transfer between trivially separable text clusters rather than between semantically meaningful unlearning targets. Please add a validation of the clusters, report which clusters are used for meta-training versus held out, and show that the targets correspond to identifiable concepts.","section":"§5, Appendix C.2"},{"comment":"The core stochastic meta-training protocol is underspecified. The environment distribution p(E) over (z, m, c) is never defined: I could not find the number of meta-training targets, the number of sampled environments S, the distribution over inner-loop operators, the distribution over query conditions, or the mapping from clusters to training/test targets. Without this information, the RQ2/RQ3 claims of transfer are not reproducible and it is unclear whether meta-test targets are truly disjoint from meta-training targets. This is not a minor implementation detail; the paper's central contribution is the stochastic meta-learning design, so the exact protocol must be specified.","section":"§4, §5"},{"comment":"The paper's motivation is that text-only unlearning of the standalone backbone fails to generalize to the recomposed VLM, yet the main experiments do not include a text-side-only unlearning baseline. Table 1 compares SMU only with direct VLM-side baselines (GradDiff, NPO, RMU), which themselves receive VLM-level feedback. A comparison with the same three unlearning objectives applied to the language backbone using text-only data, without VLM-level outer feedback, is needed to isolate the contribution of VLM-level feedback. Figure 1 is anecdotal and does not quantify the failure or the effect of SMU relative to the text-side-only version.","section":"§1, §5"},{"comment":"The evaluation metric 'Forget accuracy' on the forget set is a classification accuracy on target-cluster examples. Lower accuracy is interpreted as forgetting, but it may equally reflect misclassification or a collapse to a non-target class (e.g., always predicting 'not hateful'). The paper does not report abstention rates, entropy, or any probe of target knowledge (e.g., open-ended generation or membership-style tests). Given the central claim is that SMU 'removes' the target, additional metrics are needed to distinguish genuine erasure from label flipping. The authors acknowledge in Appendix D that no formal guarantee exists, but the empirical metric should still be strengthened.","section":"§3, §5.1"}],"minor_comments":[{"comment":"The GradDiff rows contain formatting errors: '2.0232.09 41.20' and '0.8731.16 40.20' lack spacing between the Forget and Retain entries, making the table hard to read.","section":"Table 1"},{"comment":"The sentence 'a bilevel framework that aligns language-backbone unlearning with VLM-level forgetting(Figure 2.' has a missing closing parenthesis. Please fix the typo.","section":"§4"},{"comment":"The claim 'compared with the strongest baseline for each metric' should be defined precisely. I reproduced the 10.52-point Forget reduction by averaging each baseline over all 8 settings and comparing to SMU, but the text does not say this. Please state the exact aggregation procedure.","section":"Abstract, §5.1"},{"comment":"The appendix says MiniLM embeddings are used 'if this encoder is unavailable' and otherwise TF-IDF is used. It is unclear which representation was actually used for the reported results. This affects reproducibility and should be stated explicitly.","section":"Appendix C.2"},{"comment":"The transfer results are presented only as figures (Fig. 4, Fig. 5) with no numeric tables. Given that these figures support two of the three main claims, please include the underlying numbers, ideally in a supplementary table, so readers can assess the magnitude of the effects.","section":"§5, RQ2/RQ3"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the cluster-based split is valid and, in my view, the most important issue to resolve. The current evidence does not establish that SMU's transfer is transfer of multimodal unlearning; it may be transfer between separable text clusters. I would also require multiple-seed results and a text-side-only baseline before reconsidering the paper. The idea is interesting and the limitations section is honest, but the empirical load-bearing points need substantial additional work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a serious referee. The central idea is sound: treat VLM unlearning as a bilevel problem where the inner loop does text-side unlearning on the language backbone and the outer loop evaluates the recomposed VLM and backprops through the unrolling. That is a legitimate new application of meta-learning, and the motivating observation—that text-side forgetting can fail once images are added—is real and clearly demonstrated. The transfer experiments to new targets and new operators are the right experiments for this kind of claim, and the image-side unlearning check in Appendix B is a thoughtful control. The authors also state their limitations plainly: no formal forgetting guarantee, frozen visual modules may limit forgetting of visually stored information, and meta-training is expensive.\n\nThe soft spots are mostly about evidence quality, not about the method's logic. Table 1 reports no error bars or seeds, and the headline 10.52 / 20.10 / 17.01 point gains are computed per-metric against whichever baseline happened to be strongest. That makes it hard to know if SMU's advantage is stable or partly driven by baseline collapse (GradDiff driving retain/test to 0 on Covid Memes is a very low bar). The forget set is built by KMeans on text embeddings with k=10, seed 42, and the paper does not validate that clusters correspond to semantically coherent concepts. The stress-test concern—that transfer across clusters may just reflect suppressing cluster-identifying text patterns rather than forgetting a concept—has some teeth, but it is not fatal: the method's target is the cluster, and generalizing to held-out clusters is a meaningful empirical claim even if the real-world semantics are fuzzy. Still, the paper should add variance estimates, a single-baseline comparison, and ideally a probing-based evaluation (e.g., asking the model to describe the target content) to rule out that low forget accuracy just means the model learned to hedge or output unparsable labels.\n\nWho is this for? Researchers working on multimodal unlearning or transferable unlearning. It is a useful methodological step, not a resolution of long-open theory questions. I would send it to peer review—the method is clear, the experiments are reasonably broad, and the limitations are acknowledged. Just require the missing quantitative details before acceptance.","headline":"A sound and useful bilevel meta-unlearning method for VLMs, with honest limitations, but the quantitative claims need error bars and a less cherry-picked baseline comparison before they are taken at face value.","tokens_in":16104,"tokens_out":1723,"would_cite":true,"duration_ms":19997,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stochastic Meta-Unlearning claims that a language-backbone initialization meta-learned with VLM-level feedback lets a few text-only unlearning steps remove a target from the recomposed vision-language model, even when image evidence is supp","keywords":["machine unlearning","vision-language models","stochastic meta-learning","multimodal forgetting","language backbone","bilevel optimization","transferable unlearning","meme classification"],"falsifier":"Re-run SMU with the forget set replaced by a randomly sampled subset of the training data (removing cluster coherence) and compare its forget-retain trade-off against the three baselines; if the advantage collapses, the method's transfer comes from the split's structure rather than from a generally unlearning-ready initialization.","tokens_in":15175,"feed_emoji":"🧹","tokens_out":8899,"duration_ms":78288,"temperature":0.7,"pith_summary":"The paper tries to establish that language-backbone unlearning can be made to stick in a full vision-language model (VLM) — even when images are present at inference — if the backbone begins from an initialization that was meta-learned with VLM-level feedback. The motivating observation is a mismatch: a target forgotten by the standalone text model can reappear once image information is supplied to the recomposed VLM, so text-only objectives are insufficient. To close that gap, the proposed Stochastic Meta-Unlearning (SMU) runs a few unlearning steps on the language backbone in an inner loop, then reinserts the updated backbone into the frozen VLM and evaluates forget/retain losses in an outer loop, using the resulting meta-gradient to adjust the starting point. The paper reports that SMU improves the forget–retain trade-off over three direct VLM-unlearning baselines on two VLMs and two meme datasets, and that the learned initialization transfers to new deletion targets and to different unlearning operators. If true, this matters because deployment-time unlearning could then be done cheaply from the text side only, without touching the visual encoder.","feed_headline":"Meta-learned start point makes text edits erase multimodal targets","feed_subtitle":"The meta-learned starting point is what makes the erasure hold once images enter the model.","key_machinery":"The central object is the meta-learned 'unlearning-ready' initialization θ*_T — the language-backbone parameters or trainable adapters that serve as the starting point for deployment-time unlearning. The carrying mechanism is a bilevel meta-unlearning loop: the inner loop applies K text-only unlearning steps (using GradDiff, NPO, or RMU objectives) to the backbone; the outer loop recomposes the updated backbone with the frozen VLM and computes a weighted loss over multimodal and text forget/retain batches; the meta-gradient with respect to the initialization flows back through the unrolled inner steps. Additionally, SMU stochastically samples the environment (forgetting target, inner operato","core_discovery":"The paper's central claim is that the failure of text-side unlearning to persist after multimodal recomposition can be repaired by changing the initialization rather than the unlearning procedure. SMU meta-learns an 'unlearning-ready' set of language-backbone parameters such that, after a few steps of ordinary text-only unlearning, the recomposed VLM forgets the target under both text-only and image-conditioned queries while retaining utility. The decisive mechanism is the outer loop: the updated backbone is reinserted into the frozen VLM, scored on multimodal forget/retain batches, and the meta-gradient is backpropagated through the unrolled inner steps. On two VLMs and two meme datasets, S","pith_inferences":["A broader principle suggests itself: in any modular model where an edit must stay local but behavior is judged globally, meta-learning the edit's starting point against system-level feedback could reconcile local updates with global outcomes — a recipe that may extend beyond language backbones to adapters, LoRA modules, or other modality pairs.","The advantage may depend on the forget set being a semantically coherent cluster; a natural stress test is to replace the cluster-based split with a random sample and see whether SMU's margin over baselines persists.","Because the paper measures accuracy rather than adversarial recovery, the method's resilience to rephrased prompts, image-only queries, or other cross-modal probes remains open; the authors themselves note there is no formal guarantee against stronger adversarial inputs.","The non-monotonic Forget accuracy during meta-training (Fig. 6) hints that the transferable initialization is a specific region of parameter space, not simply the endpoint of a descent; identifying that region could guide stopping criteria and even suggest a theory of when meta-unlearning generalizes."],"forward_implications":["At deployment, unlearning a VLM requires only text-side gradients on the language backbone; the visual encoder and projector are never updated, cutting multimodal backprop cost.","A single meta-training phase yields an initialization usable for many future deletion requests: SMU transfers to unseen targets in the experiments.","The initialization is not hard-wired to one unlearning recipe; swapping the inner operator to RMU or NPO at meta-test time still gives low Forget accuracy, so users can pick the operator after the fact.","Because image-side unlearning is shown to be unstable (Table 2), the combination of backbone-side updates with VLM-level feedback emerges as the more reliable intervention point for modular VLMs."],"fun_headline_variants":["Text-only unlearning fails when images arrive—meta-learned start fixes it","Meta-learned initialization makes text erasure stick in multimodal models","Forget text, remember images? Not without meta-unlearning the start point","Stochastic meta-unlearning: the right start makes text erasure total in VLMs"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim rests on the cluster-based forget/retain split isolating a single coherent target concept and on label accuracy faithfully measuring its removal; if either fails, the meta-learned initialization may be exploiting split structure or abstention rather than genuinely unlearning.","fun_headline_variants_meta":{"raw":{"variants":["Text-only unlearning fails when images arrive—meta-learned start fixes it","Meta-learned initialization makes text erasure stick in multimodal models","Forget text, remember images? Not without meta-unlearning the start point","Stochastic meta-unlearning: the right start makes text erasure total in VLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001004,"raw_usage":{"total_tokens":4121,"prompt_tokens":823,"completion_tokens":3298,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":3216}},"tokens_in":567,"tokens_out":3298,"duration_ms":92145,"temperature":1.0,"reasoning_tokens":3216,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T14:47:23.524876+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run SMU with the forget set replaced by a randomly sampled subset of the training data (removing cluster coherence) and compare its forget-retain trade-off against the three baselines; if the advantage collapses, the method's transfer comes from the split's structure rather than from a generally unlearning-ready initialization.","supporting_citations":[],"review_version":1}