{"id":"bf27533d-edb9-499d-a56f-73188434adb0","arxiv_id":"2608.01008","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"The paper defines attribute-level MLLM unlearning and proposes CLRP, but the method's headline forgetting gains on cloze are partly produced by test-time logit subtraction applied only to the forget and test sets.","lead":"This paper introduces a benchmark and method for making an AI model forget one specific attribute of a person, such as a birth date, while keeping other facts about that person intact. The claimed success of the method is weakened because its evaluation subtracts logits directly on the forget and test sets, which inflates the headline results.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CLRP's headline cloze-accuracy gains are driven by a test-set-only output-space logit shift (Appendix, 'Inference-Time Intervention'), not by the representation-level projection.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing flaw: the appendix reveals a test-set-specific output-space logit subtraction applied to CLRP's forget/test cloze evaluation but not to baselines. This concern lands because the paper's headline numbers—16.00-point reduction in Attribute Forget Set cloze accuracy and the 3.02-point retention improvement—are the main evidence for the central claim. The output-space subtraction directly, mechanically, lowers the measured cloze accuracy on the very sets used to define forgetting. Moreover, Table 2's explicit statement that 'cloze accuracy is identical across all variants' is strong independent evidence that the cloze metric is not measuring the representation-level intervention at all; it is measuring the fixed inference-time blocker. This is not an internal inconsistency in the sense of a broken derivation, but it is a correctness risk in the empirical argument: the method as evaluated includes an oracle that knows which queries are forget/test queries, which is incompatible with Definition 1's requirement that forgetting hold under held-out reformulations rather than prompt-specific blocking. A legitimate inference-time unlearning method could, in principle, include a query classifier or refusal mechanism, but then it must be evaluated as such and applied uniformly to all methods. Here, the operation is hidden in the appendix and asymmetrically applied. The concrete test—removing the −4.0 shift or applying it to baselines—would settle the matter. I concur with the REJECT verdict; no adjustment is needed.","tokens_in":26987,"tokens_out":4973,"duration_ms":52722,"concrete_test":"Remove the Appendix 'Inference-Time Intervention' logit subtraction (−4.0) from CLRP's answer-slot interface and recompute the 5% Attribute Forget Set and Test Set cloze accuracies for LLaVA-1.5-7B (long and short) and Qwen2.5-VL-3B. If CLRP's cloze accuracy rises from 4.00%/0.00% toward the 14.50–20.00% baseline range, the reported forgetting gain is an artifact of the output-space blocker. As a control, apply the same −4.0 correct-answer logit shift to the strongest baseline (GD) on the same sets; if GD's Attribute Forget Set cloze accuracy falls to ~4%, the original comparison is unfair. Also report cloze accuracy in Table 2's ablations with the output-space operation disabled, to check whether representation-level variants differ once the blocker is removed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that CLRP achieves strong attribute-level forgetting—rests primarily on Attribute Forget Set / Test Set cloze accuracy (e.g., 20.00%→4.00% in Table 1). The appendix's 'Inference-Time Intervention' states that, in the answer-slot interface, the implementation 'realizes target suppression in output space by subtracting 4.0 from the corresponding logits on the Attribute Forget Set and held-out Test Set; no such operation is applied to a retain set.' This is an evaluation-only, correct-answer logit penalty applied only to CLRP and only to the target sets. It directly lowers the measured cloze accuracy that constitutes the headline result; it is not a representation-level effect, and it presupposes an oracle that knows which queries are forget/test queries. Baselines are not given this operation, so the comparison is not method-vs-method. Table 2 corroborates the concern: cloze accuracy is omitted because it is 'identical across all variants,' meaning the output-space blocker, not the projection or layer selection, determines the cloze numbers. Without this operation, the 16-point forgetting reduction is not shown to come from CLRP; with it applied equally to baselines, the advantage may vanish.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces attribute-level multimodal unlearning as a finer-grained deletion task than profile-level unlearning, constructs the AMU-Bench benchmark, and proposes CLRP, a training-free method that first uses activation patching to localize a causally relevant layer and then applies a retain-aware contrastive projection to attenuate the target attribute subspace. Experiments on LLaVA-1.5-7B/13B and Qwen2.5-VL-3B report strong Attribute Forget Set cloze reductions (e.g., 20.00% to 4.00%) and modest Attribute Retain Set improvements over four training-based baselines. The appendix describes an additional 'Inference-Time Intervention' that subtracts 4.0 from correct-answer logits on the Attribute Forget Set and held-out Test Set only, which directly affects the headline cloze metric.","tokens_in":27270,"tokens_out":6390,"duration_ms":70922,"significance":"Attribute-level unlearning is a useful and timely problem, and the AMU-Bench construction with three attribute types, multiple forget ratios, and separate same-profile retain sets is a valuable contribution. If the reported results were valid, CLRP would be a notable training-free intervention with practical appeal. However, the central empirical claim is not supported as presented because the main cloze metric is influenced by an output-space logit offset applied only to CLRP and only to the forget/test sets, and because the ablation table shows cloze accuracy is invariant to all representation-level variants. The benchmark itself may still be of interest, but the method comparison in the current form is not fair.","major_comments":[{"comment":"The appendix states that 'the implementation realizes target suppression in output space by subtracting 4.0 from the corresponding logits on the Attribute Forget Set and held-out Test Set; no such operation is applied to a retain set.' This is a test-set-specific logit penalty applied only to CLRP. It directly lowers the cloze accuracies that are the headline results (e.g., Table 1: LLaVA long-text 20.00% to 4.00%; the abstract's 16.00-point claim), and it requires oracle knowledge of which examples are forget/test queries. This makes the comparison to baselines unfair and conflicts with Definition 1's requirement that forgetting generalize to held-out reformulations rather than be produced by an answer-slot blocker. Without this operation, or with it applied equally to all methods, the reported cloze gains are not shown to originate from the representation-level projection.","section":"Appendix, 'Inference-Time Intervention'; Table 1"},{"comment":"Table 2 explicitly omits cloze accuracy because it is 'identical across all variants.' Since cloze accuracy is the central metric used for the paper's main claims, this means causal layer selection, projection rank, and attenuation coefficient have no measurable effect on the headline metric; the only component that changes cloze is the output-space logit offset. Consequently, the paper's causal narrative—that CLRP's retain-aware projection removes the target subspace while preserving same-identity evidence—is not validated by the metric on which the state-of-the-art claim rests. The authors should report cloze accuracy with the logit offset removed and demonstrate that the representation-level intervention alone changes cloze behavior.","section":"Table 2; Appendix, 'Ablation Study'"},{"comment":"The Real-Person Retain Set is part of the evaluation of 'unrelated multimodal knowledge' and 'general utility' per Definition 1, but the appendix tables show large degradation for CLRP on this set, especially for Qwen2.5-VL-3B at 15% forget ratio. For example, long-text classification accuracy drops to 59.27% versus GA 73.63% and PO 71.67%; short-text is 61.75% versus GA 73.63%. These results are not discussed in the main text and appear inconsistent with the abstract's claim of preserving general utility and improved retention. The paper should aggregate and analyze Real-Person Retain Set results and reconcile these deficits with the claimed overall preservation.","section":"Tables 14-16; Real-Person Retain Set"},{"comment":"The hyperparameter search ranks configurations by J=(1−F)+0.5A+0.5S, where F is the Attribute Forget Set mean and A is the Attribute Retain Set mean. Thus k and epsilon are selected on the same evaluation metrics reported as final results (Table 6). Although the chosen (20,0.01) is nearly tied with the best (20,0.00), this selection procedure is circular and overstates generalizability. A separate validation split or fixed default hyperparameters should be used when reporting final performance.","section":"Appendix, 'Hyperparameter Search'; Eq. (9)"}],"minor_comments":[{"comment":"The text has many missing spaces (e.g., 'activationpatching', 'andshort-text', 'Selective,ControlledandDomain-Agnostic'). This appears to be a formatting artifact and should be corrected.","section":"Throughout"},{"comment":"The estimator has many fixed constants (common-direction ranks, retain-residual ranks, weights, shrinkage coefficients, projection rank k, retain regularization, attenuation coefficient). No sensitivity analysis or ablation is shown for most of these constants, so it is unclear how robust the method is to their choices.","section":"Appendix, 'Retain-Aware Projection Estimation'"},{"comment":"Main results are single runs without error bars or significance tests. The claimed 2-3 point retention improvements and classification differences may be within run-to-run noise, particularly with small evaluation sets.","section":"Tables 1, 7-13"},{"comment":"The definition excludes 'prompt-specific blocking' as insufficient, but the appendix's output-space logit subtraction is a form of answer-specific blocking at evaluation time. The relationship between this implementation and the definition's generalization requirement should be clarified or removed.","section":"Definition 1 and Inference-Time Intervention"}],"recommendation":"reject","confidential_remarks":"The central evaluation issue is severe: the headline cloze result is produced by an output-space logit offset applied only to CLRP and only to the forget/test sets, and the ablation table confirms that cloze accuracy is invariant to the representation-level method. This is not a minor oversight but a fundamental fairness and validity problem for the main claim. The benchmark contribution is potentially valuable, so a substantially revised version that removes the output-space blocker, re-runs all baselines under a common protocol, and separately validates the representation-level effect could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: this paper defines a genuinely useful task (attribute-level MLLM unlearning with same-profile retain sets) and constructs a benchmark around it, but the central method claim is undercut by an evaluation oracle. The appendix states that on the answer-slot interface, CLRP subtracts 4.0 from correct-answer logits on the Attribute Forget Set and Test Set only. That directly manufactures the reported 20% to 4% cloze reduction. Baselines are not given the same operation, so the comparison is not method-vs-method.\n\nGive credit where it is earned. Splitting attributes into long-text, numeric, and short-text categories and adding an Attribute Retain Set that checks preservation of non-target facts about the same profile is a real step beyond profile-level benchmarks like MLLMU-Bench, FIUBench, and CLEAR. The observation that within-profile collateral damage is invisible to entity-level retain sets is correct and worth publishing. The benchmark's stratified results (numeric easier to forget than short-text or long-text) are descriptive and likely robust.\n\nThe method itself is a sensible combination of known pieces: activation patching to pick a layer, then a retain-aware subspace projection. The ablations are honestly reported—indeed, Table 2 notes cloze accuracy is identical across variants, which essentially confirms that the output-space blocker, not the projection, drives the cloze numbers.\n\nSoft spots, in proportion. The logit subtraction is the load-bearing flaw; it is an output-space answer blocker, not a representation-level intervention, and it presupposes an oracle that knows which queries are forget/test. The hyperparameter search also optimizes an objective defined on the same forget/retain metrics being reported, which inflates the trade-off numbers. And there is no released code or data, so the benchmark is not yet independently reproducible.\n\nThe task and benchmark deserve serious attention. A careful referee should ask the authors to rerun everything with the logit operation either removed or applied uniformly to baselines, and to report the delta explicitly. As submitted, the comparative claims for CLRP should not stand, but the benchmark contribution is sufficient to justify peer review rather than desk rejection.","headline":"Attribute-level unlearning benchmark is a real contribution, but CLRP's headline cloze gains are produced by a test-set-only logit subtraction, so the method comparison doesn't hold.","tokens_in":27785,"tokens_out":1285,"would_cite":true,"duration_ms":15116,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single-layer, training-free projection can erase one attribute of a person from a multimodal AI model while leaving the same person's other facts intact.","keywords":["attribute-level unlearning","multimodal large language models","machine unlearning","activation patching","subspace projection","CLRP","AMU-Bench","privacy"],"falsifier":"Re-run the 5% LLaVA-1.5-7B conditions with the 4.0 output-score suppression removed from CLRP and applied equally to the baselines; if CLRP's Attribute Forget and Test cloze accuracies lose most of their advantage, the central forgetting result is an artifact of evaluation-side answer blocking.","tokens_in":26875,"feed_emoji":"🎯","tokens_out":10684,"duration_ms":100797,"temperature":0.7,"pith_summary":"The paper introduces a finer-grained form of machine unlearning for multimodal large language models: instead of deleting an entire profile or identity, the model should forget one specified attribute, such as a birth date, employer, or hobby description, while still answering other questions about the same person. To test this, the authors build AMU-Bench, a benchmark of profile-attribute pairs spanning long-text, numeric, and short-text targets, and find that existing unlearning methods trade off badly between forgetting and same-profile retention. Their remedy is CLRP, a training-free method that uses activation patching to locate the layer causally responsible for the target attribute, then applies a retain-aware projection that removes the attribute's subspace from that layer's hidden states while preserving directions shared with non-target facts. On LLaVA-1.5-7B and Qwen2.5-VL-3B at a 5% forget ratio, the paper reports that CLRP cuts Attribute Forget Set cloze accuracy by up to 16 percentage points, from 20.00% to 4.00%, and raises Attribute Retain Set cloze accuracy by up to 3.02 points, from 14.77% to 17.79%, relative to the strongest baseline.","feed_headline":"Training-free method drops leaked attribute answers from 20% to 4%","feed_subtitle":"Erases one fact about an identity while leaving the same person's other facts intact, no retraining.","key_machinery":"The central mechanism is a two-stage, training-free intervention called CLRP. Stage one, causal layer localization, uses activation patching to select the layer $\\ell^*$ whose attention-head output, when patched into a paired retain run, most restores target-answer evidence; this determines where to intervene. Stage two, retain-aware contrastive projection, estimates a low-rank subspace $V$ from contrastive target/retain representations at that layer, whitening retain variation by $W = Q \\Lambda^{-1/2} Q^\\top$ and taking the leading eigenvectors of $W S_f W$; the operator $P = (1+\\epsilon)I - V V^\\top$ then scales down the target subspace and slightly scales up everything else. A forward hoo","core_discovery":"The central claim is that attribute-level unlearning can be achieved without retraining or parameter updates by intervening at a single causally selected layer. CLRP first pairs target queries with same-identity retain queries and uses activation patching to find the attention head whose restoration most strongly brings back target-answer evidence; that head is used only to pick the layer. At that layer, a contrastive feature map builds target-conditioned and retain-conditioned statistics, a whitening transform ranks directions by target variation relative to retain variation, and the top-k directions form a subspace V; the operator $P = (1+\\epsilon)I - V V^\\top$ is applied to the hidden sta","pith_inferences":["The appendix's evaluation subtracts 4.0 from the correct-answer logits on the Attribute Forget and held-out Test cloze sets only; an independent replication that removes that output-space adjustment, or applies it equally to all baselines, would clarify how much of the reported forgetting gain comes from the projection itself.","CLRP needs paired target and retain queries for the same identity, which suggests a user-facing 'forget this attribute' API is plausible, but its selectivity may weaken when a user cannot articulate a clean retain question.","Because the intervention is localized to one layer, composing it with other training-free safety or privacy projections is a natural next test; the question is whether two such projections interfere when applied at different layers.","AMU-Bench uses fictitious profiles, so applying the same attribute-level protocol to real-person data would test whether the observed selectivity transfers beyond synthetic identities."],"forward_implications":["Attribute-level unlearning becomes a distinct, measurable task: a request can name a single profile-attribute pair, and success requires both held-out reformulation of the forgotten attribute and preservation of the same profile's other attributes.","A deployed multimodal assistant could forget a single sensitive field, such as salary, birth date, or employer, without retraining, while still recognizing and answering other questions about the same person.","Evaluation of multimodal unlearning should separate same-profile retention from retention of unrelated profiles, because the former is systematically harder and is masked by profile-level benchmarks.","Forgetting difficulty depends on attribute type: numeric facts are much easier to suppress than long-text or short-text facts, so unlearning scores should be reported per attribute category rather than only as aggregates.","Because CLRP changes no weights, the intervention can be applied or removed at inference time and could be combined with other frozen-model interventions."],"supporting_citations":[{"why":"Supplies the activation patching / causal tracing technique used to select the layer mediating target-attribute disclosure.","marker":"(Meng et al. 2022)"},{"why":"Provides iterative nullspace projection, the subspace-removal lineage that CLRP's retain-aware projection builds on.","marker":"(Ravfogel et al. 2020)"},{"why":"Provides the covariance-based closed-form concept-erasure estimator behind the retain-relative whitening.","marker":"(Belrose et al. 2023)"},{"why":"Provides the contrastive activation-difference construction used to form target and retain feature statistics.","marker":"(Turner et al. 2024)"},{"why":"MLLMU-Bench supplies the fictitious profiles and split structure that AMU-Bench rebuilds at attribute granularity.","marker":"(Liu et al. 2024)"},{"why":"Defines the LLaVA architecture family on which CLRP is evaluated.","marker":"(Liu et al. 2023)"},{"why":"Defines the Qwen2.5-VL architecture on which CLRP is evaluated.","marker":"(Wu et al. 2025)"},{"why":"Provides the machine-unlearning formulation and the gradient-ascent baseline CLRP must beat.","marker":"(Bourtoule et al. 2020)"},{"why":"Provides the gradient-difference baseline used in the forgetting-retention comparisons.","marker":"(Liu, Liu, and Stone 2022)"},{"why":"Provides the preference-optimization baseline and reference-model construction used in the comparisons.","marker":"(Maini et al. 2024)"}],"fun_headline_variants":["Attribute unlearning: drop 80% of leaks with no retraining","One-layer causal intervention erases a single attribute","Selective forgetting: keep identity, drop the sensitive fact","Training-free CLRP: unlearning without touching parameters","From 20% to 4%: targeted forgetting for multimodal LLMs"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The central claim depends on the evaluation being allowed to suppress the correct answers on the Attribute Forget and held-out Test cloze questions directly at scoring time; if that suppression is removed, or applied equally to all methods, the reported forgetting gains may not come from the projection itself.","fun_headline_variants_meta":{"raw":{"variants":["Attribute unlearning: drop 80% of leaks with no retraining","One-layer causal intervention erases a single attribute","Selective forgetting: keep identity, drop the sensitive fact","Training-free CLRP: unlearning without touching parameters","From 20% to 4%: targeted forgetting for multimodal LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001004,"raw_usage":{"total_tokens":4084,"prompt_tokens":749,"completion_tokens":3335,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":3250}},"tokens_in":493,"tokens_out":3335,"duration_ms":26497,"temperature":1.0,"reasoning_tokens":3250,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:34:25.233143+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 5% LLaVA-1.5-7B conditions with the 4.0 output-score suppression removed from CLRP and applied equally to the baselines; if CLRP's Attribute Forget and Test cloze accuracies lose most of their advantage, the central forgetting result is an artifact of evaluation-side answer blocking.","supporting_citations":[],"review_version":1}