{"id":"d0d2234e-7c12-47af-8866-33f1ba016a9b","arxiv_id":"2607.02897","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Existing MLLM unlearning methods reduce private-attribute leakage on entangled images but substantially harm co-occurring public figures and landmarks, with private knowledge often re-emerging after public finetuning.","lead":"PPE-Bench tests whether multimodal AI can forget a private person in a photo without also forgetting the celebrity or landmark next to them. Existing unlearning methods cut private leakage but often damage that entangled public knowledge, and forgotten facts can reappear after later public finetuning.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged synthetic-proxy limit; the directional claim is internally supported.","rationale":"The paper's central empirical finding is that, under private-public visual co-occurrence, standard MLLM unlearning objectives trade off forget-set suppression against damage to co-present public knowledge and remain vulnerable to relearning. This is directly measured on the constructed benchmark with clear axes (forget / retain / adjacent) and is reproducible in principle from the released data and described methods. The reader's CONDITIONAL verdict already correctly downgrades from ACCEPT precisely because of the synthetic proxy, keyword scoring, and missing error bars. No stronger internal flaw (e.g., circular evaluation, confounded baselines, or contradictory results) appears in the text. Therefore the stress-test does not move the verdict; it confirms that the weakest assumption is external validity rather than soundness of the reported PPE-Bench results.","tokens_in":16439,"tokens_out":546,"duration_ms":60201,"concrete_test":"Take 50 real multi-person photos that contain a non-celebrity face co-occurring with a known public figure or landmark; construct analogous forget/adjacent/retain QA pairs; run the same GA and GD unlearning pipelines used in §4 on a vanilla model finetuned on those images; if adjacent public-info accuracy still falls by a comparable relative amount while forget accuracy drops, the directional claim transfers; if adjacent accuracy remains near vanilla levels, the synthetic entanglement effect is artifactual.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest assumption (synthetic Gemini-generated images + fixed celebrity/landmark pool as proxy for real private-public entanglement) is the main external-validity soft spot, but it is not load-bearing against the paper's actual strongest claim. That claim is scoped to PPE-Bench itself: existing unlearning methods (GA, NPO, KL-Min, GD) reduce forget-set keyword accuracy while also dropping adjacent public-info accuracy, and private answers can reappear after public-only relearning (Tables 1, Figs. 3–5, §4.2–4.5). Within this controlled synthetic regime the measurements are consistent across two MLLMs, seen/unseen splits, and multiple objectives; the entanglement failure mode is demonstrated, not merely asserted. No internal inconsistency, missing control, or mis-specified objective undermines the reported trade-off on the benchmark. The synthetic construction therefore limits how far one should generalize the quantitative numbers to real social-media photos, but it does not invalidate the existence of the failure mode the paper claims to expose.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces PPE-Bench to evaluate multimodal LLM unlearning when private target individuals are visually co-present with public celebrities and landmarks in the same images. It constructs 1,800 synthetic entangled images (from MLLMU-Bench profiles + Gemini generation) paired with forget/retain/adjacent QA splits, then shows that standard methods (GA, NPO, KL-Min, GD) reduce forget-set keyword accuracy (and BLEU/ROUGE-L) on both seen and unseen images while also degrading adjacent public-information accuracy; private answers can reappear after subsequent public-only finetuning. Two mitigation objectives (PIP with adjacent/retain losses, and target-guided blurring via masks Tt/Tc) are proposed and shown to improve public retention at some cost to forgetting strength. Experiments use Qwen3-VL-4B and LLaVA-1.5-7B.","tokens_in":16683,"tokens_out":1074,"duration_ms":26549,"significance":"If the reported trade-off and relearning vulnerability hold, PPE-Bench supplies a practically relevant evaluation axis missing from prior MLLM unlearning suites (CLEAR, MLLMU-Bench, PEBench), which used isolated single-subject images and fully disjoint forget/retain sets. The open data release, dual-model results, seen/unseen splits, and explicit adjacent-set metric constitute concrete, reusable contributions that can drive entanglement-aware unlearning research. The two simple mitigations (PIP and blurring) further illustrate that the failure mode is partially addressable. External validity is limited by the synthetic construction, yet the directional claim is well-supported inside the controlled regime and is therefore useful for the community.","major_comments":[{"comment":"§4.3 / Figure 3: The central claim that existing methods “substantially harm adjacent public information” is demonstrated only on the entangled images themselves. A control that unlearns the same private targets on non-entangled (single-subject) images and then measures public-figure/landmark accuracy on clean public images would isolate how much of the observed drop is caused by visual entanglement versus generic side-effects of the unlearning objectives. Without it the entanglement-specific interpretation remains suggestive rather than fully isolated.","section":"§4.3 / Figure 3"},{"comment":"§4.1 Evaluation Metrics and Tables 1/3: Keyword presence accuracy is a coarse proxy for free-form generation; a model that refuses, hallucinates a different date, or produces a near-paraphrase can all receive the same binary score. An error analysis or secondary LLM-as-judge / exact-match metric on a stratified sample would confirm that the large ACC drops truly reflect loss of private attributes rather than surface-form artifacts.","section":"§4.1 / Tables 1, 3"},{"comment":"§3.4.2 Eqs. (2)–(4): Target-guided blurring presupposes per-image target bounding boxes (or masks) for every forget and adjacent sample. Real right-to-be-forgotten requests typically supply only the identity or a few images, not dense annotations. The paper should either demonstrate an automatic face-detection pipeline that recovers comparable performance or clearly scope the method as an oracle upper bound.","section":"§3.4.2"}],"minor_comments":[{"comment":"Throughout: “LLaV A” is inconsistently spaced; standardize to “LLaVA”.","section":null},{"comment":"Figure 2 caption and §3.1: the total image count is stated as 1,800 (1,500 train + 300 test) yet Table 2 lists 1,500 images; reconcile the numbers.","section":"Figure 2 / Table 2"},{"comment":"§4.6 / Figure 6: hyper-parameter sweeps are shown only for PIP; a brief note on whether the same λr range was used for GD/KL would aid reproducibility.","section":"§4.6"},{"comment":"Appendix A.2 prompt: the generation instruction does not constrain lighting, pose diversity or occlusion; a short qualitative failure-mode gallery would help readers judge visual realism.","section":"Appendix A.2"},{"comment":"References: several arXiv preprints (e.g., PEBench, MMUNLEARNER) lack final venue or version dates; update where possible.","section":"References"}],"recommendation":"minor_revision","confidential_remarks":"The entanglement axis is a genuine incremental advance over PEBench/MLLMU/CLEAR and fits cs.CR / privacy-oriented ML venues. The synthetic-proxy limitation is already flagged by the authors and does not undermine the internal results; I would not reject on that ground alone. Data/code link is live, which is a plus."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing worth knowing is that this paper correctly diagnoses a hole in CLEAR, MLLMU-Bench, and PEBench—single-person synthetic faces and fully separated forget/retain sets—and builds a cleaner test for the entangled case. That is the real contribution.\n\nWhat is new is the adjacent set: same images as the forget set, but questions only about the co-occurring celebrity or landmark. They generate 1,800 Gemini images (100 fictitious people from MLLMU + 28 celebs + 40 landmarks), 12k train QAs, seen/unseen test splits, and evaluate three axes plus a relearning probe after public-only finetuning. Two models (Qwen3-VL-4B, LLaVA-1.5-7B), keyword ACC + BLEU/ROUGE-L. Tables 1/3 and Figures 3–5 are consistent: GA/NPO/KL/GD drop forget accuracy hard, but adjacent public accuracy falls with them; GD keeps retain utility best; PIP and target-guided blurring (Tt/Tc masks) trade some forgetting for better public retention; private answers often rebound after public relearning. The two mitigations are simple and honestly presented as partial fixes, not silver bullets. Data link is given; construction is independent of the unlearning losses, so circularity is low.\n\nSoft spots are real but secondary. Everything is synthetic (Gemini + fixed celebrity/landmark pool + manual filter); keyword matching is coarse; no error bars; free parameters (λr, λa, per-method LRs) are tuned on the same setup. The paper itself flags the single-target/single-landmark limit. Those constrain how far you should export the exact numbers to real social-media photos, but they do not undercut the directional claim on the benchmark itself. The stress-test note is right: the synthetic proxy is an external-validity limit, not an internal contradiction.\n\nThis is for people working on multimodal unlearning or privacy tooling who need a harder evaluation protocol. It is not a theory paper and not a new unlearning algorithm. I would bring it to reading group, cite the benchmark when I need an entanglement stress test, and send it to peer review. It deserves referee time; the main ask is tighter stats and a clearer statement of the synthetic-scope limit.","headline":"PPE-Bench is a useful, well-scoped fix for a real gap in MLLM unlearning evaluation: private targets co-occurring with public figures/landmarks, with clear evidence that standard methods damage the public side and can relearn private answers.","tokens_in":17346,"tokens_out":579,"would_cite":true,"duration_ms":6321,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Existing MLLM unlearning methods can cut private leakage yet often damage public figures and landmarks that share the same photo as the forgotten person.","keywords":["MLLM unlearning","private-public entanglement","machine unlearning","visual privacy","multimodal benchmarks","right to be forgotten","adjacent set"],"falsifier":"Train and evaluate a strong unlearning method on PPE-Bench-style entangled images: if forget-set keyword accuracy falls near zero while adjacent public-figure and landmark accuracy stays close to the vanilla model, and private attributes remain suppressed after public-only relearning, the claim that existing methods substantially harm adjacent public information would not hold for that method.","tokens_in":17279,"feed_emoji":"🔒","tokens_out":583,"duration_ms":20115,"temperature":0.7,"pith_summary":"Multimodal large language models trained on web data can memorize private details from photos, so machine unlearning is used to remove that knowledge without full retraining. Prior unlearning tests mostly use simplified single-person images and keep the forget and retain data fully separate. This paper argues those tests miss a common real-world case: a private person often appears with a public celebrity or in front of a famous landmark, so scrubbing the private target can also harm public context that should stay. PPE-Bench builds images that place each fictitious private individual next to a public figure and a landmark, then scores models on three linked goals—forgetting private attributes, keeping unrelated utility, and preserving answers about the co-occurring public content. Across standard unlearning methods the private answers drop, but accuracy on the adjacent public questions falls sharply as well, and private facts can reappear after later fine-tuning only on public data. Two simple remedies—an adjacent-set preservation loss and target-guided image blurring—raise public retention, at some cost to forgetting strength.","feed_headline":"Unlearning private faces also erases public landmarks","feed_subtitle":"A new benchmark shows celebrities and landmarks suffer when models forget the person next to them.","key_machinery":"PPE-Bench: a multimodal unlearning benchmark whose every image entangles a forget-target individual with a public figure and landmark, then splits queries into forget, retain, and adjacent sets so private removal and public preservation are measured on the same visual scene.","core_discovery":"When private individuals, public celebrities, and landmarks share the same image, standard MLLM unlearning methods reduce answers about the private target but substantially degrade accuracy on questions about the co-occurring public figure and landmark; the forgotten private knowledge can also re-emerge after subsequent fine-tuning on public information alone.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Unlearning private faces also erases co-appearing public figures","Private unlearning in shared images degrades public landmarks too","MLLM unlearning harms public celebrities next to forgotten people","Forget one private face and public landmarks lose accuracy","Entangled unlearning wipes private targets but scars public context"],"cache_read_input_tokens":128,"weakest_assumption_plain":"Synthetic images that place fictitious faces next to a fixed pool of celebrities and landmarks are realistic enough that measured drops on adjacent public questions will transfer to real social-media photos and real identities.","fun_headline_variants_meta":{"raw":{"variants":["Unlearning private faces also erases co-appearing public figures","Private unlearning in shared images degrades public landmarks too","MLLM unlearning harms public celebrities next to forgotten people","Forget one private face and public landmarks lose accuracy","Entangled unlearning wipes private targets but scars public context"]},"model":"grok-4.5","effort":"low","cost_usd":0.004882,"raw_usage":{"total_tokens":1377,"prompt_tokens":749,"num_sources_used":0,"completion_tokens":65,"cost_in_usd_ticks":48820000,"prompt_tokens_details":{"text_tokens":749,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":563,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":749,"tokens_out":65,"duration_ms":4716,"temperature":1.0,"reasoning_tokens":563,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-12T06:16:30.296696+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train and evaluate a strong unlearning method on PPE-Bench-style entangled images: if forget-set keyword accuracy falls near zero while adjacent public-figure and landmark accuracy stays close to the vanilla model, and private attributes remain suppressed after public-only relearning, the claim that existing methods substantially harm adjacent public information would not hold for that method.","supporting_citations":[],"review_version":1}