REVIEW 3 major objections 3 minor 2 cited by
From Learning to Unlearning: Biomedical Security Protection in Multimodal Large Language Models
T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Biomedical multimodal language models retain private and incorrect knowledge even after machine unlearning, according to a new benchmark.
desk verdict Proposes the first biomedical multimodal unlearning benchmark; the idea is right, but the abstract alone can't support the empirical claim, and synthetic-data realism is the key thing to check. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying machinery is the benchmark construction itself. The data-generation pipeline takes ordinary biomedical multimodal training samples and synthesizes (a) private patient information, such as identifiers or clinical details that should never be emitted, and (b) factual errors, such as wrong dosages or outdated guidelines, embedding both into the training set in a way that a model can learn them. The Unlearning Efficiency Score then aggregates, across subsets, how completely each unlearning method removes the target knowledge relative to how well it preserves normal task performance, yielding a single number for direct comparison.
What would settle it
Take two biomedical MLLMs: one unlearned on MLLMU-Med's synthetic harmful samples and one unlearned on a corresponding set built from real, de-identified clinical records and verified medical errors. If the two models show substantially different leakage and error rates on real-world prompts—for instance, if the synthetic-trained model looks safe but the real-trained model still emits private tokens—the benchmark's synthetic pipeline is not measuring the right thing. A second check: probe the unlearned model with paraphrases of the exact synthetic harmful samples and count how many private or
Extended reading notes
Core claim
The central claim is a measurement result plus a measurement instrument. The instrument is MLLMU-Med, a benchmark assembled from a data-generation pipeline that injects synthetic private patient data and synthetic factual errors into the training distribution of a biomedical multimodal model, then asks unlearning methods to scrub them while preserving normal behavior. The result is that none of the five evaluated unlearning methods reliably does the scrubbing: under the proposed Unlearning Efficiency Score, they leave substantial harmful knowledge behind in both the privacy-protection and incorrectness-removal scenarios. The paper's claim is that this shortfall is real and measurable, and th
Load-bearing premise
The whole measurement stands on the assumption that synthetic private data and synthetic factual errors behave like the real private information and real mistakes that actually appear in biomedical training data, so that scores on MLLMU-Med predict what a deployed model would leak or misstate.
Editorial extensions
If this is right
- Existing unlearning approaches show limited effectiveness on biomedical MLLMs, so harmful knowledge is likely to remain after unlearning in current practice.
- MLLMU-Med can serve as a standardized evaluation set for future unlearning methods in this domain.
- The Unlearning Efficiency Score provides a single composite measure combining removal success and capability preservation.
- The dataset generation pipeline can create training sets containing synthetic private data and factual errors without needing real patient records.
Reading between the lines
- A natural next stress test is to check whether unlearning that scores well on MLLMU-Med also removes knowledge learned from real, non-synthetic clinical text; if it does not, the benchmark's synthetic pipeline may be overestimating unlearning success.
- The same benchmark structure could transfer to other high-stakes multimodal domains, such as finance or legal documents, where private detail and factual error are similarly dangerous.
- The paper's limited-effectiveness result suggests that current unlearning methods were tuned for single-modal text or simpler classification, and that designs exploiting the multimodal structure, such as aligning image and text forgetting, may be needed.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript (arXiv:2508.04192, abstract only) introduces MLLMU-Med, claimed to be the first benchmark for evaluating machine unlearning in biomedical multimodal large language models (MLLMs). The benchmark is constructed from a data generation pipeline that injects synthetic private data and factual errors into training sets. Two scenarios are considered: privacy protection and incorrectness removal. The authors also propose an Unlearning Efficiency Score for overall unlearning performance and report experiments with five unlearning approaches, concluding that current methods have limited effectiveness in removing harmful knowledge from biomedical MLLMs. The abstract does not include equations, protocol details, dataset statistics, or numerical results.
Significance. If the central claims are correct, the paper would provide a reusable benchmark and a composite evaluation metric for a practically important but underexplored problem. The finding that five existing unlearning approaches perform poorly on biomedical MLLMs would be a valuable, falsifiable result that could motivate further research. However, the significance is conditional on the synthetic data being a faithful proxy for real-world harmful biomedical content, and on the Unlearning Efficiency Score being a valid and unbiased measure. At present, the abstract provides no evidence on either point, so the significance cannot be confirmed from the abstract alone.
major comments (3)
- [Abstract] The central claim that existing unlearning methods show 'limited effectiveness' on biomedical MLLMs rests on the external validity of the synthetic data pipeline. The abstract states that the pipeline 'effectively integrates synthetic private data and factual errors into the training set,' but gives no evidence that these synthetic samples resemble real private patient information or real factual errors in terms of distribution, memorability, or learnability. If the synthetic samples are easier or harder to unlearn than natural occurrences, the reported conclusion may be an artifact of the benchmark. The manuscript should include distributional analysis, human evaluation, or external validation showing that synthetic harmful samples are representative proxies.
- [Abstract] The proposed 'Unlearning Efficiency Score' is introduced without a definition, formula, or specification of its components and weights. Since the score is new and is used to reach the headline conclusion, the absence of its formulation makes the result non-reproducible and potentially circular if weights are tuned to a desired outcome. The full manuscript must provide the exact score definition, justification for weights, and sensitivity analysis.
- [Abstract] The evaluation of five unlearning approaches is mentioned, but no quantitative results, dataset sizes, error bars, or statistical significance are reported in the abstract. The claim of 'limited effectiveness' is unverifiable without the underlying numbers and protocol details. The manuscript should report per-method scores, standard deviations across runs, and a clear description of the evaluation protocol.
minor comments (3)
- [Abstract] The phrase 'the first benchmark' should be supported by a literature review in the full text showing that no prior biomedical MLLM unlearning benchmark exists; otherwise, the novelty claim is not yet substantiated.
- [Abstract] The abstract uses 'security' in the title and first sentence but the described scenarios are privacy and correctness; the relationship between these concepts could be clarified for readers.
- [Abstract] The term 'Unlearning Efficiency Score' would benefit from a brief intuitive explanation in the abstract, even if the formal definition appears in the main text.
Circularity Check
No circularity identified in the abstract: the benchmark construction and evaluation are empirical, not definitionally forced.
full rationale
This review is limited to the abstract, which contains no equations, no fitted parameters, and no derivation chain that could reduce to its own inputs. The paper proposes a benchmark (MLLMU-Med) via a synthetic data generation pipeline and reports that five unlearning methods show limited effectiveness. This is an empirical measurement, not a tautology: the finding that current methods perform poorly could in principle have been the opposite, and nothing in the abstract defines the result into existence. The proposed Unlearning Efficiency Score is mentioned but no formula or weighting scheme is given, so there is no basis to claim its definition forces the conclusion. The skeptic's concern about synthetic-to-real transfer is a question of external validity and construct validity, not circularity: even if the synthetic benchmark does not generalize to real private data, the benchmark's reported results are not equivalent to its assumptions by construction. There are no visible self-citations or imported uniqueness theorems. Accordingly, no circular step can be exhibited, and the appropriate score is 0.
Assumptions & free parameters
free parameters (2)
- Unlearning Efficiency Score formulation =
not specified (abstract only)
- Harmful sample injection ratio =
not specified (abstract only)
assumptions (2)
- domain assumption Synthetic harmful content faithfully represents real private patient information and real factual errors in biomedical training data
- domain assumption Model responses to the benchmark scenarios reveal whether harmful knowledge was actually removed
invented entities (2)
-
MLLMU-Med benchmark
-
Unlearning Efficiency Score
Cite this review
Pith. "Pith review of From Learning to Unlearning: Biomedical Security Protection in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/STU5327F
@misc{pith2026250804192,
author = {Pith},
title = {Pith review of: From Learning to Unlearning: Biomedical Security Protection in Multimodal Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/STU5327F}},
note = {Machine review of arXiv:2508.04192}
}
read the original abstract
The security of biomedical Multimodal Large Language Models (MLLMs) has attracted increasing attention. However, training samples easily contain private information and incorrect knowledge that are difficult to detect, potentially leading to privacy leakage or erroneous outputs after deployment. An intuitive idea is to reprocess the training set to remove unwanted content and retrain the model from scratch. Yet, this is impractical due to significant computational costs, especially for large language models. Machine unlearning has emerged as a solution to this problem, which avoids complete retraining by selectively removing undesired knowledge derived from harmful samples while preserving required capabilities on normal cases. However, there exist no available datasets to evaluate the unlearning quality for security protection in biomedical MLLMs. To bridge this gap, we propose the first benchmark Multimodal Large Language Model Unlearning for BioMedicine (MLLMU-Med) built upon our novel data generation pipeline that effectively integrates synthetic private data and factual errors into the training set. Our benchmark targets two key scenarios: 1) Privacy protection, where patient private information is mistakenly included in the training set, causing models to unintentionally respond with private data during inference; and 2) Incorrectness removal, where wrong knowledge derived from unreliable sources is embedded into the dataset, leading to unsafe model responses. Moreover, we propose a novel Unlearning Efficiency Score that directly reflects the overall unlearning performance across different subsets. We evaluate five unlearning approaches on MLLMU-Med and find that these methods show limited effectiveness in removing harmful knowledge from biomedical MLLMs, indicating significant room for improvement. This work establishes a new pathway for further research in this promising field.
Forward citations
Cited by 2 Pith papers
-
Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models
Knowledge holes, severe response degradation on benign inputs that share patterns with forgotten content, are shown to exist in unlearned multimodal LLMs and are partially repaired by SPAR on LLaVA-1.5-7B.
-
Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models
The paper defines attribute-level MLLM unlearning and proposes CLRP, but the method's headline forgetting gains on cloze are partly produced by test-time logit subtraction applied only to the forget and test sets.
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.