{"id":"2b20cb5c-7559-487a-8344-8d14e82c5d43","arxiv_id":"2508.04192","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper introduces MLLMU-Med, described as the first benchmark for machine unlearning in biomedical multimodal LLMs, and reports that five existing unlearning methods only partially remove private or incorrect knowledge.","lead":"This paper introduces a new benchmark for testing whether medical AI models that read both images and text can forget private patient data or misinformation that slipped into their training sets. The reported result, that five existing unlearning methods do the job poorly, matters because leaking patient data in medicine is a safety and legal risk.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated synthetic-to-real transfer: MLLMU-Med's harmful content may not represent real biomedical private data or factual errors, making the claimed limited unlearning effectiveness not yet generalizable.","rationale":"The central claim is that five unlearning approaches are limited in removing harmful knowledge from biomedical MLLMs. This is a measurable finding only if MLLMU-Med measures the right construct: the difficulty of unlearning naturally occurring private information and factual errors. The reader identified the same premise as the weakest assumption. I agree: the synthetic generation pipeline is the crux. The proposed test is the standard external-validity check: compare against a realistic surrogate with the same model/methods. If performance is consistent, the concern is settled; if not, the benchmark needs refinement. Given abstract-only review, no other aspects can be evaluated, so the verdict remains UNVERDICTED; this concern does not change the reader's verdict but sharpens what evidence would be needed.","tokens_in":889,"tokens_out":3367,"duration_ms":38020,"concrete_test":"Construct a small realistic surrogate dataset from de-identified clinical notes (e.g., MIMIC-III) and published biomedical factual corrections; embed these as harmful samples into the same base model; run the same five unlearning methods; compare their post-unlearning performance (privacy and error removal) to their MLLMU-Med scores. If the relative ranking of methods or the magnitude of retained harmful knowledge differs by more than a pre-registered threshold (e.g., 0.1 in the Unlearning Efficiency Score), the synthetic pipeline does not faithfully predict real-world behavior. This directly tests whether MLLMU-Med's synthetic content is representative.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's validity hinges on the assumption that its synthetic private data and factual errors are faithful proxies for harmful content in real biomedical training corpora. The abstract states a 'novel data generation pipeline that effectively integrates synthetic private data and factual errors into the training set,' but supplies no evidence that these samples match the distribution, memorability, or learnability of natural occurrences. If the synthetic samples are easier to detect or unlearn because they are artificial (e.g., templated, surface-level, or statistically distinct), the five unlearning methods' measured 'limited effectiveness' could be an artifact of the benchmark rather than a property of real-world unlearning. Conversely, if synthetic samples are too contrived, the methods may appear worse than they would on real data. Without a distribution-shift analysis or external validation against realistic harmful content, the central claim that existing methods are inadequate for biomedical MLLMs is unsupported. This concern is independent of the benchmark's internal consistency; it concerns the external validity of the benchmark and the generalization of the reported finding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript (arXiv:2508.04192, abstract only) introduces MLLMU-Med, claimed to be the first benchmark for evaluating machine unlearning in biomedical multimodal large language models (MLLMs). The benchmark is constructed from a data generation pipeline that injects synthetic private data and factual errors into training sets. Two scenarios are considered: privacy protection and incorrectness removal. The authors also propose an Unlearning Efficiency Score for overall unlearning performance and report experiments with five unlearning approaches, concluding that current methods have limited effectiveness in removing harmful knowledge from biomedical MLLMs. The abstract does not include equations, protocol details, dataset statistics, or numerical results.","tokens_in":1150,"tokens_out":1393,"duration_ms":18746,"significance":"If the central claims are correct, the paper would provide a reusable benchmark and a composite evaluation metric for a practically important but underexplored problem. The finding that five existing unlearning approaches perform poorly on biomedical MLLMs would be a valuable, falsifiable result that could motivate further research. However, the significance is conditional on the synthetic data being a faithful proxy for real-world harmful biomedical content, and on the Unlearning Efficiency Score being a valid and unbiased measure. At present, the abstract provides no evidence on either point, so the significance cannot be confirmed from the abstract alone.","major_comments":[{"comment":"The central claim that existing unlearning methods show 'limited effectiveness' on biomedical MLLMs rests on the external validity of the synthetic data pipeline. The abstract states that the pipeline 'effectively integrates synthetic private data and factual errors into the training set,' but gives no evidence that these synthetic samples resemble real private patient information or real factual errors in terms of distribution, memorability, or learnability. If the synthetic samples are easier or harder to unlearn than natural occurrences, the reported conclusion may be an artifact of the benchmark. The manuscript should include distributional analysis, human evaluation, or external validation showing that synthetic harmful samples are representative proxies.","section":"Abstract"},{"comment":"The proposed 'Unlearning Efficiency Score' is introduced without a definition, formula, or specification of its components and weights. Since the score is new and is used to reach the headline conclusion, the absence of its formulation makes the result non-reproducible and potentially circular if weights are tuned to a desired outcome. The full manuscript must provide the exact score definition, justification for weights, and sensitivity analysis.","section":"Abstract"},{"comment":"The evaluation of five unlearning approaches is mentioned, but no quantitative results, dataset sizes, error bars, or statistical significance are reported in the abstract. The claim of 'limited effectiveness' is unverifiable without the underlying numbers and protocol details. The manuscript should report per-method scores, standard deviations across runs, and a clear description of the evaluation protocol.","section":"Abstract"}],"minor_comments":[{"comment":"The phrase 'the first benchmark' should be supported by a literature review in the full text showing that no prior biomedical MLLM unlearning benchmark exists; otherwise, the novelty claim is not yet substantiated.","section":"Abstract"},{"comment":"The abstract uses 'security' in the title and first sentence but the described scenarios are privacy and correctness; the relationship between these concepts could be clarified for readers.","section":"Abstract"},{"comment":"The term 'Unlearning Efficiency Score' would benefit from a brief intuitive explanation in the abstract, even if the formal definition appears in the main text.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"This review is based solely on the abstract because the full text was not available. The central claim is plausible but entirely unverified. I would recommend the editor obtain a full-text review before making an editorial decision; in the absence of a full text, 'uncertain' is the only responsible recommendation. The external-validity concern about synthetic-to-real transfer is the main substantive risk, and it should be addressed with explicit evidence in any revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2508.04192. The paper is an abstract-only snapshot of a benchmark, MLLMU-Med, for machine unlearning in biomedical multimodal LLMs. The gap is real: there is no shared evaluation resource for unlearning in this safety-critical niche, and the two scenarios (private patient data leakage and factual-error removal) are the right ones to target. If the authors ship the benchmark with code and data, this could be a useful resource for the field.\n\nWhat the paper does well from what I can see: it identifies a concrete missing artifact, proposes a data-generation pipeline to synthesize private and incorrect content, and evaluates five existing unlearning methods. The finding that those methods are limited on biomedical MLLMs is plausible and worth establishing quantitatively. The proposed Unlearning Efficiency Score is a reasonable way to summarize performance, though its formulation matters.\n\nThe soft spots are proportionate to the fact that I could only read the abstract. There are no dataset statistics, no protocol details, no error bars. The empirical claim that \"these methods show limited effectiveness\" cannot be verified from the abstract alone. The bigger conceptual risk is the one the stress-test flagged: the synthetic harmful content has to be a faithful proxy for real private information and real factual errors in biomedical training data. If the generated samples are templated or statistically distinct, the benchmark could be measuring artificial unlearning difficulty. That is not a fatal flaw—most benchmarks use synthetic data—but the authors need to provide distribution-shift analysis or external validation against realistic examples. I would also want to see the Unlearning Efficiency Score defined explicitly, with no hidden tuning of weights to make the conclusion look stronger.\n\nOverall, this is a paper that deserves a serious referee. The idea is timely, the niche is underserved, and the contribution is well-scoped. I would send it to peer review and ask for the full benchmark, code, and analysis. I wouldn't cite it yet based on the abstract, but I'd keep an eye on it.\n\nRecommendation: accept for peer review; the central claims are structural and testable once the artifact is available.","headline":"Proposes the first biomedical multimodal unlearning benchmark; the idea is right, but the abstract alone can't support the empirical claim, and synthetic-data realism is the key thing to check.","tokens_in":1585,"tokens_out":1587,"would_cite":false,"duration_ms":17961,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Biomedical multimodal language models retain private and incorrect knowledge even after machine unlearning, according to a new benchmark.","keywords":["machine unlearning","multimodal large language models","biomedical AI","privacy protection","factual error removal","benchmark","unlearning efficiency score"],"falsifier":"Take two biomedical MLLMs: one unlearned on MLLMU-Med's synthetic harmful samples and one unlearned on a corresponding set built from real, de-identified clinical records and verified medical errors. If the two models show substantially different leakage and error rates on real-world prompts—for instance, if the synthetic-trained model looks safe but the real-trained model still emits private tokens—the benchmark's synthetic pipeline is not measuring the right thing. A second check: probe the unlearned model with paraphrases of the exact synthetic harmful samples and count how many private or","tokens_in":818,"feed_emoji":"🩺","tokens_out":3776,"duration_ms":42404,"temperature":0.7,"pith_summary":"This paper argues that current machine-unlearning methods are not up to the task of removing harmful knowledge from biomedical multimodal large language models, and that the reason the field has not noticed is the absence of a dedicated benchmark. To fix that, it introduces MLLMU-Med, the first benchmark built specifically to test unlearning of private patient information and of incorrect factual knowledge in biomedical MLLMs, together with a composite Unlearning Efficiency Score. On this benchmark, five existing unlearning approaches all show limited effectiveness, leaving both privacy-leaking and wrong-knowledge responses largely intact. If the paper is right, the benchmark becomes the standard measuring stick for a class of safety problems that model retraining is too expensive to solve.","feed_headline":"Benchmark shows unlearning leaves harmful knowledge in biomedical AI","feed_subtitle":"MLLMU-Med tests five unlearning methods on privacy and misinformation; all underperform, so the safety gap is now measurable.","key_machinery":"The carrying machinery is the benchmark construction itself. The data-generation pipeline takes ordinary biomedical multimodal training samples and synthesizes (a) private patient information, such as identifiers or clinical details that should never be emitted, and (b) factual errors, such as wrong dosages or outdated guidelines, embedding both into the training set in a way that a model can learn them. The Unlearning Efficiency Score then aggregates, across subsets, how completely each unlearning method removes the target knowledge relative to how well it preserves normal task performance, yielding a single number for direct comparison.","core_discovery":"The central claim is a measurement result plus a measurement instrument. The instrument is MLLMU-Med, a benchmark assembled from a data-generation pipeline that injects synthetic private patient data and synthetic factual errors into the training distribution of a biomedical multimodal model, then asks unlearning methods to scrub them while preserving normal behavior. The result is that none of the five evaluated unlearning methods reliably does the scrubbing: under the proposed Unlearning Efficiency Score, they leave substantial harmful knowledge behind in both the privacy-protection and incorrectness-removal scenarios. The paper's claim is that this shortfall is real and measurable, and th","pith_inferences":["A natural next stress test is to check whether unlearning that scores well on MLLMU-Med also removes knowledge learned from real, non-synthetic clinical text; if it does not, the benchmark's synthetic pipeline may be overestimating unlearning success.","The same benchmark structure could transfer to other high-stakes multimodal domains, such as finance or legal documents, where private detail and factual error are similarly dangerous.","The paper's limited-effectiveness result suggests that current unlearning methods were tuned for single-modal text or simpler classification, and that designs exploiting the multimodal structure, such as aligning image and text forgetting, may be needed."],"forward_implications":["Existing unlearning approaches show limited effectiveness on biomedical MLLMs, so harmful knowledge is likely to remain after unlearning in current practice.","MLLMU-Med can serve as a standardized evaluation set for future unlearning methods in this domain.","The Unlearning Efficiency Score provides a single composite measure combining removal success and capability preservation.","The dataset generation pipeline can create training sets containing synthetic private data and factual errors without needing real patient records."],"supporting_citations":[],"fun_headline_variants":["Unlearning fails to scrub biomedical AI models","Benchmark exposes gaps in biomedical unlearning","MLLMU-Med: unlearning falls short in medical AI","Five unlearning methods miss harmful biomedical data","Biomedical AI unlearning benchmark reveals weak scrubbing"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The whole measurement stands on the assumption that synthetic private data and synthetic factual errors behave like the real private information and real mistakes that actually appear in biomedical training data, so that scores on MLLMU-Med predict what a deployed model would leak or misstate.","fun_headline_variants_meta":{"raw":{"variants":["Unlearning fails to scrub biomedical AI models","Benchmark exposes gaps in biomedical unlearning","MLLMU-Med: unlearning falls short in medical AI","Five unlearning methods miss harmful biomedical data","Biomedical AI unlearning benchmark reveals weak scrubbing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1094,"prompt_tokens":816,"completion_tokens":278,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":205}},"tokens_in":560,"tokens_out":278,"duration_ms":3643,"temperature":1.0,"reasoning_tokens":205,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:47:41.322516+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two biomedical MLLMs: one unlearned on MLLMU-Med's synthetic harmful samples and one unlearned on a corresponding set built from real, de-identified clinical records and verified medical errors. If the two models show substantially different leakage and error rates on real-world prompts—for instance, if the synthetic-trained model looks safe but the real-trained model still emits private tokens—the benchmark's synthetic pipeline is not measuring the right thing. A second check: probe the unlearned model with paraphrases of the exact synthetic harmful samples and count how many private or","supporting_citations":[],"review_version":1}