{"id":"b73a7a6b-8ab8-4cb3-9075-71ff223e8268","arxiv_id":"2506.15699","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"BLUR adds forget-retain overlapping combined queries and relearning data to four unlearning benchmarks, and shows existing unlearning methods degrade on them.","lead":"BLUR is a new benchmark for testing whether LLM unlearning methods still work when the data to forget and the data to keep overlap, as happens in real use. It shows that current methods, including recent ones, lose much of their retain quality on such overlapping queries, and that simple gradient ascent often matches or beats them.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 2-3x retain-quality drop on combined queries lacks a base-model combined-query baseline; the drop may reflect the query format itself rather than unlearning failure.","rationale":"The reader's weakest assumption points at length and truncation artifacts in ROUGE-L; I agree and sharpen it: the benchmark lacks a base-model control on the combined queries themselves. A benchmark measures unlearning-induced degradation only if the metric is stable for the base model under the same query modification; here that stability is not demonstrated, and the reported base-model perplexity differences suggest the format is not neutral. My proposed test would settle whether the 2-3x drop in Table 5 is an artifact. This does not move the verdict because the reader already assigned CONDITIONAL on essentially this basis; my concern is a more precise statement of the same issue, not a new reason to reject the paper. The relearning portion of the paper does not depend on the combined-query metric and is less exposed to this concern.","tokens_in":808,"tokens_out":1714,"duration_ms":52876,"concrete_test":"Run the same retain-quality metric on the base model without any unlearning: compute len(LCS(RetAns_base, ComAns_base))/len(RetAns_base) using the same decoding settings as the retain-only and combined evaluations, and compare the base model's combined-vs-retain-only drop with each unlearning method. Additionally, split ComAns_un at the question delimiter and compute ROUGE-L recall on the retain-only segment; if the base model's drop is comparable in magnitude, or if the segmented score recovers, the headline 2-3x drop is an artifact of the concatenated query format rather than a demonstration of forget-retain interference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 defines retain quality as ROUGE-L recall between RetAns_base and the unlearned model's answer, with denominator len(RetAns_base). For combined queries, the paper never reports the corresponding base-model quantity: len(LCS(RetAns_base, ComAns_base))/len(RetAns_base). The central claim that retain performance drops significantly on combined queries is therefore an uncontrolled comparison: it contrasts (unlearned model, combined query) with (unlearned model, retain-only query), but never measures how the base model itself is affected by concatenation. Table 5 already hints the format is not neutral: base-model perplexity on combined queries is higher than on retain-only queries for WMDP (2.1232 vs 1.3758). If the base model also shows low ROUGE-L recall against its own retain-only answer on combined queries, then the unlearned model's low score is not evidence of an unlearning-specific failure. The limitations section concedes that the combined queries are 'simple concatenations,' so representativeness and interpretation of the main quantitative result rest on an untested assumption. This is load-bearing because Table 5 is the empirical basis for the abstract's claim that existing methods drop significantly under forget-retain overlap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BLUR, a benchmark for evaluating LLM unlearning under forget-retain overlap. It augments four existing unlearning benchmarks (TOFU, WHP, WMDP, RWKU) with two kinds of stress tests: (i) combined evaluation queries that concatenate forget and retain questions or insert unlearned keywords into retain MCQs, and (ii) relearning datasets of varying relevance to the forget set. The authors evaluate six unlearning methods (GA, GA+KL, NPO, NPO+KL, SCRUB, and RMU where applicable). The main empirical claims are that retain quality drops substantially on combined queries, that benign relearning recovers forgotten knowledge roughly in proportion to relearn-set relevance, and that simple methods such as gradient ascent often match or outperform more recent methods. The paper concludes that existing benchmarks overstate unlearning effectiveness.","tokens_in":14184,"tokens_out":6405,"duration_ms":58279,"significance":"BLUR addresses an important gap: current unlearning benchmarks evaluate forget and retain sets in isolation, while real deployments mix them. The dataset construction is a useful contribution, and the release of code and data is a strength. The crossover queries in Appendix C are a particularly valuable extension. If the central results are confirmed, they would change how the community compares unlearning methods. However, the main quantitative evidence currently rests on a metric comparison without a necessary baseline, and the claims of 'significant' differences are not supported by any variance or statistical testing. The resource itself is likely to be used, but the paper's headline conclusions need revision.","major_comments":[{"comment":"The central comparison of retain quality on combined queries is uncontrolled. The metric is defined as len(LCS(RetAns_base, RetAns_un))/len(RetAns_base) for retain-only queries and len(LCS(RetAns_base, ComAns_un))/len(RetAns_base) for combined queries, but the paper never reports the corresponding base-model baseline len(LCS(RetAns_base, ComAns_base))/len(RetAns_base). Combined queries are roughly twice as long and contain two separate questions, so the base model itself may produce longer or truncated outputs that yield lower LCS against its own retain-only answer. The paper's own perplexity numbers for WMDP show the base model behaves differently on combined queries (2.1232 vs 1.3758 for retain-only), so the query format is not neutral. Without this baseline, the observed 2-3x drop on WHP and RWKU may be partly an artifact of the metric or of output truncation, not evidence of an unlearning-specific failure. This is load-bearing because the abstract's claim of significant drops rests on this table. The authors should report the base-model combined-query score, control generation length/truncation, or supplement with a judgment-based evaluation as in Appendix C.","section":"Section 4.1, Table 5"},{"comment":"The paper repeatedly uses the word 'significantly' (e.g., 'significantly lower' retain quality, 'significantly worse' performance) without any error bars, multiple seeds, or statistical tests. Tables 5 and 8 report single runs per method and dataset. This is a particular problem for the comparative claim that 'simple approaches perform better on average than more recent methods,' because method rankings can flip with different hyperparameter choices or random seeds. The authors should provide standard deviations across seeds and, where appropriate, paired significance tests for the primary quantitative comparisons.","section":"Sections 4.1 and 4.3, Tables 5 and 8"},{"comment":"The limitations section concedes that the combined forget-retain queries are 'simple concatenations' and that more realistic variants are left to future work. Since the main quantitative evidence for the paper's central claim comes from these concatenated queries, the representativeness of the stress test is asserted rather than demonstrated. The crossover questions in Appendix C are a stronger test, but they are only constructed for WHP and are not integrated into the main quantitative results. The authors should either extend the crossover-style queries to the other datasets or add an explicit base-model baseline to the concatenation experiment so that the simple-concatenation results can be interpreted as an upper bound on format-induced degradation.","section":"Section 5, Limitations and Section 4.1"},{"comment":"The main text provides no hyperparameter settings (learning rates, number of epochs, KL regularization weights) for GA, GA+KL, NPO, NPO+KL, SCRUB, or RMU, nor generation parameters such as maximum output length and temperature for the generations used in Tables 5, 7, and 8. Since the paper is a benchmark paper whose findings depend on these method configurations, these details must appear in the main text or a clearly referenced appendix; pointing to code alone is not sufficient for reproducibility in a scientific record.","section":"Sections 4.1-4.3"}],"minor_comments":[{"comment":"There is a typo in the second paragraph: 'undesriable data' should be 'undesirable data'.","section":"Introduction"},{"comment":"The numeric formatting in the table is inconsistent, with stray spaces in entries such as '1 .5342' and '1.4915 1 .5342 1 .5397'; these should be cleaned up for readability.","section":"Table 5"},{"comment":"The caption spells the metric as 'Rogue-L recall'; this should be 'ROUGE-L recall' for consistency with the rest of the paper.","section":"Table 8 caption"},{"comment":"The text says 'We show the results in Table 10 for WMDP' but the corresponding in-main-text table is labeled Table 7; the cross-reference appears to be off by one relative to the appendix tables.","section":"Section 4.2"},{"comment":"The 'C-Value method' for keyword extraction is mentioned without a citation; a reference should be provided.","section":"Section 3.2"},{"comment":"The column headers for retain quality ('Retain vs Retain' and 'Retain vs Combined') are ambiguous without an explicit statement that the numerator in both cases uses the base-model answer to the retain-only question; please clarify in the caption.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the missing base-model combined-query baseline could undermine the headline result if the base model itself exhibits low ROUGE-L recall on combined queries. The authors' heavy reliance on their own prior work (Thaker et al. [9] and Hu et al. [7]) is a mild concern but not a disqualifier. The paper fits the scope of cs.LG, and the benchmark could be a useful community resource after the quantitative claims are made rigorous."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: BLUR is a useful benchmark resource, but the headline claim—that current unlearning methods drop significantly when evaluated under forget-retain overlap—is not yet supported by the evidence. The critical missing piece is a base-model baseline on the combined queries themselves. I largely agree with the reader's conditional verdict, and one of the stress-test concerns lands.\n\nWhat is genuinely new is the packaged artifact: combined forget/retain queries for TOFU, WHP, WMDP, and RWKU; keyword-swapped MCQ variants; and relearning data at three levels of relevance, with public code and data. The relearning results are the strongest part. Across datasets and methods, the degree of forget-knowledge recovery scales sensibly with the relevance of the benign relearn text, and the ordering is consistent even with low-relevance text. The paper also honestly concedes in the limitations section that the combined queries are simple concatenations and that the desired behavior of a properly unlearned model remains an open question.\n\nThe main soft spot is Section 4.1. Retain quality on combined queries is measured as ROUGE-L recall between the base model's retain-only answer and the unlearned model's combined-query answer, normalized by the retain-only answer length. The paper never reports the base model's own combined-query score against its retain-only answer. That is a needed control: Table 5 already shows base perplexity rises on combined queries (WMDP 2.12 vs 1.38), suggesting the query format itself is not neutral. Without that baseline, some of the observed 2-3x drop could be a concatenation artifact rather than an unlearning-specific failure. On top of that, there are no error bars or multiple seeds, and 'significantly' is used without statistical tests. These are not fatal objections to the benchmark's usefulness, but they do mean the abstract overstates what the data show.\n\nNone of this kills the paper. The relearning datasets and combined-query sets are reusable and worth having. The fix is straightforward: add base-model combined-query baselines, run multiple seeds, and report error bars or tests. With those, the main claim would be much harder to dismiss.\n\nWho this is for: anyone building or evaluating LLM unlearning methods. It deserves a serious referee; send it out, but expect the review to focus on the evaluation section and likely ask for that missing baseline.","headline":"BLUR is a useful benchmark resource, but the headline claim of significant performance drops under forget-retain overlap is not yet supported because the combined-query evaluation lacks a base-model baseline and error bars.","tokens_in":14760,"tokens_out":3220,"would_cite":true,"duration_ms":30851,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that standard LLM unlearning benchmarks mislead because their forget and retain queries barely overlap, and shows that under BLUR's combined queries and benign relearning, existing unlearning methods lose retain quality…","keywords":["machine unlearning","LLM unlearning","forget-retain overlap","benchmark","relearning attack","Rouge-L recall","gradient ascent","robust evaluation"],"falsifier":"Run a matched-length control on the same models: build combined queries from two retain questions with the same total length as the forget+retain pairs and measure Rouge-L recall on both. If retain+retain pairs show a similar drop, the 2-3x degradation in BLUR is a length artifact; if they do not, the drop is caused by forget-retain overlap.","tokens_in":13697,"feed_emoji":"🧹","tokens_out":7606,"duration_ms":74710,"temperature":0.7,"pith_summary":"The paper sets out to show that current LLM unlearning benchmarks paint a false picture, because their forget and retain queries are nearly unrelated and therefore easy for an unlearning method to keep apart. It introduces BLUR, a benchmark that adds realistic overlap: combined queries that mix one forget question with one retain question, forget keywords inserted into retain multiple-choice questions, and relearning datasets of low, medium, and high relevance to the forget set. Evaluated across four existing unlearning testbeds, BLUR finds that retain quality drops sharply on the combined queries, with 2-3x reductions on WHP and RWKU, and that benign relearning recovers a substantial share of supposedly unlearned knowledge. The paper's central empirical claim is that this kind of robust evaluation changes method rankings, with simple gradient ascent matching or beating more recent approaches like NPO and SCRUB. A fair reader should care because if the claim holds, reported unlearning results overstate how ready these methods are for deployment.","feed_headline":"BLUR: overlapping forget-retain data sinks unlearning scores","feed_subtitle":"Combined queries cut retain quality by 2-3x and benign relearning revives forgotten knowledge; simple gradient ascent often wins.","key_machinery":"The central mechanism is the forget-retain overlap perturbation, implemented in BLUR as three evaluation instruments: combined queries formed by concatenating one forget question with one retain question; insertion of unlearn-set keywords into incorrect multiple-choice answers of retain questions; and relearning datasets D_hi, D_mid, and D_low, of high, medium, and low relevance to the forget set, used to finetune the unlearned model. These instruments are paired with Rouge-L recall, the length-normalized longest-common-subsequence similarity between base model answers and unlearned model answers, which the paper uses to quantify both forget quality and retain quality.","core_discovery":"BLUR's central discovery is that the prevailing evaluation protocol hides a failure mode: unlearning methods that score well on disjoint forget and retain sets degrade when the two kinds of knowledge appear in the same query or when the model is finetuned on benign text related to the forget set. Across TOFU, WHP, WMDP, and RWKU, concatenating a retain question to a forget question reduces Rouge-L recall on the retain portion relative to the same retain question asked alone, with the sharpest drops on WHP and RWKU. Relearning on text of increasing relevance to the forget set produces increasing Rouge-L recovery of forgotten answers, and even nonsensical Lorem Ipsum text causes some recovery. On the retain-MCQ perturbation, gradient-based methods are largely robust to swapped-in forget keywords, while a representation-perturbation method (RMU) is not. The authors conclude that robust evaluation can rank simple gradient ascent above more recent methods, and that benchmark design, not just method design, determines measured unlearning progress.","pith_inferences":["Editorial inference: because combined queries are simple concatenations and generation length is not controlled, part of the reported Rouge-L drop may be a length artifact; a retain+retain concatenation control of the same total length would separate genuine interference from truncation effects.","Editorial inference: the benchmark structure suggests a practical acceptance rule for future unlearning methods: retain quality on combined queries should be close to retain quality on isolated queries, and forget knowledge should not resurface after relearning on unrelated text; BLUR ships the data needed to test both.","Editorial inference: the keyword-insertion test could be extended beyond multiple-choice questions, since distractor keywords might also derail open-ended generations on retain topics, a failure mode the current accuracy-based metric would miss."],"forward_implications":["Benchmark numbers obtained with disjoint forget and retain sets should not be read as deployment readiness; BLUR shows that simply appending a retain question to a forget question can cut measured retain quality by 2-3x.","Unlearning methods are vulnerable to benign relearning: finetuning on innocuous text, even text with no semantic content, partially restores supposedly forgotten knowledge, and more relevant relearn text restores more.","Robustness to forget-retain overlap is method-specific: gradient-based methods withstand forget-keyword insertion into retain MCQs, while representation-perturbation methods such as RMU do not.","On a benchmark that includes overlap, simpler baselines such as gradient ascent can match or beat newer methods such as NPO and SCRUB, so recent progress claims need to be re-examined under this evaluation."],"supporting_citations":[{"why":"Supplies the TOFU benchmark and its Forget10 and Retain90 splits used for the TOFU combined queries.","marker":"[13]"},{"why":"Supplies the Who's Harry Potter unlearning task and the Harry Potter trivia that BLUR's forget set builds on.","marker":"[14]"},{"why":"Supplies the WMDP hazardous-knowledge benchmark, whose bio and chem questions BLUR converts to QA form.","marker":"[15]"},{"why":"Supplies the RWKU real-world knowledge unlearning benchmark and its forget_level2 split.","marker":"[12]"},{"why":"Establishes that disparate forget and retain sets overstate unlearning progress, motivating BLUR's combined queries and keyword insertion.","marker":"[9]"},{"why":"Demonstrates that benign relearning can recover unlearned knowledge, providing the basis for BLUR's relearning suite.","marker":"[7]"},{"why":"Defines Negative Preference Optimization, one of the methods whose rankings shift under BLUR.","marker":"[29]"},{"why":"Defines SCRUB, another method whose rankings shift under BLUR.","marker":"[30]"},{"why":"Defines gradient ascent, the simple baseline that often matches or outperforms newer methods on BLUR.","marker":"[28]"}],"fun_headline_variants":["BLUR benchmark exposes unlearning failures from overlapping data","Overlapping forget-retain data torpedoes LLM unlearning scores","BLUR reveals relearning attacks resurrect forgotten LLM knowledge","Simple unlearning beats fancy methods on realistic overlap benchmark","Benchmark design, not method choice, dictates unlearning success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central empirical result assumes that Rouge-L recall between base and unlearned model answers is a fair measure of retain quality, so the lower scores on longer combined queries reflect genuine forget-retain interference rather than artifacts of output length or truncation.","fun_headline_variants_meta":{"raw":{"variants":["BLUR benchmark exposes unlearning failures from overlapping data","Overlapping forget-retain data torpedoes LLM unlearning scores","BLUR reveals relearning attacks resurrect forgotten LLM knowledge","Simple unlearning beats fancy methods on realistic overlap benchmark","Benchmark design, not method choice, dictates unlearning success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3500,"prompt_tokens":1006,"completion_tokens":2494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":2412}},"tokens_in":622,"tokens_out":2494,"duration_ms":21063,"temperature":1.0,"reasoning_tokens":2412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:57:13.736152+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a matched-length control on the same models: build combined queries from two retain questions with the same total length as the forget+retain pairs and measure Rouge-L recall on both. If retain+retain pairs show a similar drop, the 2-3x degradation in BLUR is a length artifact; if they do not, the drop is caused by forget-retain overlap.","supporting_citations":[{"cited_title":"Towards un- bounded machine unlearning","cited_arxiv_id":null,"evidence_quote":"Defines SCRUB, another method whose rankings shift under BLUR."},{"cited_title":"Tofu: A task of fictitious unlearning for llms","cited_arxiv_id":null,"evidence_quote":"Supplies the TOFU benchmark and its Forget10 and Retain90 splits used for the TOFU combined queries."},{"cited_title":"The wmdp benchmark: Measuring and reducing malicious use with unlearning","cited_arxiv_id":null,"evidence_quote":"Supplies the WMDP hazardous-knowledge benchmark, whose bio and chem questions BLUR converts to QA form."},{"cited_title":"Rwku: Benchmarking real-world knowledge unlearning for large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the RWKU real-world knowledge unlearning benchmark and its forget_level2 split."},{"cited_title":"Position: Llm unlearning benchmarks are weak measures of progress","cited_arxiv_id":null,"evidence_quote":"Establishes that disparate forget and retain sets overstate unlearning progress, motivating BLUR's combined queries and keyword insertion."},{"cited_title":"Unlearning or obfuscating? jogging the memory of unlearned llms via benign relearning","cited_arxiv_id":null,"evidence_quote":"Demonstrates that benign relearning can recover unlearned knowledge, providing the basis for BLUR's relearning suite."},{"cited_title":"Negative preference optimization: From catastrophic collapse to effective unlearning","cited_arxiv_id":null,"evidence_quote":"Defines Negative Preference Optimization, one of the methods whose rankings shift under BLUR."},{"cited_title":"Eternal sunshine of the spotless net: Selective forgetting in deep networks","cited_arxiv_id":null,"evidence_quote":"Defines gradient ascent, the simple baseline that often matches or outperforms newer methods on BLUR."}],"review_version":1}