{"id":"4ec03fc0-1b31-4fa1-8307-c1e483bc6cd9","arxiv_id":"2509.05316","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Entity-aligned sampling (MELU) is stabler than 1:1 or cyclic retain-set sampling for LLM unlearning, but the paper's diverse-neighbor claim is contradicted by its own Balanced results.","lead":"This paper tests how to build the 'retain' data used while teaching an LLM to forget, and finds the common one-to-one sampling is unreliable while pairing each forgotten entity with its own related data is more stable. It proposes a sampling method, MELU, and extends a person-unlearning dataset with indirect neighbors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"1:1 sampling 'inefficiency' claim is confounded: 1:1 sees 98 retain samples/epoch vs cyclic's 1801, and the paper's own footnote shows 1:1 catches up at epoch 100.","rationale":"The reader's weakest_assumption focuses on sister-model data generation, but that is explicitly listed as a limitation and mainly affects the test set and Nind construction. The epoch-budget confound is more damaging because it undermines a headline best practice using the paper's own numbers. The paper's footnote 6 essentially admits the fix. This doesn't require rejecting the paper—MELU vs cyclic may still hold—but the abstract and conclusions must be tempered, and the 1:1 experiments rerun at matched retain exposure. I agree with the reader's CONDITIONAL verdict, hence UNCHANGED, though on different grounds.","tokens_in":20212,"tokens_out":3498,"duration_ms":39194,"concrete_test":"Replicate the DPO and NPO sampling-practice experiments (Table 1) with 1:1 random and 1:1 seq run for ~74 epochs (matching Cyclic's total retain exposure: 1801/98×4 ≈ 73.5), and plot FE/MU-T versus total retain samples seen. If 1:1 reaches cyclic/MELU-level FE and MU-T, the '1:1 is inefficient' claim is an epoch-budget artifact; if it remains lower at matched exposure, the claim stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 defines 1:1 seq/random as drawing |Dr|=|Df|=98 retain samples per epoch, while Cyclic uses the full 1801-sample Dr pool. All sampling experiments run for 4 epochs, so cyclic/MELU process ~7204 retain exposures versus 392 for 1:1 — an ~18x difference. Section 5.2 then reports 1:1 'fails to produce meaningful forgetting' and concludes 1:1 sampling is inefficient and yields poor results. But footnote 6 records a DPO run with 1:1 random at epoch 100 reaching FE 0.79 and MU-T 0.78, close to cyclic/MELU values (e.g., DPO Cyclic FE 0.70/MU-T 0.80; MELU 0.82/0.79). That shows the comparison is not controlled for total retain updates or compute; the inefficiency conclusion may be an artifact of a fixed epoch budget rather than a property of 1:1 sampling. Because this directly underlies best-practice claim (2) and the abstract's 'standard 1:1 sampling methods are inefficient and yield poor results,' it is the most load-bearing weakness. The sister-model assumption in §4.1 is a real limitation and is acknowledged, but it affects dataset validity rather than the sampling comparison itself.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates two aspects of LLM unlearning practice: retain-set composition (direct, indirect, and balanced neighbor sets) and sampling strategies (1:1 sequential/random, cyclic, and the proposed Modular Entity-Level Unlearning, MELU). Using an extension of the WPU dataset with 20 forget entities, the authors add an indirect-neighbor set and a test set generated with LLaMA 3.3 70B, then evaluate Gradient Difference, DPO, and NPO on LLaMA 3.1 8B. The paper claims three best practices: (1) diverse neighbor sets balance forget efficacy (FE) and model utility (MU-T); (2) 1:1 sampling is inefficient and yields poor results; (3) MELU provides more stable and effective unlearning than cyclic sampling. The contributions are a systematic comparison of these practices and an open-source implementation.","tokens_in":20616,"tokens_out":3566,"duration_ms":43431,"significance":"If the claims held, the paper would give practitioners concrete, low-cost guidance on retain-set design and sampling, an area where the field currently relies on heuristics. The manuscript is transparent: code is released, the data-construction pipeline is described in detail, per-entity results are reported, and limitations are stated explicitly. However, the load-bearing claims overreach the evidence. The 1:1 versus cyclic/MELU comparison is confounded by an 18x difference in retain-set exposures, and the Balanced setting does not actually support the 'diverse neighbor sets' claim—the paper's own results contradict it. Without repeated seeds or significance tests, the stability claims are not established. The work is a useful starting point but currently reads as an exploratory study rather than a validated set of best practices.","major_comments":[{"comment":"The claim that 'standard 1:1 sampling methods are inefficient and yield poor results' (abstract; §1) is not supported because the comparison is not controlled for total retain updates. In §4.2, 1:1 seq/random draw |Dr|=|Df|=98 retain samples per epoch, while Cyclic and MELU use the full 1801-sample retain pool. With 4 epochs, 1:1 sees ~392 retain exposures versus ~7204 for Cyclic/MELU—an ~18x difference. Footnote 6 reports that 1:1 random reaches FE 0.79 and MU-T 0.78 at epoch 100, close to Cyclic/MELU at 4 epochs. The inefficiency conclusion is therefore an artifact of the fixed epoch budget, not a property of the sampling scheme. To support claim (2), the authors need either equal retain-exposure comparisons or a principled compute/epoch budget with ablations over epochs.","section":"§4.2, §5.2, footnote 6"},{"comment":"The conclusion 'By including a diverse range of neighbors - we can improve the balance between forget efficacy and model utility' (§6) is contradicted by Table 1. For GD, Balanced FE 0.97/MU-T 0.55 versus Direct 0.96/0.34 and Indirect 0.89/0.65: Balanced improves FE over Indirect but degrades MU-T relative to Indirect; it does not dominate. For DPO, Balanced FE 0.57/MU-T 0.79 versus Direct 0.57/0.76 and Indirect 0.71/0.77: Balanced does not improve FE over Direct and has lower FE than Indirect. For NPO, Balanced 0.61/0.74 is not Pareto-better than Indirect 0.60/0.78. Section 5.1 itself states 'Balanced fails to achieve better FE and MU-T.' The claim that diverse neighbor sets balance the trade-off needs a defined aggregation or Pareto criterion, or the conclusion must be revised.","section":"§5.1, Table 1, §6"},{"comment":"All results are based on a single run per configuration; no seeds, error bars, or significance tests are reported. This is especially load-bearing for the 'stability' claims about MELU. For example, in NPO the difference between Cyclic FE 0.63/MU-T 0.78 and MELU 0.65/0.79 is within likely run-to-run variation for an 8B model with LoRA. The per-target 'improvements' (e.g., Amy Clampitt FE +20%) are also single draws. Without variance estimates, the paper cannot support the 'more stable and effective' wording in claim (3). At minimum, the authors should report repeated seeds and dispersion; ideally, a paired test across seeds for the per-target comparisons.","section":"§4.2, §5.2, Appendix A.4"},{"comment":"The indirect-neighbor set and the test set are generated with LLaMA 3.3 70B, and the unlearning target is LLaMA 3.1 8B. The paper assumes 'models of the same family would likely share similar pre-training knowledge.' This is acknowledged as a limitation, but it is not merely a generalizability caveat: it affects the validity of the Nind retain set and MU-T evaluation for every comparison. If the 3.3-generation data reflects family-shared knowledge, then the retain/test sets are not independent probes of unlearning quality. The authors should test the assumption explicitly (e.g., compare with a non-family model or a human-authored subset) or at least re-frame all conclusions as family-specific.","section":"§4.1, §7 Limitations"}],"minor_comments":[{"comment":"Typo: 'Siene' should be 'Seine.' Also, the figure caption does not explain what 'iconic tower' refers to; consider clarifying.","section":"Figure 5"},{"comment":"The table lacks clear grouping and a caption describing how FE and MU-T are computed. The row labels 'GA' and 'NPO' appear without distinguishing the unlearning setting from the data-practice rows.","section":"Table 1"},{"comment":"The sentence 'we already achieve better stability (FE and MU-T) with cyclic and MELU with the same number of epochs' is correct but does not address the confound raised in footnote 6. Please clarify that the comparison is under a fixed epoch budget and state whether that budget is meant to be the intended practical regime.","section":"§5.2"},{"comment":"The description of syntactic similarity uses an example 'When was Benedetto Varchi born?' and 'When was Donald Trump born?' but the paper later notes it could not build a full Ns set. This should be cross-referenced so the reader does not assume Ns was included in the experiments.","section":"§2.2"},{"comment":"The statement 'we couldn't answer why these setups provides more stability' is an honest acknowledgment, but it contradicts the causal explanation in §5.2 ('high-variance gradients... lower variance per batch'). Please either provide evidence for the variance explanation or label it as a hypothesis.","section":"§6"},{"comment":"PPL-F and PPL-T columns contain values like '3.09 × 10^82' for GD. These are not meaningful for readers; consider clipping or reporting in log space.","section":"Appendix A.4"}],"recommendation":"major_revision","confidential_remarks":"I am not questioning the authors' good faith; the manuscript is transparent about code and limitations. The main issue is that the two headline best-practice claims—one about 1:1 sampling and one about diverse neighbor sets—are contradicted or confounded by the paper's own data. The 1:1 conclusion can likely be repaired with controlled experiments; the Balanced-claim inconsistency requires either a sharper evaluation criterion or a softened conclusion. Given that the paper is an ECML-PKDD workshop submission, the current evidence is strengthened by the reproducible code and per-entity analysis, but it needs a substantive revision before it can be promoted as validated best practice. I would encourage the authors to add multiple seeds and a proper matching of retain exposures."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The paper gives the field a new and genuinely useful empirical look at retain-set construction and sampling for LLM unlearning. It extends WPU with indirect neighbors, builds a dedicated test set, compares direct/indirect/balanced retain sets, and proposes MELU, a simple entity-aligned sampling rule. The MELU stability improvement over cyclic under DPO is plausible and worth copying. Code is released, the limitations section is candid, and the comparison of 1:1, cyclic, and MELU under three algorithms is new. That is real work.\n\nBut two of the three 'validated best practices' in the abstract do not survive contact with the paper's own tables.\n\nFirst, the 1:1 inefficiency claim. Section 4.2 gives 1:1 seq and random only 98 retain samples per epoch, while cyclic and MELU use the full 1801-sample pool. At 4 epochs, that is roughly 392 retain exposures for 1:1 versus 7204 for cyclic/MELU—an 18x difference. So of course 1:1 forgets less. Footnote 6 then reports a DPO 1:1 random run at epoch 100 reaching FE 0.79 and MU-T 0.78, close to cyclic's 0.70/0.80 and MELU's 0.82/0.79. The 'inefficient and poor results' conclusion is mostly an artifact of the fixed epoch budget, not a property of the sampling rule. The fix is to compare at matched total retain updates or report learning curves, and to soften the abstract.\n\nSecond, the diverse-neighbor claim. Section 5.1 says Balanced 'fails to achieve better FE and MU-T'—the opposite of best practice (1). The conclusion rewords it as 'relying solely on any single neighbor set is suboptimal,' but the data do not clearly support that either: direct was better for GD, indirect better for MU-T, balanced worst overall. The diversity story needs to be told more carefully.\n\nAlso: no seeds, no error bars, no significance tests—and that matters because unlearning runs are known to be high-variance. And the MMLU numbers are odd: the base model at 12.42% is barely above random for MMLU-Pro, so the 'stable MMLU' evidence is hard to interpret. The sister-model data-generation assumption is a real limitation, acknowledged in the paper, but it hurts dataset validity more than the sampling comparison itself.\n\nBottom line: this is a useful workshop-level contribution with reproducible infrastructure, but the headline claims overstate what the evidence supports. It deserves peer review, and the referee ask is straightforward: multi-seed runs, matched-update comparisons, an MMLU sanity check, and rewording of claims (1) and (2).","headline":"A useful, honestly reported empirical sweep of retain-set sampling for LLM unlearning, but two of the three headline 'best practices' are not supported by the paper's own data: the 1:1 inefficiency claim is confounded by an epoch-budget mismatch, and the diverse-neighbor claim contradicts the Balanced result.","tokens_in":21066,"tokens_out":3414,"would_cite":true,"duration_ms":38669,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard LLM unlearning practice—single-type neighbor sets and 1:1 forget-retain pairing—is suboptimal; entity-aligned MELU sampling is more stable and effective, the paper argues.","keywords":["LLM unlearning","machine unlearning","retain set composition","neighbor sets","1:1 sampling","cyclic sampling","MELU","entity-level unlearning"],"falsifier":"Regenerate the indirect-neighbor retain set and the test set with a model outside the LLaMA family (or with the target model itself) and rerun the seven settings: if MELU's 12-point DPO forget-efficacy advantage over cyclic disappears, or the single-neighbor rankings change, the results hinge on the same-family data-generation assumption. A cheaper check is to run cyclic and MELU under DPO across several random seeds and verify that the per-target FE≥0.9 count (3 vs 1) and the roughly 0.82 vs 0.70 forget-efficacy gap replicate; the paper's own note that 1:1 random reaches 0.79 forget efficacy","tokens_in":20146,"feed_emoji":"🧠","tokens_out":9302,"duration_ms":91358,"temperature":0.7,"pith_summary":"This paper asks whether two de facto standards of LLM unlearning are actually good practice: building the retain set from a single kind of neighbor data, and pairing forget and retain samples in a strict 1:1 ratio. On the Wikipedia Person Unlearning benchmark, extended with indirect and syntactically similar neighbors plus a dedicated test set, the authors evaluate three unlearning algorithms (Gradient Difference, DPO, NPO) across seven data and sampling configurations. They find that no single neighbor set is optimal—direct neighbors favor forgetting for gradient-based methods, indirect neighbors protect model utility, and neither alone balances both—and that 1:1 sampling (sequential or random) barely forgets anything, with forget efficacy around 0.35 across all three algorithms. Their positive proposal is Modular Entity-Level Unlearning (MELU), which cycles each forgotten entity only over retain samples of the same entity; it outperforms cyclic sampling under DPO, raising forget efficacy by 12 points while holding model utility near baseline. If the findings hold, unlearning practitioners should abandon 1:1 sampling and construct retain sets with multiple neighbor types.","feed_headline":"Pair forget data with its own neighbors for reliable LLM unlearning","feed_subtitle":"Study shows 1:1 sampling barely forgets; entity-aligned retain pairs match or beat cyclic across three algorithms.","key_machinery":"Modular Entity-Level Unlearning (MELU), a sampling strategy that pairs each forget target only with its own retain samples—direct and indirect neighbors of that entity, plus randomly assigned general-knowledge samples—instead of cycling forget samples against the whole retain pool. It carries the paper's main positive claim. The paper attributes MELU's stability to a more consistent learning signal: low-variance gradients from related forget-retain pairs, versus the high-variance gradients of cyclic sampling. The neighbor-set taxonomy (direct, indirect, syntactic) is the second load-bearing structure, organizing the data-practice comparison.","core_discovery":"On the paper's own terms, the discovery is that retain-set composition and forget-retain pairing are first-order levers in LLM unlearning, not implementation details. Concretely: (1) a single neighbor set is suboptimal—direct neighbors favor forget efficacy for gradient-based methods, indirect neighbors favor model utility, and the balanced mix behaves differently per algorithm; (2) the standard 1:1 sampling (sequential and random) leaves forget efficacy near 0.35 across all three algorithms, far below the 0.63–0.97 achieved by cyclic and MELU; (3) MELU, which pairs each forget entity only with its own direct and indirect retain samples plus randomly assigned general-knowledge samples, is mo","pith_inferences":["If MELU's advantage really comes from lower-variance gradients, the same entity-aligned principle should transfer to sequential unlearning and to other model families; a direct test is measuring per-batch gradient variance under cyclic versus MELU.","The 1:1 result may be a repetition or epoch effect rather than an inherent ceiling: the paper notes that 1:1 random reaches 0.79 forget efficacy at 100 epochs, so the practical claim is that 1:1 is inefficient at standard epoch budgets, not that it cannot work.","Because the indirect neighbors were generated under the same-family knowledge assumption, the indirect-versus-direct rankings are the least portable result; regenerating the neighbor and test data with a cross-family generator is the natural external-validity check.","Selecting only five indirect entities per target leaves the balanced setting fairly thin; knowledge-graph-defined connections, as the authors themselves suggest, could change the balanced-set results."],"forward_implications":["Unlearning benchmarks should stop defaulting to a single neighbor set; the paper shows direct-only and indirect-only retain sets push the forget-efficacy/model-utility trade-off in opposite directions.","Practitioners should treat 1:1 forget-retain sampling as a known-weak baseline: it under-uses the retain set and leaves forget efficacy near pre-unlearning levels for GD, DPO, and NPO alike.","MELU is a drop-in replacement for cyclic sampling that requires no new algorithm or compute, and it improves per-target stability and forget efficacy, especially with preference-based methods.","Per-entity evaluation matters: targets with high memorization scores resist all settings, so aggregate forget-efficacy/model-utility numbers alone can hide where unlearning fails.","With proper sampling (cyclic or MELU), general model utility stays stable on MMLU, whereas changes in data composition produce inconsistent MMLU shifts."],"supporting_citations":[{"why":"Supplies the WPU dataset the study extends: 100 little-known Wikipedia figures, the direct neighbor set, and the general knowledge set.","marker":"[21]"},{"why":"TOFU benchmark that introduced indirect neighbor sets and the 1:1 random sampling convention the paper critiques.","marker":"[22]"},{"why":"SemEval-2025 winning solution whose 1:n cyclic sequential approach defines the cyclic sampling comparison baseline.","marker":"[25]"},{"why":"Introduced chunked sequential unlearning and cyclic iteration, the instability motivation for studying sampling.","marker":"[12]"},{"why":"Introduced the syntactic-similarity neighbor set, which the authors adapt for retain-set question generation.","marker":"[4]"},{"why":"The LLaMA 3 herd: the 8B Instruct model unlearned in all experiments and the 70B model that generates indirect neighbors and test questions.","marker":"[10]"},{"why":"Gradient Difference, the untargeted unlearning algorithm used in the gradient-based runs.","marker":"[19]"},{"why":"DPO, the preference-optimization loss used for targeted unlearning with refusal responses.","marker":"[27]"},{"why":"NPO, the bounded negative-preference objective used for untargeted unlearning runs.","marker":"[38]"}],"fun_headline_variants":["Diverse neighbors, not one set, improve LLM unlearning","1:1 sampling fails; modular entity-level pairing wins","Single neighbor set is suboptimal for unlearning","Modular sampling beats standard for LLM unlearning"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The indirect neighbor set and test set rest on the assumption that LLaMA 3.3 70B and LLaMA 3.1 8B, being the same model family, share similar pre-training knowledge; if that fails, the retain data and evaluation are biased toward the target model and the forget-efficacy and model-utility comparisons do not measure what they claim.","fun_headline_variants_meta":{"raw":{"variants":["Diverse neighbors, not one set, improve LLM unlearning","1:1 sampling fails; modular entity-level pairing wins","Single neighbor set is suboptimal for unlearning","Modular sampling beats standard for LLM unlearning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000405,"raw_usage":{"total_tokens":1965,"prompt_tokens":789,"completion_tokens":1176,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":1110}},"tokens_in":533,"tokens_out":1176,"duration_ms":9559,"temperature":1.0,"reasoning_tokens":1110,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:49:56.315411+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Regenerate the indirect-neighbor retain set and the test set with a model outside the LLaMA family (or with the target model itself) and rerun the seven settings: if MELU's 12-point DPO forget-efficacy advantage over cyclic disappears, or the single-neighbor rankings change, the results hinge on the same-family data-generation assumption. A cheaper check is to run cyclic and MELU under DPO across several random seeds and verify that the per-target FE≥0.9 count (3 vs 1) and the roughly 0.82 vs 0.70 forget-efficacy gap replicate; the paper's own note that 1:1 random reaches 0.79 forget efficacy","supporting_citations":[{"cited_title":"In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=B41hNBoWLo","cited_arxiv_id":null,"evidence_quote":"TOFU benchmark that introduced indirect neighbor sets and the 1:1 random sampling convention the paper critiques."},{"cited_title":"In: Rosen- thal, S., Rosá, A., Ghosh, D., Zampieri, M","cited_arxiv_id":null,"evidence_quote":"SemEval-2025 winning solution whose 1:n cyclic sequential approach defines the cyclic sampling comparison baseline."},{"cited_title":"In: Che, W., Nabende, J., Shutova, E., Pilehvar, M.T","cited_arxiv_id":null,"evidence_quote":"Introduced the syntactic-similarity neighbor set, which the authors adapt for retain-set question generation."},{"cited_title":"In: Pro- ceedings of the 37th International Conference on Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"DPO, the preference-optimization loss used for targeted unlearning with refusal responses."},{"cited_title":"In: First Conference on Language Modeling (2024), https://openreview.net/forum?id=MXLBXjQkmb Standard vs","cited_arxiv_id":null,"evidence_quote":"NPO, the bounded negative-preference objective used for untargeted unlearning runs."}],"review_version":1}