{"id":"4228c109-ccfe-4df4-be3e-34817218bad9","arxiv_id":"2507.22499","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LoReUn, a plug-in loss-based reweighting strategy, improves approximate machine unlearning by focusing updates on hard-to-forget low-loss data points.","lead":"Machine unlearning removes unwanted data's influence from a trained model, and this paper shows that the model's own loss values reveal which data points are hardest to forget. The authors add a loss-based reweighting step, LoReUn, that makes existing unlearning methods forget faster and closer to full retraining, including safer Stable Diffusion outputs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Fig. 2 loss–difficulty correlation is confounded: 'success' means 'prediction becomes wrong,' so already-misclassified high-loss points count as forgotten without any change.","rationale":"The reader's weakest assumption was that the Sec. 4 correlation between loss and unlearning difficulty is insufficiently supported and may be confounded. I agree with that concern, but the specific confound is sharper: the definition of 'successfully forgotten' as 'prediction becomes wrong' makes the correlation partly mechanical. High-loss points are disproportionately points that are already wrong before unlearning; low-loss points are disproportionately points that are correct and must be flipped. Thus the direction of the reweighting — upweighting small-loss points — can be explained without invoking intrinsic difficulty. This is load-bearing because it is the paper's central empirical claim and the basis for transferring the method to generative models. The check I propose is direct and inexpensive: condition on initial prediction correctness and see whether the loss gap survives. The paper does have genuine strengths: the method is simple, the experiments are broad, the run-time overhead is small, and the reweighting is evaluated against external metrics rather than the weighting objective itself. The NSFW reduction results are suggestive even if sample counts are small. However, those strengths do not resolve the confound, because the classification experiments are the main evidence for the loss–difficulty premise. I therefore keep the reader's CONDITIONAL verdict but would attach the additional condition that the paper must control for initial prediction correctness (or change the success definition to require a flip from correct to wrong). If the control shows the correlation disappears, the central claim as stated should be rejected or substantially reframed.","tokens_in":23129,"tokens_out":8245,"duration_ms":113086,"concrete_test":"Re-run the Fig. 2 analysis restricted to forgetting points that the original model classifies correctly (or, for class-wise forgetting, correctly as the target class); within that subset, compare original-model loss of points that flip to wrong after unlearning versus points that remain correct. If the loss gap is no longer significant, the Sec. 4 relation is an artifact and LoReUn's motivation collapses. As a secondary check, compare LoReUn against a binary control that reweights initially-correct points with weight 1 and initially-wrong points with weight 0; if the control matches LoReUn's Tab. 2 metrics, the exponential loss adds nothing beyond initial correctness.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing premise of Sec. 4 is that original-model loss ℓo correlates with unlearning difficulty. But the operational definition of success in Fig. 2 and Fig. A1 is 'predictions become wrong after unlearning.' Under this definition, a point the original model already misclassifies is counted as successfully forgotten with no change, and such points necessarily have high ℓo. On CIFAR-10, the original model errs on roughly 5% of the forgetting set, and those errors populate the high-loss tail. Low-loss points are mostly points the model classifies correctly and that must be flipped to be counted. The loss gap between 'success' and 'fail' is therefore partly a restatement of 'already wrong vs. correct before unlearning,' not evidence about intrinsic unlearning difficulty. Worse, for 10% random-data forgetting the gold standard Retrain has UA = 94.51 (Tab. 2), so exact unlearning should keep most random-forgetting predictions correct; 'becoming wrong' is the wrong proxy for that setting. Because w = exp(−ℓ_eval/τ) upweights exactly the low-loss, initially-correct points, the observed gains may be a correctness-aware weighting rather than a difficulty-aware one. The transfer of the claim to diffusion models, where forgetting is concept erasure rather than per-point label flip, is thus not supported by the classification evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LoReUn, a plug-and-play reweighting strategy for gradient-based machine unlearning. The central empirical premise, developed in Sec. 4, is that a data point's loss on the original model implicitly reflects how hard it is to unlearn: points with larger loss are easier to forget, and points with smaller loss are harder. Based on this, LoReUn assigns weight w = exp(-l_eval/tau), normalized per batch, to each forgetting-sample loss term, with static (original-model) and dynamic (unlearned-model) variants. The method is evaluated on CIFAR-10, SVHN, CIFAR-100, and Tiny ImageNet classification tasks, on class-wise forgetting for DDPM and Stable Diffusion, and on NSFW concept erasure from Stable Diffusion using I2P prompts and the NudeNet detector. The paper reports that LoReUn reduces the gap to exact retraining on accuracy-related metrics and improves the tug-of-war trade-off, while adding minimal runtime overhead.","tokens_in":23393,"tokens_out":5235,"duration_ms":62945,"significance":"If the loss-difficulty premise were established, LoReUn would be a valuable low-cost improvement: it is simple, model-agnostic, applicable to several gradient-based unlearning objectives, and the generation experiments address a practically important safety application (reducing NSFW outputs). The paper also evaluates against many baselines and includes additional datasets and ablations, which is a strength. However, the load-bearing evidence for the premise is currently confounded, and the reported improvements over strong baselines are often modest (e.g., random-forgetting Avg.G for GAR-m decreases from 1.32 to 0.99, and class-wise Avg.G from 0.31 to 0.28). Because the weighting rule is derived directly from the confounded correlation, the significance of the method is conditional on a corrected analysis. The paper does not provide machine-checked proofs or released code, but the experiments are described in sufficient detail to be largely reproducible.","major_comments":[{"comment":"The operational definition of 'success to forget' as 'predictions become wrong after unlearning' confounds the reported loss-difficulty correlation. A point that the original model already misclassifies has high loss on the original model and is counted as successfully forgotten even if the unlearning update leaves its prediction unchanged; on CIFAR-10 these initially wrong points populate the high-loss tail, while low-loss points are mostly initially correct and must be flipped to be counted as successes. The observed gap between success and fail is therefore partly a restatement of 'already wrong vs. correct before unlearning' rather than evidence about intrinsic unlearning difficulty. In the random-forgetting setting this is especially problematic because Retrain has UA = 94.51 in Table 2, so an exact unlearned model is expected to keep most predictions on the forgetting set correct, and 'becoming wrong' cannot be the success criterion there.","section":"Sec. 4, Fig. 2"},{"comment":"The easy/hard forgetting-set comparison inherits the same confound. The sets are selected by loss on the original model, so the hard set consists of points the original model classifies correctly; for methods whose forgetting loss drives predictions away from the original label (RL, SalUn, GAR), such points are harder to flip by construction. This does not establish that loss is a proxy for intrinsic memorization or entanglement, as claimed. The authors should re-analyze with a Retrain-relative definition of forgetting, for example by measuring whether the unlearned model's prediction changes relative to the exact retrained model, or by measuring loss increase on the forgetting set relative to Retrain, and should report confidence intervals or significance tests for the Fig. 2 and Fig. 3 comparisons.","section":"Sec. 4, Fig. 3"},{"comment":"The weight function w = exp(-l_eval/tau) upweights low-loss, initially correct points. Given the confound described above, the improved Avg.G and ToW in Table 2 may reflect a correctness-aware weighting that focuses gradient on points whose labels must be flipped, rather than a difficulty-aware weighting tied to the paper's premise. The authors should test the monotone loss-difficulty assumption directly, for example by correlating per-point original loss with per-point forgetting success measured against Retrain, and by ablating with alternative weight functions based on the original model's margin or prediction correctness to see whether the observed gains persist.","section":"Sec. 5, Eq. (6)"},{"comment":"The transfer of the classification evidence to diffusion models is not currently supported. Fig. A1 reports class-level average loss versus unlearning accuracy for ten ImageNette classes without error bars or a significance test, and the NSFW concept-erasure setting does not have per-point labels whose predictions can be flipped in the classification sense. The paper should provide per-prompt loss-difficulty evidence in the generation setting, or explicitly soften the claim that the same mechanism drives the generation results and frame the generation gains as an empirical property of the weighting heuristic.","section":"Sec. 6.2, Fig. A1 and Table 3"}],"minor_comments":[{"comment":"The sentence 'Our dynamic strategy (LoReUn-s) outperforms the static one (LoReUn-d)' reverses the variant names; LoReUn-s is the static variant and LoReUn-d is the dynamic variant.","section":"Sec. 6.2, text after Fig. 4"},{"comment":"The grouping of the '+RL', '+SalUn', '+GAR', and '+GAR-m' rows under LoReUn-s and LoReUn-d is ambiguous; the row labels should explicitly repeat the variant name for each block so the reader can tell which results correspond to which variant.","section":"Table 2"},{"comment":"The exponential fit that produces the estimated loss curve in Fig. A2b is reported only qualitatively; please report the fitted functional form, coefficients, number of seeds, fit error, and whether the same fitted curve is used for both LoReUn-s weights and the reference loss in Eq. (10).","section":"Appendix B"},{"comment":"The loss distributions for success and fail groups do not include error bars or significance tests; please add standard errors across the random seeds and a paired test of the success-vs-fail loss difference.","section":"Sec. 4, Fig. 2"},{"comment":"The notation in Algorithm 1 line 1, computing reference losses l(theta_o; D_f), is inconsistent with the per-point definition of l_eval(theta; x, y); please use per-point notation and specify how the reference losses are aggregated.","section":"Algorithm 1 and Sec. 5"},{"comment":"The sentence 'AsLoReUn requires careful tuning' appears to contain a missing space; additionally, the temperature sensitivity shown in Appendix C.3 (ToW declines sharply at small tau for RL-based models) is important enough to be stated in the main text, not only in the limitations paragraph.","section":"Conclusion and Limitations"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is currently confounded by the success criterion in Fig. 2, and the weighting formula is derived directly from that claim. I would ask the authors to provide a corrected analysis using Retrain-relative success or a loss-change measure, and to report significance tests. If the corrected correlation disappears, the paper's framing would need to change substantially; if it survives, the method is a useful empirical contribution. The scope of experiments is broad and the application to NSFW removal is timely, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe main thing you should know: the paper's load-bearing correlation in Fig. 2 is not as clean as the authors claim. 'Success' is defined as a prediction being wrong after unlearning, so points the original model already misclassifies (necessarily high-loss) get counted as successfully forgotten without changing anything. On CIFAR-10 that is roughly 5% of the forgetting set, and those points populate the high-loss tail. The gap between 'success' and 'fail' in loss is therefore partly a restatement of 'already wrong vs. correct before unlearning,' not evidence of intrinsic unlearning difficulty. I think the stress-test note is right about this, and the paper needs to redo that analysis—for example, restrict to points initially classified correctly, or use a per-point change metric.\n\nThat said, the method itself is a reasonable heuristic. Upweighting low-loss points (which are mostly correctly classified) in the forgetting loss directly targets the points that need to be flipped, and that may be why it helps. The static and dynamic variants are simple, add negligible compute, and the experiments span classification and diffusion, including NSFW removal with NudeNet and an adversarial attack evaluation. The reported gains are modest in random forgetting (Avg.G 0.99 vs 1.32 for GAR-m) but more convincing in class-wise forgetting (SalUn 0.22 to 0.05). The related work on per-sample difficulty via memorization and entanglement is properly cited, and the diffusion loss approximation via exponential fitting on 50 samples and 10 timesteps is clever and honestly documented, though the fit error is not quantified.\n\nThe soft spots: no code released; significance tests are missing for the Fig. 2 correlation and the headline improvements; the diffusion baselines omit UCE and Ablating Concepts, which are standard for concept erasure; and the temperature sensitivity shown in Fig. A3 suggests results may be cherry-picked without a clear selection rule. Also, the transfer of the 'loss reflects difficulty' story to concept erasure is not supported by the classification evidence alone.\n\nBottom line: this is a useful practical plug-in for gradient-based unlearning, especially for text-to-image safety, and anyone working in that area will find the two variants worth testing. But the paper overinterprets its own motivating correlation. A serious referee could improve it substantially. I'd send it to review with a request for major revision, not a desk reject.","headline":"Fig. 2's loss–difficulty correlation is partly an artifact of counting 'already wrong' as success; the reweighting heuristic still shows modest gains and deserves a revision, not a desk reject.","tokens_in":23922,"tokens_out":3849,"would_cite":true,"duration_ms":45040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a data point's loss on the original model encodes how hard it is to unlearn, and that reweighting the forgetting loss by an exponential decay in that loss closes much of the gap between approximate and exact…","keywords":["machine unlearning","data reweighting","loss-based weighting","exact unlearning gap","diffusion model safety","NSFW concept erasure","image classification unlearning","random labeling unlearning"],"falsifier":"Measure, on a held-out model and dataset, the correlation between original-model loss and whether a data point is forgotten after a fixed unlearning run; if hard-to-forget points do not concentrate at the low-loss end, or the correlation flips, then LoReUn's weight assignment should be inverted. A direct test is to run LoReUn with high-loss samples weighted more and show that the reported gains persist, which would disprove the claimed monotone relationship.","tokens_in":22899,"feed_emoji":"🧹","tokens_out":5991,"duration_ms":66532,"temperature":0.7,"pith_summary":"The paper claims that the loss a model assigns to a data point carries information about how hard that point is to unlearn: points with large loss on the original model are usually forgotten successfully, while small-loss points tend to survive unlearning. Building on that empirical pattern, the authors propose LoReUn, a plug-in reweighting scheme that gives more weight to small-loss, hard-to-forget samples during gradient-based unlearning, using $w = \\exp(-\\ell_{\\mathrm{eval}}/\\tau)$ normalized within each batch. They report that LoReUn reduces the gap between approximate unlearning and exact retraining on image classification and generation, and that it cuts the number of harmful NSFW images produced by Stable Diffusion under I2P prompts. A careful reader would care because it suggests a cheap, universal way to improve approximate unlearning without extra inference cost.","feed_headline":"Small-loss data are hardest to forget; LoReUn targets them","feed_subtitle":"A plug-in weight function pushes gradient onto hard samples and cuts harmful diffusion outputs.","key_machinery":"The machinery is an exponential decay weight function $w(\\theta;x,y) = \\exp(-\\ell_{\\mathrm{eval}}(\\theta;x,y)/\\tau)$, normalized to sum to one within each forgetting batch, multiplied into the forgetting loss term. LoReUn-s evaluates $\\ell_{\\mathrm{eval}}$ on the original model; LoReUn-d evaluates it on the current unlearned model. For diffusion models, the evaluation loss is averaged over timesteps, and to keep the dynamic estimate cheap and low-variance, timesteps are importance-sampled according to the original model's loss scale. The weight function's job is to concentrate optimization on samples whose small loss marks them as well-memorized and therefore hard to forget.","core_discovery":"The central discovery is an empirical regularity and an algorithm built on it. Across several unlearning methods, data points that are successfully forgotten have, on average, higher loss on the original model than data points that fail to be forgotten; the same pattern appears in class-wise forgetting for diffusion models, where low-loss classes are harder to erase. The paper's claim is that this loss signal can be used directly: reweighting the forgetting loss so that low-loss, hard samples receive more gradient makes approximate unlearning behave more like exact retraining, improving both forgetting and retaining metrics. The authors argue the signal is implicit in the loss already being computed, so the reweighting adds minimal overhead.","pith_inferences":["The monotone loss-difficulty link suggests a broader design principle: any cheap proxy for memorization, such as prediction confidence or influence estimates, could replace loss in the weight function; a testable extension is to compare $\\exp(-\\ell/\\tau)$ weights against confidence-based or influence-based weights on the same benchmarks.","Because LoReUn-d tracks the evolving loss on the unlearned model, it resembles a curriculum that progressively focuses on lingering hard points; one could couple the temperature $\\tau$ to a schedule or to per-sample uncertainty to reduce sensitivity to tuning, which the paper lists as a limitation.","The paper does not evaluate language or graph domains; transferring the weight function to LLM unlearning would require defining sequence-level loss and may fail if low loss there reflects short length rather than memorization, so that transfer is an open question rather than a consequence.","An inverse design, upweighting high-loss data to poison unlearning, is a possible attack surface the paper does not discuss: if loss is observable to an adversary, it tells them which points the unlearner will struggle with."],"forward_implications":["Plugging LoReUn into gradient-based unlearning methods such as RL, SalUn, GAR, and GAR-m moves their unlearning-accuracy, retaining-accuracy, testing-accuracy, and membership-inference gaps with Retrain toward zero, with only a few minutes of added runtime.","In class-wise forgetting, LoReUn reaches near-zero unlearning accuracy much earlier in training, so the same number of epochs yields faster and more stable forgetting than the base method alone.","In text-to-image models, LoReUn erases the nudity concept with fewer harmful images than ESD, FMN, and SalUn, while keeping FID close to or better than the baselines and preserving the prompt's subject.","LoReUn also improves gradient ascent unlearning without access to a retaining set, which means the reweighting idea is not tied to having retaining data available."],"supporting_citations":[{"why":"Supplies the observation that unlearning can fail on worst-case forgetting sets, which motivates treating points individually.","marker":"[8]"},{"why":"Argues that data should be treated individually according to how well the original model memorizes it, the premise LoReUn operationalizes.","marker":"[1]"},{"why":"Analyzes how entanglement and memorization affect unlearning difficulty, the difficulty notion LoReUn reads off from loss.","marker":"[59]"},{"why":"Provides the random-labeling objective that LoReUn rewrites through per-sample weights in classification.","marker":"[14]"},{"why":"Provides the SalUn baseline and the diffusion unlearning objective that LoReUn is plugged into.","marker":"[7]"},{"why":"Provides the gradient-ascent unlearning formulation that LoReUn improves without a retaining set.","marker":"[49]"},{"why":"Supplies the I2P prompt set used to test whether harmful NSFW images are removed.","marker":"[43]"},{"why":"Supplies the NudeNet detector used to count remaining harmful images in the safety evaluation.","marker":"[2]"}],"fun_headline_variants":["Low-loss samples are the stubborn ones; LoReUn targets them","Reweight by loss to make unlearning actually forget","LoReUn: reweight unlearning loss toward the hard data","Loss signals which data resist forgetting; LoReUn hits those","Use loss to find what won't unlearn, then reweight it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on an empirical correlation: data with smaller loss on the original model are harder to forget, and data with larger loss are easier to forget; if that ordering breaks down, the reweighting moves gradient to the wrong points.","fun_headline_variants_meta":{"raw":{"variants":["Low-loss samples are the stubborn ones; LoReUn targets them","Reweight by loss to make unlearning actually forget","LoReUn: reweight unlearning loss toward the hard data","Loss signals which data resist forgetting; LoReUn hits those","Use loss to find what won't unlearn, then reweight it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000345,"raw_usage":{"total_tokens":1829,"prompt_tokens":820,"completion_tokens":1009,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":436,"completion_tokens_details":{"reasoning_tokens":921}},"tokens_in":436,"tokens_out":1009,"duration_ms":8789,"temperature":1.0,"reasoning_tokens":921,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:36:37.122427+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, on a held-out model and dataset, the correlation between original-model loss and whether a data point is forgotten after a fixed unlearning run; if hard-to-forget points do not concentrate at the low-loss end, or the correlation flips, then LoReUn's weight assignment should be inverted. A direct test is to run LoReUn with high-loss samples weighted more and show that the reported gains persist, which would disprove the claimed monotone relationship.","supporting_citations":[{"cited_title":"Golatkar, A","cited_arxiv_id":null,"evidence_quote":"Provides the random-labeling objective that LoReUn rewrites through per-sample weights in classification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SalUn baseline and the diffusion unlearning objective that LoReUn is plugged into."},{"cited_title":"Thudi, G","cited_arxiv_id":null,"evidence_quote":"Provides the gradient-ascent unlearning formulation that LoReUn improves without a retaining set."},{"cited_title":"Schramowski, M","cited_arxiv_id":null,"evidence_quote":"Supplies the I2P prompt set used to test whether harmful NSFW images are removed."},{"cited_title":"Bedapudi","cited_arxiv_id":null,"evidence_quote":"Supplies the NudeNet detector used to count remaining harmful images in the safety evaluation."}],"review_version":1}