{"id":"e0f4eb99-43cb-415c-8d41-cf7da8d94732","arxiv_id":"2412.00383","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An influence-function-based parameter editing framework performs instance removal, query modification, and response correction on PEFT adapters without retraining.","lead":"LLMEraser uses influence functions to update only the small LoRA adapter weights, letting an LLM forget, correct, or modify specific training instances without retraining. The authors test it on recommendation and multimodal tasks, reporting results close to full retraining at roughly a 30x speedup.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim of effective unlearning is unsupported because no forgetting metric is reported; utility preservation alone cannot distinguish unlearning from a no-op.","rationale":"The reader's verdict is CONDITIONAL, and I agree that the paper should not be accepted as-is. However, the reader's stated weakest assumption is the Hessian invertibility and optimality condition in Eq. 8. That is a relevant mathematical concern, but the more fundamental gap is the absence of any direct forgetting measure. Even if the Hessian were invertible and the Taylor expansion exact, the experiments still would not prove that unlearning occurs—they only show that utility is preserved, which any conservative edit could achieve. The paper's title and abstract explicitly claim an 'unlearning framework,' and Section 4.2.1 says LLMEraser 'directly estimates the parameter changes between the retrained model and the original model,' but the retrained model's behavior on the deleted instances is never compared. This is not an internal inconsistency, but it is a missing critical measurement. The concrete test I propose would settle it: run a membership-inference or extraction attack on the target instances for Original, Retrain, and LLMEraser. If LLMEraser does not exhibit forgetting comparable to Retrain, the central claim collapses. If it does, the CONDITIONAL verdict could be upgraded. The reader did mention 'forgetting itself is never measured' in their rationale, so they saw the issue, but they did not elevate it to the weakest-assumption slot. My concern is therefore a partial agreement: same overall verdict, but a different emphasis on the most load-bearing weakness. I also acknowledge the reader's Hessian concern: it is plausible and should be verified, but it is secondary to the missing forgetting evaluation.","tokens_in":574,"tokens_out":1828,"duration_ms":78698,"concrete_test":"Run a membership-inference attack (e.g., a calibrated loss-based MIA) or an extraction probe (e.g., exact-match or ROUGE on the target responses) on the target instances for three models: Original, Retrain, and LLMEraser. For IR, the MIA AUC or extraction success rate on the removed set should drop substantially for Retrain compared with Original; LLMEraser should match that drop, not merely preserve utility on the retained set. For QM and RC, probe the model's predictions on the corrupted versus corrected versions of the modified instances; LLMEraser should no longer prefer the corrupted answer. If LLMEraser's forgetting metrics are statistically indistinguishable from Original, the central claim fails regardless of utility matching.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LLMEraser implements instance-wise unlearning for LLMs with near-retrain utility. However, every reported evaluation measures utility preservation (AUC, HitRatio@1, accuracy) on the retained or corrected data, and none measures whether the target instance's influence has actually been removed. For the IR task, the only comparison is the AUC gap between LLMEraser and Retrain (Table 2, 0.0038), which shows that LLMEraser approximates Retrain's utility on the remaining data—not that the forgotten instances are forgotten. A method that simply leaves the model unchanged would also achieve near-original utility on the retained data, yet it would not unlearn anything. The same issue affects QM and RC: the reported gains show that the corrupted model's utility is restored after parameter editing, but they do not show that the model no longer relies on the specific noisy or outdated tokens. Without membership inference, extraction tests, or other direct forgetting evaluations on the target instances, the central claim of 'unlearning' is unverified. This is more load-bearing than the Hessian-invertibility concern raised by the reader: even if the influence-function computation were mathematically exact, the paper still would not demonstrate that forgetting occurs. Appendix D.1's error analysis only bounds the parameter-change approximation error; it says nothing about the effect on the model's knowledge of the target data. Therefore, the evidence supports 'utility-preserving parameter editing' but not 'unlearning.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces LLMEraser, an influence-function-based framework for instance-wise unlearning in parameter-efficient fine-tuned large language models (PEFT/LLMs). The approach categorizes unlearning requests into Instance Removal, Query Modification, and Response Correction, and computes adapter parameter updates by solving a finite-sum quadratic program that approximates the inverse Hessian-vector product. The authors evaluate LLMEraser on LLM-based recommendation (TallRec, LLaRA) and multimodal LLM (LLaVA 1.5) tasks, reporting utility metrics (AUC, HitRatio@1, accuracy) and runtime against Retrain, SISA, RecEraser, Gradient Ascent, and E2URec. The reported results show near-retrain utility and 31.25x speedup versus retraining.","tokens_in":23572,"tokens_out":8808,"duration_ms":78946,"significance":"The paper's taxonomy of instance-wise unlearning tasks and its unified influence-function treatment of removal and correction are valuable conceptual contributions. If the forgetting behavior were verified, LLMEraser would be a significant practical tool for privacy-preserving editing of deployed PEFT adapters. The method is mathematically well-grounded in standard influence-function machinery, and the reformulation of the Hessian-vector-product computation as a mini-batch convex optimization problem is a plausible scalability improvement. The authors release code, which aids reproducibility. However, the experimental validation as presented only demonstrates utility preservation and does not directly show that target instances have been forgotten, which weakens the central claim as it stands.","major_comments":[{"comment":"The experiments measure only utility (AUC, HitRatio@1, accuracy) on retained or corrected test data; there is no direct evaluation of whether the influence of the target instances has been removed. For instance removal, a no-op 'Original' model would also attain near-Retrain utility on the test set, so the small gap between LLMEraser and Retrain (0.0038 AUC in Table 2) does not demonstrate forgetting. The same holds for QM and RC: the reported improvements over the Corrupted baseline show that test utility is restored, but not that the model no longer relies on the specific noisy or outdated tokens. Please add forgetting-specific metrics—e.g., loss or accuracy on the target instances, membership inference, or extraction tests—and report them for each task type. Without such evidence, the central claim of the paper is not supported.","section":"§4.2.1, Table 2; §4.2.2, Tables 3–5; §4.3, Figure 3"},{"comment":"The derivation of the parameter change assumes the old adapter \\hat{\\Theta} is the exact minimizer of the empirical risk (Eq. 3), so that \\nabla R(Z;\\hat{\\Theta})=0, and that the Hessian \\nabla^2 R(Z;\\hat{\\Theta}) is invertible. These assumptions are unlikely to hold for LoRA adapters trained with early-stopped SGD: the final checkpoint is not a stationary point, and the Hessian of an over-parameterized model is typically singular. The error analysis in Appendix D.1 only bounds the Taylor-expansion residual under these assumptions; it does not quantify the error introduced when the assumptions are violated. Please provide empirical checks (e.g., the gradient norm at the checkpoint, an estimate of the Hessian's smallest eigenvalue, or a small-scale experiment where the Hessian can be computed exactly) or provide a modified error bound that relaxes the assumptions.","section":"§3.3, Eq. (12); Appendix D.1"},{"comment":"The runtime comparison reports a single timing per method, without standard deviation or number of runs. The claimed 31.25x speedup is a point estimate, and the runtime of LLMEraser depends on the chosen optimizer hyperparameters and the convergence criterion for solving Eq. (12). Please report the mean and standard deviation of wall-clock time over several runs, state the exact settings (optimizer, learning rate, number of iterations, LoRA rank) used for the timing, and clarify whether the reported time includes data loading, gradient computation, and the optimization loop.","section":"§4.4, Table 6; Appendix C"}],"minor_comments":[{"comment":"The branches for RC and IM use the wrong sets: line 6 uses S_IM instead of S_RC, and line 7 uses S_RC instead of the query-modification set; please correct these labels.","section":"Appendix B, Algorithm 1, lines 6–8"},{"comment":"The notation S_IM (or SIM) appears in Eq. (13), but the text and Section 3.2 define the set S_QM for query modification; please align the notation.","section":"§3.2, Eq. (13)"},{"comment":"The phrase 'direct estimation of the parameter changes between the retrained model and the original model' overstates what influence functions compute; the method provides a first-order local approximation to the parameter change, not the actual difference between two separately trained models. Please rephrase.","section":"§4.2.1"},{"comment":"All metrics are reported as point estimates without error bars; reporting means and standard deviations over multiple random seeds (and data splits) would make the comparisons statistically meaningful.","section":"Tables 2–5 and Figure 3"},{"comment":"The Limitations section acknowledges the need for training set access and gradient information, but it does not mention the absence of direct forgetting verification in the experiments; please add a sentence noting that target-instance loss or membership inference evaluations are needed.","section":"§5, Limitations"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the ICLR 2025 version, and the referee report follows the standard criteria. The most pressing concern is the missing direct evaluation of forgetting, which is central to the unlearning claim; I recommend requiring the authors to add such experiments before considering acceptance. The paper may also benefit from positioning its contribution relative to the broader model-editing literature, but this is not a blocker."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of 2412.00383. The core idea is simple: apply influence functions to LoRA adapters so you can edit adapter weights to remove or correct training instances without retraining. That is a sensible and useful application, and the paper does clean engineering around it. The IR/QM/RC taxonomy is useful, and the finite-sum reformulation of the inverse-Hessian-vector product makes the approach feasible on 7B models. The experiments cover LLM4Rec and MLLM tasks, and they show the method matches retrained utility closely (0.0038 AUC gap) with a real speedup (31x). The code is released. Those are legitimate contributions.\n\nThe soft spot is not the math, which is standard influence-function machinery; it is the evaluation of the central claim. Every reported result is utility on retained or corrected data. Nothing measures whether the target instances have actually been forgotten. The paper says an instance is 'unlearned' when the model no longer reflects it, but a no-op also preserves utility on the retained set. Without membership inference, extraction, or any targeted forgetting test, the evidence supports 'utility-preserving parameter editing,' not 'unlearning.' That is not a fatal flaw in the method, but it is a mismatch between claim and evidence. The Hessian invertibility concern is real but secondary: for over-parameterized LoRA adapters trained with early stopping, the Hessian is likely singular and not at the true optimum. The appendix acknowledges this by citing second-order sufficient conditions, but that is assumed, not verified. The pseudocode in Algorithm 1 also has clear task-labeling errors (the RC branch uses QM terms, and the IM branch uses RC terms) that should have been caught before submission. No error bars anywhere, which is minor but matters when the gaps are as small as 0.0038.\n\nThe paper is honest in its limitations section about Taylor expansion error and the need for the training set, and there is no circularity in the parameter updates. I would send this to a serious referee, but the verdict should be conditional: add direct forgetting metrics, verify or regularize the Hessian, report seed variance, and fix the algorithm listing. The method itself is plausible and useful; the evaluation just does not yet carry the word 'unlearning.' If the authors add that, it is a solid subfield contribution.","headline":"A useful influence-function framework for editing LoRA adapters, but the unlearning claim needs direct forgetting tests before it's supported.","tokens_in":24207,"tokens_out":3788,"would_cite":true,"duration_ms":34866,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMEraser shows that instance removal, query modification, and response correction for LoRA-finetuned LLMs can each be reduced to a single influence-function-induced adapter edit that tracks retraining within a few percent of performance.","keywords":["machine unlearning","large language models","influence functions","parameter-efficient fine-tuning","LoRA","Hessian-vector product","instance-wise unlearning","data privacy"],"falsifier":"Train a LoRA adapter on a standard benchmark, stop it before convergence, delete 5% of training instances, and apply LLMEraser; if the edited adapter's AUC on the held-out set differs from full retraining by substantially more than the reported 0.0038 gap (for instance, 0.02 or more), the near-retrain claim fails for non-converged adapters. A direct numerical-rank check of the LoRA Hessian at the adapter weights would also reveal whether the invertibility condition behind Equation 8 actually holds.","tokens_in":23123,"feed_emoji":"⚡","tokens_out":7847,"duration_ms":64224,"temperature":0.7,"pith_summary":"This paper proposes LLMEraser, a framework for unlearning individual training instances from a large language model that was fine-tuned with LoRA adapters. It argues that three kinds of unlearning—deleting an instance, editing its query, or correcting its response—can all be expressed as small parameter adjustments to the adapter, and that influence functions can compute those adjustments directly. On benchmark recommendation and multimodal tasks, the edited adapter performs within 0.0038 AUC of a fully retrained model for instance removal, while running about 31 times faster. If correct, this gives a practical way to honor data-deletion requests without storing and retraining on data shards.","feed_headline":"Erase LLM training data with one adapter edit, 31x faster","feed_subtitle":"Influence functions compute the exact weight change for removal, query fixes, and response corrections—no retraining needed.","key_machinery":"The mechanism is the influence-function parameter-change formula, Equation 8: $\\Delta\\Theta_\\delta \\approx \\frac{1}{n}\\big(\\nabla^2_\\Theta R(Z;\\hat\\Theta)\\big)^{-1}\\big(G(x,y) - G(x+\\delta_x, y+\\delta_y)\\big)$, which converts an unlearning request into a difference of loss gradients times the inverse Hessian of the empirical risk on the training set. The paper's algorithmic contribution is to avoid forming the inverse Hessian: it solves the equivalent convex quadratic program $F(\\Delta)=\\frac12 \\Delta^\\top \\nabla^2_\\Theta R(Z;\\hat\\Theta)\\Delta - \\langle b,\\Delta\\rangle$ using SGD/Adam with Hessian-vector products, so each iteration costs $O(p)$ instead of $O(p^2)$ or $O(p^3)$.","core_discovery":"The paper's central claim is that instance-wise unlearning in LoRA-tuned LLMs reduces to one operation: computing the parameter change $\\Delta\\Theta$ a retrained model would undergo, and adding it to the existing adapter. For instance removal, the change is $\\Delta\\Theta_{\\mathrm{IR}} \\approx \\frac{1}{n}\\big(\\nabla^2_\\Theta R(Z;\\hat\\Theta)\\big)^{-1}\\sum_{(x,y)\\in S_{\\mathrm{IR}}} G(x,y)$; query modification and response correction use the same inverse-Hessian-times-gradient form with the gradient of the edited instance subtracted. The paper then converts the expensive inverse-Hessian-vector product into the solution of a convex finite-sum quadratic program, solved with mini-batch Adam and Hessian-vector products, cutting per-step cost from $O(p^2)$ to $O(p)$. On LLM4Rec and MLLM benchmarks, the resulting adapter lands within 0.0038 AUC of full retraining for instance removal, within 0.024 and 0.048 average accuracy on the two response-correction benchmarks, and within 0.0079 HitRatio@1 for query modification, while taking about $1.4\\times 10^3$ seconds on a query-modification task where retraining takes $5.4\\times 10^4$ seconds.","pith_inferences":["If the influence-function estimate stays accurate for adapters trained to convergence, the framework could extend to other PEFT methods such as prefix tuning or adapters, since the derivation only requires a differentiable loss with respect to trainable parameters.","The assumption that the old adapter is at the empirical-risk minimizer suggests the method may degrade for early-stopped or under-converged checkpoints; a testable extension would be to measure the AUC gap as a function of training epochs or LoRA rank.","Because the formulas only need gradients at the removed and replacement instances, the framework points toward a lightweight audit trail: a provider could log per-instance gradients during fine-tuning and later answer ‘what would the model look like without this record?’ without keeping the full training data.","The unified IR/QM/RC taxonomy could be applied to safety scenarios, such as removing a harmful output while correcting the same query toward a safe response, combining two edit types in one parameter change."],"forward_implications":["Instance removal, query modification, and response correction can all be handled by one framework without retraining or altering the model architecture.","Adapter edits cost $O(p)$ time and space per step, scaling linearly with the number of LoRA parameters.","On the reported benchmarks, the unlearned model stays within 0.0038 AUC (instance removal), 0.024 and 0.048 average accuracy (response correction on two MLLM benchmarks), and 0.0079 HitRatio@1 (query modification) of full retraining.","Unlearning is roughly 31 times faster than retraining on the query-modification task ($1.4\\times 10^3$ seconds versus $5.4\\times 10^4$ seconds on a single A100 GPU)."],"supporting_citations":[{"why":"Supplies the influence-function derivation (Equation 6) that the paper extends to unlearning tasks.","marker":"Koh & Liang, 2017"},{"why":"Defines LoRA adapters, the parameter-efficient target that LLMEraser edits.","marker":"Hu et al., 2022"},{"why":"Provides the O(p) Hessian-vector-product trick used to avoid forming the Hessian.","marker":"Pearlmutter, 1994"},{"why":"The authors' earlier work that inspires the finite-sum quadratic-program reformulation in Section 3.3.","marker":"Ding et al., 2025"},{"why":"The Adam convergence result used in Appendix D.2 to guarantee the mini-batch solver converges.","marker":"Zhang et al., 2022"},{"why":"The stochastic-estimation baseline whose cumulative errors motivate LLMEraser's new algorithm.","marker":"Agarwal et al., 2016"}],"fun_headline_variants":["One adapter edit erases LLM data, no retraining","Influence functions enable precise LLM unlearning","LLM eraser: single weight change removes training data","Unlearn LLM queries by tweaking one adapter, 31x faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the fine-tuned adapter sits exactly at the optimum of the training loss, with an invertible Hessian, so that a first-order Taylor expansion accurately predicts the retrained weights.","fun_headline_variants_meta":{"raw":{"variants":["One adapter edit erases LLM data, no retraining","Influence functions enable precise LLM unlearning","LLM eraser: single weight change removes training data","Unlearn LLM queries by tweaking one adapter, 31x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000339,"raw_usage":{"total_tokens":1890,"prompt_tokens":982,"completion_tokens":908,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":847}},"tokens_in":598,"tokens_out":908,"duration_ms":7685,"temperature":1.0,"reasoning_tokens":847,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T05:25:53.518306+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a LoRA adapter on a standard benchmark, stop it before convergence, delete 5% of training instances, and apply LLMEraser; if the edited adapter's AUC on the held-out set differs from full retraining by substantially more than the reported 0.0038 gap (for instance, 0.02 or more), the near-retrain claim fails for non-converged adapters. A direct numerical-rank check of the LoRA Hessian at the adapter weights would also reveal whether the invertibility condition behind Equation 8 actually holds.","supporting_citations":[{"cited_title":"Understanding black-box predictions via influence functions","cited_arxiv_id":null,"evidence_quote":"Supplies the influence-function derivation (Equation 6) that the paper extends to unlearning tasks."},{"cited_title":"Delayed Feedback Modeling with Influence Functions","cited_arxiv_id":"2502.01669","evidence_quote":"The authors' earlier work that inspires the finite-sum quadratic-program reformulation in Section 3.3."}],"review_version":1}