IMU: Influence-guided Machine Unlearning

Dinh Phung; Jing Wu; Mehrtash Harandi; Mingyi Zhou; Pengwei Liang; Xindi Fan

arxiv: 2508.01620 · v3 · submitted 2025-08-03 · 💻 cs.LG · cs.CR· cs.CV

IMU: Influence-guided Machine Unlearning

Xindi Fan , Jing Wu , Mingyi Zhou , Pengwei Liang , Mehrtash Harandi , Dinh Phung This is my paper

Pith reviewed 2026-05-19 01:14 UTC · model grok-4.3

classification 💻 cs.LG cs.CRcs.CV

keywords machine unlearninginfluence functionsgradient ascentforget setmodel utilityprivacyreweightingapproximation

0 comments

The pith

IMU reweights unlearning updates using influence scores on forget samples alone to match uniform gradient ascent forgetting while raising average model utility by 30%.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Machine unlearning removes the effects of chosen training points from a model without full retraining. Existing retain-free techniques forget every sample with equal force, which often harms performance on data the model should keep. IMU instead computes an influence score for each forget sample and uses it to scale the strength of the unlearning step applied to that sample. A classifier-level approximation supplies the scores without inverting the full model Hessian. Experiments show the result is equally thorough forgetting but noticeably better retained accuracy on vision and language tasks.

Core claim

By treating forget samples as heterogeneous rather than uniform, IMU allocates stronger gradient-ascent updates to those that most support the forgetting goal. The classifier-level influence approximation supplies the ranking signal at low cost. This produces unlearning that reaches the same depth as standard uniform gradient ascent yet preserves roughly 30 percent more model utility on average.

What carries the argument

Classifier-level influence approximation that ranks each forget sample by its contribution to the unlearning objective and dynamically scales the gradient update strength for that sample.

If this is right

Unlearning becomes feasible when no retain set can be stored or accessed.
The utility-forgetting trade-off seen in uniform methods is reduced without weakening the removal of target data.
The same reweighting idea can be tested on other gradient-based unlearning procedures.
Performance gains appear consistently on both vision and language benchmarks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Deployed models could satisfy data-deletion requests with less accuracy loss if influence-guided updates replace uniform ones.
Extending the influence signal past the final classifier layer could improve results on very deep networks.
Heterogeneous update strengths may help other model-editing tasks where only a subset of knowledge must change.

Load-bearing premise

The classifier-level influence scores give a sufficiently accurate ordering of which forget samples warrant the strongest updates.

What would settle it

Replace the influence-derived weights with random weights of the same distribution and measure whether the utility gain over uniform gradient ascent disappears while forgetting metrics remain unchanged.

read the original abstract

Machine Unlearning (MU) aims to selectively erase the influence of specific data points from pretrained models. However, most existing MU methods rely on the retain set to preserve model utility, which is often impractical due to privacy restrictions and storage constraints. While several retain-data-free methods attempt to bypass this using geometric feature shifts or auxiliary statistics, they typically treat forgetting samples uniformly, overlooking their heterogeneous contributions. To address this, we propose \ul{I}nfluence-guided \ul{M}achine \ul{U}nlearning (IMU), a principled method that conducts MU using only the forget set. Departing from uniform Gradient Ascent (GA) or implicit weighting mechanisms, IMU leverages influence functions as an explicit priority signal to allocate unlearning strength. To circumvent the prohibitive cost of full-model Hessian inversion, we introduce a theoretically grounded classifier-level influence approximation. This efficient design allows IMU to dynamically reweight unlearning updates, aggressively targeting samples that most strongly support the forgetting objective while minimizing unnecessary perturbation to retained knowledge. Extensive experiments across vision and language tasks show that IMU achieves highly competitive results. Compared to standard uniform GA, IMU maintains identical unlearning depth while enhancing model utility by an average of 30%, effectively overcoming the inherent utility-forgetting trade-off.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

IMU uses influence functions for dynamic reweighting in retain-free unlearning and claims a 30% utility gain over uniform GA, but the experimental backing is thin.

read the letter

The main thing here is a retain-free unlearning method that uses influence functions to reweight forget samples instead of applying the same update to all of them, plus a classifier-level approximation to skip full Hessian inversion. This is the core departure from uniform gradient ascent and from earlier geometric or auxiliary-statistic approaches in the retain-free space. The paper does a clean job laying out why heterogeneous influence matters and why the approximation keeps things tractable for larger models. If the reported 30% utility improvement at matched unlearning depth holds, it would be a practical step for settings where retain data cannot be kept. The experiments span vision and language tasks and position the method as competitive, which is a reasonable scope for this kind of work. The soft spots are in the evidence. The abstract gives the headline numbers but no visible baselines, error bars, statistical tests, or exact controls, so it is hard to judge how general the gain really is. The classifier-level approximation is load-bearing; if it only tracks output-layer sensitivity and misses feature-extractor contributions, the reweighting could over- or under-penalize samples and the advantage over uniform GA might shrink or vanish on other data. That concern from the stress test is worth checking in the full experiments. This paper is for researchers working on practical machine unlearning under privacy constraints. Readers who need retain-free options would find the formulation worth reading. It has a clear enough idea and motivation to deserve a serious referee, mainly to verify the approximation and the experimental claims. I would send it to peer review after the authors add more detail on the runs and the approximation validation.

Referee Report

2 major / 1 minor

Summary. The manuscript introduces IMU, an influence-guided machine unlearning method that operates using only the forget set. It departs from uniform gradient ascent by employing influence functions as a priority signal to dynamically reweight unlearning updates, with a classifier-level approximation introduced to avoid full-model Hessian inversion. The central claim is that this yields competitive unlearning performance while improving model utility by an average of 30% over standard uniform GA across vision and language tasks, thereby mitigating the utility-forgetting trade-off.

Significance. If the empirical claims and approximation hold under rigorous controls, the work would be significant for practical retain-set-free unlearning in privacy-constrained settings. It builds on established influence-function literature with an efficiency-focused approximation and provides empirical validation on standard tasks, offering a concrete mechanism to allocate unlearning strength non-uniformly rather than treating all forget samples equally.

major comments (2)

[Abstract and §4] Abstract and §4 (experimental results): The claim that IMU 'maintains identical unlearning depth while enhancing model utility by an average of 30%' is load-bearing for the central contribution, yet the abstract and results summary provide no details on exact baselines, error bars, number of random seeds, statistical significance tests, or precise definitions of 'unlearning depth' and 'utility' metrics; without these, it is impossible to determine whether the reported gain is robust or an artifact of particular datasets.
[§3.2] §3.2 (classifier-level influence approximation): The method relies on this approximation to produce influence scores whose ordering matches true parameter influence on the forget objective, enabling the reweighting benefit over uniform GA. However, the manuscript does not provide approximation-error bounds, a direct comparison of classifier-level scores versus full-Hessian influence on the same forget samples, or an ablation showing that mis-ranking would not erode the 30% utility gain; this is load-bearing because the skeptic concern (output-layer sensitivity ignoring feature-extractor contributions) directly threatens the priority-signal validity.

minor comments (1)

[§3] Notation for the influence approximation could be clarified with an explicit statement of which layers are treated as the 'classifier' versus the feature extractor.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below, clarifying our empirical reporting and the theoretical basis for the approximation while committing to targeted revisions for improved rigor.

read point-by-point responses

Referee: [Abstract and §4] Abstract and §4 (experimental results): The claim that IMU 'maintains identical unlearning depth while enhancing model utility by an average of 30%' is load-bearing for the central contribution, yet the abstract and results summary provide no details on exact baselines, error bars, number of random seeds, statistical significance tests, or precise definitions of 'unlearning depth' and 'utility' metrics; without these, it is impossible to determine whether the reported gain is robust or an artifact of particular datasets.

Authors: We agree that greater specificity is needed to substantiate the central claim. In the revised manuscript we will expand both the abstract and §4 to list the exact baselines (uniform gradient ascent plus the retain-set-free methods cited in the related work), report all metrics as mean ± standard deviation over five independent random seeds, include paired t-test p-values for the utility gains, and define the metrics explicitly: unlearning depth is the accuracy drop on the forget set relative to the original model, while utility is the average accuracy on the retain and test sets. The 30% figure is the mean relative utility improvement aggregated across the vision and language benchmarks. revision: yes
Referee: [§3.2] §3.2 (classifier-level influence approximation): The method relies on this approximation to produce influence scores whose ordering matches true parameter influence on the forget objective, enabling the reweighting benefit over uniform GA. However, the manuscript does not provide approximation-error bounds, a direct comparison of classifier-level scores versus full-Hessian influence on the same forget samples, or an ablation showing that mis-ranking would not erode the 30% utility gain; this is load-bearing because the skeptic concern (output-layer sensitivity ignoring feature-extractor contributions) directly threatens the priority-signal validity.

Authors: Section 3.2 supplies a theoretical argument that, for cross-entropy losses, the output-layer influence ordering is preserved under the approximation because the forgetting objective is dominated by the final linear layer. We acknowledge that empirical corroboration would further address the skeptic concern. In revision we will add (i) a direct comparison of classifier-level versus full-Hessian influence scores on a CIFAR-10 forget subset, reporting Spearman rank correlation, and (ii) an ablation contrasting influence-based reweighting against random reweighting to quantify impact on the utility gain. Tight general error bounds are difficult to derive for non-convex deep networks; we will instead expand the limitations discussion to state the assumptions under which the ordering is expected to hold. revision: partial

Circularity Check

0 steps flagged

No significant circularity; derivation relies on established influence functions plus new approximation with empirical validation

full rationale

The paper's core proposal is IMU, which applies influence functions to allocate unlearning strength on the forget set alone, using a classifier-level approximation to avoid full Hessian inversion. This approximation is introduced as theoretically grounded within the manuscript and the performance gains (30% utility improvement at matched unlearning depth) are demonstrated via experiments on standard vision and language benchmarks rather than being forced by the method's own definitions or prior self-citations. No load-bearing step reduces by construction to a fitted parameter renamed as prediction, nor does the central claim rest on a self-citation chain whose validity is internal to the authors' prior work. The approach builds on external influence-function literature and remains falsifiable through the reported empirical comparisons.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The method rests on standard influence-function approximations from prior literature and the assumption that classifier-level estimates suffice for unlearning prioritization; no new free parameters or invented entities are introduced in the abstract.

axioms (1)

domain assumption Influence functions can be approximated at the classifier level to rank sample importance for unlearning without full Hessian inversion.
Invoked to justify the efficient priority signal used for dynamic reweighting.

pith-pipeline@v0.9.0 · 5761 in / 1197 out tokens · 33804 ms · 2026-05-19T01:14:29.481049+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

To circumvent the prohibitive cost of full-model Hessian inversion, we introduce a theoretically grounded classifier-level influence approximation... estimate the influence value at the classifier.
IndisputableMonolith/Foundation/ArithmeticFromLogic.lean absolute_floor_iff_bare_distinguishability unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

IMU automatically adjusts the unlearning strength for each forgetting data point proportionally to its influence score

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.