A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning

Haoran Wang; Jie Zhang; JingHao Zheng; Tao Li; Xiaolin Huang; Xinwen Cheng; Zhehao Huang; Zhengbao He

REVIEW 4 major objections 7 minor 1 cited by

A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that continual learning and machine unlearning can be driven by one four-component steepest-descent update on KL divergence, and provides an efficient implementation that learns, unlearns, and preserves knowledge in a…

desk verdict The task-agnostic CLU benchmarks and the four-component taxonomy are worth attention, but the central steepest-descent derivation has a wrong Hessian in the proof and the four-component update does not follow as stated. read the letter →

arxiv 2505.15178 v1 pith:4TGFFWEN submitted 2025-05-21 cs.LG

Zhehao Huang , Xinwen Cheng , Jie Zhang , Jinghao Zheng , Haoran Wang , Zhengbao He , Tao Li , Xiaolin Huang This is my paper

classification cs.LG

keywords continuallearningmachineunlearningKLdivergenceminimizationsteepestdescentHessianapproximationweightsaliencytask-agnosticCLUfast-slowupdate

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that continual learning (absorbing new tasks) and machine unlearning (removing the influence of old data) are not separate processes but two faces of a single optimization problem. It derives the steepest-descent step that minimizes the Kullback-Leibler divergence between the current model and an oracle model trained on the remaining data, and shows this step decomposes into four parts: learning new knowledge, unlearning targeted data, preserving retained knowledge, and a weight-saliency modulator. To keep old knowledge stable, the update is run inside a remain-preserving manifold whose metric is the KL divergence on the remaining data, which introduces an inverse remaining-Hessian correction. The proposed UG-CLU method implements this direction with a fast-slow weight mechanism, adaptive loss coefficients, and a balanced saliency mask, and reports that it learns more accurately, unlearns completely, and stays closest to the oracle's output distribution in its experiments.

What carries the argument

The load-bearing machinery is Assumption 4, which says that at every iteration the current parameters exactly minimize a weighted sum of the remaining loss and the task-request loss; this is what lets the proof expand the parameter gap $\theta_k-\theta^*$ with Taylor series. On top of that, the remain-preserving manifold metric ($D_{KL}$ on the remaining set's output distribution) turns the update into a second-order step, and the fast-slow weight mechanism saves the framework from explicitly forming $(H_k^R)^{-1}$: a fast task update followed by inner-loop fine-tuning on the remaining set makes the outer update behave as if multiplied by the inverse remaining Hessian. The balanced weight saliency mask then selects which parameters move, approximating the middle Hessian product by a gradient-ratio threshold.

What would settle it

Take a trained model during a CLU task and check the gradient-balance condition behind Assumption 4: record iterates $\theta_k$ and solve for non-negative coefficients $\varepsilon_k$ that make $\nabla L^R(\theta_k)+\nabla L^Q(\theta_k;\varepsilon_k)=0$; if no such coefficients exist, or if their magnitudes blow up, across many architectures and tasks, the derived four-component update is not a faithful steepest-descent direction. A second direct check is to compare the predicted update direction against the actual best step that minimizes KL divergence to the oracle on a small proxy dataset.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is Proposition 2: under the assumption that every iterate $\theta_k$ is an exact minimizer of a reweighted remaining-plus-request loss, the steepest descent direction for approximate continual learning-unlearning, measured by the KL divergence between the model and the oracle on remaining, learning, and unlearning data, is approximately $$\theta_{k+1}-\theta_k \approx -\frac{\alpha_k}{p_R+1}(H_k^R)^{-1}\cdot \frac12(H_k^L p_L+H_k^U p_U)(H_*^R)^{-1}\left[\nabla L^L(\theta_k;1-\varepsilon_k^L)+\nabla L^U(\theta_k;-\varepsilon_k^U)\right].$$ This is the unified update: the first factor is the remaining-loss curvature that protects old knowledge, the middle factor is the saliency modulator, and the two gradients are the learning and unlearning requests, with the unlearning term acting as gradient ascent. When either request dataset is empty, the update reduces to a pure continual-learning or pure machine-unlearning step, which is how the paper frames existing gradient-based CL and MU methods as special cases of one framework.

Load-bearing premise

The load-bearing premise is Assumption 4: at every training step, the current model parameters are an exact global minimizer of some weighted combination of the remaining data loss and the current request loss; if that fails, the Taylor-expansion estimate of the parameter gap that generates the whole update formula has no basis.

Editorial extensions

If this is right

Gradient-based CL and MU methods that previously looked unrelated, such as replay, regularization, parameter selection, and gradient ascent, become special cases of the same four-component update, obtained by dropping parts of the decomposition.
A single implementation can serve both learning and unlearning requests by switching only the sign and sample weighting of the task loss, so a CLU agent does not need separate architectures or per-task snapshots.
Fine-grained unlearning at the level of cross-task classes or individual samples becomes possible, since the framework is task-agnostic and does not require unlearning whole tasks.
The experiments imply that balancing plasticity and stability can be quantified by output KL divergence to the oracle model, and that UG-CLU achieves the closest match to that oracle while maintaining zero unlearning accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If Assumption 4 holds only approximately, the four-component update is still a plausible correction direction; a testable extension is to measure how the learned $\varepsilon_k$ coefficients deviate from the assumption's exact-minimizer requirement during training.
The fast-slow weight trick is a general recipe: any bi-level objective where an inner loop fine-tunes on a preserved set can be reinterpreted as an implicit Hessian-inverse step, which suggests the mechanism could transfer to meta-learning or continual fine-tuning outside classification.
The framework suggests a practical diagnostic: track the relative magnitude of the four components (R, L, U, S); if catastrophic forgetting appears, it should be observable as a growing contribution of the learning and unlearning parts relative to the remaining part, rather than as generic accuracy loss.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

UG-CLU is a serious attempt to unify continual learning and unlearning, and the paper's most durable contribution is probably the task-agnostic CLU setting with its two benchmarks (cross-task class unlearning and interclass confusion). Those are genuinely new and give the community a concrete way to measure fine-grained unlearning. The four-component decomposition also works well as a taxonomy: Table 1 recasts most gradient-based CL and MU methods into (R), (L), (U), and (S), which is a useful common vocabulary even if it is mostly a reorganization of known ideas.

The empirical side is solid in its own terms. UG-CLU beats the baselines on learning accuracy, unlearning accuracy, and KL divergence to the oracle in most settings, and the ablation shows each module contributes. The fast-slow weight approximation is a reasonable way to avoid explicit Hessian computation, and the balanced saliency mask is a sensible extension of earlier parameter-selection ideas.

The theory, though, has a load-bearing error. In the proof of Proposition 1 (Appendix A.2, eq A14), the authors Taylor-expand ∇L^L(θ*;1) around θ_k using the Hessian H^R_* of the remaining loss, not H^L_k of the learning loss. The correct expansion uses H^L_k. This changes the assembled equation from 0 ≈ 2H^R_*Δ_k − ... to 0 ≈ (H^R_* + H^L_k)Δ_k − ..., and the factor 1/2 disappears. So the Δ estimate in (A15) is not a consequence of Assumption 4, and Proposition 2, Corollary 1, and Corollary 2 inherit the problem. The four-component decomposition as derived does not actually follow. This is not just a matter of Assumption 4 being strong (it is) or the heuristics being loose (they are); the algebra itself does not work.

Independently, Assumption 4 is a big ask—every iterate is an exact minimizer of a reweighted loss—and the Gaussian-gradient-noise justification does not establish it. Then the implementation replaces the theoretical ε coefficients with heuristics (eq 15–16) and tunes λ and γ on the same evaluation benchmarks (Appendix D), which further weakens the 'derived' claim. The baselines are also selected by grid-searching CL×MU combinations on test-time KL, which gives the comparisons a favorable slant. Algorithm A1 line 26 uses θ^Q_{K_in} where it should be θ^R_{K_in}; a typo, but one that would confuse any implementer.

So my honest read: treat this as a strong empirical paper with a promising framing, not as a paper that provides a rigorous theoretical foundation for CLU. The benchmarks and the taxonomy deserve attention; the theory needs a corrected derivation, and the decomposition may look different once the algebra is fixed. If I were an editor, I would send it to review, because the empirical contribution and the new setting are valuable and the theory is potentially fixable. I would ask the authors to correct the Taylor expansion, either properlly derive the decomposition or relabel it as a heuristic motivation, justify or soften Assumption 4, and move hyperparameter tuning to a validation set.

For your own work: cite it for the task-agnostic benchmarks, not for the steepest-descent derivation.

Referee Report

4 major / 7 minor

Summary. The paper proposes a unified gradient-based framework for continual learning-unlearning (CLU), aiming to connect continual learning (CL) and machine unlearning (MU) through a common optimization view. The authors model the oracle model of CLU as the minimizer of the remaining-data loss, then derive a steepest-descent direction for minimizing the KL divergence between the current model and the oracle, which they decompose into four components: remaining-data preservation, learning, unlearning, and weight-saliency modulation. They further introduce a remain-preserving manifold using the remaining-data Hessian, and design the UG-CLU algorithm with a fast-slow weight update, sample-wise adaptive coefficients, and a balanced weight saliency mask. The paper also proposes task-agnostic CLU benchmarks for cross-task category unlearning and random-sample (interclass confusion) unlearning, with experiments on CIFAR-10 and TinyImageNet showing that UG-CLU achieves high learning accuracy, low unlearning accuracy, and low output KL divergence to the oracle, often at competitive runtime.

Significance. If the theoretical derivation were correct, the unified framework would be a substantial contribution: it would give a common gradient-based perspective on CL and MU, provide a principled four-component decomposition, and justify a practical algorithm. The proposed task-agnostic CLU benchmarks address a real gap in the literature, and the empirical results are strong, with UG-CLU consistently matching or beating existing CLU and combined CL+MU baselines. The paper is also unusually complete in reporting runtimes, ablations, and sensitivity analyses. However, the central theoretical claim rests on a derivation that contains a load-bearing algebraic error, and on an extremely strong assumption about every iterate being an exact minimizer of a reweighted loss. As a result, the claimed 'theoretical foundation' for the four-component update and for the UG-CLU algorithm is not established, which substantially reduces the significance of the paper in its current form.

major comments (4)

[Appendix A.2, Eq. (A14)-(A15)] The proof of Proposition 1 contains an algebraic error in the Taylor expansion of ∇L^L(θ*;1). The manuscript writes ∇L^L(θ*;1) ≈ ∇L^L(θ_k;1) − ∇²L^R(θ*)Δ_k, but expanding ∇L^L(θ*;1) around θ_k gives ∇L^L(θ*;1) = ∇L^L(θ_k;1) + ∇²L^L(θ_k)(θ*−θ_k) + o(Δ_k) = ∇L^L(θ_k;1) − H^L_k Δ_k + o(Δ_k), with H^L_k = ∇²L^L(θ_k), not H^R_*. Under the stationarity condition at θ_k, the correct assembling yields (H^R_* + H^L_k)Δ_k ≈ ∇L^L(θ_k;1−ε^L_k) + ∇L^U(θ_k;−ε^U_k), not 2H^R_*Δ_k with the factor 1/2 and only H^R_* in the inverse. Since Eq. (A15) is used directly in Proposition 2, Corollary 1, and Corollary 2, the four-component decomposition in Eqs. (6)-(7), the saliency term (H^L_k p_L + H^U_k p_U)(H^R_*)^{-1}, and the factor 1/2 are not consequences of Assumption 4. This is not a minor typo: it invalidates the central theoretical derivation. The authors should either correct the expansion and re-derive the update direction, or clearly state that the four-component update is a heuristic built on an ad-hoc approximation rather than the steepest-descent direction of the KL objective.
[Section 3.3, Assumption 4 (Eq. 2)] Assumption 4 postulates that for every iterate θ_k there exist non-negative coefficients ε^Q_k such that θ_k exactly minimizes L^R + ε^L L^L + ε^U L^U. This is a very strong condition; it requires each SGD iterate to be a global (or at least exact local) minimizer of a per-sample reweighted empirical loss. The justification given (Gaussian gradient noise, [47]) does not imply such exactness for deep networks. Moreover, Section 5.3 admits that solving for ε is computationally intractable and replaces it with heuristics (Eqs. 15-16), meaning the theory's key quantity is not actually used in the algorithm. If Assumption 4 fails in practice, the Taylor-based estimate Δ_k in Eq. (A15) has no basis. The paper should either verify the assumption empirically (e.g., checking the gradient balance condition on the iterates) or relax the theory to an approximate optimality condition with a provable error bound.
[Algorithm A1, line 26] The outer-loop update in the pseudocode uses θ^Q_{K_in}, but the fast-slow derivation (Proposition 3 and Eq. (19)) requires the fine-tuned remaining minimizer θ^R_{K_in}, which is computed in lines 22-25. As written, θ^Q_{K_in} is not defined, since θ^Q carries the subscript k−1 from the fast step. This obscures the actual algorithm and contradicts the claim that the implementation follows Eq. (19). If this is a typo, it should be corrected to θ^R_{K_in}; otherwise the reported algorithm does not implement the described method.
[Section 5.3 / Appendix D] The weighting coefficients ε̃^L and ε̃^U in Eqs. (15)-(16) are heuristic, and the temperature λ and the saliency threshold γ are tuned by searching on the same CIFAR-10 and TinyImageNet benchmarks used for the final evaluation. This is a form of test-set hyperparameter selection, which gives UG-CLU an advantage over baselines that do not receive per-dataset tuning of comparable degrees of freedom. The paper should use held-out validation splits or additional datasets to demonstrate that the reported performance does not depend on benchmark-specific tuning. At minimum, the sensitivity analysis in Appendix E.1 should be extended to show that the method is robust across a wider range of λ and γ, not just a single value.

minor comments (7)

[Section 5.3, Eqs. (15)-(16)] Please clarify how ε̃ is used in the actual gradient update. For learning, the text says updates are weighted by 1−ε̃^L_k, but Eq. (19) writes ∇L^Q(θ_k; ε̃^Q_k) without specifying whether the argument is ε̃ or 1−ε̃. This makes it hard to reproduce the exact loss weighting.
[Section 4.2, Eq. (5)] The notation L^L(θ; ε^L_k) and L^U(θ; ε^U_k) should be defined more carefully; in particular, whether ε is a scalar or a per-sample vector and how it multiplies the sum of per-sample losses. The boundary conditions ε^L_k=0/1 and ε^U_k=1/0 are stated for vectors but used as scalars in the text.
[Appendix A.4, Eq. (A41)] The step (θ^R_k−θ_k)^T ⊗ T ⊗ (θ^R_k−θ_k) ≈ β²(∇L^Q(θ_k))² and then setting (∇L^Q(θ_k))² = ∇L^Q(θ_k) is dimensionally inconsistent. The second-order term approximation needs a more careful justification or a reference.
[Section 4.1, Eq. (4)] The symbols p_R, p_L, p_U are used both to denote the data partitions D^R, D^L, D^U and as scalar weights in Eqs. (6)-(7). Please disambiguate, for instance by using p_R = |D^R|/|D| or similar.
[Table 2] There is a typo in the L2UL row: '86,43' should be '86.43'. Also, several table headers have inconsistent spacing ('Table. 2', 'T ables. 3,4'), which should be fixed in the final version.
[Section 1] The paper claims to 'pioneer task-agnostic CLU', but UniCLUN [3] is evaluated on task-agnostic settings in Tables 3-5. Please clarify the precise novelty relative to UniCLUN's setup, or adjust the claim.
[Section 5.2, Proposition 3] The quantity δ_k in the condition β_k < sqrt(δ_k/|∇L^R(θ_k)−[∇L^R(θ_k)]²|) is not defined in the main text. Please define δ_k and explain why this condition is needed for the approximation in Eq. (11).

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the four-component update is derived from Assumption 4 and Taylor expansions rather than from its own conclusion; the main defect is an algebraic error in Appendix A.2, which is a correctness issue, not a circularity.

full rationale

The derivation chain is not circular. Proposition 1 and Proposition 2 start from an assumed optimality condition (Assumption 4, Eq. 2/5) and attempt to solve for Delta_k = theta_k - theta* via Taylor expansion (Appendix A.2, Eqs. A14-A15). This is a genuine inference from an assumption about the trajectory, not a restatement of the target update. The coefficients epsilon^L_k and epsilon^U_k are free parameters of that assumption; the later heuristic estimates (Eqs. 15-16) are explicitly disclosed as approximate, with the paper saying 'we constructs a feasible path between theoretical derivation and practical feasibility ... proposing a heuristic approximation strategy.' The implementation gap is therefore acknowledged rather than disguised as a prediction. Tuning the temperature and threshold on the benchmark (Appendix D) weakens the empirical claims as predictions, but the theoretical decomposition itself does not reduce to those fitted values. The self-citation to the NeurIPS 2024 conference version [23] is present but not load-bearing: the proofs are re-derived in Appendix A, and no uniqueness claim is imported from the authors' prior work. The serious problem is a proof error, not circularity. In Eq. (A14), the paper expands -nabla L^L(theta*;1) around theta_k using H^R_* instead of the correct request Hessian H^L_k, changing the assembled equation from 0 approx 2H^R_* Delta_k - ... to 0 approx (H^R_* + H^L_k) Delta_k - ...; consequently the claimed Delta_k approx (1/2)(H^R_*)^{-1}[...] and all downstream four-component directions do not follow. That is an internal correctness defect and should be scored as such, not as a circular reduction of the result to its inputs.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a small set of domain assumptions and one strong postulate (Assumption 4) that is not proven. The free parameters listed are the hyperparameters that are actually tuned on the evaluation benchmarks, so the quantitative predictions of the framework are partially fitted rather than fully derived.

free parameters (3)

λ (temperature scalars λ_L, λ_U) = searched in [0.0, 2.0]
Controls smoothness of sample-wise adaptive coefficients in eqs (15) and (16); chosen by search on the benchmark datasets (Appendix D).
γ (saliency mask threshold) = searched in [0.5, 3.0]
Hard threshold in eq (17) that decides which weights are updated; tuned on the same benchmarks used for evaluation.
β_L, β_U, β_R (inner-loop learning rates) = learning tasks [0.1, 0.01], unlearning [0.01, 0.001], remaining [0.1, 0.01]
Fast-slow update step sizes in Algorithm A1; values are selected by grid search and affect the effective Hessian-approximated direction.

assumptions (4)

domain assumption Training and test data are i.i.d. (Assumption 1)
Invoked to justify generalization and test-set evaluation; standard but not always valid in continual/unlearning streams.
domain assumption Loss is locally convex near the converged optimum (Assumption 2)
Needed for second-order Taylor expansions in Appendix A.2/A.3; deep network losses are non-convex globally.
domain assumption All unlearning data are subsets of historical training data (Assumption 3)
Restricts the CLU setting to previously seen data, simplifying data-ownership verification; this is a scope constraint, not a theorem.
ad hoc to paper Every iterate θ_k is the exact minimizer of a reweighted loss L^R + ε^L L^L + ε^U L^U with non-negative ε (Assumption 4, eq 2)
The load-bearing premise for the derivation of Propositions 1 and 2; it postulates that the entire optimization trajectory lies on the path of minimizers of reweighted objectives, which is not established for SGD on deep networks.

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning." pith.science (2026). https://pith.science/paper/4TGFFWEN

@misc{pith2026250515178,
  author       = {Pith},
  title        = {Pith review of: A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4TGFFWEN}},
  note         = {Machine review of arXiv:2505.15178}
}

read the original abstract

Recent advancements in deep models have highlighted the need for intelligent systems that combine continual learning (CL) for knowledge acquisition with machine unlearning (MU) for data removal, forming the Continual Learning-Unlearning (CLU) paradigm. While existing work treats CL and MU as separate processes, we reveal their intrinsic connection through a unified optimization framework based on Kullback-Leibler divergence minimization. This framework decomposes gradient updates for approximate CLU into four components: learning new knowledge, unlearning targeted data, preserving existing knowledge, and modulation via weight saliency. A critical challenge lies in balancing knowledge update and retention during sequential learning-unlearning cycles. To resolve this stability-plasticity dilemma, we introduce a remain-preserved manifold constraint to induce a remaining Hessian compensation for CLU iterations. A fast-slow weight adaptation mechanism is designed to efficiently approximate the second-order optimization direction, combined with adaptive weighting coefficients and a balanced weight saliency mask, proposing a unified implementation framework for gradient-based CLU. Furthermore, we pioneer task-agnostic CLU scenarios that support fine-grained unlearning at the cross-task category and random sample levels beyond the traditional task-aware setups. Experiments demonstrate that the proposed UG-CLU framework effectively coordinates incremental learning, precise unlearning, and knowledge stability across multiple datasets and model architectures, providing a theoretical foundation and methodological support for dynamic, compliant intelligent systems.

Figures

Figures reproduced from arXiv: 2505.15178 by the authors.

**Figure 2.** The visualization diagram for the optimization process of the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Three core modules of our proposed UG-CLU. The implicit online Hessian approximation derives the task optimization direction modulated by the Hessian through a fast-slow weight update mechanism, reducing computational overhead. The sample-wise adaptive coefficient leverages task properties and loss magnitude adaptation to re-weight the sample loss, effectively approximating the optimality conditions of the iterative… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: We adapt the interclass confusion unlearning setup from [ [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

SoK: Machine Unlearning for Large Language Models
cs.LG 2025-06 conditional novelty 5.0 of 10

A new taxonomy for LLM unlearning distinguishes removal-intended from suppression-intended methods, and argues that gradient ascent methods functionally behave like suppression.

Reference graph

Works this paper leans on

80 extracted references · 68 canonical work pages · cited by 1 Pith paper

[47]

Stochastic gradient descent as approximate bayesian inference,

S. Mandt, M. D. Hoffman, and D. M. Blei, “Stochastic gradient descent as approximate bayesian inference,”J. Mach. Learn. Res., vol. 18, pp. 134:1–134:35, 2017. 4

work page 2017
[1]

Scaling laws for neural language models,

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”ArXiv, vol. abs/2001.08361, 2020. 1

arXiv 2001
[2]

Continual learning and private unlearning,

B. Liu, Q. Liu, and P . Stone, “Continual learning and private unlearning,” inConference on Lifelong Learning Agents. PMLR, 2022, pp. 243–254. 1, 2, 3, 10

work page 2022
[3]

A unified framework for continual learning and unlearning,

R. Chatterjee, V . Chundawat, A. Tarun, A. Mali, and M. Mandal, “A unified framework for continual learning and unlearning,”

work page
[4]

A continual learning survey: Defying forgetting in classification tasks,

M. D. Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, pp. 3366–3385, 2019. 1, 2

work page 2019
[5]

A comprehensive survey of continual learning: Theory, method and application,

L. Wang, X. Zhang, H. Su, and J. Zhu, “A comprehensive survey of continual learning: Theory, method and application,”IEEE transactions on pattern analysis and machine intelligence, vol. PP , 2023. 1, 2, 4

work page 2023
[6]

Deep class-incremental learning: A survey,

D.-W. Zhou, Q. Wang, Z. Qi, H.-J. Ye, D. chuan Zhan, and Z. Liu, “Deep class-incremental learning: A survey,”ArXiv, vol. abs/2302.03648, 2023. 1, 2

arXiv 2023
[7]

Machine unlearn- ing,

L. Bourtoule, V . Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, “Machine unlearn- ing,”2021 IEEE Symposium on Security and Privacy (SP), pp. 141– 159, 2019. 1, 2

work page 2021

Show all 80 references

[8]

Exploring the landscape of machine unlearning: A survey and taxonomy,

T. Shaik, X. Tao, H. Xie, L. Li, X. Zhu, and Q. Li, “Exploring the landscape of machine unlearning: A survey and taxonomy,”arXiv preprint arXiv:2305.06360, 2023. 1, 2, 3

2023 arXiv
[9]

Machine unlearning: Solutions and challenges,

J. Xu, Z. Wu, C. Wang, and X. Jia, “Machine unlearning: Solutions and challenges,”IEEE Transactions on Emerging Topics in Computa- tional Intelligence, 2024. 1, 2, 3

2024
[10]

Overcoming catastrophic forgetting in neural networks,

J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Des- jardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska- Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,”Proceed- ings of the Nationa...
[11]

Online structured laplace approximations for overcoming catastrophic forgetting,

H. Ritter, A. Botev, and D. Barber, “Online structured laplace approximations for overcoming catastrophic forgetting,”Advances in Neural Information Processing Systems (NIPS), 2018. 1, 3, 7

2018
[12]

Learning without forgetting,

Z. Li and D. Hoiem, “Learning without forgetting,”IEEE Transac- tions on Pattern Analysis and Machine Intelligence, vol. 40, pp. 2935– 2947, 2016. 1, 3, 7, 10

2016
[13]

Learning a unified classifier incrementally via rebalancing,

S. Hou, X. Pan, C. C. Loy, Z. Wang, and D. Lin, “Learning a unified classifier incrementally via rebalancing,”2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 831–839, 2019. 1, 3

2019
[14]

Continual learning with tiny episodic memories,

A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P . K. Dokania, P . H. S. Torr, and M. Ranzato, “Continual learning with tiny episodic memories,”ArXiv, vol. abs/1902.10486, 2019. 1, 3, 4, 7, 10

1902 arXiv
[15]

Dark experience for general continual learning: a strong, simple baseline,

P . Buzzega, M. Boschini, A. Porrello, D. Abati, and S. Calderara, “Dark experience for general continual learning: a strong, simple baseline,”Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2022 (NeurIPS), 2020....

2022
[16]

icarl: Incremental classifier and representation learning,

S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “icarl: Incremental classifier and representation learning,”2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533–5542, 2016. 1, 3, 7

2017
[17]

Gradient episodic memory for continual learning,

D. Lopez-Paz and M. Ranzato, “Gradient episodic memory for continual learning,”Advances in neural information processing sys- tems, vol. 30, 2017. 1, 3, 7

2017
[18]

Mofo: Momentum-filtered optimizer for mitigating forgetting in llm fine-tuning,

Y. Chen, S. Wang, Z. Lin, Z. Qin, Y. Zhang, T. Ding, and R. Sun, “Mofo: Momentum-filtered optimizer for mitigating forgetting in llm fine-tuning,” 2024. 1, 3, 6, 7, 9, 10

2024
[19]

Hft: Half fine-tuning for large language models,

T. Hui, Z. Zhang, S. Wang, W. Xu, Y. Sun, and H. Wu, “Hft: Half fine-tuning for large language models,” 2024. 1, 3

2024
[20]

Low dimensional trajectory hypothesis is true: Dnns can be trained in tiny subspaces,

T. Li, L. Tan, Z. Huang, Q. Tao, Y. Liu, and X. Huang, “Low dimensional trajectory hypothesis is true: Dnns can be trained in tiny subspaces,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 45, pp. 3411–3420, 2022. 1

2022
[21]

Lora: Low-rank adaptation of large language models,

J. E. Hu, Y. Shen, P . Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” International Conference on Learning Representations (ICLR), 2022. 1

2022
[22]

Hesscale: Scalable computation of hessian diagonals,

M. Elsayed and A. R. Mahmood, “Hesscale: Scalable computation of hessian diagonals,” 2022. 2, 7

2022
[23]

Unified gradient-based machine unlearning with remain geom- etry enhancement,

Z. Huang, X. Cheng, J. Zheng, H. Wang, Z. He, T. Li, and X. Huang, “Unified gradient-based machine unlearning with remain geom- etry enhancement,”Advances in Neural Information Processing Sys- tems (NIPS), 2024. 2

2024
[24]

Catastrophic interference in con- nectionist networks: The sequential learning problem,

M. McCloskey and N. J. Cohen, “Catastrophic interference in con- nectionist networks: The sequential learning problem,”Psychology of Learning and Motivation, vol. 24, pp. 109–165, 1989. 2

1989
[25]

Connectionist models of recognition memory: con- straints imposed by learning and forgetting functions

R. Ratcliff, “Connectionist models of recognition memory: con- straints imposed by learning and forgetting functions.”Psycholog- ical review, vol. 97 2, pp. 285–308, 1990. 2

1990
[26]

Continual learning through synaptic intelligence,

F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,”Proceedings of machine learning research, vol. 70, pp. 3987–3995, 2017. 2

2017
[27]

Descent-to-delete: Gradient-based methods for machine unlearning,

S. Neel, A. Roth, and S. Sharifi-Malvajerdi, “Descent-to-delete: Gradient-based methods for machine unlearning,”Algorithmic Learning Theory, 2021. 3, 4

2021
[28]

Remember what you want to forget: Algorithms for machine unlearning,

A. Sekhari, J. Acharya, G. Kamath, and A. T. Suresh, “Remember what you want to forget: Algorithms for machine unlearning,” Advances in Neural Information Processing Systems (NIPS), 2021. 3

2021
[29]

Making ai forget you: Data deletion in machine learning,

A. A. Ginart, M. Y. Guan, G. Valiant, and J. Y. Zou, “Making ai forget you: Data deletion in machine learning,”Advances in Neural Information Processing Systems (NIPS), 2019. 3

2019
[30]

Cer- tified data removal from machine learning models,

C. Guo, T. Goldstein, A. Y. Hannun, and L. van der Maaten, “Cer- tified data removal from machine learning models,”International Conference on Machine Learning (ICML), 2020. 3

2020
[31]

Understanding black-box predictions via influence functions,

P . W. Koh and P . Liang, “Understanding black-box predictions via influence functions,” inInternational Conference on Machine Learning (ICML), 2017. 3

2017
[32]

Eternal sunshine of the spotless net: Selective forgetting in deep networks,

A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotless net: Selective forgetting in deep networks,”IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9301–9309, 2019. 3

2019
[33]

Machine unlearning of features and labels,

A. Warnecke, L. Pirch, C. Wressnegger, and K. Rieck, “Machine unlearning of features and labels,”Annual Network and Distributed System Security Symposium, 2023. 3, 7, 10

2023
[34]

Model sparsity can simplify machine unlearning,

J. Jia, J. Liu, P . Ram, Y. Yao, G. Liu, Y. Liu, P . Sharma, and S. Liu, “Model sparsity can simplify machine unlearning,” inNeural Information Processing Systems (NIPS), 2023. 3, 10

2023
[35]

Amnesiac machine learn- ing,

L. Graves, V . Nagisetty, and V . Ganesh, “Amnesiac machine learn- ing,” inAAAI Conference on Artificial Intelligence (AAAI), 2021. 3, 7, 10

2021
[36]

Un- rolling sgd: Understanding factors influencing machine unlearn- ing,

A. Thudi, G. Deza, V . Chandrasekaran, and N. Papernot, “Un- rolling sgd: Understanding factors influencing machine unlearn- ing,”European Symposium on Security and Privacy (EuroS&P), pp. 303–319, 2022. 3, 4, 7, 10

2022
[37]

Can bad teaching induce forgetting? unlearning in deep net- works using an incompetent teacher,

V . S. Chundawat, A. K. Tarun, M. Mandal, and M. S. Kankanhalli, “Can bad teaching induce forgetting? unlearning in deep net- works using an incompetent teacher,”AAAI Conference on Artificial Intelligence (AAAI), 2023. 3

2023
[38]

Natural gradient works efficiently in learning,

S. Amari, “Natural gradient works efficiently in learning,”Neural Computation, vol. 10, pp. 251–276, 1998. 3

1998
[39]

New insights and perspectives on the natural gradient method,

J. Martens, “New insights and perspectives on the natural gradient method,”Journal of Machine Learning Research, vol. 21, pp. 146:1– 146:76, 2014. 3, 4

2014
[40]

Natural gradient methods: Perspectives, efficient- scalable approximations, and analysis,

R. Shrestha, “Natural gradient methods: Perspectives, efficient- scalable approximations, and analysis,” 2023. 3

2023
[41]

Learning with selective forgetting,

T. Shibata, G. Irie, D. Ikami, and Y. Mitsuzumi, “Learning with selective forgetting,” inInternational Joint Conference on Artificial Intelligence, 2021. 3

2021
[42]

Re-evaluating continual learning scenarios: A categorization and case for strong baselines,

Y.-C. Hsu, Y.-C. Liu, and Z. Kira, “Re-evaluating continual learning scenarios: A categorization and case for strong baselines,”ArXiv, vol. abs/1810.12488, 2018. 3, 10

2018 arXiv
[43]

Three scenarios for continual learning,

G. M. van de Ven and A. S. Tolias, “Three scenarios for continual learning,”ArXiv, vol. abs/1904.07734, 2019. 3, 10

1904 arXiv
[44]

Approximate data deletion from machine learning models: Algorithms and evaluations,

Z. Izzo, M. A. Smart, K. Chaudhuri, and J. Y. Zou, “Approximate data deletion from machine learning models: Algorithms and evaluations,”ArXiv, vol. abs/2002.10077, 2020. 4

2002 arXiv
[45]

The elements of statistical learning: Data mining, inference, and prediction,

D. Ruppert, “The elements of statistical learning: Data mining, inference, and prediction,”Journal of the American Statistical Asso- ciation, vol. 99, pp. 567 – 567, 2004. 4

2004
[46]

The loss surfaces of multilayer networks,

A. Choroma ´nska, M. Henaff, M. Mathieu, G. B. Arous, and Y. Le- Cun, “The loss surfaces of multilayer networks,” inInternational Conference on Artificial Intelligence and Statistics, 2014. 4 SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANAL YSIS AND MACHINE INTELLIGENCE 14

2014
[48]

A unified and general framework for continual learning,

Z. Wang, Y. Li, L. Shen, and H. Huang, “A unified and general framework for continual learning,”ArXiv, vol. abs/2403.13249,

arXiv
[49]

Steepest descent algorithms for optimization under unitary matrix constraint,

T. E. Abrudan, J. Eriksson, and V . Koivunen, “Steepest descent algorithms for optimization under unitary matrix constraint,” IEEE Transactions on Signal Processing, vol. 56, pp. 1134–1147, 2008. 4

2008
[50]

Fisher sam: Information geometry and sharpness aware minimisation,

M. Kim, D. Li, S. X. Hu, and T. M. Hospedales, “Fisher sam: Information geometry and sharpness aware minimisation,” in International Conference on Machine Learning (ICML), 2022. 4

2022
[51]

Stochastic newton and cubic newton methods with simple local linear-quadratic rates,

D. Kovalev, K. Mishchenko, and P . Richt ´arik, “Stochastic newton and cubic newton methods with simple local linear-quadratic rates,”ArXiv, vol. abs/1912.01597, 2019. 4

1912 arXiv
[52]

Geometric modeling in probability and statistics,

O. Calin and C. Udris ¸te, “Geometric modeling in probability and statistics,” inArXiv, 2014. 4

2014
[53]

Salun: Em- powering machine unlearning via gradient-based weight saliency in both image classification and generation,

C. Fan, J. Liu, Y. Zhang, D. Wei, E. Wong, and S. Liu, “Salun: Em- powering machine unlearning via gradient-based weight saliency in both image classification and generation,”International Confer- ence on Learning Representations (ICLR), 2023. 6, 7, 9, 10

2023
[54]

Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection,

T. Hoang, S. Rana, S. Gupta, and S. Venkatesh, “Learn to unlearn for deep neural networks: Minimizing unlearning interference with gradient projection,”IEEE/CVF Winter Conference on Appli- cations of Computer Vision (WACV), 2023. 6

2023
[55]

Machine un- learning in learned databases: An experimental analysis,

M. Kurmanji, E. Triantafillou, and P . Triantafillou, “Machine un- learning in learned databases: An experimental analysis,”Proceed- ings of the ACM on Management of Data, vol. 2, pp. 1 – 26, 2023. 7, 10

2023
[56]

Towards un- bounded machine unlearning,

M. Kurmanji, P . Triantafillou, and E. Triantafillou, “Towards un- bounded machine unlearning,”Towards Unbounded Machine Un- learning (NIPS), 2023. 7, 10

2023
[57]

Learning to unlearn: Instance-wise unlearning for pre-trained classifiers,

S. Cha, S. Cho, D. Hwang, H. Lee, T. Moon, and M. Lee, “Learning to unlearn: Instance-wise unlearning for pre-trained classifiers,” ArXiv, vol. abs/2301.11578, 2023. 7, 10

2023 arXiv
[58]

On tiny episodic memories in continual learning,

A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P . K. Doka- nia, P . H. S. Torr, and M. Ranzato, “On tiny episodic memories in continual learning,”ArXiv, 2019. 7

2019
[59]

Random sampling with a reservoir,

J. S. Vitter, “Random sampling with a reservoir,”ACM Trans. Math. Softw., vol. 11, pp. 37–57, 1985. 7

1985
[60]

Learning to learn without forgetting by maximizing transfer and minimizing interference,

M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro, “Learning to learn without forgetting by maximizing transfer and minimizing interference,”ArXiv, vol. abs/1810.11910,

arXiv
[61]

Continual learning with deep generative replay,

H. Shin, J. K. Lee, J. Kim, and J. Kim, “Continual learning with deep generative replay,” inNeural Information Processing Systems,
[62]

Note on the quadratic penalties in elastic weight consolidation,

F. Husz ´ar, “Note on the quadratic penalties in elastic weight consolidation,”Proceedings of the National Academy of Sciences, vol. 115, no. 11, Feb. 2018. 7

2018
[63]

Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction,

Y. Wu, L.-K. Huang, R. Wang, D. Meng, and Y. Wei, “Meta continual learning revisited: Implicitly enhancing online hessian approximation via variance reduction,” inInternational Conference on Learning Representations (ICLR), 2024. [Online]. Available: https://openreview.net/for...

2024
[64]

Lookahead opti- mizer: k steps forward, 1 step back,

M. R. Zhang, J. Lucas, G. E. Hinton, and J. Ba, “Lookahead opti- mizer: k steps forward, 1 step back,”Advances in Neural Information Processing Systems (NIPS), 2019. 8

2019
[65]

On first-order meta- learning algorithms,

A. Nichol, J. Achiam, and J. Schulman, “On first-order meta- learning algorithms,”ArXiv, vol. abs/1803.02999, 2018. 8

2018 arXiv
[66]

Bilevel continual learning,

A. Shaker, F. Alesiani, S. Yu, and W. Yin, “Bilevel continual learning,” 2020. 8

2020
[67]

Fast yet effective machine unlearning,

A. K. Tarun, V . S. Chundawat, M. Mandal, and M. S. Kankanhalli, “Fast yet effective machine unlearning,”IEEE transactions on neural networks and learning systems, vol. PP , 2021. 8

2021
[68]

The shapley value in machine learning,

B. Rozemberczki, L. Watson, P . Bayer, H.-T. Yang, O. Kiss, S. Nils- son, and R. Sarkar, “The shapley value in machine learning,” International Joint Conference on Artificial Intelligence (IJCAI), 2022. 9

2022
[69]

Learn from downstream and be yourself in multimodal large language model fine-tuning,

W. Huang, J. Liang, Z. Shi, D. Zhu, G. Wan, H. Li, B. Du, D. Tao, and M. Ye, “Learn from downstream and be yourself in multimodal large language model fine-tuning,”ArXiv, vol. abs/2411.10928, 2024. 9

2024 arXiv
[70]

Fast machine unlearning without retraining through selective synaptic dampening,

J. Foster, S. Schoepf, and A. Brintrup, “Fast machine unlearning without retraining through selective synaptic dampening,”AAAI Conference on Artificial Intelligence (AAAI), 2024. 9, 18

2024
[71]

Towards adversarial evaluations for inexact machine unlearning,

S. Goel, A. Prabhu, A. Sanyal, S. N. Lim, P . H. S. Torr, and P . Ku- maraguru, “Towards adversarial evaluations for inexact machine unlearning,” inArXiv, 2022. 10, 19

2022
[72]

Systematic evaluation of privacy risks of machine learning models,

L. Song and P . Mittal, “Systematic evaluation of privacy risks of machine learning models,” inUSENIX Security Symposium, 2020. 10

2020
[73]

Membership inference attacks from first principles,

N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer, “Membership inference attacks from first principles,” 2022. 10, 18

2022
[74]

New insights on reducing abrupt representation change in online continual learning,

L. Caccia, R. Aljundi, N. Asadi, T. Tuytelaars, J. Pineau, and E. Belilovsky, “New insights on reducing abrupt representation change in online continual learning,”The Tenth International Con- ference on Learning Representations (ICLR), 2022. 10

2022
[75]

Learning multiple layers of features from tiny images,

A. Krizhevsky, “Learning multiple layers of features from tiny images,” inArXiv, 2009. 11, 19

2009
[76]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,”IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778, 2015. 11, 19

2015
[77]

Tiny imagenet visual recognition challenge,

Y. Le and X. S. Yang, “Tiny imagenet visual recognition challenge,” inArXiv, 2015. 11, 19

2015
[78]

Swin transformer: Hierarchical vision transformer using shifted windows,

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,”IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9992–10 002, 2021. 11, 19

2021
[79]

Decoupled weight decay regulariza- tion,

I. Loshchilov and F. Hutter, “Decoupled weight decay regulariza- tion,” 2019. 19 SUBMITTED TO IEEE TRANSACTIONS ON PATTERN ANAL YSIS AND MACHINE INTELLIGENCE 15 APPENDIXA DETAILEDPROOF A.1 Proof of Steepest Descent in Section. 3.4 Proof.We form the following optimization probl...

2019
[80]

(A15) By plugging (A15) into (A10), we can get ∇L(θk) =G L(θk) ≈ 1 2 H L k H R ∗ −1 h ∇LL(θk;1−ε L k ) +∇L U (θk;−ε U k ) i

By performing the Taylor expansion and droppingo(∆ k) terms, we have ⇒∆ k ≈ 1 2 h ∇2LR(θ∗) i−1 h ∇LL(θk;1−ε L k ) +∇L U (θk;−ε U k ) i = 1 2 H R ∗ −1 h ∇LL(θk;1−ε L k ) +∇L U (θk;−ε U k ) i . (A15) By plugging (A15) into (A10), we can get ∇L(θk) =G L(θk) ≈ 1 2 H L k H R ∗ −1 h...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [47]

Stochastic gradient descent as approximate bayesian inference,

S. Mandt, M. D. Hoffman, and D. M. Blei, “Stochastic gradient descent as approximate bayesian inference,”J. Mach. Learn. Res., vol. 18, pp. 134:1–134:35, 2017. 4

work page 2017

[2] [1]

Scaling laws for neural language models,

J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,”ArXiv, vol. abs/2001.08361, 2020. 1

arXiv 2001

[3] [2]

Continual learning and private unlearning,

B. Liu, Q. Liu, and P . Stone, “Continual learning and private unlearning,” inConference on Lifelong Learning Agents. PMLR, 2022, pp. 243–254. 1, 2, 3, 10

work page 2022

[4] [3]

A unified framework for continual learning and unlearning,

R. Chatterjee, V . Chundawat, A. Tarun, A. Mali, and M. Mandal, “A unified framework for continual learning and unlearning,”

work page

[5] [4]

A continual learning survey: Defying forgetting in classification tasks,

M. D. Lange, R. Aljundi, M. Masana, S. Parisot, X. Jia, A. Leonardis, G. G. Slabaugh, and T. Tuytelaars, “A continual learning survey: Defying forgetting in classification tasks,”IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, pp. 3366–3385, 2019. 1, 2

work page 2019

[6] [5]

A comprehensive survey of continual learning: Theory, method and application,

L. Wang, X. Zhang, H. Su, and J. Zhu, “A comprehensive survey of continual learning: Theory, method and application,”IEEE transactions on pattern analysis and machine intelligence, vol. PP , 2023. 1, 2, 4

work page 2023

[7] [6]

Deep class-incremental learning: A survey,

D.-W. Zhou, Q. Wang, Z. Qi, H.-J. Ye, D. chuan Zhan, and Z. Liu, “Deep class-incremental learning: A survey,”ArXiv, vol. abs/2302.03648, 2023. 1, 2

arXiv 2023

[8] [7]

Machine unlearn- ing,

L. Bourtoule, V . Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot, “Machine unlearn- ing,”2021 IEEE Symposium on Security and Privacy (SP), pp. 141– 159, 2019. 1, 2

work page 2021