pith. sign in

arxiv: 2605.18610 · v1 · pith:IHQBVTQ5new · submitted 2026-05-18 · 💻 cs.CV · cs.AI· cs.LG

CATA: Continual Machine Unlearning via Conflict-Averse Task Arithmetic

Shen Lin , Junhao Dong , Rongjie Chen , Xiaoyu Zhang , Li Xu , Xiaofeng Chen This is my paper

Pith reviewed 2026-05-20 10:38 UTC · model grok-4.3

classification 💻 cs.CV cs.AIcs.LG
keywords continual machine unlearningvision-language modelstask arithmeticconflict-averse aggregationforgetting persistencemachine unlearningsequential updates
0
0 comments X

The pith

CATA performs sign-aware conflict-averse aggregation on historical unlearning task vectors to sustain forgetting effects in sequential requests for vision-language models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes CATA for continual machine unlearning in vision-language models that receive multiple sequential forget requests over time. It represents each request as an unlearning task vector and aggregates these vectors using a sign-aware conflict-averse method to suppress updates that conflict with prior forgetting. This targets the challenges of effective knowledge removal, preserving utility on kept data, and ensuring persistence so forgotten items do not re-emerge. Readers should care because practical model deployment involves ongoing privacy and copyright compliance needs that single unlearning methods cannot meet.

Core claim

CATA represents each forget request as an unlearning task vector. By maintaining historical task vectors and performing sign-aware conflict-averse aggregation, CATA suppresses conflicting update components that may weaken previous forgetting effects. Extensive experiments under both single-shot and continual settings show that CATA outperforms baselines in terms of forgetting effectiveness, model fidelity, and forgetting persistence.

What carries the argument

sign-aware conflict-averse aggregation of historical unlearning task vectors; this mechanism combines past and current forgetting directions while canceling out sign-inconsistent components that would reverse earlier unlearning

If this is right

  • Sequential unlearning requests maintain their effectiveness without undoing prior removals.
  • Model utility on retained knowledge stays high despite multiple updates.
  • Forgotten knowledge remains suppressed even after further continual unlearning steps.
  • The method works for both single forget requests and long sequences of them.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The technique could extend to non-vision language models or other continual editing scenarios.
  • It may lower the need for expensive full retraining when complying with data deletion laws.
  • Future work could explore dynamic weighting of historical vectors instead of uniform aggregation.

Load-bearing premise

Sign-aware aggregation of historical unlearning task vectors will reliably prevent re-emergence of previously forgotten knowledge across an arbitrary number of sequential requests without degrading retained utility.

What would settle it

Running a sequence of unlearning requests with CATA and then checking if prompts can still elicit responses based on the supposedly forgotten knowledge.

Figures

Figures reproduced from arXiv: 2605.18610 by Junhao Dong, Li Xu, Rongjie Chen, Shen Lin, Xiaofeng Chen, Xiaoyu Zhang.

Figure 1
Figure 1. Figure 1: Illustration of knowledge re￾emergence. Accuracy on target classes decreases after unlearning but partially recovers at the final step, indicating that previously forgotten knowl￾edge may be restored by later updates. In continual machine unlearning, a critical chal￾lenge is the knowledge re-emergence problem, where knowledge removed in earlier unlearning steps becomes accessible again after subsequent req… view at source ↗
Figure 2
Figure 2. Figure 2: An overview of our proposed CATA method. Each incoming forget set is converted into a [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Scalability evaluation on ImageNet-1K in continual unlearning. Each subplot shows the [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study of the scaling factor λ on ImageNet-1K in continual unlearning. Impact of the top-k% selection [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study of the top-k% selection on ImageNet-1K in continual unlearning. 6 Conclusion In this paper, we study continual machine unlearning for vision-language models, where sequential forget requests introduce unique challenges in effectiveness, fidelity, and persistence. Unlike single￾shot unlearning, this setting requires the model to remove newly specified target knowledge while preserving retaine… view at source ↗
read the original abstract

Vision-language models (VLMs) have shown remarkable ability in aligning visual and textual representations, enabling a wide range of multimodal applications. However, their large-scale training data inevitably raises concerns about privacy, copyright, and undesirable content, creating a strong need for machine unlearning. While existing studies mainly focus on single-shot unlearning, practical VLM deployment often involves sequential removal requests over time, giving rise to continual machine unlearning. In this work, we make the first attempt to study continual unlearning for VLMs and identify three key challenges in this setting: effectiveness in removing target knowledge, fidelity in preserving retained model utility, and persistence in preventing knowledge re-emergence under sequential updates. To address these challenges, we propose CATA, a conflict-averse task arithmetic method that represents each forget request as an unlearning task vector. By maintaining historical task vectors and performing sign-aware conflict-averse aggregation, CATA suppresses conflicting update components that may weaken previous forgetting effects. Extensive experiments under both single-shot and continual settings show that CATA outperforms baselines in terms of forgetting effectiveness, model fidelity, and forgetting persistence.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 3 minor

Summary. The manuscript proposes CATA, a conflict-averse task arithmetic method for continual machine unlearning in vision-language models. Each forget request is represented as an unlearning task vector; historical vectors are maintained and aggregated via sign-aware rules to suppress conflicting components that could weaken prior forgetting. The work targets three challenges—effectiveness of removal, fidelity to retained utility, and persistence against re-emergence under sequential requests—and reports outperformance over baselines in both single-shot and continual experimental settings.

Significance. If the empirical claims hold under closer scrutiny, the paper would constitute a useful first exploration of continual unlearning for VLMs, extending task-arithmetic ideas to a practically relevant sequential setting. The focus on persistence and the explicit handling of update conflicts addresses a gap left by single-shot unlearning methods. The approach is lightweight and does not require full retraining, which is a practical strength.

major comments (3)
  1. [§3.2] §3.2 (aggregation rule): The sign-aware conflict-averse aggregation is defined on task vectors but supplies no error bound or analysis of how successive linear combinations behave in the highly non-linear, high-dimensional parameter space of VLMs; this directly bears on the persistence claim.
  2. [Table 4] Table 4 (continual setting): Persistence metrics are reported for a modest number of sequential requests; the experiments do not include a stress test that increases sequence length to probe whether suppressed directions re-activate through higher-order interactions.
  3. [§4.3] §4.3 (baseline adaptation): It is unclear how the single-shot unlearning baselines were extended to the continual regime; without explicit adaptation details or ablations, the reported gains in effectiveness and fidelity cannot be fully attributed to the proposed aggregation.
minor comments (3)
  1. [Abstract] Abstract: The summary asserts outperformance on three axes but does not include any numerical values or dataset names; adding one or two key metrics would improve readability.
  2. [Figure 2] Figure 2: The diagram of the aggregation step would benefit from explicit annotation of the sign-aware operation and the role of historical vectors.
  3. [§3.1] Notation: The definition of the unlearning task vector (presumably Δθ_forget) is introduced without an equation number; assigning one would aid cross-referencing.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their thoughtful comments and positive evaluation of our work's significance. We address each of the major comments below, indicating where revisions will be made to the manuscript.

read point-by-point responses

  1. Referee: [§3.2] §3.2 (aggregation rule): The sign-aware conflict-averse aggregation is defined on task vectors but supplies no error bound or analysis of how successive linear combinations behave in the highly non-linear, high-dimensional parameter space of VLMs; this directly bears on the persistence claim.

    Authors: We agree that a theoretical analysis of the aggregation rule in the non-linear VLM parameter space would strengthen the persistence claims. However, providing rigorous error bounds for successive linear combinations in high-dimensional non-linear spaces is a complex undertaking that lies beyond the empirical focus of this paper. Task arithmetic methods commonly rely on the linear approximation assumption, as seen in prior work. In the revision, we will include a dedicated discussion section addressing the limitations of this approximation and how our empirical results on persistence mitigate concerns about higher-order interactions. We believe this provides a balanced view without overclaiming theoretical guarantees. revision: partial

  2. Referee: [Table 4] Table 4 (continual setting): Persistence metrics are reported for a modest number of sequential requests; the experiments do not include a stress test that increases sequence length to probe whether suppressed directions re-activate through higher-order interactions.

    Authors: We acknowledge that the current experiments use a moderate number of sequential unlearning requests. To more rigorously test the persistence under longer sequences, we will extend the experiments in Table 4 to include stress tests with increased sequence lengths. This will help demonstrate whether the conflict-averse aggregation continues to prevent re-emergence of forgotten knowledge in more demanding continual settings. We expect to report these additional results in the revised manuscript. revision: yes

  3. Referee: [§4.3] §4.3 (baseline adaptation): It is unclear how the single-shot unlearning baselines were extended to the continual regime; without explicit adaptation details or ablations, the reported gains in effectiveness and fidelity cannot be fully attributed to the proposed aggregation.

    Authors: We apologize for any ambiguity in the presentation. The single-shot baselines were extended to the continual setting by sequentially applying each unlearning method to the model for each new forget request, without incorporating the historical task vectors or the sign-aware aggregation proposed in CATA. This naive sequential application serves as the direct comparison. We will revise Section 4.3 to explicitly describe this adaptation procedure and include additional ablation studies that isolate the contribution of the conflict-averse aggregation. This will clarify how the performance improvements are attributable to our method. revision: yes

Circularity Check

0 steps flagged

No significant circularity: CATA extends task arithmetic with new aggregation rules supported by empirical results

full rationale

The paper introduces CATA as a method that represents each forget request as an unlearning task vector and applies sign-aware conflict-averse aggregation of historical vectors to suppress conflicts. This construction draws on prior task arithmetic literature without any self-referential definitions, fitted parameters renamed as predictions, or load-bearing self-citations that reduce the central claim to its own inputs. No equations in the abstract or described approach exhibit reduction by construction (e.g., no claim that the aggregation rule is derived from the target persistence metric itself). The persistence and fidelity claims are presented as outcomes of the proposed aggregation heuristic and are evaluated empirically rather than asserted via circular derivation. The approach is self-contained against external benchmarks of task arithmetic and continual learning, yielding an honest non-finding of circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central proposal rests on the domain assumption that unlearning requests can be represented as additive task vectors whose conflicts can be resolved by sign-aware aggregation without introducing new side effects.

axioms (1)
  • domain assumption Unlearning requests can be represented as task vectors that are additive and combinable via sign-aware rules
    Core mechanism described for CATA; invoked when stating how historical vectors are maintained and aggregated.

pith-pipeline@v0.9.0 · 5740 in / 1192 out tokens · 49131 ms · 2026-05-20T10:38:28.372745+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages

  1. [1]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervi- sion,” inInternational Conference on Machine Learning, 2021, pp. 8748–8763

    work page 2021

  2. [2]

    Trustworthy ai: From principles to practices,

    B. Li, P. Qi, B. Liu, S. Di, J. Liu, J. Pei, J. Yi, and B. Zhou, “Trustworthy ai: From principles to practices,”ACM Computing Surveys, vol. 55, no. 9, pp. 1–46, 2023

    work page 2023

  3. [3]

    Allies teach better than enemies: Inverse adversaries for robust knowledge distillation,

    J. Dong, R. Z. Moayedi, Y .-S. Ong, and S.-M. Moosavi-Dezfooli, “Allies teach better than enemies: Inverse adversaries for robust knowledge distillation,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

    work page 2026

  4. [4]

    Confound from all sides, distill with resilience: Multi-objective adversarial paths to zero-shot robustness,

    J. Dong, J. Liu, X. Qu, and Y .-S. Ong, “Confound from all sides, distill with resilience: Multi-objective adversarial paths to zero-shot robustness,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 624–634

    work page 2025

  5. [5]

    Tug-of-war no more: Harmonizing accuracy and robustness in vision-language models via stability-aware task vector merging,

    J. Dong, X. Qu, C. Zhang, S. Q. Rong, N. D. Thai, W. Pan, X. Li, T. Liu, P. Koniusz, and Y .-S. Ong, “Tug-of-war no more: Harmonizing accuracy and robustness in vision-language models via stability-aware task vector merging,” inThe Fourteenth International Conference on Learning Representations, 2026

    work page 2026

  6. [6]

    Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher,

    V . S. Chundawat, A. K. Tarun, M. Mandal, and M. Kankanhalli, “Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 37, 2023, pp. 7210–7217

    work page 2023

  7. [7]

    Erm-ktp: Knowledge-level machine unlearning via knowledge transfer,

    S. Lin, X. Zhang, C. Chen, X. Chen, and W. Susilo, “Erm-ktp: Knowledge-level machine unlearning via knowledge transfer,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 20 147–20 155

    work page 2023

  8. [8]

    Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary,

    M. Chen, W. Gao, G. Liu, K. Peng, and C. Wang, “Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 7766–7775

    work page 2023

  9. [9]

    Gdr-gma: Machine unlearning via direction- rectified and magnitude-adjusted gradients,

    S. Lin, X. Zhang, W. Susilo, X. Chen, and J. Liu, “Gdr-gma: Machine unlearning via direction- rectified and magnitude-adjusted gradients,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 9087–9095

    work page 2024

  10. [10]

    Robust machine unlearning for quantized neural networks via adaptive gradient reweighting with similar labels,

    Y . Tong, Y . Wang, J. Yuan, and C. Hu, “Robust machine unlearning for quantized neural networks via adaptive gradient reweighting with similar labels,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 20 603–20 612

    work page 2025

  11. [11]

    Reminiscence attack on residuals: Exploiting approximate machine unlearning for privacy,

    Y . Xiao, Q. Ye, L. Hu, H. Zheng, H. Hu, Z. Liang, H. Li, and Y . Jiao, “Reminiscence attack on residuals: Exploiting approximate machine unlearning for privacy,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 3058–3068

    work page 2025

  12. [12]

    Safe-clip: Removing nsfw concepts from vision-and-language models,

    S. Poppi, T. Poppi, F. Cocchi, M. Cornia, L. Baraldi, and R. Cucchiara, “Safe-clip: Removing nsfw concepts from vision-and-language models,” inEuropean Conference on Computer Vision, 2024, pp. 340–356

    work page 2024

  13. [13]

    Single image unlearning: Efficient machine unlearning in multimodal large language models,

    J. Li, Q. Wei, C. Zhang, G. Qi, M. Du, Y . Chen, S. Bi, and F. Liu, “Single image unlearning: Efficient machine unlearning in multimodal large language models,” inProceedings of the 38th International Conference on Neural Information Processing Systems, 2024, pp. 35 414–35 453

    work page 2024

  14. [14]

    Targeted unlearning with single layer unlearning gradient,

    Z. Cai, Y . Tan, and M. S. Asif, “Targeted unlearning with single layer unlearning gradient,” in International Conference on Machine Learning, 2025, pp. 6257–6290

    work page 2025

  15. [15]

    Cliperase: Efficient unlearning of visual-textual associations in clip,

    T. Yang, L. Dai, X. Wang, M. Cheng, Y . Tian, and X. Zhang, “Cliperase: Efficient unlearning of visual-textual associations in clip,” inProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, 2025, pp. 30 438–30 452

    work page 2025

  16. [16]

    Zero-shot class unlearning in clip with synthetic samples,

    A. Kravets and V . P. Namboodiri, “Zero-shot class unlearning in clip with synthetic samples,” in 2025 IEEE/CVF Winter Conference on Applications of Computer Vision, 2025, pp. 6456–6464

    work page 2025

  17. [17]

    Targeted forgetting of image subgroups in clip models,

    Z. Zhang, G. Liu, C. Fleming, R. R. Kompella, and C. Xu, “Targeted forgetting of image subgroups in clip models,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 9870–9880

    work page 2025

  18. [18]

    Visually grounded continual learning of compositional phrases,

    X. Jin, J. Du, A. Sadhu, R. Nevatia, and X. Ren, “Visually grounded continual learning of compositional phrases,” inProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 2020, pp. 2018–2029. 10

    work page 2020

  19. [19]

    Preventing zero-shot transfer degradation in continual learning of vision-language models,

    Z. Zheng, M. Ma, K. Wang, Z. Qin, X. Yue, and Y . You, “Preventing zero-shot transfer degradation in continual learning of vision-language models,” in2023 IEEE/CVF International Conference on Computer Vision, 2023, pp. 19 068–19 079

    work page 2023

  20. [20]

    Boosting continual learning of vision-language models via mixture-of-experts adapters,

    J. Yu, Y . Zhuge, L. Zhang, P. Hu, D. Wang, H. Lu, and Y . He, “Boosting continual learning of vision-language models via mixture-of-experts adapters,” in2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 23 219–23 230

    work page 2024

  21. [21]

    Continual vision-language representation learning with off-diagonal information,

    Z. Ni, L. Wei, S. Tang, Y . Zhuang, and Q. Tian, “Continual vision-language representation learning with off-diagonal information,” inProceedings of the 40th International Conference on Machine Learning, 2023, pp. 26 129–26 149

    work page 2023

  22. [22]

    Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms,

    J. Dong, P. Koniusz, X. Qu, and Y .-S. Ong, “Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms,” inProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V . 1, 2025, pp. 236–247

    work page 2025

  23. [23]

    C-CLIP: Multimodal continual learning for vision- language model,

    W. Liu, F. Zhu, L. Wei, and Q. Tian, “C-CLIP: Multimodal continual learning for vision- language model,” inThe Thirteenth International Conference on Learning Representations,

    work page

  24. [24]

    Available: https://openreview.net/forum?id=sb7qHFYwBc

    [Online]. Available: https://openreview.net/forum?id=sb7qHFYwBc

    work page

  25. [25]

    Multidelete for multimodal machine unlearning,

    J. Cheng and H. Amiri, “Multidelete for multimodal machine unlearning,” inEuropean Confer- ence on Computer Vision, 2024, pp. 165–184

    work page 2024

  26. [26]

    Editing models with task arithmetic,

    G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” inThe Eleventh International Conference on Learning Representations, 2023. [Online]. Available: https://openreview.net/forum?id=6t0Kwf8-jrj

    work page 2023

  27. [27]

    Imagenet: A large-scale hierarchical image database,

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009, pp. 248–255

    work page 2009

  28. [28]

    Learning multiple layers of features from tiny images,

    A. Krizhevsky, “Learning multiple layers of features from tiny images,”Master’s thesis, Univer- sity of Tront, 2009

    work page 2009

  29. [29]

    Machine unlearning of features and labels,

    A. Warnecke, L. Pirch, C. Wressnegger, and K. Rieck, “Machine unlearning of features and labels,” inProceedings 2023 Network and Distributed System Security Symposium, 2023

    work page 2023

  30. [30]

    Unrolling sgd: Understanding factors influencing machine unlearning,

    A. Thudi, G. Deza, V . Chandrasekaran, and N. Papernot, “Unrolling sgd: Understanding factors influencing machine unlearning,” in2022 IEEE 7th European Symposium on Security and Privacy, 2022, pp. 303–319

    work page 2022

  31. [31]

    Eternal sunshine of the spotless net: Selective forgetting in deep networks,

    A. Golatkar, A. Achille, and S. Soatto, “Eternal sunshine of the spotless net: Selective forgetting in deep networks,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 9304–9312

    work page 2020

  32. [32]

    An information theoretic approach to machine unlearning,

    J. Foster, K. Fogarty, S. Schoepf, Z. Dugue, C. Öztireli, and A. Brintrup, “An information theoretic approach to machine unlearning,” 2024. [Online]. Available: https://arxiv.org/abs/2402.01401

    work page arXiv 2024

  33. [33]

    Zero-shot machine unlearning,

    V . S. Chundawat, A. K. Tarun, M. Mandal, and M. Kankanhalli, “Zero-shot machine unlearning,” IEEE Transactions on Information Forensics and Security, vol. 18, pp. 2345–2354, 2023

    work page 2023

  34. [34]

    Food-101–mining discriminative components with random forests,

    L. Bossard, M. Guillaumin, and L. Van Gool, “Food-101–mining discriminative components with random forests,” inEuropean Conference on Computer Vision, 2014, pp. 446–461

    work page 2014

  35. [35]

    An analysis of single-layer networks in unsupervised feature learning,

    A. Coates, A. Ng, and H. Lee, “An analysis of single-layer networks in unsupervised feature learning,” inProceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, 2011, pp. 215–223

    work page 2011

  36. [36]

    Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models,

    A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz, “Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models,” inProceedings of the 33rd International Conference on Neural Information Processing Systems, 2019, pp. 9453–9463. 11 A Theoretical Analysis As a simple analysis, we...

    work page arXiv 2019