Pith. sign in

REVIEW 3 major objections 5 minor 3 references

Benchmarking Unlearning for Vision Transformers

T0 review · 3 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper benchmarks machine unlearning on Vision Transformers and finds that CNN-derived methods — NegGrad+, fine-tuning, SalUn — transfer to ViT and Swin-T, with NegGrad+ the most robust and SalUn unreliable against membership inference a

desk verdict A useful, reproducible benchmark substrate for VT unlearning, but the headline transfer claim is cushioned by a pretrained reference model; the numbers are still worth having. read the letter →

arxiv 2602.20114 v2 pith:TIL32H2A submitted 2026-02-23 cs.CV cs.AI

Kairan Zhao , Iurie Luca , Peter Triantafillou This is my paper · ORCID
classification cs.CVcs.AI
keywords machineunlearningvisiontransformersbenchmarkmemorizationNegGrad+ToWmetricsmembershipinferencecontinual
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether machine-unlearning methods designed for convolutional networks still work when the model being forgotten is a Vision Transformer. It benchmarks fine-tuning, gradient-ascent NegGrad+, and saliency-selective SalUn — each wrapped in a memorization-guided framework — across ViT and Swin-T families, four datasets, multiple model sizes, and single-shot and continual protocols. The central claim is that CNN-derived unlearning transfers to Vision Transformers and can be equally or more effective, with a key caveat: which method wins depends on the architecture and on the proxy used to estimate memorization. A sympathetic reader would care because privacy requests will increasingly target transformer-based vision models, and this provides the first reference point for how well existing algorithms cope. The paper also establishes that VTs memorize training data in the same long-tailed way as CNNs and that cheap memorization proxies remain trustworthy for VTs.

What carries the argument

The load-bearing mechanism is the RUM meta-algorithm, which partitions the forget set by memorization scores (low, medium, high) using a cheap proxy, then applies a base unlearning method sequentially from low to high memorization. Three base methods are wrapped inside RUM: fine-tuning (continue training on retain data), NegGrad+ (gradient ascent on forget examples combined with fine-tuning on retain), and SalUn (saliency-based parameter selection). Performance is scored with ToW and ToW-MIA, two composite metrics that multiply terms measuring how close the unlearned model is to a reference retrained model on forget accuracy (or MIA vulnerability), retain accuracy, and test accuracy. The mem

What would settle it

Retrain the reference model θ_r for ViT-Small and Swin-Tiny on CIFAR-10 with random initialization (no ImageNet pretraining) and recompute the ToW and ToW-MIA values shown in Table 13; if the VT-versus-CNN advantages on CIFAR-10 shrink or reverse, the paper's conclusion that pretraining helps unlearning is an artifact of the reference-model definition rather than of true forgetting.

Watch

Extended reading notes

Core claim

The paper's central discovery is that state-of-the-art unlearning algorithms inherited from CNNs do not collapse when moved to Vision Transformers. On the contrary, it shows that on harder datasets the ranking can favor VTs: with the Holdout Retraining proxy, Swin-Tiny using NegGrad+ reaches ToW 0.975 and ToW-MIA 0.902 on CIFAR-100, better than the corresponding ResNet results, while ViT-Small responds best to simple fine-tuning. At the same time, SalUn, which is competitive on the ToW accuracy-matching metric, is consistently weak on ToW-MIA — the metric reflecting resistance to membership inference — especially on CIFAR-100 and ImageNet-1K validation, making it unreliable for privacy-sensi

Load-bearing premise

The benchmark defines its success metrics relative to a 'retrained-from-scratch' model, but for Vision Transformers that reference is initialized from ImageNet pretraining and then fine-tuned on the retain set — so every headline comparison inherits the assumption that a pretrained-then-fine-tuned model is a faithful stand-in for a model that never saw the forget set.

Editorial extensions

If this is right

  • Practitioners can apply existing CNN unlearning methods to Vision Transformers immediately: NegGrad+ with Holdout Retraining is a robust default, and plain fine-tuning is a strong, cheap baseline for ViT architectures.
  • SalUn should be avoided when privacy protection against membership inference is required, as its ToW-MIA scores drop sharply on complex datasets (0.582 on CIFAR-10 for ViT-Small with Confidence; 0.647 on ImageNet-1K validation for Swin-Small with Holdout Retraining).
  • Memorization-guided unlearning is feasible for VTs without expensive retraining: Confidence and Holdout Retraining both give usable ranking signals, so future VT unlearning algorithms can reuse the proxy machinery developed for CNNs.
  • Architecture matters for method choice: ViT's global attention favors fine-tuning, Swin's local, windowed attention favors gradient-ascent-based NegGrad+, and there is a capacity sweet spot around ViT-Small and Swin-Tiny.
  • Continual unlearning with a strong method-proxy pair (NegGrad+ with Holdout Retraining) shows minimal degradation over five or even ten sequential forget operations on CIFAR-10/CIFAR-100, suggesting the protocol is stable for repeated privacy requests.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'retrained-from-scratch' reference model for VTs is actually ImageNet-pretrained and then fine-tuned on the retain set; if the intended target is a model that truly never saw the forget set, the reported ToW/ToW-MIA numbers on simple datasets likely overstate how much 'forgetting' is happening, since the reference already carries world knowledge from pretraining.
  • Holdout Retraining, despite lower correlation with true memorization, often outperforms Confidence as an unlearning driver — a sign that proxy fidelity is not the only factor, and that cheap proxies approximating decision-boundary shifts could be a productive research direction.
  • The architecture-specific pairings (ViT+fine-tuning, Swin+NegGrad+) suggest a design principle for future VT-specific unlearning methods: exploit the attention structure — targeted parameter updates in windowed attention for concentrated forgetting, and global regularization for diffuse ViT representations.
  • A stress test that would sharpen the benchmark: forget sets drawn adversarially (single-class clusters, or only high-memorization examples) rather than stratified low/medium/high, to see whether the sequential low-to-high ordering itself drives stability, or whether it is the proxy's ability to rank examples.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents what it describes as the first comprehensive benchmark of machine unlearning (MU) for Vision Transformers (ViT, Swin-T), comparing three representative MU algorithms (Fine-tune, NegGrad+, SalUn) instantiated within the RUM framework across four datasets (CIFAR-10, CIFAR-100, SVHN, ImageNet-1K validation) and multiple capacities, under both single-shot and continual unlearning protocols. It also studies memorization patterns and the validity of CNN-derived memorization proxies for VTs. The main empirical claim is that CNN-derived MU algorithms transfer to VTs and can be equally or more effective than on CNNs, with NegGrad+ being the most robust, Fine-tune best on ViT, and SalUn unreliable under ToW-MIA.

Significance. If the central claim is supported, the paper would provide a useful reference baseline for a gap in the MU literature: VTs are increasingly used but lack systematic MU evaluation. The work has clear practical value: detailed hyperparameter tables, three-seed runs with 95% confidence intervals, a vanilla-vs-RUM ablation, and public code support reproducibility. The memorization-proxy analysis (Table 1) and the architecture-level observations (ViT vs Swin-T) are interesting and could inform future method design. However, the significance is conditional on the evaluation reference model being appropriate, which is the focus of the major comments.

major comments (3)
  1. [§3.3, Eqs. (2)–(3); Appendix A.2.2, Table 8] The ToW and ToW-MIA metrics are defined relative to θ_r, described as a 'retrained-from-scratch' model. For VTs, however, θ_r is obtained by fine-tuning ImageNet-pretrained timm weights on the retain set (Table 8), not by training from random initialization. Table 2 shows the consequence: on CIFAR-10, ViT-Small and Swin-Tiny θ_r have >90% forget-set accuracy, while ResNet-18's θ_r is around 50%. Because both θ_o and θ_r share pretrained features, the 'Original' ToW values (e.g., 0.891–0.940 in Table 13) are already high without any unlearning. The reported VT advantage and the claim that CNN-derived algorithms transfer successfully may therefore be an artifact of the pretrained reference rather than evidence of actual forgetting. The text discloses the fine-tuning procedure but does not reconcile it with the from-scratch definition in Eqs. (2)–(3). This is a load-bearing internal mismatc
  2. [§4.4, Table 4] The ImageNet-1K 'larger-scale' experiment uses the validation split and states 'treat the remaining 47k images as retain/test set,' reporting ToW and ToW-MIA with two terms (retain/test accuracy and forget accuracy). This means the same 47k images are used for both the retain term and the test term of Eqs. (2)–(3). Consequently, the second and third factors in ToW are identical by construction, and the 'test' component is not evaluated on unseen data. This conflates retain and test and can inflate the reported ToW values, weakening the scalability conclusion. A separate held-out test set (or at least a clear statement that the 47k set serves only as a retain set, with the 'test' term removed) is needed to make the ImageNet-scale results interpretable.
  3. [§4.2, Key Takeaways; §6, insight (v)] The headline claim—'SOTA MU algorithms from CNNs can be equally (if not more) effective for VTs'—is measured exclusively against the fine-tuned θ_r. Given the first major comment, the high VT scores on simple datasets may reflect the proximity of the pretrained initialization rather than successful unlearning. For the claim to be supported, the benchmark must demonstrate that the unlearned models are close to a model that genuinely never saw the forget set. The paper's own Table 15 shows that after unlearning, VT forget accuracies are often much higher than θ_r's (e.g., Fine-tune on CIFAR-100 ViT-Small has 86.1% vs θ_r's 69.3%), which would lower ToW if θ_r were from-scratch. The current formulation obscures this. This is not an external-contrarian objection but an internal inconsistency between the stated definition of θ_r and its implementation.
minor comments (5)
  1. [Abstract and §1] The claim of being 'the first to attempt' benchmarking MU for VTs is too strong given that Cadet et al. (2024) and Cho et al. (2024) already include ViT-Tiny in their evaluations. Suggest rephrasing to 'the first systematic benchmark focused on VTs across families, capacities, and protocols.'
  2. [§2.3] There are formatting and typographical issues in the proxy enumeration: 'Learning Events Proxies(Confidence...' lacks spacing, and the list is run-on. Please clean up.
  3. [Appendix A.2.2, Table 9 caption] 'RUMF meta-algorithm' appears to be a typo for 'RUM framework.' Please correct.
  4. [Table 15 caption] 'Holdout Retaining' should be 'Holdout Retraining.'
  5. [§4.5 and Appendix A.7] The continual-unlearning results are reported only for the single best-performing configuration (NegGrad+ + HR). This is reasonable, but the paper should state explicitly that the conclusions about continual stability may not generalize to other algorithms or proxies.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark compares against an external retrained reference; the θ_r pretraining mismatch is a validity concern, not a definitional tautology.

full rationale

The paper is an empirical benchmark, not a derivation, and no reported quantity reduces to its own inputs by construction. The central metrics ToW and ToW-MIA (Eqs. 2–3) compare unlearned models θ_u to an externally obtained reference θ_r; θ_r is trained independently on the retain set and is not fitted from θ_u or from the metrics themselves. Memorization proxies are first validated by Spearman correlation against Feldman memorization scores (Table 1) before being used to partition forget sets, so proxy effectiveness is not assumed by construction. The RUM framework is self-cited (Zhao et al., 2024), but it is prior published work and the paper includes a vanilla-versus-RUM ablation (Table 12), so the main algorithm comparisons do not rest solely on the self-citation. The Skeptic's concern that θ_r for VTs is obtained by fine-tuning ImageNet-pretrained timm weights rather than training from scratch (Appendix A.2, Table 8) is a substantive threat to the interpretation of ToW as a measure of forgetting, but it is not circular: high ToW values may reflect pretraining transfer rather than unlearning, yet no equation or fitted parameter is being renamed as a prediction. Correctness risks of this kind fall outside the circularity definition required here. No load-bearing step in the paper is equivalent to its input by definition.

Assumptions & free parameters 5 free parameters · 7 assumptions · 0 invented entities

No new theoretical entities are introduced. The central claim rests on several empirical/measurement choices: the theta_r reference model, proxy-based forget-set construction, composite metrics, and the ImageNet validation-as-retain/test setup. These are domain assumptions rather than invented mechanisms.

free parameters (5)
  • SalUn saliency threshold gamma = 0.1 (ViT-Small), 0.3 (Swin-Tiny)
    Tuned per architecture in Table 9; the paper states ViT 'requires' 0.1 while Swin matches CNN-style 0.3. This tuning affects SalUn's relative ranking.
  • Unlearning learning rate = Fine-tune 1e-4; NegGrad+ 2e-5; SalUn 5e-5 (ViT) / 2e-4 (Swin)
    Chosen per method and architecture (Table 9); different rates could change method rankings.
  • Unlearn epochs per RUM partition = 5,5,10 for main protocol; 1,1,1 for continual
    Selected without justification in Table 9/Table 10; central to the RUM sequential protocol.
  • NegGrad+ beta = 0.97
    Fixed for all NegGrad+ runs (Table 9); adopted from prior work but not swept here.
  • Forget-set size and partition sizes = |D_f|=3000, M=3, N=1000; continual: 600, 3 partitions of 200
    Design choice in Section 4.2; affects all results and has no external justification.
assumptions (7)
  • domain assumption theta_r, the retrained model, is the correct gold standard for unlearning
    ToW/ToW-MIA (Eqs. 2-3) define forgetting as closeness to theta_r, but for VTs theta_r is fine-tuned from ImageNet pretrained weights (Table 8), not trained from scratch.
  • domain assumption Memorization proxies preserve the ordering of true Feldman memorization
    RUM partitions the forget set by proxy values; Table 1 shows correlations of only 0.45-0.64 for Holdout Retraining on ViT, so this ordering is only partially supported.
  • domain assumption ToW and ToW-MIA product metrics are valid summaries of unlearning quality
    These composite metrics are taken from prior work (Zhao et al. 2024) and are not independently validated here.
  • domain assumption A membership-inference classifier trained on retain/test losses generalizes to forget-set loss evaluation
    ToW-MIA assumes the MIA confidence on D_f reflects training membership; the paper does not calibrate this classifier.
  • ad hoc to paper The ImageNet-1K validation split can serve as both retain and test set for unlearning experiments
    Section 4.4 uses 50k validation images, removes 3k forget examples, and treats the remaining 47k as both retain and test, so there is no held-out test set.
  • domain assumption Fine-tuning timm ImageNet-pretrained weights is the appropriate training regime for VTs
    All VT models are initialized from ImageNet pretraining (Appendix A.2); this is what drives the high CIFAR-10 baseline and is not varied.
  • domain assumption Three runs with 95% confidence intervals are sufficient to support the ranking claims
    All main results are averaged over 3 runs; some CIs are wide, especially for SalUn, but the rankings are still treated as robust.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Unlearning for Vision Transformers." pith.science (2026). https://pith.science/paper/TIL32H2A

@misc{pith2026260220114,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Unlearning for Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TIL32H2A}},
  note         = {Machine review of arXiv:2602.20114}
}
read the original abstract

Machine unlearning (MU) refers to the post-training capability to remove (the influence of) training examples that are incorrect, biased, or leak sensitive/private information. MU is now widely regarded as critical for building safe and fair AI. In parallel, research into transformer architectures for computer vision has been highly successful: Vision Transformers (VTs) increasingly emerge as strong alternatives to CNNs. Yet, MU research for vision tasks has largely centered on CNNs, not VTs. While MU benchmarks have been developed for LLMs, diffusion models, and CNNs, none currently exist for VTs. This work is the first to attempt this, benchmarking MU algorithm performance across different VT families (ViT, Swin-T, and DINOv2) and at different capacities. The work employs (i) different datasets, selected to assess the impacts of dataset scale and complexity; (ii) different MU algorithms, selected to represent fundamentally different approaches for MU; and (iii) both single-shot and continual unlearning protocols. Additionally, it focuses on benchmarking MU algorithms that leverage training data memorization, since leveraging memorization has been recently discovered to significantly improve the performance of previously SOTA algorithms. En route, the work characterizes how VTs memorize training data relative to CNNs, and assesses the impact of different memorization proxies on performance. The benchmark uses unified evaluation metrics that capture two complementary notions of forget quality along with accuracy on unseen (test) data and on retained data. Overall, this work offers a benchmarking basis, enabling reproducible, fair, and comprehensive comparisons of existing (and future) MU algorithms on VTs. Importantly, for the first time, it sheds light on how well existing algorithms work in VT settings, establishing a promising reference performance baseline.

Figures

Figures reproduced from arXiv: 2602.20114 by the authors.

Figure 1
Figure 1. MU performance comparison on CIFAR-100. geted unlearning. In terms of algorithm performance, Fig￾ure 1 (and [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Continual Unlearning Performance Across Five Steps on CIFAR-100, reported with 95% confidence intervals. mains stable across steps, with no evidence of substan￾tial accumulated degradation in either ToW or ToW-MIA. These results reinforce the earlier observation that suitable method-proxy pairs can support continual unlearning with￾out progressive performance degradation. Key Takeaway. Minimal (if any) degradation o… view at source ↗
Figure 3
Figure 3. Memorization histograms across different architectures on CIFAR-100 (a) ResNet-18 (b) ViT-Small (c) Swin-Tiny [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Memorization histograms across different architectures on CIFAR-10 Conf(xi , yi) = 1 E X E e=1 Pθe (y = yi | xi), (4a) MaxConf(xi) = 1 E X E e=1 max y Pθe (y | xi), (4b) Ent(xi) = 1 E X E e=1 X y Pθe (y | xi) log Pθe (y | xi), (4c) BA(xi , yi) = 1 E X E e=1 1  arg max…
Figure 5
Figure 5. Figure 5: Architecture performance comparison on CIFAR-10 the Holdout Retraining proxy. A.8. Results for SVHN A.8.1. DETAILED TOW AND TOW-MIA RESULTS ACROSS VTS FOR SVHN To extend our study to a different dataset, we repeat the ex￾periments on SVHN using both ViT-Small and Swin-…
Figure 6
Figure 6. Figure 6: Continual Unlearning Performance Across Multiple Steps on CIFAR-10. A.8.2. CONTINUAL UNLEARNING RESULTS IN VTS FOR SVHN We follow the same procedure outlined in Section 4.5 to se￾quentially apply NegGrad+ and Holdout Retraining on the SVHN dataset over five consecutive…
Figure 7
Figure 7. Figure 7: MU performance comparison on SVHN 18 [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Continual Unlearning Performance Across Five Steps on SVHN 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 linked inside Pith

  1. [10]

    Retrain” modelsθ r as well as the respective “Unlearned

    and ResNet-50 (CIFAR-100). These metrics support the comparative analysis shown in Figures 1 and 5. Further breakdowns of ToW and ToW-MIA (by individual terms) for Vision Transformer architectures across datasets are re- ported in Table 15. A.7. Continual Unlearning Results in VTs for CIFAR-10, CIFAR-100 Table 16 and Table 17 presents detailed ToW and ToW...

  2. [2023]

    right to be forgotten

    URLhttps://arxiv.org/abs/2302. 09880. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Bar- rass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu B...

    arXiv 2024

  3. [2024]

    URLhttps://arxiv.org/abs/2410. 01276. Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In2015 IEEE Sympo- sium on Security and Privacy, pages 463–480, 2015. doi: 10.1109/SP.2015.35. Nicholas Carlini, ´Ulfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learn- ing: Metrics and appl...

    arXiv 2015

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.