{"id":"a39f29b8-4a64-4794-801c-f72df931248f","arxiv_id":"2602.20114","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CNN-derived unlearning methods largely transfer to Vision Transformers: Fine-tune works best on ViT, NegGrad+ on Swin, while SalUn fails privacy-style metrics.","lead":"This paper benchmarks machine-unlearning algorithms on Vision Transformers (ViT and Swin) across several datasets and model sizes, comparing their behavior to CNNs. It finds that CNN-style unlearning methods mostly transfer to transformers, with fine-tuning and NegGrad+ strongest and SalUn weak on privacy metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VT 'retrained' reference θ_r is fine-tuned from ImageNet weights, not trained from scratch, so high ToW on simple datasets can reflect pretraining transfer rather than forgetting; the headline transfer claim is not yet supported.","rationale":"The reader's weakest assumption is exactly the load-bearing issue: the θ_r reference for VTs is fine-tuned from pretrained ImageNet weights, not trained from scratch, and all headline metrics are measured against it. This concern is internal to the paper's own definition: Eqs. 2–3 define θ_r as retrained-from-scratch, while Appendix A.2.2 specifies ImageNet-pretrained initialization. The effect is concrete—Table 2 shows >90% forget accuracy for VT θ_r on CIFAR-10 vs ~50% for ResNet-18—and it directly saturates ToW/ToW-MIA, making the 'Original' baseline already high and unlearning gains appear marginal (as the paper itself notes in A.5.1). Without controlling for this, the central claim that CNN-derived algorithms are equally or more effective on VTs is not established for simple datasets; it may only show that pretrained models have transferable features. The concern does not invalidate the benchmark's other contributions (memorization patterns, proxy correlations, continual unlearning observations), and the paper's code release and RUM-vanilla ablation are positive evidence. The appropriate verdict remains CONDITIONAL: the central claim requires re-evaluation with a from-scratch θ_r or an explicit acknowledgment that the benchmark targets fine-tuned-pretrained VT unlearning, not unlearning to a random-initialized retrained model.","tokens_in":21767,"tokens_out":2580,"duration_ms":26705,"concrete_test":"Re-run the CIFAR-10 experiments with θ_r trained from random initialization for ViT-Small and Swin-Tiny, using the same retain set, optimizer, epochs, and hyperparameters as Table 8 (but without ImageNet-pretrained weights). Compute ToW and ToW-MIA for Original, Fine-tune, NegGrad+, and SalUn, and report θ_r's forget-set accuracy. If forget-set accuracy drops from >90% to around 50% and the 'Original' ToW drops from ~0.9 to the ResNet-18 level, then the central transfer claim is a pretraining artifact. Also report the individual Δa terms on D_f, D_r, and D_test for both θ_r variants.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('SOTA MU algorithms from CNNs can be equally/more effective for VTs') is measured by ToW/ToW-MIA, which are defined relative to θ_r, described as 'retrained-from-scratch' (Eqs. 2–3). However, for VTs θ_r is obtained by fine-tuning ImageNet-pretrained timm weights on the retain set (Appendix A.2.2, Table 8), not by training from random initialization. Table 2 shows the consequences: on CIFAR-10, ViT-Small and Swin-Tiny θ_r achieve >90% forget-set accuracy, while ResNet-18's θ_r is around 50%. This means that on simple datasets both θ_o and θ_r classify forget examples almost perfectly because of shared pretrained features, so 'Original' ToW values (e.g., 0.891–0.940) are already high without any unlearning. The reported VT advantage and the claim that existing algorithms transfer successfully may therefore be an artifact of the pretrained reference model rather than evidence of actual forgetting. The paper discloses the fine-tuning procedure but does not reconcile it with the from-scratch definition of θ_r; this is an internal mismatch in the evaluation protocol. Consequently, the headline comparison between VTs and CNNs is not measuring the same quantity on both sides.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents what it describes as the first comprehensive benchmark of machine unlearning (MU) for Vision Transformers (ViT, Swin-T), comparing three representative MU algorithms (Fine-tune, NegGrad+, SalUn) instantiated within the RUM framework across four datasets (CIFAR-10, CIFAR-100, SVHN, ImageNet-1K validation) and multiple capacities, under both single-shot and continual unlearning protocols. It also studies memorization patterns and the validity of CNN-derived memorization proxies for VTs. The main empirical claim is that CNN-derived MU algorithms transfer to VTs and can be equally or more effective than on CNNs, with NegGrad+ being the most robust, Fine-tune best on ViT, and SalUn unreliable under ToW-MIA.","tokens_in":22093,"tokens_out":3239,"duration_ms":31559,"significance":"If the central claim is supported, the paper would provide a useful reference baseline for a gap in the MU literature: VTs are increasingly used but lack systematic MU evaluation. The work has clear practical value: detailed hyperparameter tables, three-seed runs with 95% confidence intervals, a vanilla-vs-RUM ablation, and public code support reproducibility. The memorization-proxy analysis (Table 1) and the architecture-level observations (ViT vs Swin-T) are interesting and could inform future method design. However, the significance is conditional on the evaluation reference model being appropriate, which is the focus of the major comments.","major_comments":[{"comment":"The ToW and ToW-MIA metrics are defined relative to θ_r, described as a 'retrained-from-scratch' model. For VTs, however, θ_r is obtained by fine-tuning ImageNet-pretrained timm weights on the retain set (Table 8), not by training from random initialization. Table 2 shows the consequence: on CIFAR-10, ViT-Small and Swin-Tiny θ_r have >90% forget-set accuracy, while ResNet-18's θ_r is around 50%. Because both θ_o and θ_r share pretrained features, the 'Original' ToW values (e.g., 0.891–0.940 in Table 13) are already high without any unlearning. The reported VT advantage and the claim that CNN-derived algorithms transfer successfully may therefore be an artifact of the pretrained reference rather than evidence of actual forgetting. The text discloses the fine-tuning procedure but does not reconcile it with the from-scratch definition in Eqs. (2)–(3). This is a load-bearing internal mismatc","section":"§3.3, Eqs. (2)–(3); Appendix A.2.2, Table 8"},{"comment":"The ImageNet-1K 'larger-scale' experiment uses the validation split and states 'treat the remaining 47k images as retain/test set,' reporting ToW and ToW-MIA with two terms (retain/test accuracy and forget accuracy). This means the same 47k images are used for both the retain term and the test term of Eqs. (2)–(3). Consequently, the second and third factors in ToW are identical by construction, and the 'test' component is not evaluated on unseen data. This conflates retain and test and can inflate the reported ToW values, weakening the scalability conclusion. A separate held-out test set (or at least a clear statement that the 47k set serves only as a retain set, with the 'test' term removed) is needed to make the ImageNet-scale results interpretable.","section":"§4.4, Table 4"},{"comment":"The headline claim—'SOTA MU algorithms from CNNs can be equally (if not more) effective for VTs'—is measured exclusively against the fine-tuned θ_r. Given the first major comment, the high VT scores on simple datasets may reflect the proximity of the pretrained initialization rather than successful unlearning. For the claim to be supported, the benchmark must demonstrate that the unlearned models are close to a model that genuinely never saw the forget set. The paper's own Table 15 shows that after unlearning, VT forget accuracies are often much higher than θ_r's (e.g., Fine-tune on CIFAR-100 ViT-Small has 86.1% vs θ_r's 69.3%), which would lower ToW if θ_r were from-scratch. The current formulation obscures this. This is not an external-contrarian objection but an internal inconsistency between the stated definition of θ_r and its implementation.","section":"§4.2, Key Takeaways; §6, insight (v)"}],"minor_comments":[{"comment":"The claim of being 'the first to attempt' benchmarking MU for VTs is too strong given that Cadet et al. (2024) and Cho et al. (2024) already include ViT-Tiny in their evaluations. Suggest rephrasing to 'the first systematic benchmark focused on VTs across families, capacities, and protocols.'","section":"Abstract and §1"},{"comment":"There are formatting and typographical issues in the proxy enumeration: 'Learning Events Proxies(Confidence...' lacks spacing, and the list is run-on. Please clean up.","section":"§2.3"},{"comment":"'RUMF meta-algorithm' appears to be a typo for 'RUM framework.' Please correct.","section":"Appendix A.2.2, Table 9 caption"},{"comment":"'Holdout Retaining' should be 'Holdout Retraining.'","section":"Table 15 caption"},{"comment":"The continual-unlearning results are reported only for the single best-performing configuration (NegGrad+ + HR). This is reasonable, but the paper should state explicitly that the conclusions about continual stability may not generalize to other algorithms or proxies.","section":"§4.5 and Appendix A.7"}],"recommendation":"major_revision","confidential_remarks":"The paper is heavily built on the authors' previous RUM framework, and the main evaluations all use RUM-integrated algorithms. While the vanilla-vs-RUM ablation in Table 12 is helpful, the framing in the abstract and intro presents the algorithms as standard baselines without emphasizing that the reported performance is for RUM-integrated versions. This should be disclosed more prominently. The θ_r issue is the core technical concern: if the authors cannot provide a from-scratch reference, the benchmark still has value as a comparison among unlearning algorithms, but the headline claim about transferability to VTs must be substantially softened. I recommend major revision with a request that the authors either add from-scratch reference experiments on a subset (e.g., CIFAR-10 with ViT-Small) or explicitly re-scope the claims to 'closeness to a fine-tuned retain model.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a solid empirical benchmark for machine unlearning on vision transformers. The new measurements are real: memorization-proxy correlations for ViT/Swin, unlearning scores across two VT families, capacities, four datasets, two proxies, and a continual protocol, all with code, three seeds, and 95% CIs. The RUM-vs-vanilla ablation (Table 12) is a good control and shows the framework does something. The architecture-method pairings (ViT + Fine-tune, Swin + NegGrad+ + HR) are actionable and likely to be cited.\n\nThe main soft spot is the theta_r reference. Equations 2–3 call it \"retrained-from-scratch,\" but for VTs it is fine-tuned from ImageNet-pretrained timm weights (Table 8). Table 2 shows the consequence: on CIFAR-10, theta_r for ViT-Small and Swin-Tiny gets >90% forget-set accuracy, while ResNet-18's theta_r sits around 50%. That pushes \"Original\" ToW to 0.89–0.94 before any unlearning. The paper openly discusses the pretraining advantage, but it never reconciles it with the from-scratch definition. As a result, the headline \"equally (if not more) effective\" comparison between VTs and CNNs is not measuring the same quantity on both sides—high VT scores on simple datasets may reflect transferable features rather than forgetting. This needs to be addressed, either by retraining VTs from scratch where feasible or by redefining theta_r and reinterpreting the baselines.\n\nTwo smaller issues. The ImageNet-scale experiment has no held-out test set; the 47k non-forget images are treated as retain/test together, so the two-term ToW is weaker than the four-dataset comparisons. And the forget set is proxy-stratified rather than random, which limits external validity. The \"first VT benchmark\" claim is also overstated given Cho et al. and Cadet et al. already evaluated unlearning on ViTs, even if not at this scale or depth.\n\nNone of this is fatal. The paper ships code, reports hyperparameters, gives confidence intervals, and the limitations section is honest but thin. It is a good substrate for future work, and the central claims are addressable in revision.\n\nAnyone working on unlearning for vision transformers should read this. I would send it to peer review—the issues are fixable and the benchmark is worth having in the literature. I'd cite it for the measurements.","headline":"A useful, reproducible benchmark substrate for VT unlearning, but the headline transfer claim is cushioned by a pretrained reference model; the numbers are still worth having.","tokens_in":744,"tokens_out":924,"would_cite":true,"duration_ms":26410,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper benchmarks machine unlearning on Vision Transformers and finds that CNN-derived methods — NegGrad+, fine-tuning, SalUn — transfer to ViT and Swin-T, with NegGrad+ the most robust and SalUn unreliable against membership inference a","keywords":["machine unlearning","vision transformers","benchmark","memorization","NegGrad+","ToW metrics","membership inference","continual unlearning"],"falsifier":"Retrain the reference model θ_r for ViT-Small and Swin-Tiny on CIFAR-10 with random initialization (no ImageNet pretraining) and recompute the ToW and ToW-MIA values shown in Table 13; if the VT-versus-CNN advantages on CIFAR-10 shrink or reverse, the paper's conclusion that pretraining helps unlearning is an artifact of the reference-model definition rather than of true forgetting.","tokens_in":21621,"feed_emoji":"🔄","tokens_out":6141,"duration_ms":49202,"temperature":0.7,"pith_summary":"This paper asks whether machine-unlearning methods designed for convolutional networks still work when the model being forgotten is a Vision Transformer. It benchmarks fine-tuning, gradient-ascent NegGrad+, and saliency-selective SalUn — each wrapped in a memorization-guided framework — across ViT and Swin-T families, four datasets, multiple model sizes, and single-shot and continual protocols. The central claim is that CNN-derived unlearning transfers to Vision Transformers and can be equally or more effective, with a key caveat: which method wins depends on the architecture and on the proxy used to estimate memorization. A sympathetic reader would care because privacy requests will increasingly target transformer-based vision models, and this provides the first reference point for how well existing algorithms cope. The paper also establishes that VTs memorize training data in the same long-tailed way as CNNs and that cheap memorization proxies remain trustworthy for VTs.","feed_headline":"CNN unlearning methods carry over to Vision Transformers","feed_subtitle":"First benchmark measures forgetting quality across ViT and Swin-T, finds NegGrad+ most robust and fine-tuning best for ViT.","key_machinery":"The load-bearing mechanism is the RUM meta-algorithm, which partitions the forget set by memorization scores (low, medium, high) using a cheap proxy, then applies a base unlearning method sequentially from low to high memorization. Three base methods are wrapped inside RUM: fine-tuning (continue training on retain data), NegGrad+ (gradient ascent on forget examples combined with fine-tuning on retain), and SalUn (saliency-based parameter selection). Performance is scored with ToW and ToW-MIA, two composite metrics that multiply terms measuring how close the unlearned model is to a reference retrained model on forget accuracy (or MIA vulnerability), retain accuracy, and test accuracy. The mem","core_discovery":"The paper's central discovery is that state-of-the-art unlearning algorithms inherited from CNNs do not collapse when moved to Vision Transformers. On the contrary, it shows that on harder datasets the ranking can favor VTs: with the Holdout Retraining proxy, Swin-Tiny using NegGrad+ reaches ToW 0.975 and ToW-MIA 0.902 on CIFAR-100, better than the corresponding ResNet results, while ViT-Small responds best to simple fine-tuning. At the same time, SalUn, which is competitive on the ToW accuracy-matching metric, is consistently weak on ToW-MIA — the metric reflecting resistance to membership inference — especially on CIFAR-100 and ImageNet-1K validation, making it unreliable for privacy-sensi","pith_inferences":["The 'retrained-from-scratch' reference model for VTs is actually ImageNet-pretrained and then fine-tuned on the retain set; if the intended target is a model that truly never saw the forget set, the reported ToW/ToW-MIA numbers on simple datasets likely overstate how much 'forgetting' is happening, since the reference already carries world knowledge from pretraining.","Holdout Retraining, despite lower correlation with true memorization, often outperforms Confidence as an unlearning driver — a sign that proxy fidelity is not the only factor, and that cheap proxies approximating decision-boundary shifts could be a productive research direction.","The architecture-specific pairings (ViT+fine-tuning, Swin+NegGrad+) suggest a design principle for future VT-specific unlearning methods: exploit the attention structure — targeted parameter updates in windowed attention for concentrated forgetting, and global regularization for diffuse ViT representations.","A stress test that would sharpen the benchmark: forget sets drawn adversarially (single-class clusters, or only high-memorization examples) rather than stratified low/medium/high, to see whether the sequential low-to-high ordering itself drives stability, or whether it is the proxy's ability to rank examples."],"forward_implications":["Practitioners can apply existing CNN unlearning methods to Vision Transformers immediately: NegGrad+ with Holdout Retraining is a robust default, and plain fine-tuning is a strong, cheap baseline for ViT architectures.","SalUn should be avoided when privacy protection against membership inference is required, as its ToW-MIA scores drop sharply on complex datasets (0.582 on CIFAR-10 for ViT-Small with Confidence; 0.647 on ImageNet-1K validation for Swin-Small with Holdout Retraining).","Memorization-guided unlearning is feasible for VTs without expensive retraining: Confidence and Holdout Retraining both give usable ranking signals, so future VT unlearning algorithms can reuse the proxy machinery developed for CNNs.","Architecture matters for method choice: ViT's global attention favors fine-tuning, Swin's local, windowed attention favors gradient-ascent-based NegGrad+, and there is a capacity sweet spot around ViT-Small and Swin-Tiny.","Continual unlearning with a strong method-proxy pair (NegGrad+ with Holdout Retraining) shows minimal degradation over five or even ten sequential forget operations on CIFAR-10/CIFAR-100, suggesting the protocol is stable for repeated privacy requests."],"fun_headline_variants":["Vision Transformers accept CNN unlearning methods","ViT unlearning: CNN methods work, but fine-tuning is best","NegGrad+ excels on Swin-T, fine-tuning on ViT","SalUn fails privacy metric in ViT unlearning benchmark","First unlearning benchmark for ViTs: CNN methods transfer"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark defines its success metrics relative to a 'retrained-from-scratch' model, but for Vision Transformers that reference is initialized from ImageNet pretraining and then fine-tuned on the retain set — so every headline comparison inherits the assumption that a pretrained-then-fine-tuned model is a faithful stand-in for a model that never saw the forget set.","fun_headline_variants_meta":{"raw":{"variants":["Vision Transformers accept CNN unlearning methods","ViT unlearning: CNN methods work, but fine-tuning is best","NegGrad+ excels on Swin-T, fine-tuning on ViT","SalUn fails privacy metric in ViT unlearning benchmark","First unlearning benchmark for ViTs: CNN methods transfer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001131,"raw_usage":{"total_tokens":4604,"prompt_tokens":875,"completion_tokens":3729,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":3644}},"tokens_in":619,"tokens_out":3729,"duration_ms":25914,"temperature":1.0,"reasoning_tokens":3644,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T21:24:18.820206+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the reference model θ_r for ViT-Small and Swin-Tiny on CIFAR-10 with random initialization (no ImageNet pretraining) and recompute the ToW and ToW-MIA values shown in Table 13; if the VT-versus-CNN advantages on CIFAR-10 shrink or reverse, the paper's conclusion that pretraining helps unlearning is an artifact of the reference-model definition rather than of true forgetting.","supporting_citations":[],"review_version":1}