REVIEW 3 major objections 8 minor 19 references
A Survey of Adversarial Efficiency Degradation for Vision Transformer by Exploiting Input-adaptive Optimization
T0 review · 3 major / 8 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read This survey argues that input-adaptive token pruning in Vision Transformers creates a compute pathway that adversarial patches and perturbations can steer toward near-dense cost, and standardizes cross-paper comparison with an Attack…
desk verdict Useful survey of efficiency-degradation attacks on ViTs, but the central unified comparison is internally inconsistent and the tables cannot be trusted as published. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the compute policy $\pi(x)$ that maps an input to kept/removed computation (token masks, halting indices, activated components). In each framework, $\pi$ is a function of differentiable statistics: A-ViT's halting scores $h_j^l$ accumulate across layers and a token halts when the cumulative score crosses $1-\tau$; ATS samples survivors from class-token attention weighted by value norms; AdaViT uses Gumbel-Softmax decision networks over blocks/heads/patches. Because these signals are smooth functions of pixels, an adversary can push their distributions via projected gradient steps (DeSparsify) or a learned universal patch (SlowFormer) to delay halting, flatten attention, or force masks to 'activate,' restoring near-dense compute. The survey's standardized Attack Success metric $\text{AS} = (F_{\text{attack}} - F_{\min})/(F_{\max} - F_{\min})$ quantifies how much of the clean-model savings is cancelled.
What would settle it
Inspect the two original papers' reported GFLOPs for A-ViT on ViT-Small or DeiT-Small under no attack: if the true baseline is neither 3.70 nor 0.87 GFLOPs, or if DeiT-S and ViT-S differ measurably in FLOPs beyond rounding, the cross-paper comparison loses its basis.
Extended reading notes
Core claim
The paper's central claim is that input-adaptive inference methods such as token pruning compute a policy $\pi(x)$ from differentiable, image-conditioned statistics—attention maps, halting scores, mask logits—and that this policy is as steerable from pixel space as a decision boundary is in conventional adversarial attacks. SlowFormer and DeSparsify both exploit this: they optimize a patch or perturbation to shift the policy into a 'keep-more, exit-later' regime, pushing computational cost back toward the dense baseline while preserving the label. The unified tables show this effect across A-ViT, ATS, and AdaViT, with Attack Success rates up to 100% on A-ViT, and the paper concludes that efficiency robustness does not currently come with guarantees: the best defenses reduce Attack Success substantially (e.g., from 100% to 34%–37%) but leave residual compute inflation.
Load-bearing premise
The load-bearing premise is that numeric results from two independent papers—SlowFormer on ViT-Small and DeSparsify on DeiT-Small—can be transcribed and compared directly, treating the two architectures as equivalent; the survey's own tables contradict this by reporting 3.70 GFLOPs for A-ViT no-attack in one table and 0.87 in another.
Editorial extensions
If this is right
- Token-pruned ViTs deployed on edge devices carry a security risk: a universal physical patch could erase a large fraction of the intended FLOPs savings without perceptible changes.
- Because the compute policy is differentiable, defenses must either bound the policy's output (confidence-based caps, token retention limits) or desensitize it to pixel perturbations (adversarial training), both shown to cut Attack Success substantially.
- The Attack Success metric enables meaningful comparison of attacks across papers, provided the dense baseline and the pruned baseline are reported consistently.
- Efficiency attacks do not transfer well across pruning frameworks, so a single input will not degrade all ViTs equally.
- FLOPs-only reporting understates the impact; DeSparsify is reported to raise energy by about 72% and memory by about 37% under attack.
Reading between the lines
- The tables' internal inconsistency—A-ViT no-attack GFLOPs of 3.70 in Table 2 versus 0.87 in Table 3—suggests the transcribed baselines may come from different experimental setups, and any reader wanting to rely on the comparative ranking should re-pull the numbers from the original papers.
- The AS normalization $F_{\max} - F_{\min}$ means that a framework with higher clean FLOPs will score lower AS for the same absolute FLOPs added, so AS is a relative measure, not an absolute robustness gauge.
- A natural next step, not taken in the survey, is to test whether the same attacks transfer to modern token-merging methods (e.g., ToMe) or to black-box settings with randomized pruning policies.
- A defense that caps per-block active tokens could be combined with injected randomness to make it harder to reverse-engineer, since the survey notes that handcrafted thresholds are easy to attack.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript surveys adversarial efficiency-degradation attacks on input-adaptive Vision Transformers. It organizes the topic around three research questions, explains in Section 3 how token-pruning mechanisms (A-ViT, ATS, AdaViT) create a compute policy that is steerable from pixel space, and describes the two representative attacks: SlowFormer, a universal adversarial patch, and DeSparsify, a per-image bounded perturbation. The stated contribution is a unified, standardized comparison of the two attacks, operationalized through GFLOPs, accuracy loss, and an Attack Success (AS) metric defined as AS = (F_attack − F_min)/(F_max − F_min) × 100%. The paper also summarizes two defenses (adversarial training against universal patches and confidence-based token caps) and outlines limitations and future research directions. The quantitative backbone of the survey is the cross-paper comparison presented in Tables 2 and 3.
Significance. The topic is timely: efficiency-degradation attacks on adaptive ViTs are a genuine and under-organized security concern, and the mechanistic framing of Section 3.1 (the compute policy is a differentiable function of pixels, hence as steerable as a decision boundary) is a useful conceptual contribution. The AS metric is a reasonable normalization proposal, the defense summary in Section 4 is helpful, and Section 6 is honest about the limits of current attacks and defenses. I agree with the reader's report that there is no circularity concern: AS is a deterministic normalization of published GFLOPs values, not a fitted quantity. However, the survey's usefulness is conditional on the reliability of its central empirical artifact, and, as detailed below, Tables 2 and 3 contain load-bearing internal contradictions. Under corrected numbers, the qualitative conclusions may change materially, for instance whether SlowFormer restores compute close to the dense baseline on A-ViT.
major comments (3)
- [Section 5.1, Table 3] The AS column of Table 3 is not reproducible from the formula stated in Section 5.1. For the SlowFormer/A-ViT rows, the formula AS = (F_attack − F_min)/(F_max − F_min) × 100% with F_min = 0.87 (the table's own no-attack entry) and F_max = 4.60 yields (1.26 − 0.87)/(4.60 − 0.87) ≈ 10.5% for no defense and (1.01 − 0.87)/(4.60 − 0.87) ≈ 3.8% with defense, not the reported 100% and 34%. The reported 100% can be obtained only by taking F_max equal to the attacked value (1.26), a normalization that contradicts the definition used in Table 2. The DeSparsify rows deviate as well: A-ViT with defense gives (3.95 − 3.70)/(4.60 − 3.70) ≈ 27.8%, not 36.9%, and ATS with defense gives ≈ 4.7%, not 5.3%. Because the AS column is the paper's stated mechanism for making cross-paper comparisons concrete (Abstract; Section 5.1; Section 7), this non-reproducibility is load-bearing for the central claim.
- [Tables 2 and 3] The two tables give mutually incompatible numbers for the same nominal setup. Table 2 lists A-ViT no-attack GFLOPs as 3.70 for the SlowFormer/ViT-S condition, while Table 3 lists A-ViT no-attack GFLOPs as 0.87 for the same condition (SlowFormer on ViT-S with A-ViT), a factor-of-4.25 discrepancy that cannot be explained by the 'DeiT-S versus ViT-S' labeling in Section 5.1. The SlowFormer attack GFLOPs on A-ViT are likewise 4.60 in Table 2 but 1.26 in Table 3. Section 5.1 asserts that 'the GFLOPs and accuracy-loss comparisons are directly meaningful across the two sources,' but both tables cannot be correct for the same setup, so the unified comparison rests on internally contradictory data.
- [Table 2] Table 2 reports SlowFormer accuracy losses of 76.5%, 78.2%, and 76.9% on A-ViT, ATS, and AdaViT, respectively, but these values are never explained or discussed. They are difficult to reconcile with the paper's framing of efficiency-degradation attacks: Section 2.3 states that 'as a secondary objective, the adversary seeks to enhance the stealthiness of the attack by preserving the model's original prediction,' and Section 3.2 describes the accuracy term as optional. The contrast with DeSparsify's 0.1–1.2% losses is stark, and a model that loses roughly 77–78% of its accuracy is not operating in the same regime as one that loses 0.1–1.2%. As written, the accuracy-loss asymmetry confounds the efficiency comparison and the claim of a 'consistent' picture across the two attacks; the survey needs either to explain why SlowFormer's accuracy drops are so large or to restrict its comparison to the accuracy-preserving variant.
minor comments (8)
- [Section 5.1 heading] The heading 'Comparsion between patch and single efficiency adversarial attack' contains a spelling error: 'Comparsion' should be 'Comparison'.
- [Section 2.2] 'removes uniformative to-kens' should read 'removes uninformative tokens'.
- [Section 4] The phrase 'specifically against to the two attacks' should be 'specifically against the two attacks'.
- [Table 1] 'CIF AR-10' appears twice in Table 1 and should read 'CIFAR-10'.
- [Section 5.1] The sentence 'Table 2 can not only increase computation but also reduce model accuracy' has the wrong subject; it is the attacks, not the table, that increase computation.
- [Section 3.2] The surrogate loss formula for A-ViT uses a non-standard indicator symbol and an ambiguous ℓMSE target; it should be typeset conventionally with the target explicitly defined.
- [Table 3] The no-defense AS values in Table 3 for ATS (73.5%) and AdaViT (43.2%) differ slightly from the corresponding values in Table 2 (73.3% and 43.4%) for the same conditions; these should be reconciled.
- [Table 3] The dataset and configuration underlying the SlowFormer rows of Table 3 are not stated; it is unclear whether the 0.87 GFLOPs no-attack figure refers to ImageNet or CIFAR-10.
Circularity Check
No significant circularity: the AS metric is a normalization formula over published GFLOPs, and the survey's quantitative claims rest on external sources, not on self-citations.
full rationale
The paper's central artifact is the Attack Success formula AS = (F_attack - F_min)/(F_max - F_min) × 100, defined explicitly in Section 5.1. This is a normalization of published GFLOPs values, not a fitted parameter and not a prediction derived from fitted inputs. The survey transcribes attack results from SlowFormer and DeSparsify and applies this formula to compare them; the comparison is therefore a re-expression of the original papers' empirical numbers, not a derivation that reduces to its inputs by construction. The self-citations (references 7-11) support background statements about conventional adversarial attacks and adversarial patches; they are not load-bearing for the survey's comparative claim and no uniqueness theorem or ansatz is imported from the authors' prior work. The internal inconsistency between Table 2 (A-ViT no-attack 3.70 GFLOPs) and Table 3 (A-ViT no-attack 0.87 GFLOPs), and the fact that the reported AS=100% for SlowFormer/A-ViT in Table 3 does not follow from the stated formula, is a serious correctness and reproducibility defect, but it is not circularity: the formula is being misapplied or the numbers mis-transcribed, not used tautologically. Similarly, the assumption that DeiT-S and ViT-S are interchangeable is an empirical comparability assumption, not a definitional equivalence. No step in the paper's claimed derivation chain equates an output to an input by definition or by self-citation, so the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The reported GFLOPs, accuracy, and attack success numbers from SlowFormer and DeSparsify are accurate and directly comparable.
- domain assumption DeiT-S and ViT-S can be treated as the same architecture for cross-paper comparison.
- domain assumption The three token-pruning frameworks, A-ViT, ATS, and AdaViT, are representative of input-adaptive inference in ViTs.
Cite this review
Pith. "Pith review of A Survey of Adversarial Efficiency Degradation for Vision Transformer by Exploiting Input-adaptive Optimization." pith.science (2026). https://pith.science/paper/XW5IRD2D
@misc{pith2026260805217,
author = {Pith},
title = {Pith review of: A Survey of Adversarial Efficiency Degradation for Vision Transformer by Exploiting Input-adaptive Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/XW5IRD2D}},
note = {Machine review of arXiv:2608.05217}
}
read the original abstract
Vision Transformers (ViTs) increasingly rely on input-adaptive inference, such as token pruning and early halting, to meet energy and latency budgets. This survey examines a recent class of adversarial efficiency degradation attacks that target these mechanisms to increase computation without necessarily degrading accuracy. We unify and compare two representative attacks, SlowFormer (a universal adversarial patch) and DeSparsify (per-image perturbations), across three popular token-pruning frameworks: A-ViT, ATS, and AdaViT. We standardize reporting using GFLOPs, accuracy loss, and an Attack Success (AS) metric that measures how much of the model's compute savings the attack takes away. Understanding these attacks is crucial for designing countermeasures that not only mitigate risk but also remain lightweight, since deployment often occurs in low-power settings such as mobile or embedded devices. To organize our analysis, we focus on three questions: how input-adaptive optimizations (e.g., token pruning and early halting) create attack surfaces for efficiency degradation; how such attacks operate in practice and which optimizations are most vulnerable; and which defenses exist today and whether they meaningfully restore efficiency under attack.
Figures
Reference graph
Works this paper leans on
-
[1]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017
2017
-
[2]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Un- terthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020
arXiv 2010
-
[3]
Adaptive token sampling for efficient vision transformers,
M. Fayyaz, S. A. Koohpayegani, F. R. Jafari, S. Sengupta, H. R. V. Joze, E. Som- merlade, H. Pirsiavash, and J. Gall, “Adaptive token sampling for efficient vision transformers,” inEuropean conference on computer vision. Springer, 2022, pp. 396–414
work page 2022
-
[4]
Adavit: Adaptive vision transformers for efficient image recognition,
L. Meng, H. Li, B.-C. Chen, S. Lan, Z. Wu, Y.-G. Jiang, and S.-N. Lim, “Adavit: Adaptive vision transformers for efficient image recognition,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12 309–12 318
work page 2022
-
[5]
A-vit: Adaptive tokens for efficient vision transformer,
H. Yin, A. Vahdat, J. M. Alvarez, A. Mallya, J. Kautz, and P. Molchanov, “A-vit: Adaptive tokens for efficient vision transformer,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 809–10 818
work page 2022
-
[6]
Slowformer: Adversarial attack on compute and energy consumption of efficient vision trans- formers,
K. Navaneet, S. A. Koohpayegani, E. Sleiman, and H. Pirsiavash, “Slowformer: Adversarial attack on compute and energy consumption of efficient vision trans- formers,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24 786–24 797
work page 2024
-
[7]
Curse of dimensionality in adversarial examples,
N. Chattopadhyay, A. Chattopadhyay, S. S. Gupta, and M. Kasper, “Curse of dimensionality in adversarial examples,” in2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019, pp. 1–8
2019
-
[8]
Robustness against ad- versarial attacks using dimensionality,
N. Chattopadhyay, S. Chatterjee, and A. Chattopadhyay, “Robustness against ad- versarial attacks using dimensionality,” inInternational Conference on Security, Privacy, and Applied Cryptography Engineering. Springer, 2021, pp. 226–241
work page 2021
Show all 19 references
-
[9]
Robust perception for au- tonomous vehicles using dimensionality reduction,
S. Garg, N. Chattopadhyay, and A. Chattopadhyay, “Robust perception for au- tonomous vehicles using dimensionality reduction,” in2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom). IEEE, 2022, pp. 1516–1521
2022
-
[10]
Oddr: Out- lier detection & dimension reduction based defense against adversarial patches,
N. Chattopadhyay, A. Guesmi, M. A. Hanif, B. Ouni, and M. Shafique, “Oddr: Out- lier detection & dimension reduction based defense against adversarial patches,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 22 999–23 008
2025
-
[11]
Anomaly unveiled: Se- curing image classification against adversarial patch attacks,
N. Chattopadhyay, A. Guesmi, and M. Shafique, “Anomaly unveiled: Se- curing image classification against adversarial patch attacks,”arXiv preprint arXiv:2402.06249, 2024
2024 arXiv
-
[12]
Ilfo: Adversarial attack on adap- tive neural networks,
M. Haque, A. Chauhan, C. Liu, and W. Yang, “Ilfo: Adversarial attack on adap- tive neural networks,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14 264–14 273
2020
-
[13]
Skipnet: Learning dynamic routing in convolutional networks,
X. Wang, F. Yu, Z.-Y. Dou, T. Darrell, and J. E. Gonzalez, “Skipnet: Learning dynamic routing in convolutional networks,” inProceedings of the European con- ference on computer vision (ECCV), 2018, pp. 409–424
2018
-
[14]
Spatially adaptive computation time for residual networks,
M. Figurnov, M. D. Collins, Y. Zhu, L. Zhang, J. Huang, D. Vetrov, and R. Salakhutdinov, “Spatially adaptive computation time for residual networks,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 1039–1048. 14 Anadi Goyal et al
2017
-
[15]
Gradauto: Energy- oriented attack on dynamic neural networks,
J. Pan, Q. Zheng, Z. Fan, H. Rahmani, Q. Ke, and J. Liu, “Gradauto: Energy- oriented attack on dynamic neural networks,” inEuropean Conference on Com- puter Vision. Springer, 2022, pp. 637–653
2022
-
[16]
A panda? no, it’s a sloth: Slowdown attacks on adaptive multi-exit neural network inference,
S. Hong, Y. Kaya, I.-V. Modoranu, and T. Dumitra¸ s, “A panda? no, it’s a sloth: Slowdown attacks on adaptive multi-exit neural network inference,”arXiv preprint arXiv:2010.02432, 2020
2010 arXiv
-
[17]
Transslowdown: Efficiency attacks on neural machine translation systems,
S. Chen, M. Haque, Z. Song, C. Liu, and W. Yang, “Transslowdown: Efficiency attacks on neural machine translation systems,” 2022. [Online]. Available: https://openreview.net/forum?id=zfmB5vgfaCt
2022
-
[18]
Nicgslowdown: Evaluating the efficiency robustness of neural image caption generation models,
S. Chen, Z. Song, M. Haque, C. Liu, and W. Yang, “Nicgslowdown: Evaluating the efficiency robustness of neural image caption generation models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15 365–15 374
2022
-
[19]
Desparsify: Adversarial attack against token sparsification mechanisms,
O. Yehezkel, A. Zolfi, A. Baras, Y. Elovici, and A. Shabtai, “Desparsify: Adversarial attack against token sparsification mechanisms,”Advances in Neural Information Processing Systems, vol. 37, pp. 127 536–127 560, 2024
2024
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.