REVIEW 4 major objections 6 minor 20 references
MOAT: Model-Agnostic Randomized Transformations for preventing Efficiency Degradation Attacks on ViTs
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A preprocessing pipeline of random resizing, median filtering, and JPEG compression, applied to inputs before inference, suppresses adversarial efficiency attacks on token-pruning Vision Transformers and keeps GFLOPs degradation within…
desk verdict A modest but real contribution: a lightweight preprocessing defense against efficiency-degradation attacks on token-pruning ViTs, with clean numbers against the static DeSparsify attack but no adaptive-attack evaluation, so the headline claim is only partially supported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the ordered pipeline of three input transformations, each aimed at a distinct facet of the adversarial perturbation. Random resizing (scale sampled from $[0.9,1.1]$) shifts pixel coordinates and breaks the alignment between crafted noise and patch embeddings; median filtering with a $3\times3$ kernel removes localized high-frequency noise while preserving edges; JPEG compression (quality 75) discards DCT high-frequency components and introduces a rounding-based non-differentiable step that obstructs gradient flow. Together they force the token-importance distribution of an adversarially perturbed input to resemble that of the clean image, so the ATS pruning mechanism again removes tokens and computation stays near the unattacked level. The defense's parameters were chosen as a 'balanced' trade-off (as opposed to more aggressive or more accuracy-preserving settings), and the order resize→median→JPEG gave the lowest attack success rate at 7.57%.
What would settle it
Run the same DeSparsify attack on DeiT-Tiny/DeiT-Small with an adaptive variant that computes gradients via expectation over transformation across the random resize and uses a differentiable approximation or straight-through estimator for median filtering and JPEG compression. If the resulting GFLOPs exceeds the unattacked baseline by more than 3.4% (or attack success rate rises substantially above ~9%), the paper's central robustness claim is falsified.
Extended reading notes
Core claim
MOAT's central claim is that adversarial efficiency attacks on adaptive ViTs can be neutralized entirely in input space. The attack works by pushing the token-importance distribution $S^{(l)}$ toward uniform, forcing ATS to sample nearly every token and raising inference cost toward the dense ViT. MOAT suppresses this by sequentially random-resizing (scale $s\sim U(0.9,1.1)$), median-filtering ($3\times3$ kernel), and JPEG-compressing (quality 75) the input, each step disrupting a different component of the adversarial noise without changing the model. In experiments on DeiT-Tiny and DeiT-Small under the $\ell_\infty$-bounded PGD DeSparsify attack ($\epsilon=8/255$, 500 iterations), GFLOPs savings under attack recover from 12.4% to 28.2% and from 16.9% to 31.4%, respectively, while attack success rate drops to 7.5% and 8.9%, all with the same defense configuration on both models. The defense adds only 0.15–0.57% overhead, in contrast to a diffusion-based defense that multiplies cost by over one hundred times.
Load-bearing premise
The defense is assumed to hold against the specific PGD-based single-image attack used in the evaluation (DeSparsify on ATS), and the fixed hyperparameters are assumed to generalize; the paper explicitly does not test adaptive attackers who know the full defense and can use techniques such as expectation over transformation.
Editorial extensions
If this is right
- Deploying MOAT requires no retraining or architecture changes, so it can be layered onto existing efficient ViT systems with minimal integration effort.
- The same fixed pipeline transfers across models of different capacity (DeiT-Tiny and DeiT-Small) and keeps adversarial GFLOPs within 3.4% of baseline, supporting the model-agnostic claim.
- Overhead of the defense is negligible (0.15–0.57% of baseline GFLOPs), which is crucial for resource-constrained and real-time settings.
- The combination of transformations is essential: single transformations yield attack success rates of 15.4%–32.1%, while the full pipeline drops it to 7.5%.
- The defense trades off clean accuracy: clean-sample accuracy falls from 83.5% to 76.5% (DeiT-Small) and 79.0% to 67.4% (DeiT-Tiny), a cost that must be weighed against robustness.
Reading between the lines
- An obvious next test is an adaptive attacker using expectation over transformation to average gradients through the random resize, combined with a straight-through estimator for JPEG's rounding; the paper leaves this untested, and the 3.4% bound may not survive it.
- The 3.4% figure is established on only two ATS-based DeiT variants; whether it holds for other pruning mechanisms (e.g., attention-score based pruning or trained token selectors) is open and can be checked with the same pipeline.
- Since the pipeline is image-space only, it may also transfer to efficiency attacks on other input-adaptive systems (early-exit or dynamic-width networks), although the paper does not evaluate that.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes MOAT, a preprocessing pipeline consisting of random resizing, median filtering, and JPEG compression, intended to defend adaptive token-pruning Vision Transformers against adversarial efficiency-degradation attacks. The defense is model-agnostic in the sense that it operates on the input image and does not alter the model or pruning mechanism. The authors evaluate MOAT against the DeSparsify attack on ATS-based DeiT-Tiny and DeiT-Small models using 500 ImageNet validation images, reporting GFLOPs, accuracy, and attack success rate. They claim that across all evaluated ViT models, MOAT limits GFLOPs degradation under attack to within 3.4% of the unattacked model while introducing negligible overhead. The paper also includes a discussion of adaptive attacks, acknowledging that expectation-over-transformation and gradient approximation would be needed, but such attacks are not implemented.
Significance. If validated, MOAT would be a useful contribution: it is the first preprocessing-based defense specifically for efficiency-degradation attacks on token pruning, it is lightweight and training-free, and the overhead comparison against a diffusion-based defense is informative. The experimental results are internally consistent, and the authors are transparent about the threat model and the need for adaptive evaluation. However, the significance is currently limited by a substantial mismatch between the stated white-box threat model and the non-adaptive attack evaluation, by the small evaluation set (500 images) with no error bars, and by hyperparameters selected post hoc on the same test set. The central robustness claim therefore rests on evidence against a weaker adversary than the one specified in the paper.
major comments (4)
- [III.A, IV.B, IV.E] The threat model in Section III.A states that the attacker has full knowledge of the victim model and its pruning strategy, and Section IV.E further contemplates an adaptive attacker with full knowledge of the defense. However, every attack evaluation in Section IV.B is generated against the undefended ATS model, with MOAT applied only after the perturbation is crafted. No experiment optimizes the perturbation through the defense pipeline, such as expectation-over-transformation (EOT) for the random resize and a differentiable surrogate (e.g., BPDA or straight-through estimator) for the median filter and JPEG quantization. Consequently, the headline claim that GFLOPs degradation stays within 3.4% of the unattacked model is demonstrated only against a defense-unaware adversary, which is weaker than the adversary specified in the paper's own threat model. This is the primary load-bearing gap, and an adaptive-attack evaluation is needed before the central claim can be accepted.
- [IV.C, Table II] The defense hyperparameters are selected on the same 500-image evaluation set used to report the final results. Table II compares configurations by clean/adversarial accuracy and ASR, and the 'balanced' configuration is then adopted for Table I. This post hoc selection on the test set can inflate the reported effectiveness and gives no estimate of variability; the small evaluation set makes this concern concrete. Please use a held-out validation split or cross-validation for parameter selection, and report confidence intervals or multiple subsample runs for the headline GFLOPs and ASR figures.
- [IV.A, Table I] The evaluation covers only two models (DeiT-Tiny and DeiT-Small) that both use the same ATS pruning mechanism, and only 500 ImageNet validation images. The phrase 'across all evaluated ViT models' in the abstract is technically accurate but the evidence is too narrow to support the model-agnostic generality claim. At minimum, the paper should include a different token-pruning framework (e.g., AdaViT or DynamicViT) and report statistics over multiple subsamples. Additionally, only a single attack family (DeSparsify per-image PGD attack) is considered; the defense's effectiveness against other efficiency-degradation attacks, such as the universal patch attack Slowformer, remains untested.
- [IV.E] The discussion in Section IV.E argues that randomness and non-differentiability make adaptive attacks harder, but the paper stops at argument rather than providing evidence. Non-differentiability of JPEG quantization is not a robustness guarantee: standard practice is to approximate gradients with BPDA or straight-through estimators, and the random resize can be handled with EOT. The authors explicitly acknowledge that EOT would be needed yet do not run it. Please either run these adaptive attacks or substantially weaken the claimed robustness to the white-box adversary in the threat model.
minor comments (6)
- [Introduction, Figure 1] The introduction refers to 'Figure 4 (green box)' and 'Figure 4 (red box)', but the figure showing the efficiency-degradation attack is labeled Figure 1; the figure numbering appears inconsistent and should be corrected.
- [Abstract / Contributions] The abstract uses 'MOdel Agnostic randomized Transformations' while the contributions paragraph uses 'MOdel Agnostic randomized Transformations' with different capitalization than 'Model Agnostic' in the title; please unify the acronym expansion.
- [References] References [5] and [11] cite the same paper ('An image is worth 16x16 words'); duplicate entries should be merged into a single reference.
- [II.B] There are typos in the text: 'a various optimization' should be 'various optimization' and 'efficiancy' should be 'efficiency'; a careful proofread would improve readability.
- [Table III] Table III compares MOAT with a 'DDPM Rectifier' citing DiffPure [20], but DiffPure is a diffusion-based purification/certification method; please clarify what implementation of the 'Rectifier' is used and why it is a representative comparison.
- [Algorithm 1] In Algorithm 1, the center-crop/pad operation after random resizing does not specify the padding mode or fill value; this detail matters at image boundaries for the subsequent median filter and should be stated.
Circularity Check
MOAT's headline robustness number is a post-selection statistic: the defense hyperparameters were chosen to minimize ASR on the same DeSparsify/ATS attack and the same 500-image validation set that is then reported as the main result.
-
fitted input called prediction
[Section IV.C (hyperparameter selection) and Table I (Adv.+Def. rows)]
"To balance robustness and clean-sample accuracy, we evaluated multiple hyperparameter settings and all possible transformation orders. ... Based on these observations, we selected the three representative configurations shown in Table II. ... In this work, we adopt the balanced configuration, which provides a favorable trade-off between robustness and accuracy."
The balanced configuration (resize range 0.9-1.1, 3x3 median, JPEG quality 75, RES->MED->JPE order) was selected because it achieved the lowest attack success rate (7.57%) among all six orderings on the exact DeSparsify-vs-ATS evaluation. Table I then reports the selected configuration's ASR (7.5%) and GFLOPs (3.26) as evidence that 'MOAT limits GFLOPs degradation.' The reported effectiveness is a post-selection statistic on the same attack and same 500-image validation set used for selection, not an independent prediction; the defense parameters were effectively fitted to the quantity they are then used to demonstrate.
full rationale
No definitional or self-citation circularity is present: MOAT's transformations are generic preprocessing operations, not constructed from the attack equations, and the paper's citations to the authors' prior defenses ([6]-[10]) are not load-bearing. The central derivation is empirical rather than analytical. The one substantive circularity is the hyperparameter selection: Section IV.C chooses the balanced configuration by minimizing ASR on the same DeSparsify/ATS attack and 500-image set that Section IV.B reports as the headline result, so the 'within 3.4%' figure is a tuned outcome rather than a held-out prediction. Separately, Section IV.E acknowledges that adaptive attacks with full knowledge of the defense were not evaluated, despite the white-box threat model in Section III.A; this is a limitation of the empirical support, not a circular step. Overall, the defense pipeline itself is not reduced to its inputs by construction, but the headline quantitative claim is weakened by the selection-on-test-set procedure.
Assumptions & free parameters
free parameters (3)
- Resize range =
[0.9, 1.1]
- Median filter kernel size =
3x3
- JPEG quality factor =
75
assumptions (2)
- domain assumption The token pruning mechanism is vulnerable through the differentiable token-importance signal g(x), and perturbing this signal toward uniformity increases token retention.
- domain assumption JPEG compression's rounding operation is nondifferentiable, which obstructs gradient-based attack optimization.
Cite this review
Pith. "Pith review of MOAT: Model-Agnostic Randomized Transformations for preventing Efficiency Degradation Attacks on ViTs." pith.science (2026). https://pith.science/paper/Z7YD4YD2
@misc{pith2026260804680,
author = {Pith},
title = {Pith review of: MOAT: Model-Agnostic Randomized Transformations for preventing Efficiency Degradation Attacks on ViTs},
year = {2026},
howpublished = {\url{https://pith.science/paper/Z7YD4YD2}},
note = {Machine review of arXiv:2608.04680}
}
read the original abstract
To adopt the Vision Transformers (ViTs) in resource-constrained environment, token pruning is widely used to reduce computational cost without impacting accuracy. However, adversaries have developed targeted attacks against said token pruning techniques to undermine such attempts to make ViTs efficient. In this paper, we propose MOAT, a model-agnostic pre-processing defense pipeline that applies a combination of input transformations to protect efficient ViT implementations against adversarial efficiency attacks. MOAT operates directly on the input without requiring modifications to the model architecture or token pruning mechanism. Experimental results demonstrate that, across all evaluated ViT models, MOAT limits GFLOPs degradation under adversarial attacks to within 3.4% of the original unattacked model.
Figures
Reference graph
Works this paper leans on
-
[1]
Adaptive token sampling for efficient vision trans- formers,
M. Fayyaz et. al, “Adaptive token sampling for efficient vision trans- formers,” inECCV. Springer, 2022, pp. 396–414
work page 2022
-
[2]
Adavit: Adaptive vision transformers for efficient image recognition,
L. Meng et. al, “Adavit: Adaptive vision transformers for efficient image recognition,” inCVPR, 2022, pp. 12 309–12 318
work page 2022
-
[3]
Desparsify: Adversarial attack against token spar- sification mechanisms,
O. Yehezkel et. al, “Desparsify: Adversarial attack against token spar- sification mechanisms,”NeurIPS, vol. 37, pp. 127 536–127 560, 2024
work page 2024
-
[4]
Slowformer: Adversarial attack on compute and energy consumption of efficient vision transformers,
K. Navaneet et. al, “Slowformer: Adversarial attack on compute and energy consumption of efficient vision transformers,” inCVPR, 2024, pp. 24 786–24 797
work page 2024
-
[6]
Curse of dimensionality in adversarial examples,
N. Chattopadhyay, A. Chattopadhyay, S. S. Gupta, and M. Kasper, “Curse of dimensionality in adversarial examples,” in2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019, pp. 1–8
work page 2019
-
[7]
Robustness against adversarial attacks using dimensionality,
N. Chattopadhyay, S. Chatterjee, and A. Chattopadhyay, “Robustness against adversarial attacks using dimensionality,” inInternational Con- ference on Security, Privacy, and Applied Cryptography Engineering. Springer, 2021, pp. 226–241
work page 2021
-
[8]
Robust perception for autonomous vehicles using dimensionality reduction,
S. Garg, N. Chattopadhyay, and A. Chattopadhyay, “Robust perception for autonomous vehicles using dimensionality reduction,” in2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom). IEEE, 2022, pp. 1516–1521
work page 2022
-
[9]
Oddr: Outlier detection & dimension reduction based defense against adversarial patches,
N. Chattopadhyay, A. Guesmi, M. A. Hanif, B. Ouni, and M. Shafique, “Oddr: Outlier detection & dimension reduction based defense against adversarial patches,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 22 999–23 008
work page 2025
Show all 20 references
-
[10]
Anomaly unveiled: Securing image classification against adversarial patch attacks,
N. Chattopadhyay, A. Guesmi, and M. Shafique, “Anomaly unveiled: Securing image classification against adversarial patch attacks,”arXiv preprint arXiv:2402.06249, 2024
2024 arXiv
-
[11]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020
2010 arXiv
-
[12]
Vit-togo: Vision transformer accelerator with grouped token pruning,
S. Lee, K. Cho, E. Kwon, S. Park, S. Kim, and S. Kang, “Vit-togo: Vision transformer accelerator with grouped token pruning,” in2024 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2024, pp. 1–6
2024
-
[13]
Dynam- icvit: Efficient vision transformers with dynamic token sparsification,
Y . Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, “Dynam- icvit: Efficient vision transformers with dynamic token sparsification,” Advances in neural information processing systems, vol. 34, pp. 13 937– 13 949, 2021
2021
-
[14]
Ilfo: Adversarial attack on adaptive neural networks,
M. Haque, A. Chauhan, C. Liu, and W. Yang, “Ilfo: Adversarial attack on adaptive neural networks,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14 264–14 273
2020
-
[15]
Skipnet: Learning dynamic routing in convolutional networks,
X. Wang, F. Yu, Z.-Y . Dou, T. Darrell, and J. E. Gonzalez, “Skipnet: Learning dynamic routing in convolutional networks,” inProceedings of the European conference on computer vision (ECCV), 2018, pp. 409– 424
2018
-
[16]
Gradauto: Energy-oriented attack on dynamic neural networks,
J. Pan, Q. Zheng, Z. Fan, H. Rahmani, Q. Ke, and J. Liu, “Gradauto: Energy-oriented attack on dynamic neural networks,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 637–653
2022
-
[17]
A panda? no, it’s a sloth: Slowdown attacks on adaptive multi-exit neural network inference,
S. Hong, Y . Kaya, I.-V . Modoranu, and T. Dumitras ¸, “A panda? no, it’s a sloth: Slowdown attacks on adaptive multi-exit neural network inference,”arXiv preprint arXiv:2010.02432, 2020
2010 arXiv
-
[18]
Nicgslowdown: Evaluating the efficiency robustness of neural image caption generation models,
S. Chen, Z. Song, M. Haque, C. Liu, and W. Yang, “Nicgslowdown: Evaluating the efficiency robustness of neural image caption generation models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15 365–15 374
2022
-
[19]
Transslowdown: Efficiency attacks on neural machine translation systems,
S. Chen, M. Haque, Z. Song, C. Liu, and W. Yang, “Transslowdown: Efficiency attacks on neural machine translation systems,” 2022. [Online]. Available: https://openreview.net/forum?id=zfmB5vgfaCt
2022
-
[20]
Diffpure: Certifiably robust deep learning via diffusion models against adversarial attacks,
W. Nie, S. Anwar, and L. Li, “Diffpure: Certifiably robust deep learning via diffusion models against adversarial attacks,”Advances in Neural Information Processing Systems (NeurIPS), 2022
2022
-
[21]
Barrage of random transforms for adversarially robust defense,
E. Raff, J. Sylvester, S. Forsyth, and M. McLean, “Barrage of random transforms for adversarially robust defense,” in2019 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6521–6530
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.