Pith. sign in

REVIEW 4 major objections 6 minor 20 references

MOAT: Model-Agnostic Randomized Transformations for preventing Efficiency Degradation Attacks on ViTs

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A preprocessing pipeline of random resizing, median filtering, and JPEG compression, applied to inputs before inference, suppresses adversarial efficiency attacks on token-pruning Vision Transformers and keeps GFLOPs degradation within…

desk verdict A modest but real contribution: a lightweight preprocessing defense against efficiency-degradation attacks on token-pruning ViTs, with clean numbers against the static DeSparsify attack but no adaptive-attack evaluation, so the headline claim is only partially supported. read the letter →

arxiv 2608.04680 v1 pith:Z7YD4YD2 submitted 2026-08-05 cs.CR cs.CV

classification cs.CRcs.CV
keywords VisionTransformerstokenpruningefficiencydegradationattackspreprocessingdefenserandomresizingmedianfilteringJPEGcompressionmodel-agnostic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a lightweight, model-agnostic preprocessing pipeline can shield token-pruning Vision Transformers from adversarial efficiency-degradation attacks. The pipeline, called MOAT, applies random resizing, median filtering, and JPEG compression to every input image before inference, with no changes to the model or its pruning mechanism. Across the evaluated ViT models (DeiT-Tiny and DeiT-Small with ATS token sampling), it keeps attack-induced GFLOPs growth within 3.4% of the unattacked baseline and cuts attack success rate from roughly 56–60% to under 9%. A careful reader would care because efficiency attacks can silently drain power and latency on edge deployments, and current defenses (token capping, adversarial training) require model modification or retraining.

What carries the argument

The load-bearing mechanism is the ordered pipeline of three input transformations, each aimed at a distinct facet of the adversarial perturbation. Random resizing (scale sampled from $[0.9,1.1]$) shifts pixel coordinates and breaks the alignment between crafted noise and patch embeddings; median filtering with a $3\times3$ kernel removes localized high-frequency noise while preserving edges; JPEG compression (quality 75) discards DCT high-frequency components and introduces a rounding-based non-differentiable step that obstructs gradient flow. Together they force the token-importance distribution of an adversarially perturbed input to resemble that of the clean image, so the ATS pruning mechanism again removes tokens and computation stays near the unattacked level. The defense's parameters were chosen as a 'balanced' trade-off (as opposed to more aggressive or more accuracy-preserving settings), and the order resize→median→JPEG gave the lowest attack success rate at 7.57%.

What would settle it

Run the same DeSparsify attack on DeiT-Tiny/DeiT-Small with an adaptive variant that computes gradients via expectation over transformation across the random resize and uses a differentiable approximation or straight-through estimator for median filtering and JPEG compression. If the resulting GFLOPs exceeds the unattacked baseline by more than 3.4% (or attack success rate rises substantially above ~9%), the paper's central robustness claim is falsified.

Watch

Extended reading notes

Core claim

MOAT's central claim is that adversarial efficiency attacks on adaptive ViTs can be neutralized entirely in input space. The attack works by pushing the token-importance distribution $S^{(l)}$ toward uniform, forcing ATS to sample nearly every token and raising inference cost toward the dense ViT. MOAT suppresses this by sequentially random-resizing (scale $s\sim U(0.9,1.1)$), median-filtering ($3\times3$ kernel), and JPEG-compressing (quality 75) the input, each step disrupting a different component of the adversarial noise without changing the model. In experiments on DeiT-Tiny and DeiT-Small under the $\ell_\infty$-bounded PGD DeSparsify attack ($\epsilon=8/255$, 500 iterations), GFLOPs savings under attack recover from 12.4% to 28.2% and from 16.9% to 31.4%, respectively, while attack success rate drops to 7.5% and 8.9%, all with the same defense configuration on both models. The defense adds only 0.15–0.57% overhead, in contrast to a diffusion-based defense that multiplies cost by over one hundred times.

Load-bearing premise

The defense is assumed to hold against the specific PGD-based single-image attack used in the evaluation (DeSparsify on ATS), and the fixed hyperparameters are assumed to generalize; the paper explicitly does not test adaptive attackers who know the full defense and can use techniques such as expectation over transformation.

Editorial extensions

If this is right

  • Deploying MOAT requires no retraining or architecture changes, so it can be layered onto existing efficient ViT systems with minimal integration effort.
  • The same fixed pipeline transfers across models of different capacity (DeiT-Tiny and DeiT-Small) and keeps adversarial GFLOPs within 3.4% of baseline, supporting the model-agnostic claim.
  • Overhead of the defense is negligible (0.15–0.57% of baseline GFLOPs), which is crucial for resource-constrained and real-time settings.
  • The combination of transformations is essential: single transformations yield attack success rates of 15.4%–32.1%, while the full pipeline drops it to 7.5%.
  • The defense trades off clean accuracy: clean-sample accuracy falls from 83.5% to 76.5% (DeiT-Small) and 79.0% to 67.4% (DeiT-Tiny), a cost that must be weighed against robustness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An obvious next test is an adaptive attacker using expectation over transformation to average gradients through the random resize, combined with a straight-through estimator for JPEG's rounding; the paper leaves this untested, and the 3.4% bound may not survive it.
  • The 3.4% figure is established on only two ATS-based DeiT variants; whether it holds for other pruning mechanisms (e.g., attention-score based pruning or trained token selectors) is open and can be checked with the same pipeline.
  • Since the pipeline is image-space only, it may also transfer to efficiency attacks on other input-adaptive systems (early-exit or dynamic-width networks), although the paper does not evaluate that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MOAT, a preprocessing pipeline consisting of random resizing, median filtering, and JPEG compression, intended to defend adaptive token-pruning Vision Transformers against adversarial efficiency-degradation attacks. The defense is model-agnostic in the sense that it operates on the input image and does not alter the model or pruning mechanism. The authors evaluate MOAT against the DeSparsify attack on ATS-based DeiT-Tiny and DeiT-Small models using 500 ImageNet validation images, reporting GFLOPs, accuracy, and attack success rate. They claim that across all evaluated ViT models, MOAT limits GFLOPs degradation under attack to within 3.4% of the unattacked model while introducing negligible overhead. The paper also includes a discussion of adaptive attacks, acknowledging that expectation-over-transformation and gradient approximation would be needed, but such attacks are not implemented.

Significance. If validated, MOAT would be a useful contribution: it is the first preprocessing-based defense specifically for efficiency-degradation attacks on token pruning, it is lightweight and training-free, and the overhead comparison against a diffusion-based defense is informative. The experimental results are internally consistent, and the authors are transparent about the threat model and the need for adaptive evaluation. However, the significance is currently limited by a substantial mismatch between the stated white-box threat model and the non-adaptive attack evaluation, by the small evaluation set (500 images) with no error bars, and by hyperparameters selected post hoc on the same test set. The central robustness claim therefore rests on evidence against a weaker adversary than the one specified in the paper.

major comments (4)
  1. [III.A, IV.B, IV.E] The threat model in Section III.A states that the attacker has full knowledge of the victim model and its pruning strategy, and Section IV.E further contemplates an adaptive attacker with full knowledge of the defense. However, every attack evaluation in Section IV.B is generated against the undefended ATS model, with MOAT applied only after the perturbation is crafted. No experiment optimizes the perturbation through the defense pipeline, such as expectation-over-transformation (EOT) for the random resize and a differentiable surrogate (e.g., BPDA or straight-through estimator) for the median filter and JPEG quantization. Consequently, the headline claim that GFLOPs degradation stays within 3.4% of the unattacked model is demonstrated only against a defense-unaware adversary, which is weaker than the adversary specified in the paper's own threat model. This is the primary load-bearing gap, and an adaptive-attack evaluation is needed before the central claim can be accepted.
  2. [IV.C, Table II] The defense hyperparameters are selected on the same 500-image evaluation set used to report the final results. Table II compares configurations by clean/adversarial accuracy and ASR, and the 'balanced' configuration is then adopted for Table I. This post hoc selection on the test set can inflate the reported effectiveness and gives no estimate of variability; the small evaluation set makes this concern concrete. Please use a held-out validation split or cross-validation for parameter selection, and report confidence intervals or multiple subsample runs for the headline GFLOPs and ASR figures.
  3. [IV.A, Table I] The evaluation covers only two models (DeiT-Tiny and DeiT-Small) that both use the same ATS pruning mechanism, and only 500 ImageNet validation images. The phrase 'across all evaluated ViT models' in the abstract is technically accurate but the evidence is too narrow to support the model-agnostic generality claim. At minimum, the paper should include a different token-pruning framework (e.g., AdaViT or DynamicViT) and report statistics over multiple subsamples. Additionally, only a single attack family (DeSparsify per-image PGD attack) is considered; the defense's effectiveness against other efficiency-degradation attacks, such as the universal patch attack Slowformer, remains untested.
  4. [IV.E] The discussion in Section IV.E argues that randomness and non-differentiability make adaptive attacks harder, but the paper stops at argument rather than providing evidence. Non-differentiability of JPEG quantization is not a robustness guarantee: standard practice is to approximate gradients with BPDA or straight-through estimators, and the random resize can be handled with EOT. The authors explicitly acknowledge that EOT would be needed yet do not run it. Please either run these adaptive attacks or substantially weaken the claimed robustness to the white-box adversary in the threat model.
minor comments (6)
  1. [Introduction, Figure 1] The introduction refers to 'Figure 4 (green box)' and 'Figure 4 (red box)', but the figure showing the efficiency-degradation attack is labeled Figure 1; the figure numbering appears inconsistent and should be corrected.
  2. [Abstract / Contributions] The abstract uses 'MOdel Agnostic randomized Transformations' while the contributions paragraph uses 'MOdel Agnostic randomized Transformations' with different capitalization than 'Model Agnostic' in the title; please unify the acronym expansion.
  3. [References] References [5] and [11] cite the same paper ('An image is worth 16x16 words'); duplicate entries should be merged into a single reference.
  4. [II.B] There are typos in the text: 'a various optimization' should be 'various optimization' and 'efficiancy' should be 'efficiency'; a careful proofread would improve readability.
  5. [Table III] Table III compares MOAT with a 'DDPM Rectifier' citing DiffPure [20], but DiffPure is a diffusion-based purification/certification method; please clarify what implementation of the 'Rectifier' is used and why it is a representative comparison.
  6. [Algorithm 1] In Algorithm 1, the center-crop/pad operation after random resizing does not specify the padding mode or fill value; this detail matters at image boundaries for the subsequent median filter and should be stated.

Circularity Check

1 steps flagged · score 4.0 of 10

MOAT's headline robustness number is a post-selection statistic: the defense hyperparameters were chosen to minimize ASR on the same DeSparsify/ATS attack and the same 500-image validation set that is then reported as the main result.

  1. fitted input called prediction [Section IV.C (hyperparameter selection) and Table I (Adv.+Def. rows)]
    "To balance robustness and clean-sample accuracy, we evaluated multiple hyperparameter settings and all possible transformation orders. ... Based on these observations, we selected the three representative configurations shown in Table II. ... In this work, we adopt the balanced configuration, which provides a favorable trade-off between robustness and accuracy."

    The balanced configuration (resize range 0.9-1.1, 3x3 median, JPEG quality 75, RES->MED->JPE order) was selected because it achieved the lowest attack success rate (7.57%) among all six orderings on the exact DeSparsify-vs-ATS evaluation. Table I then reports the selected configuration's ASR (7.5%) and GFLOPs (3.26) as evidence that 'MOAT limits GFLOPs degradation.' The reported effectiveness is a post-selection statistic on the same attack and same 500-image validation set used for selection, not an independent prediction; the defense parameters were effectively fitted to the quantity they are then used to demonstrate.

full rationale

No definitional or self-citation circularity is present: MOAT's transformations are generic preprocessing operations, not constructed from the attack equations, and the paper's citations to the authors' prior defenses ([6]-[10]) are not load-bearing. The central derivation is empirical rather than analytical. The one substantive circularity is the hyperparameter selection: Section IV.C chooses the balanced configuration by minimizing ASR on the same DeSparsify/ATS attack and 500-image set that Section IV.B reports as the headline result, so the 'within 3.4%' figure is a tuned outcome rather than a held-out prediction. Separately, Section IV.E acknowledges that adaptive attacks with full knowledge of the defense were not evaluated, despite the white-box threat model in Section III.A; this is a limitation of the empirical support, not a circular step. Overall, the defense pipeline itself is not reduced to its inputs by construction, but the headline quantitative claim is weakened by the selection-on-test-set procedure.

Assumptions & free parameters 3 free parameters · 2 assumptions · 0 invented entities

The pipeline introduces three hyperparameters, all selected by a post hoc search over the same benchmark, but no new physical or mathematical entities. The core assumptions are standard security properties of preprocessing defenses, which are plausible but not independently verified.

free parameters (3)
  • Resize range = [0.9, 1.1]
    Random resizing scale range is a defense hyperparameter. The paper explores ranges and selects a balanced configuration; the choice affects both robustness and clean accuracy.
  • Median filter kernel size = 3x3
    Kernel size for median filtering is selected from a search over k=3 to 11 and affects noise suppression and image distortion.
  • JPEG quality factor = 75
    JPEG quality factor is selected from a search over q=20 to 90 and balances robustness against clean accuracy.
assumptions (2)
  • domain assumption The token pruning mechanism is vulnerable through the differentiable token-importance signal g(x), and perturbing this signal toward uniformity increases token retention.
    Section II.B defines the vulnerability framework using the compute policy and differentiable signals; the paper relies on this to justify both the attack and the defense.
  • domain assumption JPEG compression's rounding operation is nondifferentiable, which obstructs gradient-based attack optimization.
    Section IV.E argues that the quantization step makes attacks harder, but the attack could be approximated with a straight-through estimator or other techniques, which the paper does not test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MOAT: Model-Agnostic Randomized Transformations for preventing Efficiency Degradation Attacks on ViTs." pith.science (2026). https://pith.science/paper/Z7YD4YD2

@misc{pith2026260804680,
  author       = {Pith},
  title        = {Pith review of: MOAT: Model-Agnostic Randomized Transformations for preventing Efficiency Degradation Attacks on ViTs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z7YD4YD2}},
  note         = {Machine review of arXiv:2608.04680}
}
read the original abstract

To adopt the Vision Transformers (ViTs) in resource-constrained environment, token pruning is widely used to reduce computational cost without impacting accuracy. However, adversaries have developed targeted attacks against said token pruning techniques to undermine such attempts to make ViTs efficient. In this paper, we propose MOAT, a model-agnostic pre-processing defense pipeline that applies a combination of input transformations to protect efficient ViT implementations against adversarial efficiency attacks. MOAT operates directly on the input without requiring modifications to the model architecture or token pruning mechanism. Experimental results demonstrate that, across all evaluated ViT models, MOAT limits GFLOPs degradation under adversarial attacks to within 3.4% of the original unattacked model.

Figures

Figures reproduced from arXiv: 2608.04680 by the authors.

Figure 1
Figure 1. Efficiency-Degradation Attack on Adaptive Vision Transformers. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our Proposed Preprocessing Defence: [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Overview of the adversarial efficiency-degradation attack. The dashed [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: ASR and clean accuracy across all six defence orderings. Hatched [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 15 canonical work pages

  1. [1]

    Adaptive token sampling for efficient vision trans- formers,

    M. Fayyaz et. al, “Adaptive token sampling for efficient vision trans- formers,” inECCV. Springer, 2022, pp. 396–414

  2. [2]

    Adavit: Adaptive vision transformers for efficient image recognition,

    L. Meng et. al, “Adavit: Adaptive vision transformers for efficient image recognition,” inCVPR, 2022, pp. 12 309–12 318

  3. [3]

    Desparsify: Adversarial attack against token spar- sification mechanisms,

    O. Yehezkel et. al, “Desparsify: Adversarial attack against token spar- sification mechanisms,”NeurIPS, vol. 37, pp. 127 536–127 560, 2024

  4. [4]

    Slowformer: Adversarial attack on compute and energy consumption of efficient vision transformers,

    K. Navaneet et. al, “Slowformer: Adversarial attack on compute and energy consumption of efficient vision transformers,” inCVPR, 2024, pp. 24 786–24 797

  5. [6]

    Curse of dimensionality in adversarial examples,

    N. Chattopadhyay, A. Chattopadhyay, S. S. Gupta, and M. Kasper, “Curse of dimensionality in adversarial examples,” in2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019, pp. 1–8

  6. [7]

    Robustness against adversarial attacks using dimensionality,

    N. Chattopadhyay, S. Chatterjee, and A. Chattopadhyay, “Robustness against adversarial attacks using dimensionality,” inInternational Con- ference on Security, Privacy, and Applied Cryptography Engineering. Springer, 2021, pp. 226–241

  7. [8]

    Robust perception for autonomous vehicles using dimensionality reduction,

    S. Garg, N. Chattopadhyay, and A. Chattopadhyay, “Robust perception for autonomous vehicles using dimensionality reduction,” in2022 IEEE International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom). IEEE, 2022, pp. 1516–1521

  8. [9]

    Oddr: Outlier detection & dimension reduction based defense against adversarial patches,

    N. Chattopadhyay, A. Guesmi, M. A. Hanif, B. Ouni, and M. Shafique, “Oddr: Outlier detection & dimension reduction based defense against adversarial patches,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 22 999–23 008

Show all 20 references
  1. [10]

    Anomaly unveiled: Securing image classification against adversarial patch attacks,

    N. Chattopadhyay, A. Guesmi, and M. Shafique, “Anomaly unveiled: Securing image classification against adversarial patch attacks,”arXiv preprint arXiv:2402.06249, 2024

  2. [11]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,”arXiv preprint arXiv:2010.11929, 2020

  3. [12]

    Vit-togo: Vision transformer accelerator with grouped token pruning,

    S. Lee, K. Cho, E. Kwon, S. Park, S. Kim, and S. Kang, “Vit-togo: Vision transformer accelerator with grouped token pruning,” in2024 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2024, pp. 1–6

  4. [13]

    Dynam- icvit: Efficient vision transformers with dynamic token sparsification,

    Y . Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, “Dynam- icvit: Efficient vision transformers with dynamic token sparsification,” Advances in neural information processing systems, vol. 34, pp. 13 937– 13 949, 2021

  5. [14]

    Ilfo: Adversarial attack on adaptive neural networks,

    M. Haque, A. Chauhan, C. Liu, and W. Yang, “Ilfo: Adversarial attack on adaptive neural networks,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 14 264–14 273

  6. [15]

    Skipnet: Learning dynamic routing in convolutional networks,

    X. Wang, F. Yu, Z.-Y . Dou, T. Darrell, and J. E. Gonzalez, “Skipnet: Learning dynamic routing in convolutional networks,” inProceedings of the European conference on computer vision (ECCV), 2018, pp. 409– 424

  7. [16]

    Gradauto: Energy-oriented attack on dynamic neural networks,

    J. Pan, Q. Zheng, Z. Fan, H. Rahmani, Q. Ke, and J. Liu, “Gradauto: Energy-oriented attack on dynamic neural networks,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 637–653

  8. [17]

    A panda? no, it’s a sloth: Slowdown attacks on adaptive multi-exit neural network inference,

    S. Hong, Y . Kaya, I.-V . Modoranu, and T. Dumitras ¸, “A panda? no, it’s a sloth: Slowdown attacks on adaptive multi-exit neural network inference,”arXiv preprint arXiv:2010.02432, 2020

  9. [18]

    Nicgslowdown: Evaluating the efficiency robustness of neural image caption generation models,

    S. Chen, Z. Song, M. Haque, C. Liu, and W. Yang, “Nicgslowdown: Evaluating the efficiency robustness of neural image caption generation models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 15 365–15 374

  10. [19]

    Transslowdown: Efficiency attacks on neural machine translation systems,

    S. Chen, M. Haque, Z. Song, C. Liu, and W. Yang, “Transslowdown: Efficiency attacks on neural machine translation systems,” 2022. [Online]. Available: https://openreview.net/forum?id=zfmB5vgfaCt

  11. [20]

    Diffpure: Certifiably robust deep learning via diffusion models against adversarial attacks,

    W. Nie, S. Anwar, and L. Li, “Diffpure: Certifiably robust deep learning via diffusion models against adversarial attacks,”Advances in Neural Information Processing Systems (NeurIPS), 2022

  12. [21]

    Barrage of random transforms for adversarially robust defense,

    E. Raff, J. Sylvester, S. Forsyth, and M. McLean, “Barrage of random transforms for adversarially robust defense,” in2019 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 6521–6530

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.