Pith. sign in

REVIEW 4 major objections 6 minor 24 references

Sparse patches adversarial attacks via extrapolating point-wise information

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that sparse and patch adversarial attacks can be solved directly by trimming dense perturbations, and that the result achieves state-of-the-art success on ImageNet.

desk verdict New trimming heuristic for sparse/patch attacks that consistently beats baselines on ImageNet, but the load-bearing proxy in Eq. (6) is unvalidated and no variance is reported, so treat SOTA margins as provisional. read the letter →

arxiv 2411.16162 v1 pith:2QUHWI6B submitted 2024-11-25 cs.CV cs.LG

Yaniv Nemcovsky , Avi Mendelson , Chaim Baskin This is my paper · ORCID
classification cs.CVcs.LG
keywords adversarialattackssparsepatchpoint-wisetrimmingL0normboundMonteCarloestimationImageNetclassificationPGDoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the combinatorial problem of sparse and patch adversarial attacks can be solved directly by starting from a dense adversarial perturbation and trimming it point by point. It claims to be the first method that simultaneously optimizes the locations and the pixel values of multiple patches of any given shape, and that the same approach also improves over existing sparse attacks. If true, the method trades an exponentially large search over pixel subsets for a tractable rank-and-prune loop, giving attackers a stronger tool and defenders a harder baseline. The reported experiments on ImageNet show state-of-the-art success rates for PGDTrim across multiple architectures and L0 budgets, with 100 percent success at modest budgets.

What carries the argument

The load-bearing mechanism is the TrimStep procedure: it computes a per-pixel importance score by averaging, over Monte Carlo-sampled binary masks, the model's loss on the masked perturbation weighted by the mask, then selects the top-k pixels (or patches). The attack wraps TrimStep in a logarithmic trimming schedule, interleaving PGD optimization with Bernoulli dropout that simulates the binary-mask projection, so the dense perturbation stays robust to the next trim and the L0 gap per step stays small. This converts the combinatorial mask search into a sequence of top-k selections plus differentiable perturbation updates.

What would settle it

Take a trained ImageNet model, compute a PGDTrim mask for a given L0 budget, then replace an equal number of the lowest-ranked kept pixels with an equal-size set of medium-ranked pixels chosen for high pairwise correlation with the surviving set, keeping perturbation magnitudes fixed; if the replacement achieves equal or higher adversarial success, the point-wise independence assumption behind the ranking is untenable.

Watch

Extended reading notes

Core claim

The central claim is that the optimal sparse mask can be extrapolated from the point-wise importance of a dense perturbation. From a pre-optimized dense perturbation δ, the authors estimate for each pixel the expected attack loss over binary masks that keep that pixel, using Monte Carlo sampling against the current support. They then keep the highest-scoring pixels (or, under a patch kernel, the highest-scoring patches via a max-out procedure), and repeat an optimize-then-trim loop on a logarithmic schedule of L0 bounds. The resulting attacks, PGDTrim and PGDTrimKernel, achieve higher adversarial success rates than SparseFool, PGD_L0, GreedyFool, and Homotopy across all tested models and L0 budgets on ImageNet, reaching 100 percent success at L0 = 128 for InceptionV3 and standard ResNet50, and at L0 = 224 for a robust ResNet50.

Load-bearing premise

The entire method leans on the assumption that a pixel's importance computed from a dense perturbation stays valid as the perturbation is made sparse, so pixels that are only useful in correlated groups are never discarded by the trimming.

Editorial extensions

If this is right

  • Attackers gain a practical way to place multiple arbitrary-shaped adversarial patches in one optimization, which prior patch attacks did not support directly.
  • Defenses can no longer assume sparse-attack evaluations are complete without this baseline, since it outperforms existing sparse attacks on standard, robust, and transformer models.
  • Because the trimming step does not require differentiability over locations, the method transfers to settings like physical-world patch placement where location optimization is not differentiable.
  • Reported 100 percent success rates at low L0 budgets indicate that current ImageNet classifiers remain vulnerable to sparse modifications, and robustness evaluations should include such budgets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The top-k trim could be interpreted as a greedy step for a possibly submodular set function, suggesting a provable approximation guarantee when the loss is near-submodular over selected pixels.
  • The Monte Carlo estimator in Eq. (6) is a candidate for variance reduction (control variates or stratified sampling), which could cut the 1000-sample cost and make the attack practical for video or real-time settings.
  • One could invert PGDTrim's point-wise ranking to build saliency maps of the minimal pixel subsets a classifier relies on, giving an interpretability tool rather than only an attack.
  • The method is evaluated only on single-image attacks; running the trimming loop over a batch of images would produce a universal sparse patch, and it is an open question whether the gains persist in that setting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes sparse and patch adversarial attacks by starting from a dense adversarial perturbation and iteratively trimming it to a target L0 budget. The central idea is a TrimStep procedure that estimates per-point importance via Monte Carlo sampling over random binary masks and then keeps the top-scoring points (or the highest-scoring patches under a kernel constraint). The method is evaluated on ImageNet across five models (InceptionV3, ResNet50, robust ResNet50, Swin-B, ConvNeXt-B), reporting adversarial success rate as a function of the L0 bound for the sparse attack and for 1x1, 2x2, and 4x4 patch variants. The authors claim state-of-the-art sparse attack results and that their approach is the first direct solution to simultaneous patch-location and perturbation optimization.

Significance. If the empirical claims hold, the paper would provide a practical and general mechanism for jointly optimizing patch locations and perturbations, addressing a recognized gap in patch-based adversarial attacks. The paper has notable strengths: the method is described with explicit algorithms, a reference implementation is promised, and the evaluation covers multiple architectures including robust and transformer-based models. The sparse attack results are consistently better than the compared baselines across the reported settings. However, the central approximation on which the method rests—replacing the posterior mask distribution by the prior mask distribution in Eq. (6)—is presented without a formal bound or empirical validation, and the reported curves lack variance information despite the stochastic components (Monte Carlo sampling and restarts). These issues make the state-of-the-art claim not yet fully supported.

major comments (4)
  1. [3.1, Eq. (5)-Eq. (6)] The approximation Lδs = E_B[ℓ(M(x+δ^B_s),y)·B] ≈ E_B[ℓ(M(x+B⊙δ),y)·B] is the load-bearing step of the entire method, but it is only motivated by qualitative assumptions. No bound is given and no experiment checks whether the top-k sets selected by Eq. (6) match those selected by the true objective Eq. (5) or Eq. (4) on a tractable subproblem. Since the authors acknowledge that the approximation is exact only under assumptions that are not verified on ImageNet, the "significantly outperforms" claim is not yet backed by a correctness argument. Please provide either a quantitative bound (e.g., in terms of the Lipschitz constant of ℓ∘M and the projection error) or an empirical validation showing that the selected masks closely match the optimal masks on a simplified setting, and report the discrepancy as a function of the L0 gap.
  2. [4.1, Figs. 2-5] The reported ASR curves are single trajectories with no error bars, confidence intervals, or significance tests. The method has multiple stochastic components: MC=1000 mask samples per trim step and nrestarts=11 restarts. The authors also state that MC samples may be "sufficient" and compute Eq. (6) directly, but no criterion for sufficiency is given. Without variance reporting, the claim that the method "significantly outperforms" the baselines is not statistically supported. Please report mean and standard deviation (or full error bars) over the 11 restarts and, ideally, over several MC seeds, and indicate the number of images used for each curve.
  3. [4, Figs. 2-4] The patch-specific contribution is not compared against any existing patch attack. Figures 2-4 show the proposed patch variants only against sparse baselines and against the paper's own sparse attack; no comparison is made to prior patch attacks such as those of Nemcovsky et al., Chen et al., Wei et al., or Zolfi et al. described in the introduction. Since the paper's claimed novelty is simultaneous optimization of patch locations and perturbations, an evaluation against previous patch attacks on the same models and L0 budgets (or a clear explanation of why such a comparison is not meaningful) is necessary to support that claim.
  4. [A.2, Algorithm 2] In the point-wise evaluation loop, BCount starts at zero and is incremented by the sampled mask B. If a point is never selected in the MC samples, BCount[i] remains zero and the expression BLoss/BCount is undefined unless a special case is handled. The pseudocode does not describe such handling, which makes the algorithm as written non-reproducible. Please specify how zero-count points are treated (e.g., excluded from the top-k selection or assigned a default score).
minor comments (6)
  1. [4, Fig. 5] The caption says "ConvNextB" while the text refers to the model as "ConvNeXtVIT"; please unify the naming.
  2. [4, Experimental settings] The text has a typo: "M C= 1000" should be "MC = 1000".
  3. [A.1, Eq. (9)] The notation ∥δinit∥0 = N is stated for the initial perturbation, but the algorithm initializes δbest with a uniform draw in [-1,1]^N; please clarify whether this initialization is always full-support and whether the count is over pixels or scalar channels.
  4. [A.2, Algorithm 3] The padding and pooling operations for overlapping patches are described compactly; providing a small example or a precise definition of MaxPool, SumPool, and Pad would improve reproducibility.
  5. [References] The Koonce reference is not the standard citation for ResNet50; please cite the original He et al. paper.
  6. [4.1, patch discussion] The text says ASR "starting from ϵ0=128" for IncV3 and RN50, but the curves show the reported points; please clarify whether the curves are monotonic and whether the 100% claim is exact or rounded.

Circularity Check

0 steps flagged · score 2.0 of 10

No circularity in the derivation; Eq. (6) is an explicitly stated approximation, and the SOTA claim rests on external benchmarks.

full rationale

The paper's derivation chain is not circular. The attack objectives in Eqs. (1)-(3) are standard optimization definitions. Eq. (4) states the joint mask/perturbation objective, Eq. (5) defines an exact but intractable point-wise importance criterion, and Eq. (6) is an explicitly labeled approximation of that criterion using the dense perturbation. The mask selection in Eqs. (7)-(8) and in Algorithms 2-3 then uses this approximation, and the final perturbation is separately optimized and evaluated. No parameter is fitted to the reported adversarial success rates, no uniqueness theorem is imported from the authors' prior work, and no conclusion is assumed in the construction of the method. The paper's own Section 3.1 lists the assumptions under which the Eq. (6) approximation is expected to hold; those assumptions are unproven heuristics, which is a correctness/novelty risk rather than circularity. The only self-citation is Nemcovsky et al. (2022), used for motivation that patch attacks are realistic in physical settings; it is not load-bearing for the method or the experimental claim. The SOTA claim is supported by comparisons to external baselines (PGD_L0, GreedyFool, SparseFool, Homotopy) on ImageNet models, so the central claim has independent empirical content.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted to the target metric; the listed items are disclosed hyperparameters and heuristic assumptions. The central method rests on three assumptions about point-wise importance transfer, none of which are formally proven. No new entities or physical constants are introduced.

free parameters (4)
  • Monte Carlo samples MC = 1000
    Number of binary mask samples in TrimStep; chosen by hand, not fitted to data.
  • trim steps ntrim = 11
    Number of logarithmic trimming steps; chosen by hand.
  • PGD restarts nrestarts = 11
    Number of restarts in the PGD optimization; chosen by hand.
  • PGD step size alpha = not specified
    Step size for DPGD is used in algorithms but its value is not reported, making exact reproduction dependent on unstated choices.
assumptions (3)
  • domain assumption The attack criterion mainly depends on selecting significant points in the dense perturbation rather than a well-correlated group.
    Stated as the first assumption in Section 3.1; it underpins the validity of point-wise top-k selection.
  • domain assumption The dense perturbation is sufficiently robust to the projections B⊙δ so that the loss decrease during trimming is mainly due to removing less significant points.
    Second assumption in Section 3.1, used to justify the approximation of the posterior by the prior in Eq. 6.
  • domain assumption The L0 gap between successive trim steps is sufficiently small so that point-wise significance remains relatively unaltered.
    Third assumption in Section 3.1; the method tries to satisfy it with a logarithmic schedule, but no guarantee is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sparse patches adversarial attacks via extrapolating point-wise information." pith.science (2026). https://pith.science/paper/2QUHWI6B

@misc{pith2026241116162,
  author       = {Pith},
  title        = {Pith review of: Sparse patches adversarial attacks via extrapolating point-wise information},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QUHWI6B}},
  note         = {Machine review of arXiv:2411.16162}
}
read the original abstract

Sparse and patch adversarial attacks were previously shown to be applicable in realistic settings and are considered a security risk to autonomous systems. Sparse adversarial perturbations constitute a setting in which the adversarial perturbations are limited to affecting a relatively small number of points in the input. Patch adversarial attacks denote the setting where the sparse attacks are limited to a given structure, i.e., sparse patches with a given shape and number. However, previous patch adversarial attacks do not simultaneously optimize multiple patches' locations and perturbations. This work suggests a novel approach for sparse patches adversarial attacks via point-wise trimming dense adversarial perturbations. Our approach enables simultaneous optimization of multiple sparse patches' locations and perturbations for any given number and shape. Moreover, our approach is also applicable for standard sparse adversarial attacks, where we show that it significantly improves the state-of-the-art over multiple extensive settings. A reference implementation of the proposed method and the reported experiments is provided at \url{https://github.com/yanemcovsky/SparsePatches.git}

Figures

Figures reproduced from arXiv: 2411.16162 by the authors.

Figure 1
Figure 1. Flowchart of our sparse (top) and 2 × 2 patch (bottom) adversarial attacks trim process on Imagenet standard Resnet50 model, for attacks bounded to ϵ0 = 224. We present the adversarial inputs produced for distinct ϵ0 bounds during the process and the predicted label for each, compared to the true label. simultaneous optimization of the patches’ locations and corresponding perturbations. Moreover, the locations’ opti… view at source ↗
Figure 2
Figure 2. We compare our method to previous sparse attack works(left) and with various patch sizes [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. We compare our method to previous sparse attack works(left) and with various patch sizes [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: We compare our method to previous sparse attack works(left) and with various patch sizes [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: We compare our method to previous works on the Imagenet dataset, visual transformer [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 13 canonical work pages

  1. [1]

    Intriguing properties of neural networks

    Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013. URL http://arxiv.org/abs/1312.6199

    arXiv 2013

  2. [2]

    Goodfellow, Jonathon Shlens, and Christian Szegedy

    Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014. URL http://arxiv.org/abs/1412.6572

    arXiv 2014

  3. [3]

    Towards deep learning models resistant to adversarial attacks

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJzIBfZAb

    2018

  4. [4]

    Sparse and imperceivable adversarial attacks

    Francesco Croce and Matthias Hein. Sparse and imperceivable adversarial attacks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4724--4732, 2019

    work page 2019

  5. [5]

    Sparse adversarial attack via perturbation factorization

    Yanbo Fan, Baoyuan Wu, Tuanhui Li, Yong Zhang, Mingyang Li, Zhifeng Li, and Yujiu Yang. Sparse adversarial attack via perturbation factorization. In Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part XXII 16, pages 35--50. Springer, 2020

    work page 2020

  6. [6]

    Mind the box: l\_1 -apgd for sparse adversarial attacks on image classifiers

    Francesco Croce and Matthias Hein. Mind the box: l\_1 -apgd for sparse adversarial attacks on image classifiers. In International Conference on Machine Learning, pages 2201--2211. PMLR, 2021

    work page 2021

  7. [7]

    Greedyfool: Distortion-aware sparse adversarial attack

    Xiaoyi Dong, Dongdong Chen, Jianmin Bao, Chuan Qin, Lu Yuan, Weiming Zhang, Nenghai Yu, and Dong Chen. Greedyfool: Distortion-aware sparse adversarial attack. Advances in Neural Information Processing Systems, 33: 0 11226--11236, 2020

    work page 2020

  8. [8]

    Physical passive patch adversarial attacks on visual odometry systems

    Yaniv Nemcovsky, Matan Jacoby, Alex M Bronstein, and Chaim Baskin. Physical passive patch adversarial attacks on visual odometry systems. In Proceedings of the Asian Conference on Computer Vision, pages 1795--1811, 2022

    work page 2022

Show all 24 references
  1. [9]

    Evading real-time person detectors by adversarial t-shirt

    Kaidi Xu, Gaoyuan Zhang, Sijia Liu, Quanfu Fan, Mengshu Sun, Hongge Chen, Pin-Yu Chen, Yanzhi Wang, and Xue Lin. Evading real-time person detectors by adversarial t-shirt. arXiv preprint arXiv:1910.11099, 2019. URL http://arxiv.org/abs/1910.11099

    1910 arXiv

  2. [10]

    The translucent patch: A physical and universal attack on object detectors

    Alon Zolfi, Moshe Kravchik, Yuval Elovici, and Asaf Shabtai. The translucent patch: A physical and universal attack on object detectors. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15232--15241, 2021

    2021

  3. [11]

    Simultaneously optimizing perturbations and positions for black-box adversarial patch attacks

    Xingxing Wei, Ying Guo, Jie Yu, and Bo Zhang. Simultaneously optimizing perturbations and positions for black-box adversarial patch attacks. IEEE transactions on pattern analysis and machine intelligence, 2022 a

    2022

  4. [12]

    Shapeshifter: Robust physical adversarial attack on faster r-cnn object detector

    Shang-Tse Chen, Cory Cornelius, Jason Martin, and Duen Horng Chau. Shapeshifter: Robust physical adversarial attack on faster r-cnn object detector. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2018, Dublin, Ireland, September 10--14...

    2018

  5. [13]

    Adversarial sticker: A stealthy attack method in the physical world

    Xingxing Wei, Ying Guo, and Jie Yu. Adversarial sticker: A stealthy attack method in the physical world. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (3): 0 2711--2725, 2022 b

    2022

  6. [14]

    Sparsefool: a few pixels make a big difference

    Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. Sparsefool: a few pixels make a big difference. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9087--9096, 2019

    2019

  7. [15]

    Generative adversarial networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63 0 (11): 0 139--144, 2020

    2020

  8. [16]

    Sparse and imperceptible adversarial attack via a homotopy algorithm

    Mingkang Zhu, Tianlong Chen, and Zhangyang Wang. Sparse and imperceptible adversarial attack via a homotopy algorithm. In International Conference on Machine Learning, pages 12868--12877. PMLR, 2021

    2021

  9. [17]

    Sparse and redundant representations: from theory to applications in signal and image processing

    Michael Elad. Sparse and redundant representations: from theory to applications in signal and image processing. Springer Science & Business Media, 2010

    2010

  10. [18]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248--255. Ieee, 2009

    2009

  11. [19]

    Rethinking the inception architecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818--2826, 2016

    2016

  12. [20]

    Resnet 50

    Brett Koonce and Brett Koonce. Resnet 50. Convolutional neural networks with swift for tensorflow: image recognition and dataset categorization, pages 63--72, 2021

    2021

  13. [21]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012--10022, 2021

    2021

  14. [22]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976--11986, 2022

    2022

  15. [23]

    Robustbench: a standardized adversarial robustness benchmark

    Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020

    2010 arXiv

  16. [24]

    Do adversarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33: 0 3533--3545, 2020

    Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33: 0 3533--3545, 2020

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.