Pith. sign in

REVIEW 4 major objections 5 minor 41 references

Even a fixed X-shaped perturbation touching 1.76% of pixels destabilizes vision-language models across classification, captioning, and VQA.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Perturbing only about 1–2% of pixels along a fixed X-shaped pattern degrades CLIP zero-shot accuracy and transfers to hurt captioning and VQA in six vision-language models.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection Useful controlled stress-test attack with a genuinely new fixed-geometry sparse setting; the classification numbers are solid, but the caption/VQA claims rest on a judge protocol that needs to be disclosed before the cross-task story carries weight. the 4 major comments →

arxiv 2603.28568 v2 pith:GCEKMO4H submitted 2026-03-30 cs.CV

XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

Chengyin Hu , Jiaju Han , Xuemeng Sun , Qike Zhang , Luwei Yang , Lehan Sun , Jiahuan Long , Yiwei Wei
show 1 more author
Jiujiang Guo
This is my paper
classification cs.CV
keywords XSPAsparse adversarial perturbationvision-language modelstransferable attackzero-shot classificationimage captioningvisual question answeringstability evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a fixed X-shaped perturbation touching only about 1.76% of image pixels can destabilize vision-language models across three tasks at once. On COCO, zero-shot accuracy falls by up to 67 points on CLIP-style encoders, with one variant dropping from 97 to 30; GPT-4-based caption consistency falls by up to 58.60 points and VQA correctness by up to 44.38 points across six downstream VLMs. The authors' point is that the perturbation needs neither dense noise nor flexible patch geometry: a rigid geometric prior under a strict budget already reveals a stability gap in shared visual-textual alignment. They frame XSPA as a controlled stress test rather than a universally superior attack, noting that an adaptive saliency-based support attacks more successfully, and conceding that same-image analysis found no three-task targeted success.

Core claim

XSPA confines adversarial modification to two intersecting diagonal line segments that form an X-shaped binary mask, then optimizes a joint loss containing a CLIP classification attack term, target-text attraction, source-text suppression, and smoothness regularization on perturbation magnitude and line continuity. Updates are restricted to the mask via projection, and the optimizer uses momentum and input diversity. The center claim is empirical: with roughly 1.76% of pixels altered, the perturbation transfers to downstream VLMs and degrades zero-shot classification, image captioning, and visual question answering. The authors emphasize that XSPA should be read as a controlled robustness pr

What carries the argument

The central object is the X-shaped sparse mask: a binary support built from two diagonal line segments parameterized by center position, angles, length ratio, and line width, covering about 1.76% of pixels at 224x224. It serves as a hard constraint: every gradient update is masked and clipped so the perturbation never leaves the X. The optimization signal is a weighted sum of classification loss, target-text attraction (pulling the image embedding toward wrong captions/answers), source-text suppression (pushing away from correct semantics), and smoothness regularizers; a two-stage schedule shifts weight from classification toward semantic guidance. The smoothness regularizer is not merely co

Load-bearing premise

The headline caption and VQA numbers rest on an undisclosed GPT-4 judge protocol that must measure semantic disruption rather than odd phrasing, and the authors' own admission of no three-task targeted success leaves the cross-task mechanism unsupported.

What would settle it

Run the caption and VQA evaluations on the same 300 COCO images with a small human-rating study (or a second LLM judge with a published rubric); if the drop in human-rated semantic quality is far smaller than the GPT-4 drop, the claimed cross-task disruption is an artifact of the judge protocol.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, perturbations need not be dense or flexible: a rigid geometric prior under a ~1.76% budget is enough to shift CLIP-style alignment, so stability evaluations that only consider norm-bounded or patch attacks miss a real failure mode.
  • Cross-task transfer: the same perturbation optimized on a white-box surrogate degrades classification, captioning, and VQA on models it was never trained against, meaning a single corrupted image can poison multiple downstream decisions.
  • Attack strength scales smoothly with iteration count and budget (attack success rises from 23% to 70%), so the effect is not a threshold artifact and can be tuned predictably.
  • Smoothness regularization is functionally load-bearing: removing it drops attack success from 70% to 55.67%, so structure-preserving constraints also improve optimization, not just perceptual quality.
  • A structure-aware defense would need to detect or normalize perturbations along diagonal sparse masks; existing norm-based or patch-based defenses may miss them.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The X-shape may act as a sparse line probe that happens to cross semantically salient regions; a natural extension is to test randomized line orientations to see whether the effect depends on the X specifically or on any sparse line covering informative pixels.
  • Since the abstract concedes no three-task targeted success on the same image, the causal story that a shared embedding space transfers disruption across tasks is not established; an alternative reading is that strong classification-side disruption alone explains the downstream degradation, with cross-task semantic guidance adding little.
  • A cheap falsification of the judge dependency: re-run the caption and VQA evaluations with a different LLM judge or with human ratings on a subset; if scores diverge sharply, the headline drops are largely protocol artifacts.
  • The smooth monotonic budget and iteration curves suggest the attack could be exported to adaptive sparse masks chosen by saliency, which the paper's own matched-budget comparison already found stronger; that points toward a general family of sparse structured perturbations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces XSPA, a sparse adversarial attack that confines perturbations to a fixed X-shaped mask (two intersecting diagonal line segments) optimized on a CLIP-style surrogate and transferred to downstream vision-language models. The central claim is that even a highly constrained, low-budget, fixed-geometry perturbation can degrade zero-shot classification, image captioning, and visual question answering across multiple VLMs. Experiments on a 300-image COCO subset report large accuracy drops (e.g., 97→30 on OpenCLIP ViT-B/16, 93→40.67 on OpenAI CLIP ViT-L/14) and large GPT-4-judged drops in caption consistency and VQA correctness (up to −58.60 and −44.38 points respectively). The abstract positions XSPA as a controlled robustness stress test rather than a universally superior attack.

Significance. If the results hold, the paper contributes a novel threat model — fixed-geometry, sparse, structured perturbations — and a diagnostic perspective on VLM robustness. The classification experiments are direct, internally consistent, and supported by the iteration/budget ablations; the authors deserve credit for framing the method as a stress test rather than claiming universal superiority. However, the cross-task semantic-disruption claims rest on an undisclosed LLM-as-a-judge protocol and on analyses that the abstract mentions but the body does not present. These gaps must be closed before the paper's central claims are fully supported.

major comments (4)
  1. [§4.1 Evaluation Metrics; Tables 2 and 3] The caption-consistency and VQA-correctness scores are produced by a 'fixed prompt template and a consistent scoring rubric' using GPT-4, but the prompt, rubric, API/model version, temperature, and any post-processing are never disclosed. All headline cross-task numbers (e.g., −58.60 caption consistency, −44.38 VQA correctness) depend on this protocol. If the judge penalizes surface oddity, hallucinated phrasing, or refusal rather than semantic infidelity, the reported drops would overstate the attack's semantic effect. The authors should provide the exact judge inputs/outputs and validate the judge against human ratings or a second judge model.
  2. [Abstract vs. §4.2–§4.4] The abstract states two findings that do not appear anywhere in the body: (i) 'Matched-budget experiments show that an adaptive saliency Top-k support achieves higher attack success than the fixed X-shaped support, while XSPA provides better SSIM and LPIPS but worse PSNR and MAE' and (ii) 'Same-image analysis finds no three-task targeted success.' Neither the matched-budget comparison nor the same-image analysis is reported in the experimental sections. These are not cosmetic omissions: the first qualifies the paper's claim to not being universally superior, and the second directly bears on the 'correlated semantic failures' mechanism asserted in the abstract and introduction. Add the missing experiments or remove the claims.
  3. [§3.2–§3.4 and §4.1] The default configuration is not specified. The X-mask parameters (ρ_col, ρ_row, θ_1, θ_2, r, b) and the loss-schedule weights (λ_A^ν, λ_B^ν, γ) are introduced symbolically in Section 3, but no numerical values are provided anywhere. This makes the experiments non-reproducible and prevents an independent check of the claimed 1.76% pixel budget. Please include a complete hyperparameter table, including the values for all λs, γ, number of text-pool samples, step size α, momentum μ, and perturbation bound ε.
  4. [§4.4, Table 4 and Figures 5–6] All experimental results appear to come from a single run, with no seeds, error bars, or variance reported. With only 300 evaluation images, the differences in Table 4 (70.00 vs. 59.33 vs. 55.67 ASR) and the trends in Figures 5 and 6 could be partly attributable to run-to-run variation. The paper should report results over multiple seeds (at least for the main tables and ablations) and, if possible, statistical significance for the headline comparisons.
minor comments (5)
  1. [Abstract vs. §3.2] The abstract says XSPA changes 'about 1.04% of image pixels,' while Section 3.2 states the mask covers 'about 1.76% of all pixels.' Reconcile the discrepancy; the exact percentage matters for the sparsity claim.
  2. [§4.2, Figures 2–3] The qualitative figures show perturbations that are hard to see at reproduction size. Add magnified insets of the X-shaped region or separate perturbation maps so readers can verify the geometric constraint.
  3. [§4.1 Implementation Details] The paper says the evaluation set 'follows ITA' but does not specify which COCO-80 split, how the 300 images were selected, or whether the same images are used for all three tasks. Provide the selection procedure and the exact set or a released index.
  4. [§3.3 Eq. (14)] The two-stage schedule is described abstractly. Even if the values are deferred, state explicitly that λ_A, λ_B, and γ are hyperparameters and report their values in the implementation section.
  5. [§2.3 Related Work] Reference [21] is the GPT-4 technical report, not an LLM-as-a-judge methodology; [22] and [39] are appropriate. Consider citing a more direct judge-evaluation protocol if available.

Circularity Check

0 steps flagged

No circularity identified: XSPA's results are measured on disjoint black-box downstream models and an external GPT-4 judge, not on its own optimization objective.

full rationale

The derivation chain is self-contained. XSPA optimizes a perturbation on a surrogate CLIP model using Eq. (5)-(13), combining a classification loss, CLIP-space target-attraction/source-suppression terms, and smoothness regularization. Every headline result is then measured on different systems: Table 1 on four CLIP encoders, Table 2 on six generative VLMs judged by GPT-4, and Table 3 on the same VLMs judged by GPT-4. None of these downstream models or the judge is used to define or fit the perturbation, so the reported accuracy drops and caption/VQA score drops are not forced by construction. The cross-task semantic guidance terms use CLIP text-image cosine similarity, while the caption and VQA evaluation uses GPT-4 judgments of semantic consistency and correctness; these are different metrics, so observing transfer is a non-trivial empirical result. There are no self-citations: the reference list contains no works by the present authors, and no load-bearing claim rests on the authors' prior work. The paper even reports negative results (matched-budget Top-k outperforms XSPA; no three-task targeted success in same-image analysis), which further cuts against a self-serving construction. The main weaknesses are not circularity: the GPT-4 judge prompt and rubric are undisclosed in §4.1, and the abstract's same-image no-three-task-success analysis is absent from the body. These are reproducibility and reporting gaps that could affect interpretation of the caption/VQA numbers, but they do not make any derivation step equivalent to its inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The attack is a composed empirical pipeline, so the ledger is dominated by hand-set configuration values that are never reported (mask geometry, loss weights, schedule, text pools, judge prompt) and by domain assumptions about transfer and judge fidelity that the paper tests empirically but does not validate independently. No invented entities are introduced; the X-mask is a method artifact (the independent variable), not a postulated explanation.

free parameters (5)
  • X-mask geometry (ρ_col, ρ_row, θ₁, θ₂, r, b) = not reported
    Eqs. 2–3 define the paper's signature support via these hand-chosen values; none are specified, and no geometry ablation is reported in the body.
  • Loss weights and stage schedule (λ_clip, λ_tar, λ_src, λ_mag, λ_line, λ_A, λ_B, γ) = not reported
    Eqs. 5, 11, and 14 define the attack objective; without numeric values or an ablation of the schedule, the reported results cannot be reproduced.
  • Optimizer parameters (α, μ, ε) = not reported
    Algorithm 1; N=200 is stated, but step size α, momentum μ, and the default bound ε are never given. The budget ε is the x-axis of Fig. 6 without a default value.
  • Target/source text pools (T_tar, T_src) = not reported
    §3.3 says pools are "predefined from an attack specification" but never lists pool contents, sizes, or how target captions/answers were selected.
  • GPT-4/GPT-4o judge prompt and rubric = not reported
    §4.1 claims "a fixed prompt template and a consistent scoring rubric" without showing either; this instrument carries all caption/VQA headline numbers.
axioms (5)
  • domain assumption Transfer premise: perturbations optimized in a source CLIP encoder space degrade behavior of other VLMs with different encoders and LLMs.
    The paper's central empirical hypothesis, tested in §4.2 but not explained; it is an assumption imported from the transfer-attack literature.
  • domain assumption Judge fidelity: GPT-4-assigned consistency/correctness scores faithfully measure semantic disruption of captions and VQA answers.
    Invoked in §4.1; no human agreement, alternate judge, or rubric disclosure validates it.
  • domain assumption Subset representativeness: the 300 COCO-80 images inherited from ITA fairly represent the distribution for all three tasks.
    §4.1; image IDs and selection seeds are unlisted, so the assumption is untestable externally.
  • domain assumption Importation of DI-FGSM input diversity and MI-FGSM momentum as transferability heuristics.
    §3.4, Eqs. 16–18; accepted from refs [32] and [8] without re-validation.
  • ad hoc to paper Two-stage loss schedule (Eq. 14) prevents local oscillations on the sparse support and improves final attack strength.
    §3.3 justifies this with "avoid ineffective local oscillations"; no ablation isolates the schedule itself.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs." pith.science (2026). https://pith.science/paper/GCEKMO4H

@misc{pith2026260328568,
  author       = {Pith},
  title        = {Pith review of: XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GCEKMO4H}},
  note         = {Machine review of arXiv:2603.28568}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vision-language models (VLMs) share visual-textual representations across zero-shot classification, image captioning, and visual question answering (VQA), creating a pathway through which subtle perturbations may cause failures across tasks. We introduce X-shaped Sparse Pixel Attack (XSPA), a structured attack that restricts perturbations to two intersecting diagonal lines. Within this fixed support, XSPA jointly optimizes a classification objective, cross-task semantic guidance, perturbation magnitude, and linewise smoothness. Under the default setting, it changes about 1.04\% of image pixels. On COCO, XSPA reduces zero-shot accuracy by 52.33 points on OpenAI CLIP ViT-L/14 and 67.00 points on OpenCLIP ViT-B/16; GPT-4-based caption consistency and VQA correctness decrease by up to 58.60 and 44.25 points, respectively. Matched-budget experiments show that an adaptive saliency Top-k support achieves higher attack success than the fixed X-shaped support, while XSPA provides better SSIM and LPIPS but worse PSNR and MAE. Same-image analysis finds no three-task targeted success, indicating that strong surrogate classification effects do not reliably translate into targeted output-space failures. These results position XSPA as a controlled stress test for studying how sparse fixed geometry affects VLM robustness, rather than as a universally superior attack.

Figures

Figures reproduced from arXiv: 2603.28568 by Chengyin Hu, Jiahuan Long, Jiaju Han, Jiujiang Guo, Lehan Sun, Luwei Yang, Qike Zhang, Xuemeng Sun, Yiwei Wei.

Figure 1
Figure 1. Figure 1: Overall framework of XSPA. XSPA optimizes perturbations within a fixed X-shaped sparse mask on a surrogate CLIP model and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative zero-shot classification examples. The top row shows clean images, and the bottom row shows XSPA adversarial images. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative examples on image captioning and visual question answering. Compared with clean outputs, XSPA adversarial outputs [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Grad-CAM attention heatmaps for clean and adversarial samples. Clean examples focus more strongly on the main object, while [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Effect of perturbation budget on attack performance and [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Iteration ablation on four CLIP backbones. The attack [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 7 linked inside Pith

  1. [1]

    Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. 2020. Square Attack: A Query-Efficient Black-Box Adversarial Attack via Random Search. InEuropean Conference on Computer Vision. Springer, Cham, Switzerland, 34 pages. doi:10.1007/978-3-030-58592-1_29

    doi:10.1007/978-3-030-58592-1_29 2020

  2. [2]

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models. arXiv prepri...

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.01390 2023

  3. [3]

    Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer

    Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer

  4. [4]

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jit- sev. 2023. Reproducible Scaling Laws for Contrastive Language-Image Learn- ing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition. IEEE Computer Society, Los Alamitos, CA, US...

    arXiv 2023

  5. [5]

    Xuanming Cui, Alejandro Aparcedo, Young Kyun Jang, and Ser-Nam Lim. 2024. On the Robustness of Large Multimodal Models Against Image Adversarial Attacks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Los Alamitos, CA, USA, 24625–24634. doi:10. 1109/cvpr52733.2024.02325

    arXiv 2024

  6. [6]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. In- structBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv preprint. doi:10.48550/arxiv.2305.06500

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.06500 2023

  7. [7]

    Junhao Dong, Piotr Koniusz, Yifei Zhang, Hao Zhu, Weiming Liu, Xinghua Qu, and Yew-Soon Ong. 2025. Improving Zero-Shot Adversarial Robustness in Vision- Language Models by Closed-form Alignment of Adversarial Path Simplices. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, ...

    2025

  8. [8]

    Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018. Boosting Adversarial Attacks with Momentum. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 9185–9193. doi:10.1109/CVPR. 2018.00957

    arXiv 2018

  9. [9]

    Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. InInternational Conference on Learning Representations. OpenReview.net, Online, 11 pages

    2015

  10. [10]

    Yujun Guo, Jiaming Zhang, YuGang Jiang, and Jitao Sang. 2023. Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre- training Models. InProceedings of the IEEE/CVF International Conference on Computer Vision. IEEE Computer Society, Los Alamitos, CA, USA, 102–111. https://arxiv.org/abs/2307.14061

    Pith/arXiv arXiv 2023

  11. [11]

    Teng-Fang Hsiao, Bo-Lun Huang, Zi-Xiang Ni, Yan-Ting Lin, Hong-Han Shuai, YungHui Li, and HaoWen Cheng. 2024. Natural Light Can Also Be Dan- gerous: Traffic Sign Misinterpretation Under Adversarial Natural Light At- tacks. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. IEEE Computer Society, Los Alamitos, CA, USA, 10 p...

    arXiv 2024

  12. [12]

    Bingyao Huang and Haibin Ling. 2022. SPAA: Stealthy Projector-based Adversar- ial Attacks on Deep Image Classifiers. InIEEE Conference on Virtual Reality and 3D User Interfaces. IEEE, Los Alamitos, CA, USA, 9 pages. doi:10.1109/vr51125. 2022.00073

    arXiv 2022

  13. [13]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. InInternational Conference on Machine Learning. PMLR, Online, 13 pages

    2023

  14. [14]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. InEuropean Conference on Computer Vision. Springer, Cham, Switzerland, 740–755. doi:10.1007/978-3-319-10602-1_48

    doi:10.1007/978-3-319-10602-1_48 2014

  15. [15]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual In- struction Tuning. InAdvances in Neural Information Processing Systems. Curran Associates, Inc., Red Hook, NY, USA, 25 pages

    2023

  16. [16]

    Hanqing Liu, Shouwei Ruan, Yao Huang, Shiji Zhao, and Xingxing Wei. 2025. When Lighting Deceives: Exposing Vision-Language Models’ Illumination Vul- nerability Through Illumination Transformation Attack. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE Computer Society, Los Alamitos, CA, USA, 10485–10495

    2025

  17. [17]

    Yunqi Liu, Xue Ouyang, and Xiaohui Cui. 2025. GLEAM: Enhanced Transferable Adversarial Attacks for Vision-Language Pre-training Models via Global-Local Transformations. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE Computer Society, Los Alamitos, CA, USA, 1665– 1674

    2025

  18. [18]

    Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2018. Towards Deep Learning Models Resistant to Adversarial Attacks. InInternational Conference on Learning Representations. OpenReview.net, Online, 28 pages

    2018

  19. [19]

    Apostolos Modas, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard. 2019. SparseFool: A Few Pixels Make a Big Difference. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Los Alamitos, CA, USA, 9087–9096. doi:10.1109/cvpr.2019.00930

    arXiv 2019

  20. [20]

    Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Los Alamitos, CA, USA, 2574–2582

    2016

  21. [21]

    OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774. doi:10. 48550/arXiv.2303.08774

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774 2023

  22. [22]

    OpenAI. 2024. GPT-4o System Card. OpenAI system card. https://openai.com/ index/gpt-4o-system-card/

    2024

  23. [23]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. InInternational Conference on Machine Learning. PMLR, Online, 8748–8763

    2021

  24. [24]

    Yu Ran, Weijia Wang, Mingjie Li, Lin-Cheng Li, YuanGen Wang, and Jin Li. 2023. Cross-Shaped Adversarial Patch Attack.IEEE Transactions on Circuits and Systems for Video Technology34, 4 (2023), 2289–2303. doi:10.1109/TCSVT.2023.3307150

    arXiv 2023

  25. [25]

    Christian Schlarmann, Naman Deep Singh, Francesco Croce, and Matthias Hein

  26. [26]

    Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan- tam, Devi Parikh, and Dhruv Batra

    Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedan- tam, Devi Parikh, and Dhruv Batra. 2017. Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization. InProceedings of the IEEE Inter- national Conference on Computer Vision. IEEE Computer Society, Los Alamitos, CA, USA, 618–626. doi:10.1109/ICCV.2017.74

    doi:10.1109/iccv.2017.74 2017

  27. [28]

    Jiawei Su, Danilo Vasconcellos Vargas, and Kouichi Sakurai. 2019. One Pixel Attack for Fooling Deep Neural Networks.IEEE Transactions on Evolutionary Computation23, 5 (2019), 828–841. doi:10.1109/tevc.2019.2890858

    arXiv 2019

  28. [29]

    Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. 2023. EVA-CLIP: Improved Training Techniques for CLIP at Scale. arXiv preprint. doi:10.48550/ arxiv.2303.15389

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.15389 2023

  29. [31]

    Yanting Wang, Hongye Fu, Wei Zou, and Jinyuan Jia. 2024. MMCert: Provable Defense against Adversarial Attacks to Multi-modal Models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 24655–24664. doi:10.1109/CVPR52733. 2024.02328

    arXiv 2024

  30. [32]

    Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jihong Wang, Zhou Ren, and Alan L. Yuille. 2019. Improving Transferability of Adversarial Examples with Input Diversity. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 2730–2739. doi:10.1109/CVPR.2019.00284

    arXiv 2019

  31. [33]

    Peng Xie, Yequan Bie, Jiazhen Mao, Yangqiu Song, Yang Wang, Hao Chen, and Kani Chen. 2025. Chain of Attack: On the Robustness of Vision-Language Models Against Transfer-Based Adversarial Attacks. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Los Alamitos, CA, USA, 11970–11979. doi:10.1109/cvpr5...

    arXiv 2025

  32. [34]

    Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang-Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Fe- ichtenhofer. 2024. Demystifying CLIP Data. InInternational Conference on Learning Representations. OpenReview.net, Vienna, Austria, 33 pages. https: //openreview.net/forum?id=5BCFlnfE1g

    2024

  33. [35]

    Jiaming Zhang, Junhong Ye, Xingjun Ma, Yige Li, Yunfan Yang, Yunhao Chen, Jitao Sang, and Dit-Yan Yeung. 2025. AnyAttack: Towards Large-scale Self- supervised Adversarial Attacks on Vision-Language Models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 19900–19909. ...

    arXiv 2025

  34. [36]

    Jiaming Zhang, Qi Yi, and Jitao Sang. 2022. Towards Adversarial Attack on Vision-Language Pre-training Models. InProceedings of the ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, 5005–5013. doi:10.1145/3503161.3547801

    arXiv 2022

  35. [37]

    Jiale Zhao, Xinyang Jiang, Junyao Gao, Yuting Xue, and Cairong Zhao. 2025. One Object, Multiple Lies: A Benchmark for Cross-task Adversarial Attack on Unified Vision-Language Models. InProceedings of the IEEE/CVF International Conference on Computer Vision. IEEE Computer Society, Los Alamitos, CA, USA, 18140–18152

    2025

  36. [38]

    Yunqing Zhao, Tianyu Pang, ChaoHai Du, Xiao Yang, Chongxuan Li, NgaiMan Cheung, and Min Lin. 2023. On Evaluating Adversarial Robustness of Large Vision-Language Models. InAdvances in Neural Information Processing Systems. Curran Associates, Inc., Red Hook, NY, USA, 28 pages. https://arxiv.org/abs/ 2305.16934

    Pith/arXiv arXiv 2023

  37. [39]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685. doi:10.48550/arXiv. 2306.05685

    Pith/arXiv arXiv doi:10.48550/arxiv.2306.05685 2023

  38. [40]

    Xuyang Zhong and Chen Liu. 2025. Sparse-PGD: A Unified Framework for Sparse Adversarial Perturbations Generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, early access. doi:10.1109/TPAMI.2025.3630185

    arXiv 2025

  39. [41]

    Yiqi Zhong, Xianming Liu, Deming Zhai, Junjun Jiang, and Xiangyang Ji. 2022. Shadows Can Be Dangerous: Stealthy and Effective Physical-World Adversarial Attack by Natural Phenomenon. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Los Alamitos, CA, USA, 10 pages. doi:10.1109/cvpr52688.2022.01491 10

    arXiv 2022

  40. [2018]

    InNeurIPS Workshop

    Adversarial Patch. InNeurIPS Workshop. NeurIPS Foundation, Montreal, Canada, 6 pages

  41. [2024]

    InProceedings of the 41st Interna- tional Conference on Machine Learning (Proceedings of Machine Learning Research, Vol

    Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models. InProceedings of the 41st Interna- tional Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, Online, 43685–43704

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.