Pith. sign in

REVIEW 4 major objections 4 minor 25 references

CLIP's adversarial examples collapse under low-pass filtering, and a contrastive rectifier that pulls features toward a low-frequency anchor restores zero-shot accuracy against strong attacks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

CSR detects and repairs adversarial CLIP inputs by comparing features with a low-pass filtered copy and applying a small contrastive PGD correction, claiming SOTA robust accuracy on 16 benchmarks.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection A promising test-time defense for CLIP with a clear spectral story, but the robustness claims are oversold: no adaptive attacks, and the 'AutoAttack' is really only APGD. the 4 major comments →

arxiv 2601.19210 v2 pith:TRGTGNWB submitted 2026-01-27 cs.CV

Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP

Sen Nie , Jie Zhang , Zhuo Wang , Shiguang Shan , Xilin Chen This is my paper
classification cs.CV MSC 68T0768T45
keywords adversarial robustnessCLIPtest-time defensespectral biasfrequency analysiscontrastive learningvision-language modelszero-shot classification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that adversarial examples against CLIP are not arbitrary noise: they live in mid-to-high frequencies because CLIP's gradients and representations are hypersensitive to those bands. This spectral fragility gives a free detector—benign images keep their features under low-pass filtering, adversarial ones collapse—and a free remedy. The proposed Contrastive Spectral Rectification optimizes a small rectification perturbation at inference time that attracts the image's embedding toward the low-pass filtered feature and repels it from the adversarial embedding, then returns the best step of a few PGD iterations. Across 16 zero-shot benchmarks, the authors report roughly 7 points gain over prior test-time defenses under PGD and 18 points under AutoAttack, with a few milliseconds of extra latency, and the same rectifier transfers to segmentation, captioning, and VQA. If right, it means parameter-free test-time purification can protect frozen vision-language models without sacrificing benign accuracy.

Core claim

The central discovery is a spectral asymmetry between benign and adversarial images as seen by CLIP: when mid-to-high frequencies are progressively removed, benign images and Gaussian-corrupted images keep high cosine similarity with their original embeddings, while adversarial images (PGD or AutoAttack, even at 1/255) suffer abrupt feature collapse. The paper attributes this to CLIP's spectral bias—loss gradients and feature displacements concentrate in mid-to-high frequency bands—which makes those bands the efficient place for constrained attacks to operate. From this, the paper builds CSR: detect an input as adversarial when the cosine similarity C(x) between f(x) and f(G_r(x)) falls belo

What carries the argument

The load-bearing object is the spectral-guided contrastive objective L_rec(δ) = sim(f(x+δ), f(G_r(x))) − λ·sim(f(x+δ), f(x)), where G_r is a Gaussian low-pass filter with radius r=40. The first term is the attraction anchor: the low-pass filtered feature approximates the benign manifold. The second term is the repulsion: it pushes the rectified sample away from the adversarial embedding. This is wrapped in an input-adaptive gate (C(x)=cos(f(x), f(G_r(x))) < τ=0.85 triggers rectification) and a greedy selection that returns the iterate with the highest L_rec. The theoretical justification in the appendix shows that under local linearization, the defense gradient opposes adversarial perturbati

Load-bearing premise

That adversarial perturbations are concentrated in mid-to-high frequencies, so the low-pass-filtered image is a faithful benign anchor—if an attacker crafts smooth low-frequency perturbations, or optimizes adversarially through the low-pass gate and rectification, both the detection and the attraction anchor can be bypassed.

What would settle it

Run a low-frequency-constrained PGD attack (masking the perturbation to, say, the inner 30% of Fourier frequencies) at ℓ∞=4/255 against CLIP plus CSR. If robust accuracy stays near the clean level, the high-frequency assumption is sound; if it collapses toward zero, the defense's central premise is falsified. A second check: an adaptive attacker that optimizes the perturbation against the full CSR pipeline, including the detection gate, should be able to defeat the defense if the gate itself is not robust.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • CLIP's frozen weights can be protected at inference without fine-tuning: robust accuracy jumps from near zero to roughly 58–66% on standard benchmarks with only a few gradient steps per adversarial input.
  • The consistency score C(x) doubles as a reliable adversarial detector (reported AUC > 0.95 across 15 datasets), so the same mechanism can flag malicious inputs before downstream processing.
  • Because rectification operates on the raw image, it plugs into any CLIP-based system—zero-shot classification, segmentation, captioning, VQA—without task-specific adaptation.
  • Robustness transfers across CLIP backbones (ViT-B/32, ViT-B/16, ViT-L/14), so the defense scales with model size rather than requiring re-tuning.
  • The defense also withstands diverse attack objectives (cross-modal, targeted, label-free), suggesting it targets a structural property of the perturbation rather than a specific loss.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The core premise—that attacks live in mid-to-high frequencies—is only tested against static attacks. An attacker who constrains perturbations to low-frequency bands, or who optimizes through the low-pass filter and rectifier (adaptive attack), could invalidate both the detection gate and the attraction anchor; a direct experiment would be to run low-frequency-restricted PGD at the same ℓ∞ budget a
  • The detection capability is a byproduct that may be more broadly useful than the rectification: any CLIP-based content filter could use C(x)<τ as a cheap 'is this an adversarial example?' test before human review.
  • The report that rectified accuracy under strong attacks sometimes exceeds clean accuracy hints that the procedure acts as a general semantic denoiser, not just an anti-adversarial device; testing on Gaussian-corrupted or naturally blurred inputs could clarify whether this is a generic spectral prior.
  • The same spectral-bias argument should apply to other vision transformers trained with similar contrastive or supervised objectives; testing whether the attraction-repulsion rectification transfers to non-CLIP encoders would delimit the paper's reach.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Contrastive Spectral Rectification (CSR), a test-time defense for CLIP-style vision-language models. The method builds on the observation that adversarial examples exhibit a sharp drop in feature consistency under low-pass filtering, while benign images do not. CSR uses a Gaussian low-pass filtered embedding as a positive anchor and the original adversarial embedding as a negative anchor in a contrastive objective, solved with a few PGD steps, and applies the rectification only when a cosine-similarity gate detects the input as adversarial. Experiments across 16 classification datasets report large gains over prior test-time defenses against PGD and AutoAttack, including an average +18.1% over SOTA under AutoAttack, with modest inference overhead, and extensions to semantic segmentation, image captioning, and VQA.

Significance. If the robustness numbers hold under adaptive adversaries, CSR would be a practically valuable test-time defense: it is training-free, model-agnostic within the CLIP family, computationally light, and demonstrated on multiple tasks and backbones. The paper's strengths include a clear insight (spectral fragility of adversarial features), an open-source code release, a broad 16-dataset benchmark, and detailed appendices with ablations and AUC analysis. The central limitation is that all robustness claims are measured against non-adaptive attacks; because CSR is a differentiable defense with a thresholded gate, the white-box threat model requires an adaptive attack evaluation before the claimed robustness can be accepted.

major comments (4)
  1. [§5.2, Tables 1–2] The robustness evaluation only considers attacks generated against the undefended CLIP. CSR is an input-dependent differentiable transform (Gaussian filter, CLIP encoder, 3-step PGD, cosine gate from Eq. (5)), so a white-box attacker who knows the defense can optimize through the entire CSR pipeline, including the gate, or use gradient-free/BPDA-style attacks. The central claim of SOTA robustness (e.g., +18.1% over AutoAttack) is not established without an adaptive attack evaluation. Please provide results under end-to-end PGD/AutoAttack through CSR, or at minimum a strong transfer/BPDA baseline, and discuss how the defense behaves when τ, r, λ, and N are known to the attacker.
  2. [Appendix B, Eq. (12); §3.2] The theoretical gradient-conflict argument explicitly assumes Pδ≈δ, i.e., that the adversarial perturbation is dominated by high-frequency content. The paper's own analysis (Figure 4b) shows that low-frequency-constrained attacks are less efficient but become effective at larger budgets (e.g., ε=16/255). If an attacker crafts smooth, low-frequency perturbations, the gate C(x) in Eq. (5) may stay above τ, so no rectification is triggered, and the anchor f(x_low) is itself adversarial, pulling the rectification toward the wrong semantic content. The paper does not evaluate attacks constrained to low-frequency bands or otherwise test the Pδ≈δ assumption. Please add such an evaluation or provide a principled argument for why the assumption holds under the threat model considered.
  3. [§5.1, Implementation Details; Figure 5] Several key hyperparameters — Gaussian filter radius r=40, detection threshold τ=0.85, repulsion weight λ, budget ε=4/255, step size α=2/255, and steps N=3 — are fixed for all experiments, apparently selected on the same evaluation benchmarks. The paper reports no error bars or standard deviations for the robust accuracy numbers in Tables 1–2, and the ablation (Figure 5) shows that results vary noticeably with r and τ. It is important to report variance over repeated runs or multiple seeds, and to justify the hyperparameter choice through a validation procedure that does not use the test set, or to show that CSR is not overly sensitive to these choices.
  4. [Appendix D, AUC curves; §4.1] The detection gate is a load-bearing component: if it fails on an adversarial input, no rectification is applied and robustness falls to zero. Appendix D reports AUC values above 0.95 across 15 datasets, but no standard deviations, no false-positive rate at the operating point τ=0.85, and no per-dataset detection accuracy for benign and adversarial inputs. Since the gate directly determines the effective robustness and the benign accuracy trade-off, please report the confusion matrix at τ=0.85 with error bars, and clarify how τ is chosen.
minor comments (4)
  1. [Table 3] The reported inference overhead is TClean=4.16ms and TAvg=15.17ms. The practical overhead depends on the fraction of adversarial inputs (since only detected adversarial inputs trigger the rectification). Please report the overhead conditioned on the detected-adversarial rate, or state the assumed attack fraction.
  2. [Figure 1] The radar chart legend states 'nearly zero' for some baselines; please replace this with exact numerical values or a note that the values are below 0.5%.
  3. [Algorithm 1, line 12] Line 12 uses 'L' in 'L_best ← L' and 'x* ← x'' but L is not defined in the pseudocode; should be 'L_rec' for consistency with Eq. (6).
  4. [References] Many references are to arXiv preprints; please update to published versions where available, particularly for works that appeared at CVPR/ICML/NeurIPS in 2024–2025.

Circularity Check

0 steps flagged

No definitional circularity; robustness claims are benchmark-measured rather than constructed from fitted quantities.

full rationale

The paper's derivation chain is self-contained. The central robustness result—CSR restoring zero-shot accuracy under PGD and AutoAttack—is measured on external benchmark datasets and compared with external baselines; no fitted parameter is subsequently relabeled as a prediction. The detection score C(x)=cos(f(x),f(G(x))) (Eq. 5) and the rectification objective L_rec (Eq. 6) are defined directly from CLIP features and do not encode the ground-truth class, so optimizing them cannot by construction force the reported accuracy numbers. The Appendix B theoretical argument is explicitly conditional: with a local linearization and the high-pass projection P=I-G, it derives ⟨−∇L_sim,δ⟩ ≈ −∥J(Pδ)∥² ≤ 0, an inequality that actually holds for arbitrary δ and does not depend on assuming Pδ≈δ. The empirical observation that PGD/AutoAttack perturbations concentrate in mid-to-high frequencies is used to motivate the method, not to define the evaluation metric. Hyperparameters such as r=40, τ=0.85, and N=3 are selected through ablations and sensitivity studies, which is model selection rather than constructing the reported result from the test set. The only author-overlapping citation is V-Attack (Nie et al., 2025), used to generate adversarial examples for the segmentation/captioning/VQA extension; that is an evaluation tool, not a load-bearing uniqueness theorem or a premise whose conclusion is assumed. The lack of adaptive-attack evaluation is a robustness limitation, not a circularity: it does not make any derived quantity equal to an input by definition.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The paper's central claim depends on several hand-set hyperparameters and the empirical assumption that AEs live in mid-to-high frequencies; these are not derived from first principles. The rectification variable δ is an optimization variable, not a new physical or model entity.

free parameters (4)
  • Gaussian filter radius r = 40
    Chosen in implementation details (Sec. 5.1) to set the low-pass anchor; directly controls the positive anchor and the feature-consistency gate. No sensitivity/validation split reported.
  • detection threshold τ = 0.85
    Set in Sec. 5.1; determines which inputs are rectified. AUC reported on the same benchmark sets, giving no independent validation for the threshold.
  • repulsion weight λ = not reported
    Eq. (6) introduces λ to balance attraction/repulsion; the paper never states its value, making the method incompletely specified.
  • rectification budget, step size, steps (ε=4/255, α=2/255, N=3) = 4/255, 2/255, 3
    Chosen in Sec. 5.1. These control the strength of correction; no analysis of sensitivity to attack budget beyond fixed settings.
axioms (4)
  • domain assumption Adversarial perturbations are concentrated in mid-to-high frequencies (Pδ≈δ).
    Used in Appendix B Eq. (12) and Sec. 3.2 to argue the defense opposes adversarial gradients. Empirically supported but not guaranteed; low-frequency/smooth attacks would violate it.
  • domain assumption Low-pass filtered CLIP features remain on the natural semantic manifold for benign images.
    The attraction anchor f(x_low) is assumed to guide toward the benign manifold (Sec. 4.2). This is validated observationally on ImageNet subsets but not proven; if the anchor is off-manifold for a given class, rectification may push wrong.
  • ad hoc to paper The threat model is static: attacks are generated without knowledge of the defense.
    All evaluations use PGD/AutoAttack on the base CLIP loss; the defense itself is not treated as part of the attacked function. This is the standard but unstated non-adaptive assumption (Sec. 5.2).
  • standard math Standard math: DFT, cosine similarity, projected gradient descent.
    Used throughout; no issue.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP." pith.science (2026). https://pith.science/paper/TRGTGNWB

@misc{pith2026260119210,
  author       = {Pith},
  title        = {Pith review of: Contrastive Spectral Rectification: Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TRGTGNWB}},
  note         = {Machine review of arXiv:2601.19210}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vision-language models (VLMs) such as CLIP have demonstrated remarkable zero-shot generalization, yet remain highly vulnerable to adversarial examples (AEs). While test-time defenses are promising, existing methods fail to provide sufficient robustness against strong attacks and are often hampered by high inference latency and task-specific applicability. To address these limitations, we start by investigating the intrinsic properties of AEs, which reveals that AEs exhibit severe feature inconsistency under progressive frequency attenuation. We further attribute this to the model's inherent spectral bias. Leveraging this insight, we propose an efficient test-time defense named Contrastive Spectral Rectification (CSR). CSR optimizes a rectification perturbation to realign the input with the natural manifold under a spectral-guided contrastive objective, which is applied input-adaptively. Extensive experiments across 16 classification benchmarks demonstrate that CSR outperforms the SOTA by an average of 18.1% against strong APGD with modest inference overhead. Furthermore, CSR exhibits broad applicability across diverse visual tasks. Code is available at https://github.com/Summu77/CSR.

Figures

Figures reproduced from arXiv: 2601.19210 by Jie Zhang, Sen Nie, Shiguang Shan, Xilin Chen, Zhuo Wang.

Figure 1
Figure 1. Figure 1: Zero-shot adversarial robustness comparison. We evalu￾ate our CSR against CLIP, FARE (adversarial fine-tuning), and TTC (test-time defense) on 16 datasets grouped into General, Fine￾Grained (FG), Scene, and Domain. The radar charts show Top-1 accuracy on benign and adversarial samples under standard PGD (ℓ∞ = 1/255) and the stronger AutoAttack (ℓ∞ = 4/255). To mitigate this vulnerability, Adversarial Fine-… view at source ↗
Figure 2
Figure 2. Figure 2: Mechanism of Contrastive Spectral Rectification (CSR). Leveraging a spectral contrastive strategy, CSR exerts repulsion from the original adversarial feature (solid red star) within the ad￾versarial subspace, while inducing attraction toward the low-pass filtered feature (dashed red star)—an approximation on the benign manifold. This synergy steers the optimization toward the ground￾truth feature (solid bl… view at source ↗
Figure 3
Figure 3. Figure 3: Spectral Consistency Disparity. Cosine similarity be￾tween original and low-pass filtered embeddings across decaying bandwidth radii r. While benign examples maintain high semantic fidelity, adversarial examples exhibit rapid feature collapse. See Appendix C for consistent trends across additional datasets. apply filters with decaying bandwidth radii r to three cate￾gories: clean images, Gaussian-corrupted… view at source ↗
Figure 4
Figure 4. Figure 4: Analysis. (a) The gradient magnitude (left) and representational shift (right) are concentrated in mid-to-high frequency components, indicating that the model is most vulnerable to perturbations in these bands. (b) Attacks constrained to low frequencies are inefficient. (c) The adversarial gradient exhibits consistent negative cosine similarity with the low-frequency constraint gradient. density of loss gr… view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study on the rectification steps N, the Gaussian filter radius r, and the detection threshold τ . More in Appendix G [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Visualization of Semantic Segmentation (top), Image Captioning, and Visual Question Answering (bottom). More in Appendix H. mations and prompt optimization. In contrast, CSR demon￾strates superior efficiency. This confirms that CSR achieves the optimal trade-off between performance and computa￾tional cost, making it suitable for real-time deployment. Results on Different Backbones. To validate the archi￾te… view at source ↗
Figure 7
Figure 7. Figure 7: Additional Visualizations. CIFAR-100 CIFAR-10 FGVCAirCraft DTD Country211 STL10 OxfordPets Flowers102 Caltech256 Caltech101 EuroSAT StanfordCars [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Ablation on the Rectification Steps. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Spectral Analysis of CLIP Feature Consistency on widely used 15 Benchmark Datasets. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: AUC curves for adversarial sample detection on 15 benchmark datasets. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 13 linked inside Pith

  1. [2]

    This forces the visual feature to align closely with the target class’s text embeddingz t,ytarget . • DLR:Moreover, we utilize the targeted version of the Difference of Logits Ratio (DLR) loss: Ltar dlr =− sy −s ytarget sπ1 − 1 2 (sπ3 +s π4 ) .(15) Here, si =z ′ v ⊤zt,i represents the cosine similarities (logits), and π denotes the descending order of the...

    2024

  2. [6]

    and Hwang, W

    Han, J. and Hwang, W. D-tpt: Dimensional entropy maxi- mization for calibrating test-time prompt tuning in vision- language models.arXiv preprint arXiv:2510.09473,

    arXiv

  3. [7]

    Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Parameter-efficient fine-tuning for large models: A com- prehensive survey.Trans. Mach. Learn. Res., 2024,

    2024

  4. [8]

    Sa-attack: Improving adversarial transferability of vision- language pre-training models via self-augmentation

    He, B., Jia, X., Liang, S., Lou, T., Liu, Y ., and Cao, X. Sa-attack: Improving adversarial transferability of vision- language pre-training models via self-augmentation. arXiv preprint arXiv:2312.04913,

    Pith/arXiv arXiv

  5. [9]

    As firm as their foundations: Can open-sourced foundation mod- els be used to create adversarial examples for downstream tasks?CoRR, abs/2403.12693,

    Hu, A., Gu, J., Pinto, F., Kamnitsas, K., and Torr, P. As firm as their foundations: Can open-sourced foundation mod- els be used to create adversarial examples for downstream tasks?CoRR, abs/2403.12693,

    Pith/arXiv arXiv

  6. [11]

    One prompt word is enough to boost adversarial robustness for pre- trained vision-language models

    Li, L., Guan, H., Qiu, J., and Spratling, M. One prompt word is enough to boost adversarial robustness for pre- trained vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24408–24419, 2024a. Li, X., Zhang, W., Liu, Y ., Hu, Z., Zhang, B., and Hu, X. Language-driven anchors for zero-shot adversa...

    arXiv

  7. [12]

    Deepseek-vl: towards real-world vision-language understanding.arXiv preprint arXiv:2403.05525,

    Lu, H., Liu, W., Zhang, B., Wang, B., Dong, K., Liu, B., Sun, J., Ren, T., Li, Z., Yang, H., et al. Deepseek-vl: towards real-world vision-language understanding.arXiv preprint arXiv:2403.05525,

    Pith/arXiv arXiv

  8. [13]

    Safety at scale: A comprehensive survey of large model safety.arXiv preprint arXiv:2502.05206,

    Ma, X., Gao, Y ., Wang, Y ., Wang, R., Wang, X., Sun, Y ., Ding, Y ., Xu, H., Chen, Y ., Zhao, Y ., et al. Safety at scale: A comprehensive survey of large model safety.arXiv preprint arXiv:2502.05206,

    Pith/arXiv arXiv

  9. [14]

    Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083,

    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks.arXiv preprint arXiv:1706.06083,

    Pith/arXiv arXiv

  10. [19]

    Sui, Y ., Huang, W., Yang, W., Zhao, C., Ren, J., and Wang, J. Robust clip-guided deep thinking: A two-stage opti- mization strategy for enhancing adversarial robustness 11 Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP and reliability in lvlms. InICASSP 2025-2025 IEEE In- ternational Conference on Acoustics, Speech and Signal Processi...

    2025

  11. [21]

    P., Zhao, Q., and Chen, B

    Zhou, W., Bai, S., Mandic, D. P., Zhao, Q., and Chen, B. Revisiting the adversarial robustness of vision lan- guage models: a multimodal perspective.arXiv preprint arXiv:2404.19287, 2024a. Zhou, Y ., Xia, X., Lin, Z., Han, B., and Liu, T. Few-shot adversarial prompt learning on vision-language models. Advances in Neural Information Processing Systems, 37:...

    Pith/arXiv arXiv

  12. [22]

    14 Section BTheoretical Analysis of Gradient Conflict

    13 Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP Appendix: Table of Contents Section ARelated Work on Adversarial Attacks . . . . . . 14 Section BTheoretical Analysis of Gradient Conflict . 14 Section CDisparity in Feature Consistency . . . . . . . . . 15 Section DCSR Detection AUC on 15 Datasets . . . . . . 15 Section EAdditional Imp...

    2025

  13. [23]

    To ad- dress black-box transferability, recent works predominantly leverage augmentation and cross-modal priors

    explored universal perturba- tions capable of deceiving diverse downstream tasks. To ad- dress black-box transferability, recent works predominantly leverage augmentation and cross-modal priors. Specifically, SGA (Lu et al., 2023), SA-Attack (He et al., 2023), and DI- RAT (Gao et al.,

    2023

  14. [24]

    pioneers this line of research by optimizing Universal Adversarial Pertur- bations (UAPs) via contrastive learning, and ETU (Zhang et al., 2024c) further elevates their potency through global optimization and mix-based data augmentation. Adversarial Attacks on LVLMs.A critical vulnerabil- ity in Large Vision-Language Models (LVLMs)—such as LLaV A (Liu et ...

    2023

  15. [2008]

    Noever, D. A. and Noever, S. E. M. Reading isn’t believ- ing: Adversarial attacks on multi-modal neurons.arXiv preprint arXiv:2103.10480,

    Pith/arXiv arXiv

  16. [2009]

    Fpt-noise: Dy- namic scene-aware counterattack for test-time adversar- ial defense in vision-language models.arXiv preprint arXiv:2510.20856,

    Deng, J., Li, J., Zhao, Z., and Wang, S. Fpt-noise: Dy- namic scene-aware counterattack for test-time adversar- ial defense in vision-language models.arXiv preprint arXiv:2510.20856,

    arXiv

  17. [2010]

    One perturbation is enough: On generating uni- versal adversarial perturbations against vision-language pre-training models.CoRR, abs/2406.05491,

    Fang, H., Kong, J., Yu, W., Chen, B., Li, J., Xia, S., and Xu, K. One perturbation is enough: On generating uni- versal adversarial perturbations against vision-language pre-training models.CoRR, abs/2406.05491,

    arXiv

  18. [2017]

    Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,

    Maji, S., Rahtu, E., Kannala, J., Blaschko, M., and Vedaldi, A. Fine-grained visual classification of aircraft.arXiv preprint arXiv:1306.5151,

    Pith/arXiv arXiv

  19. [2018]

    Transferable multimodal attack on vision-language pre-training models

    Wang, H., Dong, K., Zhu, Z., Qin, H., Liu, A., Fang, X., Wang, J., and Liu, X. Transferable multimodal attack on vision-language pre-training models. In2024 IEEE Symposium on Security and Privacy (SP), pp. 1722–1740. IEEE, 2024a. Wang, S., Zhang, J., Yuan, Z., and Shan, S. Pre-trained model guided fine-tuning for zero-shot adversarial ro- bustness. InProc...

    Pith/arXiv arXiv

  20. [2019]

    Adversarial attacks against closed-source mllms via feature optimal alignment.arXiv preprint arXiv:2505.21494,

    Jia, X., Gao, S., Qin, S., Pang, T., Du, C., Huang, Y ., Li, X., Li, Y ., Li, B., and Liu, Y . Adversarial attacks against closed-source mllms via feature optimal alignment.arXiv preprint arXiv:2505.21494,

    Pith/arXiv arXiv

  21. [2021]

    V- attack: Targeting disentangled value features for con- trollable adversarial attacks on lvlms.arXiv preprint arXiv:2511.20223,

    Nie, S., Zhang, J., Yan, J., Shan, S., and Chen, X. V- attack: Targeting disentangled value features for con- trollable adversarial attacks on lvlms.arXiv preprint arXiv:2511.20223,

    arXiv

  22. [2022]

    and Balogh, A

    Su, L. and Balogh, A. Atac: Augmentation-based test- time adversarial correction for clip.arXiv preprint arXiv:2511.17362,

    Pith/arXiv arXiv

  23. [2023]

    N., and Fink, O

    Dong, H., Chatzi, E. N., and Fink, O. Towards robust multimodal open-set test-time adaptation via adaptive entropy-aware optimization. InICLR. OpenReview.net, 2025a. Dong, J., Koniusz, P., Feng, L., Zhang, Y ., Zhu, H., Liu, W., Qu, X., and Ong, Y .-S. Robustifying zero-shot vision language models by subspaces alignment. InProceedings of the IEEE/CVF Inte...

    Pith/arXiv arXiv

  24. [2024]

    Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories

    9 Test-Time Defense towards Zero-shot Adversarial Robustness of CLIP Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recogni- tion workshop, pp. 178–178. IEEE,

    2004

  25. [2025]

    Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond.arXiv preprint arXiv:2308.12966,

    Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile vision- language model for understanding, localization, text read- ing, and beyond.arXiv preprint arXiv:2308.12966,

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.