Pith. sign in

REVIEW 1 major objections 4 minor 70 references

Adversarial examples in vision-language pretrained models push embeddings off the data manifold, which makes them detectable by simple geometric distance scores.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 01:12 UTC pith:JLV66I2A

load-bearing objection A genuinely useful empirical study of classical geometric detectors for VLPs, but the theory's central condition is unverified and the abstract overstates what the tables show. the 1 major comments →

arxiv 2607.14737 v1 pith:JLV66I2A submitted 2026-07-16 cs.CV cs.LG

GeoDetect: Geometric Adversarial Detection for VLPs

Afsaneh Hasanebrahimi , Hanxun Huang , Christopher Leckie , James Bailey , Sarah Erfani This is my paper
classification cs.CV cs.LG
keywords adversarial detectionvision-language pre-trained modelsembedding geometryanisotropyoff-manifoldlocal intrinsic dimensionalityMahalanobis distanceadaptive attacks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Vision-language models embed images and text in shared spaces with a strongly anisotropic structure: representations concentrate along a few dominant directions. The paper argues that this anisotropy is what makes adversarial examples detectable: an optimal adversarial embedding, derived from KL-divergence maximization, is pushed out of the clean data manifold, so adversarial inputs sit measurably farther from randomly sampled clean points than their clean counterparts do. On that basis the paper builds GeoDetect, which applies four classical geometric scores—LID, k-NN distance, Mahalanobis distance, and kernel density estimation—to VLP embeddings and reports 96–100% AUC across classification and retrieval tasks on standard attacks, with weaker performance (roughly 50–95% AUC) when attacks are explicitly optimized against the detector. If the picture is right, detecting adversarial inputs in multimodal models does not require retraining or task-specific classifiers; simple geometry suffices.

Core claim

Adversarial examples in VLPs have larger expected distances to randomly sampled clean embeddings than clean inputs do, because perturbations push representations off the clean manifold. Under anisotropic covariance Σ and Gaussian local structure, the KL-optimal adversarial embedding is z'* = (Σ+λI)^(-1)λz, which suppresses components along dominant (tangent) directions and amplifies components in the less-used (normal) directions. Lemma 2 states this yields a manifold deviation γ > α; Theorem 1 shows that if γ > 3α, the expected-distance gap follows. GeoDetect operationalizes this via LID, k-NN, Mahalanobis, and KDE scores computed against clean reference batches.

What carries the argument

The load-bearing object is the anisotropic covariance Σ of the VLP embedding space. The paper models clean embeddings as Gaussian draws with covariance Σ and adversarial draws as the KL-optimal shift z'* = (Σ+λI)^(-1)λz, then measures manifold proximity: clean points satisfy ∥z_i − M∥ ≤ α and adversarial points satisfy ∥z'_i − M∥ ≥ γ. The condition γ > 3α converts off-manifold motion into the expected-distance inequality that geometric detectors rely on. The detector itself is the family of geometric scores (LID, k-NN, Mahalanobis, KDE) computed against a clean reference batch; LID is extracted layer-wise, and the other scores operate on the image encoder output.

Load-bearing premise

The argument stands on the unmeasured condition that real adversarial attacks push VLP embeddings off the clean manifold by a margin γ larger than three times the clean proximity α, and that the adversarial embedding distribution follows the KL-optimal Gaussian form—if either fails, the expected-distance gap in Eq. (5) has no theoretical support, and the experiments show such failures for some fused-model attack combinations.

What would settle it

Measure, for any VLP and attack, the true manifold deviation of adversarial embeddings: estimate α from clean embeddings' residuals to their local PCA subspace and γ from adversarial embeddings' residuals. If for some model or attack γ ≤ 3α while the detector still reports high AUC, the distance gap is not driven by the claimed off-manifold mechanism. Conversely, a single family of inputs that changes the model's prediction while staying within the clean manifold (γ ≈ α) would defeat GeoDetect-style scores; the paper's own ALBEF/TCL Sepmulti results (AUC ~33–57%) come close to providing that c

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Adversarial inputs can be distinguished from clean ones in VLPs using only embedding geometry, without fine-tuning or task-specific classifiers.
  • The same geometric scores transfer across zero-shot classification and image-text retrieval, and across aligned (CLIP-style) and fused (ALBEF/TCL-style) architectures.
  • Under non-adaptive attacks, the method reaches 96–100% AUC in-distribution, indicating that geometry alone is a practical low-cost defense.
  • When attacks are optimized against the detector, AUC drops into the 50–95% range, showing the boundary of a purely geometric signal.
  • Because the off-manifold signature follows from anisotropy, the same detection principle should extend to other contrastively trained multimodal models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The theory's premise that γ > 3α is never measured; the paper verifies off-manifold motion only for CLIP-CNN on ImageNet-V2. Checking this condition on fused models could explain why GeoDetect collapses to chance for ALBEF/TCL against Sepmulti and adaptive k-NN attacks (AUC ≈ 33–57%).
  • If the off-manifold distance gap is genuine, the same geometric scores could detect out-of-distribution or corrupted inputs in VLPs, not just adversarial ones—an extension the paper does not test.
  • The near-chance results under adaptive attacks suggest that any purely geometric detector can be evaded if the attacker includes the score in its loss; a more durable defense would need to combine geometry with an invariance or certified component.
  • Because the reference batch is drawn from clean data, the detector's reliability depends on the query distribution matching the reference distribution; deployment shifts could require periodic recalibration.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper proposes GeoDetect, a task-agnostic, geometry-based adversarial-example detection method for vision-language pre-trained models (VLPs). The authors observe that VLP embedding spaces are more anisotropic than unimodal vision model spaces and argue theoretically that adversarial embeddings are pushed off the clean data manifold, so adversarial examples have larger expected distance to random clean points than clean examples do (Theorem 1). Based on this, GeoDetect applies LID, k-NN, Mahalanobis, and KDE scores to image or multimodal embeddings and uses thresholds or logistic regression to detect adversarial inputs. The method is evaluated on zero-shot classification (CIFAR, ImageNet, STL-10, Food-101) and image-text retrieval (Flickr30k, COCO) for CLIP-ViT, CLIP-CNN, ALBEF, and TCL under Sep-Attack, Co-Attack, and adaptive attacks.

Significance. If the central claim were fully supported, the paper would provide a useful, theoretically motivated, model-agnostic detector for VLPs that requires no fine-tuning. The empirical scope is broad: multiple architectures (aligned and fused), multiple datasets, several attack families, and adaptive attacks. Strengths include a simple and reproducible pipeline, public code, and a clear presentation of geometric metrics. However, the blanket claim in the abstract of 'consistently' and 'reliably' detecting AEs is contradicted by the paper's own numerical results in several key cells, and the theory's central premise (γ>3α) is never directly measured. The paper is therefore not yet at the level of a strong, fully supported claim, but the underlying idea and much of the empirical evaluation are worth pursuing in a major revision.

major comments (1)
  1. The appendix is referenced extensively (A–E) for proofs, attack details, and ablations, but the appendix content is not provided in the submitted version. This prevents verification of Lemma 1, Lemma 2, Theorem 1, and the exact adaptive-attack construction. Please ensure the appendix is included in any revised submission.
minor comments (4)
  1. [Eq. (1)] The notation {z_j}^n_{j=1} and later {z_j}^n_{j≠i} is inconsistent. Define the reference batch unambiguously.
  2. [Sec. 3.1] The paper says 'all without requiring fine-tuning', but the pipeline fits thresholds and logistic-regression weights on a calibration split. Please clarify that no model fine-tuning is required, while some detector hyperparameters are fitted.
  3. [Sec. 4.3] The abbreviations 'N-adap.' and 'Adap.' in Table 3 are not expanded in the caption. Spell out 'non-adaptive' and 'adaptive'.
  4. [Sec. 3.2] The isotropic measures I1 and I2 are defined for an embedding matrix Z, but Fig. 2a reports them 'across iterations'. Specify what quantity is computed per iteration and how the embeddings are subsampled.

Circularity Check

0 steps flagged

No significant circularity: the central theorem is conditional and the detection results are measured rather than fitted.

full rationale

The paper's derivation chain is: Assumption 1 (anisotropic covariance) and Assumption 2 (Gaussian clean/adversarial embeddings plus manifold proximity) lead to Lemma 1 (an optimal adversarial embedding of the form z'* = (Σ+λI)^{-1}λz), then Lemma 2 (such embeddings move off the clean manifold), and then Theorem 1 (if γ > 3α, adversarial embeddings have larger expected distance to random clean points). Theorem 1 is explicitly conditional on the unmeasured threshold γ > 3α; it does not assert that every attack satisfies this condition, and the paper does not fit any parameter from detection labels into this inequality. The empirical detection scores (LID, k-NN, Mahalanobis, KDE) are computed directly from embeddings, with thresholds or logistic-regression coefficients fit on a calibration split and evaluated on a held-out split. The reported AUCs are therefore direct measurements of the geometric scores, not predictions forced by the theorem. The fact that α and γ are never measured, and that the off-manifold premise is only qualitatively verified for CLIP-CNN on ImageNet-V2 (Fig. 3), is a genuine evidential gap and a correctness risk — especially given the near-chance ALBEF/TCL Sepmulti and adaptive k-NN cells — but it is not circularity: a failed premise is different from a conclusion that is equivalent to its input. The self-citations, notably [38] for LID methodology, are to established detection tools and do not carry the central derivation; [63] is an external source for the optimal-embedding inspiration. No equation in the main text reduces to its own input by construction, and no fitted parameter is renamed as a prediction. Accordingly, the correct circularity finding is 'no significant circularity'.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The theoretical chain (Sec 3.2) rests on two stated assumptions (anisotropy, manifold proximity + Gaussianity) plus an implicit attack-model claim (Lemma 1: real AEs resemble the KL-optimal embedding). None of the theorem's thresholds (α, γ, λ) is estimated from data. The detector itself contributes standard hyperparameters (k, KDE bandwidth) and a calibration classifier fit on 80% of the same test distribution, so the reported AUC measures within-attack separability rather than generalization to unseen attack families.

free parameters (5)
  • LID neighborhood size k = 100 (CLIPCNN/ViT), 40 (ALBEF/TCL)
    Tuning hyperparameter chosen per model family (Sec 4.1); LID scores and thus detection depend on it, with sensitivity deferred to App C.3.
  • k-NN neighborhood size k = 10
    Chosen by hand (Sec 4.1); the k-NN score is the mean distance to the k nearest references.
  • KDE bandwidth = 0.1
    Gaussian kernel bandwidth set globally (Sec 4.1); KDE scores depend directly on it.
  • Calibration thresholds / LID logistic-regression weights = fit on 80% of test-set scores
    Threshold τ and LR weights are fitted on the calibration split (Sec 3.1, 4.1), so reported AUC measures separability within known attack types, not zero-shot detection of unseen attack families.
  • Theorem bounds α, γ, λ
    Never estimated. Theorem 1 requires γ > 3α but no measurement of manifold proximity (α) or off-manifold separation (γ) is given for any model or attack; λ appears in Lemma 1's closed form.
axioms (4)
  • domain assumption VLP embedding covariance is anisotropic (Σ ≠ cI, eigenvalues σ1 ≫ σ2 ≫ ... ≫ σD)
    Assumption 1, Sec 3.2. Empirically supported by Fig 2 (isotropy measures and cER) but asserted as an assumption for the theory.
  • domain assumption Clean embeddings lie within α of a manifold, and clean/adversarial embeddings are Gaussian: p(z)∼N(μz,Σ), q(z')∼N(μz',Σ')
    Assumption 2, Sec 3.2. The 'locally Gaussian' justification is imported from [68] and is not verified on VLP embedding neighborhoods.
  • domain assumption Real adversarial attacks behave like the KL-maximizing optimal embedding z'* = (Σ+λI)^{-1}λz
    Lemma 1, Sec 3.2. Verified only qualitatively in Fig 3 for CLIP-CNN on ImageNet-V2; the Tab 2–3 failures for Sepmulti on fused VLPs suggest it does not hold universally.
  • domain assumption The defender has a clean reference batch of the same distribution as the query data
    Sec 3.1, following [38]. All scores are computed against clean reference embeddings assumed available to the defender.

pith-pipeline@v1.3.0-alltime-deepseek · 15659 in / 23614 out tokens · 215508 ms · 2026-08-02T01:12:21.918750+00:00 · methodology

0 comments
read the original abstract

Vision-language pre-trained models (VLPs) are widely used in real-world applications. However, they remain vulnerable to adversarial attacks. Although adversarial detection methods have demonstrated success in single-modality settings (either vision or language), their effectiveness and reliability in multimodal models such as VLPs remain largely unexplored. In this work, we study the geometry of VLP embedding spaces and observe structured anisotropy that differs from unimodal vision models. Our theoretical analysis shows that under this anisotropic structure, adversarial attacks increase the expected geometric separation between clean and adversarial examples (AEs). Specifically, we demonstrate that AEs consistently exhibit greater expected distances to randomly sampled points than their clean counterparts, indicating that AEs tend to push representations out of manifold regions. Building on these insights, we propose GeoDetect, which leverages these off-manifold deviations via geometric scores to identify AEs. Through comprehensive evaluations, we show that our approach reliably detects AEs across diverse VLP architectures and threat settings, covering unimodal and multimodal attacks as well as adaptive attacks, thereby providing a robust and practical approach to improving the safety and reliability of these models.

Figures

Figures reproduced from arXiv: 2607.14737 by Afsaneh Hasanebrahimi, Christopher Leckie, Hanxun Huang, James Bailey, Sarah Erfani.

Figure 1
Figure 1. Figure 1: Pipeline of geometric score extraction for GeoDetect. Recent research has explored adversarial training as a strategy to enhance the zero-shot robustness of VLPs [40, 48, 54]. However, adversarial training is computationally expensive [39, 55] and often involves a trade-off between model performance and robustness [52, 64]. Detecting AEs presents a more flexible alternative by allowing the model to identif… view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of isotropy metrics and effective ranking for VLPs vs. supervised model. Iteration refers to the batch index during evaluation. We begin by empirically verifying that VLP embeddings are anisotropic, concentrated along a few dominant directions, which motivates our assumptions. Building on this, we show that the principal directions of adversarial embeddings differ significantly from those of cle… view at source ↗
Figure 3
Figure 3. Figure 3: Verification of Lemma 2: adversarial data in VLPs are off-manifold. Σ ̸= cI for any scalar constant c, indicating anisotropy property. Consequently, its eigenvalues vary significantly across dimensions (σ1 ≫ σ2 ≫ ... ≫ σD). Assumption 2 (Manifold Proximity). Clean embeddings reside on a manifold \mathcal {M} , such that the distance of a data point z_i from the manifold satisfies a proximity condition \|z_… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

70 extracted references · 5 linked inside Pith

  1. [1]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Agrawal, A., Batra, D., Parikh, D., Kembhavi, A.: Don’t just assume; look and answer: Overcoming priors for visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    2018

  2. [2]

    Applied Intelligence53(1), 509–531 (2023)

    Aldahdooh, A., Hamidouche, W., Déforges, O.: Revisiting model’s uncertainty and confidences for adversarial example detection. Applied Intelligence53(1), 509–531 (2023)

    2023

  3. [3]

    In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015) 16 A

    Amsaleg, L., Chelly, O., Furon, T., Girard, S., Houle, M.E., Kawarabayashi, K.i., Nett, M.: Estimating local intrinsic dimensionality. In: Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2015) 16 A. Hasanebrahimi et al

    2015

  4. [4]

    Transactions of the Association for Computational Linguistics4, 385–399 (2016)

    Arora, S., Li, Y., Liang, Y., Ma, T., Risteski, A.: A latent variable model approach to pmi-based word embeddings. Transactions of the Association for Computational Linguistics4, 385–399 (2016)

    2016

  5. [5]

    In: Proceedings of the International Conference on Machine Learning (ICML) (2018)

    Athalye, A., Carlini, N., Wagner, D.: Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In: Proceedings of the International Conference on Machine Learning (ICML) (2018)

    2018

  6. [6]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)35(8), 1798–1828 (2013)

    Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)35(8), 1798–1828 (2013)

    2013

  7. [7]

    In: European Conference on Computer Vision (ECCV) (2014)

    Bossard, L., Guillaumin, M., Van Gool, L.: Food-101–mining discriminative compo- nents with random forests. In: European Conference on Computer Vision (ECCV) (2014)

    2014

  8. [8]

    Annals of Statistics38(5), 2916–2957 (2010)

    Botev, Z.I., Grotowski, J.F., Kroese, D.P.: Kernel density estimation via diffusion. Annals of Statistics38(5), 2916–2957 (2010)

    2010

  9. [9]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2022)

    Bryniarski, O., Hingun, N., Pachuca, P., Wang, V., Carlini, N.: Evading adver- sarial example detection defenses with orthogonal projected gradient descent. In: Proceedings of the International Conference on Learning Representations (ICLR) (2022)

    2022

  10. [10]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Chen, H., Ding, G., Liu, X., Lin, Z., Liu, J., Han, J.: IMRAM: Iterative matching with recurrent attention memory for cross-modal image-text retrieval. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    2020

  11. [11]

    In: European Conference on Computer Vision (ECCV) (2020)

    Chen, Y.C., Li, L., Yu, L., El Kholy, A., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: UNITER: Universal image-text representation learning. In: European Conference on Computer Vision (ECCV) (2020)

    2020

  12. [12]

    In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (2011)

    Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (2011)

    2011

  13. [13]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    Cohen, G., Sapiro, G., Giryes, R.: Detecting adversarial samples using influence functions and nearest neighbors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020)

    2020

  14. [14]

    IEEE Transactions on Information Theory13(1), 21–27 (1967)

    Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Transactions on Information Theory13(1), 21–27 (1967)

    1967

  15. [15]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2009)

    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A large-scale hierarchical image database. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2009)

    2009

  16. [16]

    arXiv preprint arXiv:1703.00410 (2017)

    Feinman, R., Curtin, R.R., Shintre, S., Gardner, A.B.: Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410 (2017)

    Pith/arXiv arXiv 2017

  17. [17]

    Information Fusion91, 424–444 (2023)

    Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., Hussain, A.: Multimodal senti- ment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges and future directions. Information Fusion91, 424–444 (2023)

    2023

  18. [18]

    arXiv preprint arXiv:1702.06280 (2017)

    Grosse, K., Manoharan, P., Papernot, N., Backes, M., McDaniel, P.: On the (statis- tical) detection of adversarial examples. arXiv preprint arXiv:1702.06280 (2017)

    Pith/arXiv arXiv 2017

  19. [19]

    arXiv preprint arXiv:2312.04403 (2023)

    Han, D., Jia, X., Bai, Y., Gu, J., Liu, Y., Cao, X.: OT-Attack: Enhancing adversarial transferability of vision-language models via optimal transport optimization. arXiv preprint arXiv:2312.04403 (2023)

    Pith/arXiv arXiv 2023

  20. [20]

    arXiv preprint arXiv:2312.04913 (2023) GeoDetect 17

    He, B., Jia, X., Liang, S., Lou, T., Liu, Y., Cao, X.: SA-Attack: Improving adversarial transferability of vision-language pre-training models via self-augmentation. arXiv preprint arXiv:2312.04913 (2023) GeoDetect 17

    Pith/arXiv arXiv 2023

  21. [21]

    In: IEEE International Conference on Data Mining Workshops (2013)

    Houle, M.E.: Dimensionality, discriminability, density and distance distributions. In: IEEE International Conference on Data Mining Workshops (2013)

    2013

  22. [22]

    In: Similarity Search and Applications (2017)

    Houle, M.E.: Local intrinsic dimensionality i: an extreme-value-theoretic foundation for similarity applications. In: Similarity Search and Applications (2017)

    2017

  23. [23]

    In: IEEE International Conference on Data Mining Workshops (2012)

    Houle, M.E., Kashima, H., Nett, M.: Generalized expansion dimension. In: IEEE International Conference on Data Mining Workshops (2012)

    2012

  24. [24]

    In: Proceedings of the Annual ACM Symposium on Theory of Computing (2002)

    Karger, D.R., Ruhl, M.: Finding nearest neighbors in growth-restricted metrics. In: Proceedings of the Annual ACM Symposium on Theory of Computing (2002)

    2002

  25. [25]

    In: International Joint Conference on Neural Networks (IJCNN) (2020)

    Kherchouche, A., Fezza, S.A., Hamidouche, W., Déforges, O.: Detection of adversar- ial examples in deep neural networks with natural scene statistics. In: International Joint Conference on Neural Networks (IJCNN) (2020)

    2020

  26. [26]

    Krizhevsky, A.: Learning multiple layers of features from tiny images. Tech. rep., University of Toronto (2009)

    2009

  27. [27]

    Springer Science & Business Media (2006)

    Lee, J.M.: Riemannian manifolds: an introduction to curvature. Springer Science & Business Media (2006)

    2006

  28. [28]

    Advances in Neural Information Processing Systems (NeurIPS) (2018)

    Lee, K., Lee, K., Lee, H., Shin, J.: A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances in Neural Information Processing Systems (NeurIPS) (2018)

    2018

  29. [29]

    In: Proceedings of the International Conference on Machine Learning (ICML) (2025)

    Levi, M.Y., Gilboa, G.: The double-ellipsoid geometry of CLIP. In: Proceedings of the International Conference on Machine Learning (ICML) (2025)

    2025

  30. [30]

    In: Proceedings of the International Conference on Machine Learning (ICML) (2022)

    Li, J., Li, D., Xiong, C., Hoi, S.: BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: Proceedings of the International Conference on Machine Learning (ICML) (2022)

    2022

  31. [31]

    Advances in Neural Information Processing Systems (NeurIPS) (2021)

    Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems (NeurIPS) (2021)

    2021

  32. [32]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Li, L., Guan, H., Qiu, J., Spratling, M.: One prompt word is enough to boost adversarial robustness for pre-trained vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  33. [33]

    In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020)

    Li, L., Ma, R., Guo, Q., Xue, X., Qiu, X.: BERT-attack: Adversarial attack against BERT using BERT. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020)

    2020

  34. [34]

    Advances in Neural Information Processing Systems (NeurIPS) (2022)

    Liang, V.W., Zhang, Y., Kwon, Y., Yeung, S., Zou, J.Y.: Mind the gap: Understand- ing the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Systems (NeurIPS) (2022)

    2022

  35. [35]

    In: European Conference on Computer Vision (ECCV) (2014)

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: European Conference on Computer Vision (ECCV) (2014)

    2014

  36. [36]

    In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2023)

    Lu, D., Wang, Z., Wang, T., Guan, W., Gao, H., Zheng, F.: Set-level guidance attack: Boosting adversarial transferability of vision-language pre-training models. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2023)

    2023

  37. [37]

    Advances in Neural Information Processing Systems (NeurIPS) (2019)

    Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: Pretraining task-agnostic vi- siolinguistic representations for vision-and-language tasks. Advances in Neural Information Processing Systems (NeurIPS) (2019)

    2019

  38. [38]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2018) 18 A

    Ma, X., Li, B., Wang, Y., Erfani, S.M., Wijewickrema, S., Schoenebeck, G., Song, D., Houle, M.E., Bailey, J.: Characterizing adversarial subspaces using local intrin- sic dimensionality. In: Proceedings of the International Conference on Learning Representations (ICLR) (2018) 18 A. Hasanebrahimi et al

    2018

  39. [39]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2018)

    Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learn- ing models resistant to adversarial attacks. In: Proceedings of the International Conference on Learning Representations (ICLR) (2018)

    2018

  40. [40]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2023)

    Mao, C., Geng, S., Yang, J., Wang, X., Vondrick, C.: Understanding zero-shot adversarial robustness for large-scale models. In: Proceedings of the International Conference on Learning Representations (ICLR) (2023)

    2023

  41. [41]

    Resonance4(6), 20–26 (1999)

    McLachlan, G.J.: Mahalanobis distance. Resonance4(6), 20–26 (1999)

    1999

  42. [42]

    Advances in Neural Information Processing Systems (NeurIPS) (2022)

    Ming, Y., Cai, Z., Gu, J., Sun, Y., Li, W., Li, Y.: Delving into out-of-distribution detection with vision-language representations. Advances in Neural Information Processing Systems (NeurIPS) (2022)

    2022

  43. [43]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2018)

    Mu, J., Viswanath, P.: All-but-the-top: Simple and effective postprocessing for word representations. In: Proceedings of the International Conference on Learning Representations (ICLR) (2018)

    2018

  44. [44]

    arXiv preprint arXiv:1807.03748 (2018)

    van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

    Pith/arXiv arXiv 2018

  45. [45]

    In: Proceedings of the International Conference on Machine Learning (ICML) (2021)

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: Proceedings of the International Conference on Machine Learning (ICML) (2021)

    2021

  46. [46]

    In: European Signal Processing Conference (EUSIPCO) (2007)

    Roy, O., Vetterli, M.: The effective rank: A measure of effective dimensionality. In: European Signal Processing Conference (EUSIPCO) (2007)

    2007

  47. [47]

    In: Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW) (2023)

    Schlarmann, C., Hein, M.: On the adversarial robustness of multi-modal foundation models. In: Proceedings of the IEEE International Conference on Computer Vision Workshops (ICCVW) (2023)

    2023

  48. [48]

    In: Proceedings of the International Conference on Machine Learning (ICML) (2024)

    Schlarmann, C., Singh, N.D., Croce, F., Hein, M.: Robust CLIP: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models. In: Proceedings of the International Conference on Machine Learning (ICML) (2024)

    2024

  49. [49]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Shah, M., Chen, X., Rohrbach, M., Parikh, D.: Cycle-consistency for robust visual question answering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    2019

  50. [50]

    EURASIP Journal on Information Security2020, 1–10 (2020)

    Sotgiu, A., Demontis, A., Melis, M., Biggio, B., Fumera, G., Feng, X., Roli, F.: Deep neural rejection against adversarial examples. EURASIP Journal on Information Security2020, 1–10 (2020)

    2020

  51. [51]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2014)

    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I.J., Fer- gus, R.: Intriguing properties of neural networks. In: Proceedings of the International Conference on Learning Representations (ICLR) (2014)

    2014

  52. [52]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2019)

    Tsipras, D., Santurkar, S., Engstrom, L., Turner, A., Madry, A.: Robustness may be at odds with accuracy. In: Proceedings of the International Conference on Learning Representations (ICLR) (2019)

    2019

  53. [53]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2020)

    Wang, L., Huang, J., Huang, K., Hu, Z., Wang, G., Gu, Q.: Improving neural language generation with spectrum control. In: Proceedings of the International Conference on Learning Representations (ICLR) (2020)

    2020

  54. [54]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Wang, S., Zhang, J., Yuan, Z., Shan, S.: Pre-trained model guided fine-tuning for zero-shot adversarial robustness. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  55. [55]

    In: Proceedings of the International Conference on Learning Representations (ICLR) (2020)

    Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., Gu, Q.: Improving adversarial robust- ness requires revisiting misclassified examples. In: Proceedings of the International Conference on Learning Representations (ICLR) (2020)

    2020

  56. [56]

    In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024) GeoDetect 19

    Wang, Z., Li, X., Zhu, H., Xie, C.: Revisiting adversarial training at scale. In: Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024) GeoDetect 19

    2024

  57. [57]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)45(10), 12113–12132 (2023)

    Xu, P., Zhu, X., Clifton, D.A.: Multimodal learning with transformers: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)45(10), 12113–12132 (2023)

    2023

  58. [58]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Xu, X., Chen, X., Liu, C., Rohrbach, A., Darrell, T., Song, D.: Fooling vision and language models despite localization and attention mechanism. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    2018

  59. [59]

    In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

    Yang, J., Duan, J., Tran, S., Xu, Y., Chanda, S., Chen, L., Zeng, B., Chilimbi, T., Huang, J.: Vision-language pre-training with triple contrastive learning. In: Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

    2022

  60. [60]

    National Science Review11(12), nwae403 (2024)

    Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on multimodal large language models. National Science Review11(12), nwae403 (2024)

    2024

  61. [61]

    Advances in Neural Information Processing Systems (NeurIPS) (2023)

    Yin, Z., Ye, M., Zhang, T., Du, T., Zhu, J., Liu, H., Chen, J., Wang, T., Ma, F.: VLATTACK: Multimodal adversarial attacks on vision-language tasks via pre- trained models. Advances in Neural Information Processing Systems (NeurIPS) (2023)

    2023

  62. [62]

    Transactions of the Association for Computational Linguistics2, 67–78 (2014)

    Young, P., Lai, A., Hodosh, M., Hockenmaier, J.: From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics2, 67–78 (2014)

    2014

  63. [63]

    In: International Conference on Data Mining (ICDM) (2024)

    Zhang, C., Jin, M., Yu, Q., Liu, C., Xue, H., Jin, X.: Goal-guided generative prompt injection attack on large language models. In: International Conference on Data Mining (ICDM) (2024)

    2024

  64. [64]

    In: Proceedings of the International Conference on Machine Learning (ICML) (2019)

    Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., Jordan, M.: Theoretically princi- pled trade-off between robustness and accuracy. In: Proceedings of the International Conference on Machine Learning (ICML) (2019)

    2019

  65. [65]

    In: European Conference on Computer Vision (ECCV) (2024)

    Zhang, J., Ma, X., Wang, X., Qiu, L., Wang, J., Jiang, Y.G., Sang, J.: Adversarial prompt tuning for vision-language models. In: European Conference on Computer Vision (ECCV) (2024)

    2024

  66. [66]

    In: Proceedings of the ACM International Conference on Multime- dia (ACM MM) (2022)

    Zhang, J., Yi, Q., Sang, J.: Towards adversarial attack on vision-language pre- training models. In: Proceedings of the ACM International Conference on Multime- dia (ACM MM) (2022)

    2022

  67. [67]

    In: Proceedings of the ACM International Conference on Multimedia (ACM MM) (2024)

    Zhang, Y., Xie, R., Chen, J., Sun, X., Wang, Y.: PIP: Detecting adversarial examples in large vision-language models via attention patterns of irrelevant probe questions. In: Proceedings of the ACM International Conference on Multimedia (ACM MM) (2024)

    2024

  68. [68]

    In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2007)

    Zhao, D., Lin, Z., Tang, X.: Laplacian pca and its applications. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (2007)

    2007

  69. [69]

    Advances in Neural Information Processing Systems (NeurIPS) (2024)

    Zhou, Y., Xia, X., Lin, Z., Han, B., Liu, T.: Few-shot adversarial prompt learning on vision-language models. Advances in Neural Information Processing Systems (NeurIPS) (2024)

    2024

  70. [70]

    In: Proceedings of the ACM International Conference on Multimedia (ACM MM) (2023)

    Zhou, Z., Hu, S., Li, M., Zhang, H., Zhang, Y., Jin, H.: AdvCLIP: Downstream- agnostic adversarial examples in multimodal contrastive learning. In: Proceedings of the ACM International Conference on Multimedia (ACM MM) (2023)

    2023