Pith. sign in

REVIEW 5 major objections 3 minor 44 references

Semantically Guided Adversarial Testing of Vision Models Using Language Models

T0 review · 5 major / 3 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pretrained language and vision-language models can select adversarial target labels by semantic similarity, beating static lexical resources such as WordNet.

desk verdict Plausible incremental idea for adversarial target selection, but the evidence as presented is thin and the headline comparison may be confounded by target difficulty. read the letter →

arxiv 2508.11341 v1 pith:IFBID2JM submitted 2025-08-15 cs.CV cs.CRcs.LG

Katarzyna Filus , Jorge M. Cruz-Duarte This is my paper · ORCID
classification cs.CVcs.CRcs.LG
keywords targetedadversarialattackstargetlabelselectionsemanticsimilaritylanguagemodelsvision-languageCLIPBERTbenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In targeted adversarial attacks on image classifiers, the attacker chooses a wrong label to force, and that choice largely determines whether the attack succeeds. This paper claims that pretrained language and vision-language models — BERT, TinyLLAMA, and CLIP — can select those target labels by semantic similarity to the true class, producing both easy (semantically close) and hard (semantically distant) attack scenarios. Evaluated on three vision models and five attack methods, the semantically chosen targets consistently work as adversarial targets and beat a static lexical database, WordNet, especially when the true and target classes are far apart. The authors argue this makes adversarial testing interpretable, reproducible, and scalable: the target-ranking itself can be tested before running any attack, as an a priori check of the similarity source.

What carries the argument

The mechanism is cross-modal semantic similarity: take the ground-truth class name and every candidate target class name, embed them with a pretrained text encoder (BERT, TinyLLAMA, or CLIP), and rank candidate targets by embedding similarity to the ground-truth label. The ranked list defines the attack scenarios — most similar labels as one end, least similar as the other — and can be evaluated statically, without running attacks, as an a priori test of the similarity source. What carries the argument is the assumption that similarities computed in a text-pretrained space transfer to the difficulty of fooling a vision classifier.

What would settle it

Take a vision model and an attack not among the three models and five attacks tested, choose targets by the paper's semantic rankings and by WordNet, and compare success rates on semantically distant pairs. If the lexical or random targets match or beat the embedding-based targets, the generality claim fails; likewise, if the rank correlation between embedding similarity and empirical per-target success is near zero across all classes, the proxy is not doing the work.

Watch

Extended reading notes

Core claim

The central claim is that target-label selection, not just the perturbation method, is a major determinant of targeted attack success, and that cross-modal semantic similarity from pretrained models is the right tool for choosing targets. The paper builds best- and worst-case scenarios by ranking all candidate labels by their semantic similarity to the ground-truth label in embeddings from BERT, TinyLLAMA, and CLIP, then attacking vision classifiers (three architectures) with five attack methods. Across these setups, the similarity-based targets are practical adversarial targets and outperform static lexical databases such as WordNet, with the largest gap for distant class relationships — ca

Load-bearing premise

The load-bearing premise is that semantic similarity measured in a text-pretrained embedding space reliably predicts how easy it is to push a vision model to a given wrong label, for models, attacks, and datasets beyond the ones tested.

Editorial extensions

If this is right

  • If semantic similarity predicts target difficulty, attack success rates can be reported by semantic distance, making benchmark results interpretable rather than target-dependent.
  • The same target-ranking method can be applied to any new dataset or model without retraining, since it only needs class names.
  • Static a priori testing lets researchers compare similarity sources before expensive attack runs.
  • Best- and worst-case target selection brackets robustness, exposing where a model is most and least easily fooled.
  • Semantic target selection overtakes static lexical resources like WordNet for far-apart classes, where ontology links are missing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the target-ranking step only needs class names, the method should extend to very large label sets (for example, the full 21,000-class ImageNet) where lexical resources such as WordNet become sparse; the paper does not test this scale.
  • Embedding similarity can encode corpus co-occurrence and dataset bias as much as genuine semantic distance, so some of the reported 'distant' targets may be easy for non-semantic reasons; separating those two effects would sharpen the benchmark.
  • The same semantic ranking could be used to predict natural model confusions, not just adversarial ones: classes that are close in the embedding space may be the ones a vision model already confuses, which would let the a priori test double as a diagnostic for learned representations.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The paper proposes a semantics-guided framework for selecting target labels in targeted adversarial attacks on vision models. It uses embeddings from pretrained text or vision-language models (BERT, TinyLLAMA, CLIP) to pick the most and least semantically similar labels with respect to the ground-truth class, thereby defining best- and worst-case attack scenarios. The authors report experiments on three vision models and five attacks, claiming that these similarity sources 'consistently render practical adversarial targets' and surpass static lexical databases such as WordNet, especially for distant class relationships. They also propose that static target evaluation provides an a priori assessment of similarity-source quality. The central claim is an empirical one: embedding-based target selection is superior and more generalizable than static lexical selection, enabling more interpretable and standardized adversarial benchmarks.

Significance. If substantiated, the proposed framework would be a useful step toward principled adversarial target selection, which is indeed an underexamined component in targeted attack evaluation. The use of pretrained multimodal models as similarity sources is timely and potentially valuable for constructing scalable, interpretable benchmarks. However, the significance is currently prospective: the submission as provided does not give enough experimental detail to verify the headline claims, and the main comparison may be confounded by target-difficulty effects. No machine-checked proofs, code, or data are provided; the contribution is an empirical methodology that must be validated by careful controlled experiments.

major comments (5)
  1. [Abstract] The abstract makes strong empirical claims—'consistently render practical adversarial targets' and 'surpass static lexical databases'—but gives no variance, confidence intervals, statistical tests, or experiment configuration. The three vision models, five attacks, and dataset(s) are not named, and no results table is visible in the provided text. As written, the central claim is not checkable. Please provide a full experimental section with model/attack/dataset details and uncertainty quantification.
  2. [Abstract; experiments (not numbered in the provided text)] The headline comparison between embedding-based target selection and WordNet may be confounded by per-target attackability. If the embedding methods select targets that are intrinsically easier to attack (e.g., classes with low model confidence or high visual confusion with the source), the reported advantage merely reflects target difficulty, not semantic quality. This is especially relevant for the 'distant class relationships' claim: 'distant' in an embedding space may coincide with visually similar but semantically unrelated classes under the attacked model. Please control for this by including a random-target baseline matched on marginal label frequencies, or by matching selected target sets on the attacked model's own softmax margins, and report per-target success rates.
  3. [Abstract; 'particularly for distant class relationships'] The notion of 'distant class relationships' is not operationally defined. If WordNet distance is measured by path length while embedding distance is measured by cosine similarity, the two methods select different target sets by construction. The claim that embedding-based selection is better for distant classes needs a concrete definition of 'distance' and a demonstration that the advantage is not an artifact of the distance measure or of the attacked model's feature geometry. A cross-condition comparison with matched target semantic-distance bins would address this.
  4. [Abstract; 'across architectures and datasets'] The generality claim is too strong for the evidence stated. Three vision models and five attacks is a reasonable start, but 'across architectures and datasets' requires a broader and more systematic evaluation—for example, CNNs versus ViTs, multiple dataset domains, and attacks with different budgets and threat models. The current abstract reports only examples, not a systematic ablation or transfer study. Please either provide such evidence or temper the claim to the tested settings.
  5. [Abstract; 'a priori testing'] The claim that 'static testing of target labels offers a preliminary assessment of the effectiveness of similarity sources, a priori testing' is not operationalized. What is the static test? How does it relate to the downstream attack success? Without a definition and supporting experiments, this observation remains anecdotal. Please state the proposed static metric and show that it predicts the ranking of similarity sources.
minor comments (3)
  1. [Full text] The body text supplied to me is heavily corrupted or unreadable in many places; only the abstract and fragments are intelligible. If this reflects the submitted PDF, the manuscript cannot be properly reviewed and a clean version must be provided.
  2. [Abstract] Please define all acronyms and model/attack names on first use; also specify which CLIP variant, TinyLLAMA checkpoint, and BERT model are used.
  3. [Experiments (to be added)] Please include a reproducibility statement with code/data availability and attack hyperparameters (e.g., number of iterations, epsilon bound, loss function) for each of the five attacks.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical comparison of fixed similarity sources on held-out attack tasks.

full rationale

The paper's central claim is an empirical evaluation: pretrained language/vision-language models (BERT, TinyLLAMA, CLIP) are used to select adversarial target labels, and the resulting attack success is compared against WordNet-based selection. The abstract explicitly anchors the evaluation in experiments on three vision models and five attack methods, and the target-selection step is defined independently of the attack outcomes. There is no fitted parameter that is later renamed as a prediction, no result that is assumed by the construction, and no self-citation chain that supplies the load-bearing content. The only mild concern—that CLIP is both a similarity source and related to vision models—is not substantiated by any quoted reduction in the manuscript; the vision models are not specified as CLIP, and no equation makes the attack success depend on the similarity source by definition. The 'a priori testing' phrase refers to using static target properties as a screening step, not to deriving the experimental result from its own output. Hence the derivation chain is self-contained and the comparison is an independent empirical test.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper introduces no new fitted parameters or entities. It relies on the domain assumptions that language-model semantic similarity is a generalizable proxy for adversarial difficulty and that class-label rankings transfer across vision architectures and datasets.

assumptions (2)
  • domain assumption Semantic similarity from pretrained language models is a valid proxy for adversarial target difficulty in vision models.
    The entire framework rests on using text-based embeddings to rank target labels for image attacks; the abstract gives no independent justification for this transfer.
  • domain assumption Pretrained LMs generalize across vision architectures and datasets for class-label similarity rankings.
    The claim of 'consistent' results across three vision models and five attacks assumes the similarity rankings transfer; this is not established from the abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Semantically Guided Adversarial Testing of Vision Models Using Language Models." pith.science (2026). https://pith.science/paper/IFBID2JM

@misc{pith2026250811341,
  author       = {Pith},
  title        = {Pith review of: Semantically Guided Adversarial Testing of Vision Models Using Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IFBID2JM}},
  note         = {Machine review of arXiv:2508.11341}
}
read the original abstract

In targeted adversarial attacks on vision models, the selection of the target label is a critical yet often overlooked determinant of attack success. This target label corresponds to the class that the attacker aims to force the model to predict. Now, existing strategies typically rely on randomness, model predictions, or static semantic resources, limiting interpretability, reproducibility, or flexibility. This paper then proposes a semantics-guided framework for adversarial target selection using the cross-modal knowledge transfer from pretrained language and vision-language models. We evaluate several state-of-the-art models (BERT, TinyLLAMA, and CLIP) as similarity sources to select the most and least semantically related labels with respect to the ground truth, forming best- and worst-case adversarial scenarios. Our experiments on three vision models and five attack methods reveal that these models consistently render practical adversarial targets and surpass static lexical databases, such as WordNet, particularly for distant class relationships. We also observe that static testing of target labels offers a preliminary assessment of the effectiveness of similarity sources, \textit{a priori} testing. Our results corroborate the suitability of pretrained models for constructing interpretable, standardized, and scalable adversarial benchmarks across architectures and datasets.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 40 canonical work pages

  1. [1]

    Szegedy, W

    C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, ``Intriguing properties of neural networks,'' Proceedings of the 2nd International Conference on Learning Representations , 2013

    work page 2013

  2. [2]

    I. J. Goodfellow, J. Shlens, and C. Szegedy, ``Explaining and harnessing adversarial examples,'' arXiv preprint arXiv:1412.6572, 2014

    arXiv 2014

  3. [3]

    Carlini and D

    N. Carlini and D. Wagner, ``Towards evaluating the robustness of neural networks,'' in IEEE S ymposium on S ecurity and P rivacy , 2017, pp. 39--57

    work page 2017

  4. [4]

    J. Byun, S. Cho, M.-J. Kwon, H.-S. Kim, and C. Kim, ``Improving the transferability of targeted adversarial examples through object-based diverse input,'' in Proceedings of the IEEE/CVF C onference on C omputer V ision and P attern R ecognition , 2022, pp. 15\,244--15\,253

    work page 2022

  5. [5]

    Filus and J

    K. Filus and J. Domanska, ``Netsat: Network saturation adversarial attack,'' in 2023 IEEE International Conference on Big Data (BigData). 1em plus 0.5em minus 0.4em IEEE, 2023, pp. 5038--5047

    work page 2023

  6. [6]

    Gelenbe and M

    E. Gelenbe and M. Nasereddin, ``Adaptive attack mitigation for iov flood attacks,'' IEEE Internet of Things Journal, 2025

    work page 2025

  7. [7]

    Filus, J

    K. Filus, J. Doma \'n ska, and E. Gelenbe, ``Random neural network for lightweight attack detection in the iot,'' in Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems. 1em plus 0.5em minus 0.4em Springer, 2020, pp. 79--91

    work page 2020

  8. [8]

    Guerra-Manzanares, J

    A. Guerra-Manzanares, J. Medina-Galindo, H. Bahsi, and S. N \ o mm, ``Medbiot: Generation of an iot botnet dataset in a medium-sized iot network.'' in ICISSP, 2020, pp. 207--218

    work page 2020

Show all 44 references
  1. [9]

    Siavvas, I

    M. Siavvas, I. Kalouptsoglou, E. Gelenbe, D. Kehagias, and D. Tzovaras, ``Transforming the field of vulnerability prediction: Are large language models the key?'' in 2024 32nd International Conference on Modeling, Analysis and Simulation of Computer and Telecommunication Syste...

    2024

  2. [10]

    Filus and J

    K. Filus and J. Doma \'n ska, `` Software vulnerabilities in TensorFlow-based deep learning applications ,'' Computers & Security, vol. 124, p. 102948

  3. [11]

    Filus, M

    K. Filus, M. Siavvas, J. Doma \'n ska, and E. Gelenbe, ``The random neural network as a bonding model for software vulnerability prediction,'' in Symposium on modelling, analysis, and simulation of computer and telecommunication systems. 1em plus 0.5em minus 0.4em Springer, 20...

    2020

  4. [12]

    Hanif and S

    H. Hanif and S. Maffeis, ``Vulberta: Simplified source code pre-training for vulnerability detection,'' in 2022 International joint conference on neural networks (IJCNN). 1em plus 0.5em minus 0.4em IEEE, 2022, pp. 1--8

    2022

  5. [13]

    Kurakin, I

    A. Kurakin, I. Goodfellow, and S. Bengio, ``Adversarial machine learning at scale,'' arXiv preprint arXiv:1611.01236, 2016

    2016 arXiv

  6. [14]

    S. Hu, L. Ke, X. Wang, and S. Lyu, ``Tkml-ap: Adversarial attacks to top-k multi-label learning,'' in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 7649--7657

    2021

  7. [15]

    Filus and J

    K. Filus and J. Doma \'n ska, ``Similarity-driven adversarial testing of neural networks,'' Knowledge-Based Systems, vol. 305, p. 112621, 2024

    2024

  8. [16]

    Filus, M

    K. Filus, M. Romaszewski, and M. \.Z arski, ``Semantic depth matters: Explaining errors of deep vision networks through perceived class similarities,'' arXiv preprint arXiv:2504.09956, 2025

    2025

  9. [17]

    `` Dataset for the NIPS 2017 adversarial competition ,'' [online], -, available: https://github.com/cleverhans-lab/cleverhans/tree/master/cleverhans_v3.1.0/examples/nips17_adversarial_competition/dataset [Accessed: 2025-05-06]

    2017

  10. [18]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, ``Bert: Pre-training of deep bidirectional transformers for language understanding,'' in Proceedings of NAACL-HLT, 2019

    2019

  11. [19]

    Zhang, G

    P. Zhang, G. Zeng, T. Wang, and W. Lu, ``Tinyllama: An open-source small language model,'' arXiv preprint arXiv:2401.02385, 2024

    2024 arXiv

  12. [20]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., ``Learning transferable visual models from natural language supervision,'' in International Conference on Machine Learning, 2021

    2021

  13. [21]

    C. Li, H. Wang, W. Yao, and T. Jiang, ``Adversarial attacks in computer vision: a survey,'' Journal of Membrane Computing, pp. 1--18, 2024

    2024

  14. [22]

    Madry, A

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, ``Towards deep learning models resistant to adversarial attacks,'' arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  15. [23]

    Moosavi-Dezfooli, A

    S.-M. Moosavi-Dezfooli, A. Fawzi, O. Fawzi, and P. Frossard, ``Universal adversarial perturbations,'' in Proceedings of the IEEE/CVF C onference on C omputer V ision and P attern R ecognition , 2017, pp. 1765--1773

    2017

  16. [24]

    Z. Chen, B. Li, S. Wu, K. Jiang, S. Ding, and W. Zhang, ``Content-based unrestricted adversarial attack,'' Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  17. [25]

    B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, ``Learning transferable architectures for scalable image recognition,'' in Proceedings of the IEEE/CVF C onference on C omputer V ision and P attern R ecognition , 2018, pp. 8697--8710

    2018

  18. [26]

    W. Zhou, X. Hou, Y. Chen, M. Tang, X. Huang, X. Gan, and Y. Yang, ``Transferable adversarial perturbations,'' in Proceedings of the European Conference on Computer Vision , 2018, pp. 452--467

    2018

  19. [27]

    J. Wang, M. Wang, H. Wu, B. Ma, and X. Luo, ``Improving transferability of adversarial attacks with gaussian gradient enhance momentum,'' in Chinese Conference on Pattern Recognition and Computer Vision (PRCV). 1em plus 0.5em minus 0.4em Springer, 2023, pp. 421--432

    2023

  20. [28]

    Filus, J

    K. Filus, J. Doma \'n ska, and J. Klamka, ``Evaluating convolutional neural networks via measuring similarity of their first-layer filters and gabor filters,'' in 2024 32nd International Conference on Modeling, Analysis and Simulation of Computer and Telecommunication Systems ...

    2024

  21. [29]

    Bayat and I

    R. Bayat and I. Rish, ``Adversarial training with synthesized data: A path to robust and generalizable neural networks,'' in ICML 2024 Next Generation of AI Safety Workshop

    2024

  22. [30]

    Kurakin, I

    A. Kurakin, I. J. Goodfellow, and S. Bengio, ``Adversarial examples in the physical world,'' in Artificial intelligence safety and security. 1em plus 0.5em minus 0.4em Chapman and Hall/CRC, 2018, pp. 99--112

    2018

  23. [31]

    Mahmood and E

    H. Mahmood and E. Elhamifar, ``Semantic-aware multi-label adversarial attacks,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24\,251--24\,262

    2024

  24. [32]

    flipping the label

    K. R. Mopuri, V. Shaj, and R. V. Babu, ``Adversarial fooling beyond" flipping the label",'' in Proceedings of the IEEE/CVF C onference on C omputer V ision and P attern R ecognition W orkshops , 2020, pp. 778--779

    2020

  25. [33]

    Kolb, ``Experiments on the difference between semantic similarity and relatedness,'' in Proceedings of the 17th Nordic Conference of Computational Linguistics , 2009, pp

    P. Kolb, ``Experiments on the difference between semantic similarity and relatedness,'' in Proceedings of the 17th Nordic Conference of Computational Linguistics , 2009, pp. 81--88

    2009

  26. [34]

    Pedersen, S

    T. Pedersen, S. Patwardhan, J. Michelizzi et al., ``Wordnet:: Similarity-measuring the relatedness of concepts.'' 2004

    2004

  27. [35]

    Wu and M

    Z. Wu and M. Palmer, ``Verb semantics and lexical selection,'' arXiv preprint cmp-lg/9406033, 1994

    1994 arXiv

  28. [36]

    Leacock and M

    C. Leacock and M. Chodorow, ``Combining local context and wordnet similarity for word sense identification,'' WordNet: An electronic lexical database, vol. 49, no. 2, pp. 265--283, 1998

    1998

  29. [37]

    Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, ``Boosting adversarial attacks with momentum,'' in Proceedings of the IEEE/CVF C onference on C omputer V ision and P attern R ecognition , 2018, pp. 9185--9193

    2018

  30. [38]

    Uesato, B

    J. Uesato, B. O’donoghue, P. Kohli, and A. Oord, ``Adversarial risk and the dangers of evaluating against weak attacks,'' in International conference on machine learning. 1em plus 0.5em minus 0.4em PMLR, 2018, pp. 5025--5034

    2018

  31. [39]

    Sandler, A

    M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, ``Mobilenetv2: Inverted residuals and linear bottlenecks,'' in Proceedings of the IEEE/CVF C onference on C omputer V ision and P attern R ecognition , 2018, pp. 4510--4520

    2018

  32. [40]

    Tan and Q

    M. Tan and Q. Le, ``Efficientnetv2: Smaller models and faster training,'' in I nternational C onference on M achine L earning , 2021, pp. 10\,096--10\,106

    2021

  33. [41]

    K. He, X. Zhang, S. Ren, and J. Sun, ``Identity mappings in deep residual networks,'' in Proceedings of the European Conference on Computer Vision . 1em plus 0.5em minus 0.4em Springer, 2016, pp. 630--645

    2016

  34. [42]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, ``Imagenet: A large-scale hierarchical image database,'' in Proceedings of the IEEE/CVF C onference on C omputer V ision and P attern R ecognition C onference on C omputer V ision and P attern R ecognition , 2009, p...

    2009

  35. [43]

    Russakovsky, J

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein et al., ``Imagenet large scale visual recognition challenge,'' International journal of computer vision, vol. 115, pp. 211--252, 2015

    2015

  36. [44]

    M. Li, C. Deng, T. Li, J. Yan, X. Gao, and H. Huang, ``Towards transferable targeted attack,'' in Proceedings of the IEEE/CVF C onference on C omputer V ision and P attern R ecognition , 2020, pp. 641--649

    2020

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.