REVIEW 3 major objections 5 minor 55 references
CoEvoAttack uses evolutionary search on both text and image sides to generate object-region adversarial examples that transfer across captioning, detection, region categorization, and localization in unified VLMs.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-04 13:48 UTC pith:4NTJONG2
load-bearing objection Solid empirical attack paper with a weak theory section and an unvalidated representation proxy; the core attack result is credible and worth refereeing. the 3 major comments →
Two Sides of the Same Coin: Co-Evolving Search for Cross-Task Attacks on Vision-Language Models
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper claims that "jointly optimizing semantic negative guidance and localized perturbations generates adversarial examples that consistently shift source-object semantics toward target categories across vision-language tasks". Table I backs this with higher average success and CTSR-4/CTSR-3 rates than existing targeted attacks on Florence-2, OFA, and UnifiedIO-2 over four tasks.
Load-bearing premise
Eq. (11) defines the object-level visual representation as the average of visual patch features inside the target bounding box, and Eq. (12) optimizes cosine similarity between this representation and text embeddings. The entire cross-task transfer claim depends on the assumption that this patch-averaged representation is a faithful proxy for the semantics that the captioning, detection, region categorization, and localization heads actually use. If the four task heads rely on different internal feature structures, gradient updates optimized on this single proxy could overfit to region categorization and not transfer. The paper's own limitations section acknowledges that localization accuracy and object entanglement can hurt, but the proxy assumption itself is never validated, for example by ablating Eq. (11) against other pooling or feature choices.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CoEvoAttack, a co-evolutionary adversarial attack for unified vision-language models. On the text side, it evolves a population of hard negative embeddings around the source-category embedding; on the image side, it evolves a population of object-region perturbations using momentum-based gradient updates plus selection, mutation, and crossover. The objective (Eq. 12) pulls the mean-pooled object-region patch features toward the target text embedding and away from the evolved negatives. The paper claims this produces adversarial images that consistently shift source-object semantics toward target categories across image captioning, object detection, region categorization, and object localization on Florence-2, OFA, and UnifiedIO-2. It also provides two theoretical results: feasibility/non-degradation of the best image-side fitness (Proposition 1) and an increased probability of hitting high-margin adversarial regions from multiple random initializations (Theorem 1).
Significance. If the empirical claims hold, CoEvoAttack would be a useful addition to the cross-task adversarial robustness literature, showing that population-based multimodal search can outperform existing task-specific attacks such as CRAFT at inducing object-level semantic shifts. The paper has concrete strengths: it evaluates three different unified VLMs and four heterogeneous tasks, compares against several strong baselines, includes extensive ablations of the key components, and provides qualitative examples of cross-task transfer. However, the significance is currently tempered by three issues: (i) the image-side objective directly optimizes a mean-pooled visual proxy whose relationship to the representations actually used by the four task heads is not established; (ii) the theoretical results are largely formal restatements of the algorithm's bookkeeping rather than explanatory statements about co-evolution; and (iii) all quantitative claims are point estimates without error bars, seeds, or statistical tests, while code is not released. These are fixable in revision.
major comments (3)
- [§IV, Theorem 1] The cross-task transfer claim rests on the assumption that the mean-pooled patch feature inside the bounding box, z(t)_i = 1/|T| Σ_{j∈T} F_v(x+δ)_j, is a faithful proxy for the object-level semantics used by all four task heads. The image-side loss directly optimizes cosine similarity between this proxy and the target text embedding, so the objective is, by construction, aligned with the quantity the paper calls semantic alignment. But captioning decodes from cross-attention over visual tokens, detection and localization require spatial structure that mean pooling destroys, and region categorization might rely on a different feature subspace. The reported CTSR gains could therefore reflect optimization of a proxy that mostly correlates with region categorization or generic objectness rather than a genuine shared semantic shift. The limitations section acknowledges localization and object
- [§V-B, Table I] Theorem 1 computes 1-(1-p_τ)^{K_v}, the probability that at least one of K_v independent random initializations lands in A_τ. This is a property of random restart, not of co-evolutionary search: selection, mutation, crossover, and momentum updates play no role in the proof. The comparison to 'single-trajectory optimization' is also not modeled correctly, since a gradient trajectory is not a single random initialization but an initialized point improved by gradient steps. Proposition 1 is likewise a direct consequence of the projection/masking operations and the best-so-far tracking in Eq. (18); it holds for any population method with these ingredients and does not explain why the evolutionary mechanism would find stronger adversarial regions. The paper should reframe these results as sanity checks about feasibility and non-degradation, and either prove a statement that actually involves
- All reported attack success rates are point estimates. Several of the claimed gains are small, most notably for OFA: CoEvoAttack obtains avg 0.470 vs. CRAFT's 0.443, CTSR-4 0.224 vs. 0.204, and CTSR-3 0.342 vs. 0.311. Without multiple seeds, confidence intervals, or significance tests, it is not possible to assess whether these differences are reliable or merely noise from the random evolutionary operators. The paper also does not release code, which makes independent replication harder. I request either multiple random seeds with error bars or statistical tests, and at minimum a commitment to releasing code to support the reproducibility of the central empirical claim.
minor comments (5)
- [Figures 1 and 3] Typographical issues: Fig. 1 uses 'Average Time per Imagege (s)' and Fig. 3's caption has 'Rigion to Category' for 'Region to Category'.
- [Eq. (2)] The text says the population is initialized 'around the source-category representation,' but Eq. (2) initializes around e^-_s, the source-category text embedding. Please clarify what 'representation' refers to and whether the semantic anchor is the text embedding or the object visual representation.
- [Eq. (8) and notation] Eq. (8) writes E^- = E^(U), but later E^- is used as a set {e^-_k}. Overloading the symbol is confusing; use a distinct symbol for the final population.
- [Appendix C] The appendix gives concrete hyperparameters such as 'negative population evolved for 20 steps with an L2 radius of 0.05,' but the main text's notation R_t and θ_t are not tied to these values. Please provide a consistent mapping between notation and numerical settings.
- [Limitations] The supplementary limitations section is thoughtful, but it should be summarized in the main text and, more importantly, it does not mention the unvalidated mean-pooling proxy of Eq. (11). Please address that explicitly.
Axiom & Free-Parameter Ledger
free parameters (9)
- text population size Kt =
9
- image population size Kv =
9
- text mutation strength phi_t =
0.1
- image mutation strength phi_v =
0.1
- negative similarity weight h =
0.2
- text search radius R_t =
0.05
- perturbation budget epsilon =
32/255
- step size alpha =
8/255
- beta in text fitness (Eq. 4) =
not reported
axioms (5)
- standard math Probability of at least one of K independent draws hitting a set is 1-(1-p)^K
- domain assumption Averaged visual patch features inside the bounding box form a valid object-level representation for cross-task semantics
- domain assumption The target/source text embeddings e+ and e_s are the correct anchors for adversarial semantic shift
- domain assumption Object mask M from the benchmark correctly delineates the source object
- domain assumption The unified VLM is white-box: visual encoder F_v is differentiable and accessible
read the original abstract
Vision-language models (VLMs) exhibit strong generalization across multimodal tasks but remain vulnerable to adversarial perturbations. Existing attacks typically follow single-trajectory gradient optimization or task-specific objectives, limiting search-space exploration and cross-task transferability. We propose an evolutionary-computation-guided cross-modal attack framework for unified VLMs. The framework adaptively searches both textual and visual spaces. On the textual side, it evolves hard negative semantic embeddings around the source-category representation to provide diverse cross-modal repulsion. On the visual side, it maintains a population of object-region perturbations and combines momentum-based gradient updates with evolutionary selection, mutation, and crossover to more reliably explore multiple feasible trajectories. Jointly optimizing semantic negative guidance and localized perturbations generates adversarial examples that consistently shift source-object semantics toward target categories across vision-language tasks. Theoretical analyses show that the co-evolutionary search preserves perturbation feasibility, prevents degradation of the best observed fitness, and increases the probability of reaching high-margin adversarial regions compared with single-trajectory optimization. Experiments on Florence-2, OFA, and UnifiedIO-2 demonstrate strong overall attack performance across image captioning, object detection, region categorization, and object localization. Ablation studies further verify the complementary effectiveness of text-side semantic evolution and image-side perturbation evolution, as well as the framework's efficiency and cross-task transferability.
Figures
Reference graph
Works this paper leans on
-
[1]
Kuckreja, M
K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan, ``Geochat: Grounded large vision-language model for remote sensing,'' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 27\,831--27\,840
2024
-
[2]
C. Cui, Y. Ma, X. Cao, W. Ye, Y. Zhou, K. Liang, J. Chen, J. Lu, Z. Yang, K.-D. Liao et al., ``A survey on multimodal large language models for autonomous driving,'' in Proceedings of the IEEE/CVF winter conference on applications of computer vision, 2024, pp. 958--979
2024
-
[3]
Y. Hou, X. Lin, L. Guo, F. Chen, H. Lv, Q. Zhou, and Y. Li, ``Vision side prompt learning with low-rank multimodal alignment for video paragraph captioning,'' Expert Systems with Applications, p. 130794, 2025
2025
-
[4]
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, ``Intriguing properties of neural networks,'' arXiv preprint arXiv:1312.6199, 2013
Pith/arXiv arXiv 2013
-
[5]
X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal, ``Visual adversarial examples jailbreak aligned large language models,'' in Proceedings of the AAAI conference on artificial intelligence, vol. 38, no. 19, 2024, pp. 21\,527--21\,536
2024
-
[6]
Kurakin, I
A. Kurakin, I. J. Goodfellow, and S. Bengio, ``Adversarial examples in the physical world,'' in Artificial intelligence safety and security. 1em plus 0.5em minus 0.4em Chapman and Hall/CRC, 2018, pp. 99--112
2018
-
[7]
J. Dong, P. Koniusz, X. Qu, and Y.-S. Ong, ``Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms,'' in Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, 2025, pp. 236--247
2025
-
[8]
D \' az-Rodr \' guez, J
N. D \' az-Rodr \' guez, J. Del Ser, M. Coeckelbergh, M. L. De Prado, E. Herrera-Viedma, and F. Herrera, ``Connecting the dots in trustworthy artificial intelligence: From ai principles, ethics, and key requirements to responsible ai systems and regulation,'' Information Fusion, vol. 99, p. 101896, 2023
2023
-
[9]
C. Mao, S. Geng, J. Yang, X. Wang, and C. Vondrick, ``Understanding zero-shot adversarial robustness for large-scale models,'' arXiv preprint arXiv:2212.07016, 2022
Pith/arXiv arXiv 2022
-
[10]
X. Li, W. Zhang, Y. Liu, Z. Hu, B. Zhang, and X. Hu, ``Language-driven anchors for zero-shot adversarial robustness,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 24\,686--24\,695
2024
-
[11]
F. Ma, L. Liu, and H. V. Cheng, ``Tima: Text-image mutual awareness for balancing zero-shot adversarial robustness and generalization ability,'' arXiv preprint arXiv:2405.17678, 2024
Pith/arXiv arXiv 2024
-
[12]
Z. Tao, D. Zhang, Y. Qiu, Y. Yang, Q. Zhao et al., ``Calibrating uncertainty for zero-shot adversarial clip,'' arXiv preprint arXiv:2512.12997, 2025
Pith/arXiv arXiv 2025
-
[13]
C. Schlarmann, N. D. Singh, F. Croce, and M. Hein, ``Robust clip: Unsupervised adversarial fine-tuning of vision embeddings for robust large vision-language models,'' arXiv preprint arXiv:2402.12336, 2024
Pith/arXiv arXiv 2024
-
[14]
Zhang, X
J. Zhang, X. Ma, X. Wang, L. Qiu, J. Wang, Y.-G. Jiang, and J. Sang, ``Adversarial prompt tuning for vision-language models,'' in European conference on computer vision. 1em plus 0.5em minus 0.4em Springer, 2024, pp. 56--72
2024
-
[15]
Jiang, S
W. Jiang, S. You, J. Zhan, X. Wang, H. Lei, and D. Adhikari, ``Query-efficient generation of adversarial examples for defensive dnns via multiobjective optimization,'' IEEE Transactions on Evolutionary Computation, vol. 27, no. 4, pp. 832--847, 2022
2022
-
[16]
P. N. Williams, K. Li, and G. Min, ``Evolutionary art attack for black-box adversarial example generation,'' IEEE transactions on evolutionary computation, 2024
2024
-
[17]
Y. Zhou, X. Xia, Z. Lin, B. Han, and T. Liu, ``Few-shot adversarial prompt learning on vision-language models,'' Advances in Neural Information Processing Systems, vol. 37, pp. 3122--3156, 2024
2024
-
[18]
C. Liu, Y. Wang, and L. Xu, ``Gradient regularization-based cross-prompt attacks on vision language models,'' 2024. [Online]. Available: https://openreview.net/forum?id=I05Z6KjQ9K
2024
-
[19]
X. Jia, S. Gao, S. Qin, K. Ma, X. Li, Y. Huang, W. Dong, Y. Liu, and X. Cao, ``Evolution-based region adversarial prompt learning for robustness enhancement in vision-language models,'' arXiv preprint arXiv:2503.12874, 2025
Pith/arXiv arXiv 2025
-
[20]
X. Li, M. G. Epitropakis, K. Deb, and A. Engelbrecht, ``Seeking multiple solutions: An updated survey on niching methods and their applications,'' IEEE Transactions on Evolutionary Computation, vol. 21, no. 4, pp. 518--538, 2016
2016
-
[21]
Alayrac, J
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds et al., ``Flamingo: a visual language model for few-shot learning,'' Advances in neural information processing systems, vol. 35, pp. 23\,716--23\,736, 2022
2022
-
[22]
P. Wang, A. Yang, R. Men, J. Lin, S. Bai, Z. Li, J. Ma, C. Zhou, J. Zhou, and H. Yang, ``Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework,'' in International conference on machine learning. 1em plus 0.5em minus 0.4em PMLR, 2022, pp. 23\,318--23\,340
2022
-
[23]
J. Lu, C. Clark, S. Lee, Z. Zhang, S. Khosla, R. Marten, D. Hoiem, and A. Kembhavi, ``Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 26\,439--26\,455
2024
-
[24]
B. Xiao, H. Wu, W. Xu, X. Dai, H. Hu, Y. Lu, M. Zeng, C. Liu, and L. Yuan, ``Florence-2: Advancing a unified representation for a variety of vision tasks,'' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 4818--4829
2024
-
[25]
H. Liu, C. Li, Q. Wu, and Y. J. Lee, ``Visual instruction tuning,'' Advances in neural information processing systems, vol. 36, pp. 34\,892--34\,916, 2023
2023
-
[26]
I. J. Goodfellow, J. Shlens, and C. Szegedy, ``Explaining and harnessing adversarial examples,'' arXiv preprint arXiv:1412.6572, 2014
Pith/arXiv arXiv 2014
-
[27]
Madry, A
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, ``Towards deep learning models resistant to adversarial attacks,'' in 6th International Conference on Learning Representations, ICLR 2018 , 2018
2018
-
[28]
Carlini and D
N. Carlini and D. Wagner, ``Towards evaluating the robustness of neural networks,'' in 2017 ieee symposium on security and privacy (sp). 1em plus 0.5em minus 0.4em Ieee, 2017, pp. 39--57
2017
-
[29]
Zhang, Q
J. Zhang, Q. Yi, and J. Sang, ``Towards adversarial attack on vision-language pre-training models,'' in Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 5005--5013
2022
-
[30]
Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N.-M. M. Cheung, and M. Lin, ``On evaluating adversarial robustness of large vision-language models,'' Advances in Neural Information Processing Systems, vol. 36, pp. 54\,111--54\,138, 2023
2023
-
[31]
J. Zhao, X. Jiang, J. Gao, Y. Xue, and C. Zhao, ``One object, multiple lies: A benchmark for cross-task adversarial attack on unified vision-language models,'' in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 187--196
2025
-
[32]
J. H. Holland, Adaptation in natural and artificial systems: an introductory analysis with applications to biology, control, and artificial intelligence. 1em plus 0.5em minus 0.4em MIT press, 1992
1992
-
[33]
Storn and K
R. Storn and K. Price, ``Differential evolution--a simple and efficient heuristic for global optimization over continuous spaces,'' Journal of global optimization, vol. 11, no. 4, pp. 341--359, 1997
1997
-
[34]
Kennedy and R
J. Kennedy and R. Eberhart, ``Particle swarm optimization,'' in Proceedings of ICNN'95-international conference on neural networks, vol. 4. 1em plus 0.5em minus 0.4em ieee, 1995, pp. 1942--1948
1995
-
[35]
Hansen, ``The cma evolution strategy: a comparing review,'' Towards a new evolutionary computation: Advances in the estimation of distribution algorithms, pp
N. Hansen, ``The cma evolution strategy: a comparing review,'' Towards a new evolutionary computation: Advances in the estimation of distribution algorithms, pp. 75--102, 2006
2006
-
[36]
X. Ma, X. Li, Q. Zhang, K. Tang, Z. Liang, W. Xie, and Z. Zhu, ``A survey on cooperative co-evolutionary algorithms,'' IEEE Transactions on Evolutionary Computation, vol. 23, no. 3, pp. 421--441, 2018
2018
-
[37]
Demir, B
K. Demir, B. H. Nguyen, B. Xue, and M. Zhang, ``Multi-label black-box attacks via evolutionary structured many-objective adversarial perturbations,'' IEEE Transactions on Evolutionary Computation, 2025
2025
-
[38]
J. Su, D. V. Vargas, and K. Sakurai, ``One pixel attack for fooling deep neural networks,'' IEEE Transactions on Evolutionary Computation, vol. 23, no. 5, pp. 828--841, 2019
2019
-
[39]
H. Qiu, L. L. Custode, and G. Iacca, ``Black-box adversarial attacks using evolution strategies,'' in Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2021, pp. 1827--1833
2021
-
[40]
C. Li, H. Wang, J. Zhang, W. Yao, and T. Jiang, ``An approximated gradient sign method using differential evolution for black-box adversarial attack,'' IEEE Transactions on Evolutionary Computation, vol. 26, no. 5, pp. 976--990, 2022
2022
-
[41]
Gupta, Y.-S
A. Gupta, Y.-S. Ong, and L. Feng, ``Multifactorial evolution: Toward evolutionary multitasking,'' IEEE Transactions on Evolutionary Computation, vol. 20, no. 3, pp. 343--357, 2015
2015
-
[42]
Y. Li, X. Wu, W. Gong, M. Xu, Y. Wang, and Q. Gu, ``Evolutionary competitive multiobjective multitasking: One-pass optimization of heterogeneous pareto solutions,'' IEEE Transactions on Evolutionary Computation, vol. 29, no. 6, pp. 2757--2770, 2025
2025
-
[43]
Y. Wang, C. Hu, F. Ming, Y. Li, W. Gong, and L. Gao, ``A diversity-enhanced tri-stage framework for constrained multi-objective optimization,'' IEEE Transactions on Evolutionary Computation, 2024
2024
-
[44]
Li and W
Y. Li and W. Gong, ``Multiobjective multitask optimization with multiple knowledge types and transfer adaptation,'' IEEE Transactions on Evolutionary Computation, vol. 29, no. 1, pp. 205--216, 2024
2024
-
[45]
Basak, S
A. Basak, S. Das, and K. C. Tan, ``Multimodal optimization using a biobjective differential evolution algorithm enhanced with mean distance-based selection,'' IEEE Transactions on Evolutionary Computation, vol. 17, no. 5, pp. 666--685, 2012
2012
-
[46]
Cheng, M
R. Cheng, M. Li, K. Li, and X. Yao, ``Evolutionary multiobjective optimization-based multimodal optimization: Fitness landscape approximation and peak detection,'' IEEE Transactions on Evolutionary Computation, vol. 22, no. 5, pp. 692--706, 2017
2017
-
[47]
Lin, Z.-G
G.-Y. Lin, Z.-G. Chen, C. Liu, Y. Jiang, S. Kwong, J. Zhang, and Z.-H. Zhan, ``A landscape-aware differential evolution for multimodal optimization problems,'' IEEE Transactions on Evolutionary Computation, 2025
2025
-
[48]
Y. Li, W. Gong, and S. Li, ``Multitask evolution strategy with knowledge-guided external sampling,'' IEEE Transactions on Evolutionary Computation, vol. 28, no. 6, pp. 1733--1745, 2023
2023
-
[49]
Y. Li, W. Gong, and Q. Gu, ``Transfer task-averaged natural gradient for efficient many-task optimization,'' IEEE Transactions on Evolutionary Computation, 2024
2024
-
[50]
H. Tu, C. Cui, Z. Wang, Y. Zhou, B. Zhao, J. Han, W. Zhou, H. Yao, and C. Xie, ``How many unicorns are in this image? a safety evaluation benchmark for vision llms,'' arXiv preprint arXiv:2311.16101, 2023
Pith/arXiv arXiv 2023
-
[51]
Y. Dong, H. Chen, J. Chen, Z. Fang, X. Yang, Y. Zhang, Y. Tian, H. Su, and J. Zhu, ``How robust is google's bard to adversarial image attacks?'' arXiv preprint arXiv:2309.11751, 2023
Pith/arXiv arXiv 2023
-
[52]
Herrera, M
F. Herrera, M. Lozano, and J. L. Verdegay, ``Tackling real-coded genetic algorithms: Operators and tools for behavioural analysis,'' Artificial intelligence review, vol. 12, no. 4, pp. 265--319, 1998
1998
-
[53]
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, ``Towards deep learning models resistant to adversarial attacks,'' arXiv preprint arXiv:1706.06083, 2017
Pith/arXiv arXiv 2017
-
[54]
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll \'a r, and C. L. Zitnick, ``Microsoft coco: Common objects in context,'' in European conference on computer vision. 1em plus 0.5em minus 0.4em Springer, 2014, pp. 740--755
2014
-
[55]
Y. Dong, F. Liao, T. Pang, H. Su, J. Zhu, X. Hu, and J. Li, ``Boosting adversarial attacks with momentum,'' in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9185--9193
2018
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.