Pith. sign in

REVIEW 5 major objections 3 minor 45 references

Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models

T0 review · 5 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A universal image-text jailbreak transfers across multimodal AI models from 7B to 34B parameters.

desk verdict Plausible and useful attack engineering, but the central qualitative claim is unsecured because it rests on an unvalidated GPT-4 judge and missing baselines. read the letter →

arxiv 2506.01307 v1 pith:Y3UBUEWX submitted 2025-06-02 cs.CR cs.AI

Youze Wang , Wenbo Hu , Yinpeng Dong , Jing Liu , Hanwang Zhang , Richang Hong This is my paper
classification cs.CRcs.AI
keywords multimodaljailbreakuniversaladversarialperturbationlargelanguagemodelssafetyalignmenttransferabilityimage-textinteractionASR-Gattack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the safety alignment of multimodal large language models (MLLMs) is vulnerable specifically at the intersection of image and text inputs, and that an attacker can exploit this by distributing jailbreak information across both modalities. Its central claim is that a universal adversarial image plus a short 10-token adversarial suffix, optimized jointly through iterative image-text interaction on a small 7B surrogate model, transfers to other open MLLMs (from 7B to 34B parameters) and produces responses that more genuinely fulfill harmful instructions than either text-only or image-only jailbreak attacks. The authors measure this with ASR-G, a metric in which GPT-4 judges whether a response actually carries out the requested harmful behavior, and report higher ASR-G scores than the GCG and Visual-jailbreak baselines. A sympathetic reader would care because current safety mechanisms largely screen text and images separately, and the result suggests that cross-modal attacks can slip through these individual filters.

What carries the argument

The carrying mechanism is the alternating image-text interaction optimization loop (Algorithms 1–3). In each outer iteration, the current adversarial suffix is held fixed while the adversarial image is optimized to maximize the average per-token log-likelihood of a target harmful response; then the image is held fixed while the suffix token embeddings are optimized by greedy coordinate gradient search, which computes token-substitution candidates from the gradient and selects the best replacement by loss. Both steps use variance-tuned gradients: the update direction is the raw gradient minus the average gradient over K (image) or M (suffix) random neighboring samples, combined with a momentum term. This variance tuning is what the paper credits for making the adversarial image and suffix transfer across different MLLM architectures and sizes. The adversarial loss itself is the normalized negative log-likelihood of the target harmful response conditioned on the concatenation of the harmful query and suffix, with the adversarial image as an additional conditioning input.

What would settle it

Take a sample of model outputs from the transfer experiments, have independent human annotators rate whether each response genuinely fulfills the corresponding harmful instruction, and compare their labels with GPT-4's ASR-G labels. If human raters disagree with GPT-4 on a substantial fraction of cases, or if the ordering between the proposed method and GCG reverses under human labels, the paper's central quality claim collapses.

Watch

Extended reading notes

Core claim

The central discovery is that 'align is not enough': aligning an MLLM's safety behavior on text and images separately leaves a gap at the image-text interaction, and this gap can be weaponized. The paper proposes an attack that alternates between optimizing an adversarial image with projected gradient descent and optimizing a 10-token adversarial suffix with greedy coordinate gradient search, using the current state of the other modality as the conditioning input at each step. To keep the adversarial pair transferable across different MLLMs, both optimization steps replace the raw gradient with a variance-tuned gradient obtained by sampling neighboring inputs, which steers updates toward flatter loss landscapes. Trained on 25 harmful behaviors from the AdvBench subset, the universal image-suffix pair is then evaluated on unseen harmful prompts and transferred to models like LLaVA, MiniGPT-v2, MiniGPT4, InstructBLIP, Yi-VL, mPLUG-Owl2, MiniCPM, CogVLM, and Qwen2-VL. The paper reports that this method achieves higher ASR-G than GCG and Visual-jailbreak in both white-box and transfer settings, and that the shorter suffix lowers prompt perplexity, making the attack harder to detect.

Load-bearing premise

The load-bearing premise is that GPT-4's judgment of whether a response actually fulfills a harmful instruction (the ASR-G metric) is accurate and unbiased; the paper provides no human agreement study, calibration, or error analysis for this judge, so if the judge is too lenient or too strict, the claimed superiority of the multimodal attack over GCG on response quality could reverse.

Editorial extensions

If this is right

  • A universal adversarial image plus a 10-token suffix crafted on a 7B surrogate transfers to larger open MLLMs, including 34B-parameter models, with higher ASR-G than text-only GCG or image-only Visual-jailbreak.
  • Because the resulting suffix is half the length of GCG's 20-token suffix, the concatenated prompt has lower perplexity, which should make the attack less noticeable to perplexity-based defenses.
  • The attack generalizes to multimodal in-context learning: adding adversarial image-text demonstrations raises ASR and ASR-G on models such as MiniCPM-V2.6, Qwen2-VL, and LLaVA-NeXT.
  • Single-modality safety filters are insufficient; the authors argue defenses should be designed against cross-modal adversarial interactions, for example by cross-modal adversarial fine-tuning or multimodal input sanitization.
  • Attack success diminishes as MLLMs grow larger, so model scale provides some safety benefit, but the vulnerability persists across sizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the GPT-4-based ASR-G judge is validated against human raters, the paper's emphasis on response quality rather than mere refusal absence would give the field a more meaningful evaluation standard for jailbreak attacks.
  • A simple testable extension is to run the same alternating cross-modal optimization against a closed model such as GPT-4V through its API; the paper's in-context experiments with GPT-4O hint at partial transfer but do not claim full jailbreak.
  • Because variance-tuned gradients are borrowed from transferability research, a plausible defense is adversarial training on multimodal inputs that explicitly flattens the loss landscape; the paper proposes this idea but does not test it.
  • The authors note that resource constraints limited surrogates to 7B models; using a larger surrogate or an ensemble of surrogates might restore transfer to very large models, but this is left untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. The paper proposes a universal jailbreak attack against multimodal large language models (MLLMs) by jointly optimizing a universal adversarial suffix and a universal adversarial image. The optimization alternates between the two modalities, using a variance-tuned gradient update motivated by transferability, and evaluates the resulting attack in white-box and black-box (transfer) settings across several open MLLMs, including larger models and multimodal in-context learning scenarios. The central claims are that image-text interaction is a critical vulnerability, that the proposed attack transfers better than existing text-only and image-only jailbreak baselines, and that it produces higher-quality undesirable generations, measured by both a phrase-based ASR and a GPT-4-based ASR-G metric.

Significance. If the claims hold, the paper would be a useful contribution to MLLM safety evaluation: it demonstrates a practical universal attack surface in the interaction between modalities, provides a shorter suffix with potentially lower detection risk, and evaluates across a broad set of open MLLMs. The algorithmic description is fairly concrete, the use of two surrogate models and a held-out test set is a reasonable methodology, and the paper includes an in-context learning variant that goes beyond the original GCG setting. However, the central qualitative claim currently rests on an unvalidated GPT-4 judge, the most relevant bi-modal baselines are absent, and no statistical uncertainty is reported for the main tables; these gaps must be closed before the strength of the claimed contribution can be accepted.

major comments (5)
  1. [Section V-A2, Tables III-V] The central qualitative claim, stated in the Abstract and developed in Section V-B4, that the proposed attack yields 'higher-quality undesirable generations' rests entirely on the ASR-G metric, which uses GPT-4 to judge whether a response fulfills the harmful instruction. The prompt template and rejection-phrase set for ASR are explicitly deferred to Appendix I, but no appendix is present in the submission, and no human agreement study, calibration, or error analysis for the GPT-4 judge is reported. Because Tables III-V and Figures 2-3 use ASR-G for the main comparisons, and because Figure 8 shows that ASR and ASR-G can disagree sharply, the reported ASR-G advantage of 'Ours' over GCG could be an artifact of judge bias rather than a genuine improvement in response quality. Please provide the exact judge prompt, a human-annotation validation with agreement statistics, and an error analysis (e.g., false-positive and false-negative rates on a labeled sample of outputs across victim models).
  2. [Section V-B3, Tables II-IV] The protocol for the universal-attack scenario states that three random selections of 100 harmful behaviors were used and average ASR and ASR-G were computed, but Tables II-IV report only single numbers with no standard deviations, confidence intervals, or seed information. Observed differences are often small (e.g., Table III, LLaVA-7B surrogate on MiniGPT4-7B: ASR-G 59 vs. 64; Table IV, Yi-VL-6B ASR-G 40 vs. 37), so without variance estimates the claimed 'superior transferability' is not statistically supported. Please report per-run results, means, and standard deviations or confidence intervals, and specify the number of random seeds used for both the training-set selection and the test-set selection.
  3. [Section V-B2, V-D] The baseline set contains only GCG (text-only), Visual-jailbreak (image-only), and a simple Text&Image combination that appears only in Figure 9. Existing bi-modal jailbreak methods cited in the paper itself, notably 'Jailbreak in pieces' [40] and the bi-modal adversarial prompt of [41], are not evaluated even though they are the most directly relevant prior work for a claim about multimodal interaction attacks. Without comparisons to these image-text attack methods, the paper's central novelty and superiority claims over 'existing baselines' are not established; the main transfer tables should include at least one or two established bi-modal baselines under the same evaluation protocol.
  4. [Section IV-C and V-D] The comparison 'Ours vs. Text&Image' in Figure 9 does not isolate the effect of joint image-text interaction because the two conditions differ in multiple factors at once: the proposed method uses a 10-token suffix, alternating optimization, and variance-tuned gradients, whereas Text&Image concatenates a separately optimized 20-token GCG suffix with a separately optimized Visual-jailbreak image. The improved transferability could therefore be due to the shorter suffix, the stronger optimization, or the variance tuning rather than to the interaction itself. A controlled ablation should equalize suffix length, perturbation budgets, and optimization iterations between the joint and independent conditions (e.g., the same number of total gradient steps and the same suffix length in both arms).
  5. [Section V-B5] The only 34B evaluation reported in the main text is Yi-VL-34B (Figure 5); the claimed evaluation of LLaVA-34B is deferred to 'Appendix II', which is absent from the submission, and additional examples are deferred to 'Appendix III'. Since the abstract and introduction emphasize attacks on models 'from 7B to 34B', the LLaVA-34B results are load-bearing and must be present and fully specified (exact test selection, number of behaviors, and metric computation) before the scope claim can be verified.
minor comments (3)
  1. [Section V-A3] The implementation details mislabel the iteration counts: the text says 'total iteration T is set to 50, the image iteration H is set to 50, and the suffix iteration K is set to 20', but in Algorithms 1-3, T is the image-optimization loop, H is the suffix-optimization loop, and K/M are the neighborhood sampling sizes (Equations 3 and 6). Please align the notation and report all hyperparameters (N, K, M, b, l) consistently.
  2. [Section III-A / IV-A] There are several typographical errors, including 'underdesriable' (Section II-A), 'Text&Imiage' (Section III-A and IV-A), and the one-hot embedding notation 'e i s′' in Section IV-C2 that is not clearly defined. Please proofread and clarify the notation for the neighborhood perturbations in the suffix optimization.
  3. [Appendix references] The manuscript repeatedly refers to Appendices I-III ('the prompt template ... are provided in Appendix I', 'The results of evaluating LLaVA-34B are in Appendix II', 'More examples can be found in Appendix III'), but no appendices are included in the arXiv submission. Please either include the appendices or remove the references and place the essential content in the main text or a supplement.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the attack is optimized on a disjoint training subset and evaluated on held-out AdvBench behaviors, with the ASR-G judge external to the optimization; self-citations are not load-bearing.

full rationale

This is an empirical attack-optimization paper rather than a derivation, and its central claims do not reduce to their inputs by construction. The universal adversarial suffix and image are optimized with the token log-likelihood loss in Eq. (1) on a randomly selected 25 harmful behaviors from AdvBench, then evaluated on 100 disjoint behaviors 'ensuring no overlap between the training and test sets' (Section V-A1). This is a standard held-out protocol, so the evaluation is not a fitted-input-called-prediction. The ASR-G metric uses GPT-4 as an external judge to assess whether a response fulfills a harmful instruction (Section V-A2); although the judge prompt is deferred to a missing appendix and no human agreement study is reported, the judge is not a parameter fitted by the attack and the optimization objective does not include the judge's verdict. Any weakness in ASR-G is a measurement-validity or completeness concern, not circularity. The self-citations ([26], [29]) appear only in the related-work discussion of adversarial transferability and are not used to justify the method's effectiveness, to forbid alternatives, or to import a uniqueness theorem. There is no ansatz smuggled in via citation and no renaming of a known result as a new organization. The missing appendices and the unvalidated GPT-4 judge are legitimate reviewer concerns, but they do not constitute circular reasoning under the stated criteria.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its claims rest on hand-chosen attack hyperparameters, on transferability of gradient-based adversarial samples, on the reliability of a GPT-4 judge, and on the representativeness of a small AdvBench training subset. These are domain assumptions rather than derived facts.

free parameters (6)
  • Adversarial suffix length = 10 tokens (GCG uses 20)
    A design choice; the paper claims a shorter suffix reduces perplexity and detection risk, but no sensitivity analysis is provided.
  • Training set size for universal attack = 25 harmful behaviors from AdvBench
    The universal suffix and image are optimized on this random, unseeded subset; results may vary with the subset.
  • Image perturbation constraint = unconstrained
    The universal adversarial image is optimized without an epsilon bound, so pixel changes can be large; this may inflate transferability relative to constrained attacks.
  • Variance-tuning neighborhood size (K/M) = 5 (text); inconsistent in algorithms
    Number of neighboring samples used to estimate gradient variance; set by hand, no ablation on this value.
  • Optimization iterations T, H, N = T=50, H=50, N=20 (text); Algorithm 3 uses T and H
    Iteration counts are handwritten and inconsistent between the text and Algorithm 3, where T, H, and K are confused.
  • Search batch size and top-k = batch 128, top-k 50 for image; batch 512, top-k 256 for GCG
    Greedy coordinate search hyperparameters chosen following GCG; not tuned for the multimodal setting.
assumptions (5)
  • domain assumption Gradient-based optimization on a surrogate MLLM yields adversarial samples that transfer to other MLLMs.
    Section IV-C; the entire transfer-based attack relies on this premise, supported only by the reported experiments.
  • domain assumption GPT-4-based ASR-G accurately judges whether a response fulfills a harmful instruction.
    Section V-A2; the main quality metric uses GPT-4 without human agreement or calibration, and the prompt template is in a missing appendix.
  • ad hoc to paper Alternating image-text optimization with variance tuning finds flatter loss landscapes and improves transferability.
    Section IV-C; this is the paper's methodological premise, validated only through its own ablations and not independently.
  • domain assumption Semantic gaps between modalities allow harmful image content to evade text-based safety checkers.
    Section III-A; the paper cites a survey [35] for this gap, but does not demonstrate it applies to the safety checkers of the tested models.
  • domain assumption The 25 selected training behaviors represent the distribution of harmful behaviors in AdvBench.
    Section V-A1; no seeds, no distribution overlap analysis, and no cross-benchmark generalization test.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/Y3UBUEWX

@misc{pith2026250601307,
  author       = {Pith},
  title        = {Pith review of: Align is not Enough: Multimodal Universal Jailbreak Attack against Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3UBUEWX}},
  note         = {Machine review of arXiv:2506.01307}
}
read the original abstract

Large Language Models (LLMs) have evolved into Multimodal Large Language Models (MLLMs), significantly enhancing their capabilities by integrating visual information and other types, thus aligning more closely with the nature of human intelligence, which processes a variety of data forms beyond just text. Despite advancements, the undesirable generation of these models remains a critical concern, particularly due to vulnerabilities exposed by text-based jailbreak attacks, which have represented a significant threat by challenging existing safety protocols. Motivated by the unique security risks posed by the integration of new and old modalities for MLLMs, we propose a unified multimodal universal jailbreak attack framework that leverages iterative image-text interactions and transfer-based strategy to generate a universal adversarial suffix and image. Our work not only highlights the interaction of image-text modalities can be used as a critical vulnerability but also validates that multimodal universal jailbreak attacks can bring higher-quality undesirable generations across different MLLMs. We evaluate the undesirable context generation of MLLMs like LLaVA, Yi-VL, MiniGPT4, MiniGPT-v2, and InstructBLIP, and reveal significant multimodal safety alignment issues, highlighting the inadequacy of current safety mechanisms against sophisticated multimodal attacks. This study underscores the urgent need for robust safety measures in MLLMs, advocating for a comprehensive review and enhancement of security protocols to mitigate potential risks associated with multimodal capabilities.

Figures

Figures reproduced from arXiv: 2506.01307 by the authors.

Figure 1
Figure 1. The current safety alignment of MLLMs is not enough. We explore the jailbreak attack for MLLMs from a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. ASR-G on white-box models and black-box models. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. ASR-G on white-box models and black-box models. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Comparison of cross-modal interaction in multimodal [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 4
Figure 4. Figure 4: The detailed process is as follows: 1) Generating Universal Jailbreak Adversarial Images: Our goal is to find the x ′ such that it encourages the MLLMs to generate the target harmful responses Yi when users input the harmful instructions C(Qi : s ′ ) and x ′ , as follo…
Figure 5
Figure 5. Figure 5: ASR and ASR-G measured on Yi-VL-34B. Ours [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Illustration of multimodal in-context jailbreak attacks. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Examples of multimodal universal jailbreak attack on LLaVA-7B and LLaVA-13B [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]
Figure 8
Figure 8. Figure 8: Examples illustrating that ASR and ASR-G evaluate the same output differently. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Comparison between the proposed method and [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 16 canonical work pages

  1. [41]

    Jailbreak vision language models via bi-modal adversarial prompt,

    Z. Ying, A. Liu, T. Zhang, Z. Yu, S. Liang, X. Liu, and D. Tao, “Jailbreak vision language models via bi-modal adversarial prompt,”arXiv preprint arXiv:2406.04031, 2024

    arXiv 2024

  2. [40]

    Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models,

    E. Shayegani, Y . Dong, and N. Abu-Ghazaleh, “Jailbreak in pieces: Compositional adversarial attacks on multi-modal language models,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  3. [1]

    Gpt-4 technical report. arxiv 2303.08774,

    R. OpenAI, “Gpt-4 technical report. arxiv 2303.08774,”View in Article, vol. 2, no. 5, 2023

    arXiv 2023

  4. [2]

    Gemini: a family of highly capable multimodal models,

    G. Team, R. Anil, S. Borgeaud, Y . Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauthet al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

    arXiv 2023

  5. [3]

    A survey on multimodal large language models,

    S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,”arXiv preprint arXiv:2306.13549, 2023

    arXiv 2023

  6. [4]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, 2024

    2024

  7. [5]

    Videochat: Chat-centric video understanding,

    K. Li, Y . He, Y . Wang, Y . Li, W. Wang, P. Luo, Y . Wang, L. Wang, and Y . Qiao, “Videochat: Chat-centric video understanding,”arXiv preprint arXiv:2305.06355, 2023

    arXiv 2023

  8. [6]

    Pengi: An audio language model for audio tasks,

    S. Deshmukh, B. Elizalde, R. Singh, and H. Wang, “Pengi: An audio language model for audio tasks,”Advances in Neural Information Processing Systems, vol. 36, pp. 18 090–18 108, 2023

    2023

Show all 45 references
  1. [7]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day,

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “Llava-med: Training a large language-and-vision assistant for biomedicine in one day,”Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  2. [8]

    Universal and transferable adversarial attacks on aligned language models,

    A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,”arXiv preprint arXiv:2307.15043, 2023

    2023 arXiv

  3. [9]

    Gpt- 4 is too smart to be safe: Stealthy chat with llms via cipher,

    Y . Yuan, W. Jiao, W. Wang, J.-t. Huang, P. He, S. Shi, and Z. Tu, “Gpt- 4 is too smart to be safe: Stealthy chat with llms via cipher,”arXiv preprint arXiv:2308.06463, 2023

    2023 arXiv

  4. [10]

    Jailbreaking black box large language models in twenty queries,

    P. Chao, A. Robey, E. Dobriban, H. Hassani, G. J. Pappas, and E. Wong, “Jailbreaking black box large language models in twenty queries,”arXiv preprint arXiv:2310.08419, 2023

    2023 arXiv

  5. [11]

    Visual adversarial examples jailbreak aligned large language models,

    X. Qi, K. Huang, A. Panda, M. Wang, and P. Mittal, “Visual adversarial examples jailbreak aligned large language models,” inThe Second Workshop on New Frontiers in Adversarial Machine Learning, 2023

    2023

  6. [12]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

    2023

  7. [13]

    Yi: Open foundation models by 01.ai,

    AI, :, A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, K. Yu, P. Liu, Q. Liu, S. Yue, S. Yang, S. Yang, T. Yu, W. Xie, W. Huang, X. Hu, X. Ren, X. Niu, P. Nie, Y . Xu, Y . Liu, Y . Wang, Y . Cai, Z. Gu, Z. Liu, and Z. Dai, “Yi: Open fo...

    2024

  8. [14]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning,

    J. Chen, D. Zhu, X. Shen, X. Li, Z. Liu, P. Zhang, R. Krishnamoorthi, V . Chandra, Y . Xiong, and M. Elhoseiny, “Minigpt-v2: large language model as a unified interface for vision-language multi-task learning,” arXiv preprint arXiv:2310.09478, 2023

    arXiv

  9. [15]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “Minigpt-4: Enhancing vision-language understanding with advanced large language models,”arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  10. [16]

    Instructblip: Towards general-purpose vision- language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,”Advances in Neural Informa- tion Processing Systems, vol. 36, 2024

    2024

  11. [17]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Geet al., “Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution,”arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  12. [18]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration,

    Q. Ye, H. Xu, J. Ye, M. Yan, A. Hu, H. Liu, Q. Qian, J. Zhang, and F. Huang, “mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 040–13 051

    2024

  13. [19]

    Minicpm-v: A gpt-4v level mllm on your phone,

    Y . Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. Heet al., “Minicpm-v: A gpt-4v level mllm on your phone,”arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  14. [20]

    Cogvlm: Visual expert for pretrained language models,

    W. Wang, Q. Lv, W. Yu, W. Hong, J. Qi, Y . Wang, J. Ji, Z. Yang, L. Zhao, X. Songet al., “Cogvlm: Visual expert for pretrained language models,”arXiv preprint arXiv:2311.03079, 2023

    2023 arXiv

  15. [21]

    Cold-attack: Jail- breaking llms with stealthiness and controllability,

    X. Guo, F. Yu, H. Zhang, L. Qin, and B. Hu, “Cold-attack: Jail- breaking llms with stealthiness and controllability,”arXiv preprint arXiv:2402.08679, 2024

    2024 arXiv

  16. [22]

    Jailbreaking attack against multimodal large language model,

    Z. Niu, H. Ren, X. Gao, G. Hua, and R. Jin, “Jailbreaking attack against multimodal large language model,”arXiv preprint arXiv:2402.02309, 2024

    2024 arXiv

  17. [23]

    Cross-shaped adversarial patch attack,

    Y . Ran, W. Wang, M. Li, L.-C. Li, Y .-G. Wang, and J. Li, “Cross-shaped adversarial patch attack,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 4, pp. 2289–2303, 2024

    2024

  18. [24]

    Targeted adversarial attack against deep cross-modal hashing retrieval,

    T. Wang, L. Zhu, Z. Zhang, H. Zhang, and J. Han, “Targeted adversarial attack against deep cross-modal hashing retrieval,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 10, pp. 6159– 6172, 2023. IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO ...

    2023

  19. [25]

    Dynamics-aware adversarial attack of adaptive neural networks,

    A. Tao, Y . Duan, Y . Wang, J. Lu, and J. Zhou, “Dynamics-aware adversarial attack of adaptive neural networks,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 7, pp. 5505– 5518, 2024

    2024

  20. [26]

    Iterative adversarial attack on image- guided story ending generation,

    Y . Wang, W. Hu, and R. Hong, “Iterative adversarial attack on image- guided story ending generation,”IEEE Transactions on Multimedia, 2023

    2023

  21. [27]

    Toward robust neural image compression: Adver- sarial attack and model finetuning,

    T. Chen and Z. Ma, “Toward robust neural image compression: Adver- sarial attack and model finetuning,”IEEE Transactions on Circuits and Systems for Video Technology, vol. 33, no. 12, pp. 7842–7856, 2023

    2023

  22. [28]

    Towards adversarial attack on vision- language pre-training models,

    J. Zhang, Q. Yi, and J. Sang, “Towards adversarial attack on vision- language pre-training models,” inProceedings of the 30th ACM Inter- national Conference on Multimedia, 2022, pp. 5005–5013

    2022

  23. [29]

    Exploring transferability of multimodal adversarial samples for vision-language pre-training models with contrastive learning,

    Y . Wang, W. Hu, Y . Dong, and R. Hong, “Exploring transferability of multimodal adversarial samples for vision-language pre-training models with contrastive learning,”arXiv preprint arXiv:2308.12636, 2023

    2023 arXiv

  24. [30]

    Scalable universal adversarial watermark defending against fa- cial forgery,

    T. Qiao, B. Zhao, R. Shi, M. Han, M. Hassaballah, F. Retraint, and X. Luo, “Scalable universal adversarial watermark defending against fa- cial forgery,”IEEE Transactions on Information Forensics and Security, 2024

    2024

  25. [31]

    An unforgeable publicly verifiable watermark for large language models,

    A. Liu, L. Pan, X. Hu, S. Li, L. Wen, I. King, and S. Y . Philip, “An unforgeable publicly verifiable watermark for large language models,” inThe Twelfth International Conference on Learning Representations, 2023

    2023

  26. [32]

    A novel model watermarking for protecting generative adversarial network,

    T. Qiao, Y . Ma, N. Zheng, H. Wu, Y . Chen, M. Xu, and X. Luo, “A novel model watermarking for protecting generative adversarial network,” Computers & Security, vol. 127, p. 103102, 2023

    2023

  27. [33]

    Benchmarking trustworthiness of multimodal large language models: A comprehensive study,

    Y . Zhang, Y . Huang, Y . Sun, C. Liu, Z. Zhao, Z. Fang, Y . Wang, H. Chen, X. Yang, X. Weiet al., “Benchmarking trustworthiness of multimodal large language models: A comprehensive study,”arXiv preprint arXiv:2406.07057, 2024

    2024 arXiv

  28. [34]

    Visually adversarial attacks and defenses in the physical world: A survey,

    X. Wei, B. Pu, J. Lu, and B. Wu, “Visually adversarial attacks and defenses in the physical world: A survey,”arXiv preprint arXiv:2211.01671, 2022

    2022

  29. [35]

    How to bridge the gap between modalities: A comprehensive survey on multimodal large language model,

    S. Song, X. Li, and S. Li, “How to bridge the gap between modalities: A comprehensive survey on multimodal large language model,”arXiv preprint arXiv:2311.07594, 2023

    2023 arXiv

  30. [36]

    Towards deep learning models resistant to adversarial attacks,

    A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,”arXiv preprint arXiv:1706.06083, 2017

    2017 arXiv

  31. [37]

    Visual adversarial examples jailbreak aligned large language models,

    X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal, “Visual adversarial examples jailbreak aligned large language models,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 19, 2024, pp. 21 527–21 536

    2024

  32. [38]

    Enhancing the transferability of adversarial attacks through variance tuning,

    X. Wang and K. He, “Enhancing the transferability of adversarial attacks through variance tuning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1924–1933

    2021

  33. [39]

    Pytorch: An imperative style, high-performance deep learning library,

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antigaet al., “Pytorch: An imperative style, high-performance deep learning library,”Advances in neural information processing systems, vol. 32, 2019

    2019

  34. [42]

    Smoothllm: Defending large language models against jailbreaking attacks,

    A. Robey, E. Wong, H. Hassani, and G. J. Pappas, “Smoothllm: Defending large language models against jailbreaking attacks,”arXiv preprint arXiv:2310.03684, 2023

    2023 arXiv

  35. [43]

    Self-guard: Empower the llm to safeguard itself,

    Z. Wang, F. Yang, L. Wang, P. Zhao, H. Wang, L. Chen, Q. Lin, and K.-F. Wong, “Self-guard: Empower the llm to safeguard itself,”arXiv preprint arXiv:2310.15851, 2023

    2023 arXiv

  36. [44]

    Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training,

    Y . Yuan, W. Jiao, W. Wang, J.-t. Huang, J. Xu, T. Liang, P. He, and Z. Tu, “Refuse whenever you feel unsafe: Improving safety in llms via decoupled refusal training,”arXiv preprint arXiv:2407.09121, 2024

    2024 arXiv

  37. [45]

    Efficient adversarial training in llms with continuous attacks,

    S. Xhonneux, A. Sordoni, S. G ¨unnemann, G. Gidel, and L. Schwinn, “Efficient adversarial training in llms with continuous attacks,”arXiv preprint arXiv:2405.15589, 2024. Youze Wangreceived a B.S. and master’s degree from the School of Computer Science and Informa- tion Engine...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.