{"id":"ba71b053-7044-463d-977f-1b79e934b5a2","arxiv_id":"2506.01307","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An alternating image-text optimization produces a universal adversarial suffix and image that transfer across open multimodal LLMs more effectively than single-modality jailbreaks.","lead":"This paper proposes an attack method that adds a specially crafted image and a short text suffix to make multimodal AI chatbots produce harmful content, even when the attack was created on a different model. It reports that such two-part attacks transfer across many open-source vision-language models and often produce higher-quality harmful responses than single-modality attacks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ASR-G superiority claim rests entirely on an unvalidated GPT-4 judge whose prompt is deferred to a missing appendix; a human-label agreement test is needed before the headline comparison can be trusted.","rationale":"The Pith reader identified the unvalidated GPT-4 ASR-G judge as the weakest assumption, and I agree. The paper's strongest claim is that a single image-suffix pair yields higher-quality undesirable generations with superior transferability; the 'higher-quality' part is operationalized exclusively through ASR-G. Without a human agreement study, a released judge prompt, or an error analysis, the reported superiority on ASR-G could be an artifact of the judge's permissiveness or strictness. This concern is load-bearing because it directly attacks the central qualitative claim, not merely the optimization details. The paper does have independent support in the form of consistent ASR-G improvements in many rows of Tables III and IV, as well as a self-reported limitation that attack success diminishes with model size, which tempers the universality claim but does not rescue the judge validation gap. My proposed test would settle whether the concern lands: if human labels agree with GPT-4, the central claim survives; if not, the comparison must be redone. The reader's verdict of CONDITIONAL is appropriate, and my read does not change it.","tokens_in":21089,"tokens_out":2434,"duration_ms":26128,"concrete_test":"Select a stratified sample of 100 model outputs from Tables III and IV, spanning Ours, GCG, and Visual-jailbreak across at least five victim models. Have three independent annotators label each response as 'fulfills the harmful instruction' versus 'does not fulfill it', following the criteria described in Section V-A2. Compare the majority human label against the GPT-4 ASR-G label. If Cohen's kappa is below 0.6, or if the Ours-versus-GCG ASR-G gap reverses under human labels on any model row, the headline superiority claim requires re-analysis. In addition, the GPT-4 judge prompt, currently deferred to the missing Appendix I, must be released and an independent replication of the evaluation on a public subset should be run to confirm that the reported ASR-G numbers are reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central qualitative claim that the proposed attack brings 'higher-quality undesirable generations' across MLLMs rests solely on the ASR-G metric, which uses GPT-4 to judge whether a response fulfills a harmful instruction (Section V-A2). The paper explicitly states that the GPT-4 prompt template and the rejection-phrase set for ASR are provided in Appendix I, but no appendix is present in the arXiv submission. No human agreement study, no calibration, and no error analysis for the judge are reported. Figure 8 illustrates that ASR and ASR-G can disagree sharply, and the single worked example in Fig. 8(b) shows GPT-4 rejecting an irrelevant response, but that anecdote does not establish that the judge is unbiased across 17 victim models, two surrogate models, and three attack families. If GPT-4 is systematically more permissive toward the fluent, instruction-following outputs that the authors' method tends to produce, or systematically stricter toward GCG's longer, higher-perplexity suffixes (which the paper itself notes can trigger off-topic or multilingual responses), then the reported ASR-G advantage of Ours over GCG in Tables III, IV, and V could be a judge artifact rather than a real increase in harmful response quality. Because the entire qualitative contribution of the paper is built on this unvalidated metric, the strongest claim is unsecured until the judge is validated against human labels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a universal jailbreak attack against multimodal large language models (MLLMs) by jointly optimizing a universal adversarial suffix and a universal adversarial image. The optimization alternates between the two modalities, using a variance-tuned gradient update motivated by transferability, and evaluates the resulting attack in white-box and black-box (transfer) settings across several open MLLMs, including larger models and multimodal in-context learning scenarios. The central claims are that image-text interaction is a critical vulnerability, that the proposed attack transfers better than existing text-only and image-only jailbreak baselines, and that it produces higher-quality undesirable generations, measured by both a phrase-based ASR and a GPT-4-based ASR-G metric.","tokens_in":21360,"tokens_out":8571,"duration_ms":92513,"significance":"If the claims hold, the paper would be a useful contribution to MLLM safety evaluation: it demonstrates a practical universal attack surface in the interaction between modalities, provides a shorter suffix with potentially lower detection risk, and evaluates across a broad set of open MLLMs. The algorithmic description is fairly concrete, the use of two surrogate models and a held-out test set is a reasonable methodology, and the paper includes an in-context learning variant that goes beyond the original GCG setting. However, the central qualitative claim currently rests on an unvalidated GPT-4 judge, the most relevant bi-modal baselines are absent, and no statistical uncertainty is reported for the main tables; these gaps must be closed before the strength of the claimed contribution can be accepted.","major_comments":[{"comment":"The central qualitative claim, stated in the Abstract and developed in Section V-B4, that the proposed attack yields 'higher-quality undesirable generations' rests entirely on the ASR-G metric, which uses GPT-4 to judge whether a response fulfills the harmful instruction. The prompt template and rejection-phrase set for ASR are explicitly deferred to Appendix I, but no appendix is present in the submission, and no human agreement study, calibration, or error analysis for the GPT-4 judge is reported. Because Tables III-V and Figures 2-3 use ASR-G for the main comparisons, and because Figure 8 shows that ASR and ASR-G can disagree sharply, the reported ASR-G advantage of 'Ours' over GCG could be an artifact of judge bias rather than a genuine improvement in response quality. Please provide the exact judge prompt, a human-annotation validation with agreement statistics, and an error analysis (e.g., false-positive and false-negative rates on a labeled sample of outputs across victim models).","section":"Section V-A2, Tables III-V"},{"comment":"The protocol for the universal-attack scenario states that three random selections of 100 harmful behaviors were used and average ASR and ASR-G were computed, but Tables II-IV report only single numbers with no standard deviations, confidence intervals, or seed information. Observed differences are often small (e.g., Table III, LLaVA-7B surrogate on MiniGPT4-7B: ASR-G 59 vs. 64; Table IV, Yi-VL-6B ASR-G 40 vs. 37), so without variance estimates the claimed 'superior transferability' is not statistically supported. Please report per-run results, means, and standard deviations or confidence intervals, and specify the number of random seeds used for both the training-set selection and the test-set selection.","section":"Section V-B3, Tables II-IV"},{"comment":"The baseline set contains only GCG (text-only), Visual-jailbreak (image-only), and a simple Text&Image combination that appears only in Figure 9. Existing bi-modal jailbreak methods cited in the paper itself, notably 'Jailbreak in pieces' [40] and the bi-modal adversarial prompt of [41], are not evaluated even though they are the most directly relevant prior work for a claim about multimodal interaction attacks. Without comparisons to these image-text attack methods, the paper's central novelty and superiority claims over 'existing baselines' are not established; the main transfer tables should include at least one or two established bi-modal baselines under the same evaluation protocol.","section":"Section V-B2, V-D"},{"comment":"The comparison 'Ours vs. Text&Image' in Figure 9 does not isolate the effect of joint image-text interaction because the two conditions differ in multiple factors at once: the proposed method uses a 10-token suffix, alternating optimization, and variance-tuned gradients, whereas Text&Image concatenates a separately optimized 20-token GCG suffix with a separately optimized Visual-jailbreak image. The improved transferability could therefore be due to the shorter suffix, the stronger optimization, or the variance tuning rather than to the interaction itself. A controlled ablation should equalize suffix length, perturbation budgets, and optimization iterations between the joint and independent conditions (e.g., the same number of total gradient steps and the same suffix length in both arms).","section":"Section IV-C and V-D"},{"comment":"The only 34B evaluation reported in the main text is Yi-VL-34B (Figure 5); the claimed evaluation of LLaVA-34B is deferred to 'Appendix II', which is absent from the submission, and additional examples are deferred to 'Appendix III'. Since the abstract and introduction emphasize attacks on models 'from 7B to 34B', the LLaVA-34B results are load-bearing and must be present and fully specified (exact test selection, number of behaviors, and metric computation) before the scope claim can be verified.","section":"Section V-B5"}],"minor_comments":[{"comment":"The implementation details mislabel the iteration counts: the text says 'total iteration T is set to 50, the image iteration H is set to 50, and the suffix iteration K is set to 20', but in Algorithms 1-3, T is the image-optimization loop, H is the suffix-optimization loop, and K/M are the neighborhood sampling sizes (Equations 3 and 6). Please align the notation and report all hyperparameters (N, K, M, b, l) consistently.","section":"Section V-A3"},{"comment":"There are several typographical errors, including 'underdesriable' (Section II-A), 'Text&Imiage' (Section III-A and IV-A), and the one-hot embedding notation 'e i s′' in Section IV-C2 that is not clearly defined. Please proofread and clarify the notation for the neighborhood perturbations in the suffix optimization.","section":"Section III-A / IV-A"},{"comment":"The manuscript repeatedly refers to Appendices I-III ('the prompt template ... are provided in Appendix I', 'The results of evaluating LLaVA-34B are in Appendix II', 'More examples can be found in Appendix III'), but no appendices are included in the arXiv submission. Please either include the appendices or remove the references and place the essential content in the main text or a supplement.","section":"Appendix references"}],"recommendation":"major_revision","confidential_remarks":"I would ask the editor to ensure that the submission is complete before the next round of review: the paper contains multiple explicit references to Appendices I-III that are not present in the posted version, and one of those appendices is said to contain the prompt template for the main quality metric. This is not a criticism of the authors' intent, but a completeness check that the review process should enforce."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the attack is a sensible piece of red-team engineering, but its headline claim of higher-quality undesirable generations leans on an ASR-G metric whose judge is never validated. I would send it to review, but I would demand the missing evidence before accepting.\n\nWhat is genuinely new: the alternating image-text optimization loop, where each modality supervises the other and the gradient is variance-tuned, is not in the cited references. The paper describes the algorithm clearly and evaluates broadly: two 7B surrogates, seventeen target MLLMs up to 34B, plus a multimodal in-context variant. Reporting ASR and ASR-G separately and showing examples where they disagree is good practice. The ablation that removes suffix, image, and transfer strategy is informative and goes beyond the minimal comparison.\n\nThe soft spots are mostly missing evidence, not demonstrated failure. The ASR-G judge is the load-bearing issue: the prompt is deferred to a missing appendix, there is no human agreement study, no calibration, and no error analysis. You can see ASR and ASR-G diverge sharply in Figure 8, so a biased judge could flip the main comparison against GCG. That is a real problem.\n\nThe second soft spot is the absence of bi-modal baselines. Ref [41] is the closest competitor and is cited but never benchmarked. That weakens the claim that iterative image-text interaction is what drives the gain. Relatedly, variance tuning is a known transferability booster, and the paper does not isolate it from the interaction mechanism. Adding GCG-plus-variance-tuning as a baseline would clarify this.\n\nMinor but worth fixing: the paper says three random selections were averaged but reports no seeds, error bars, or per-run spread. The ensemble definition in Figure 5 counts success if at least one adversarial sample works, which flatters the attack; it is disclosed but should be flagged in the text. The limitations paragraph is honest about degradation on larger models and resource constraints, which helps credibility.\n\nOverall, the central mechanism is credible, but the evidence is incomplete. The paper is for MLLM safety researchers and red-team practitioners; as a red-team report it is useful, but as a scientific claim about modality interaction it needs the judge validated, code released, and proper baselines added.\n\nRecommendation: major revision. Require human validation of ASR-G, code and seeds, bi-modal baselines, and an ablation that separates variance tuning from modality interaction.","headline":"Plausible and useful attack engineering, but the central qualitative claim is unsecured because it rests on an unvalidated GPT-4 judge and missing baselines.","tokens_in":21926,"tokens_out":2360,"would_cite":true,"duration_ms":30520,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A universal image-text jailbreak transfers across multimodal AI models from 7B to 34B parameters.","keywords":["multimodal jailbreak","universal adversarial perturbation","multimodal large language models","safety alignment","adversarial transferability","image-text interaction","ASR-G","jailbreak attack"],"falsifier":"Take a sample of model outputs from the transfer experiments, have independent human annotators rate whether each response genuinely fulfills the corresponding harmful instruction, and compare their labels with GPT-4's ASR-G labels. If human raters disagree with GPT-4 on a substantial fraction of cases, or if the ordering between the proposed method and GCG reverses under human labels, the paper's central quality claim collapses.","tokens_in":20851,"feed_emoji":"⚠️","tokens_out":9328,"duration_ms":85828,"temperature":0.7,"pith_summary":"The paper tries to establish that the safety alignment of multimodal large language models (MLLMs) is vulnerable specifically at the intersection of image and text inputs, and that an attacker can exploit this by distributing jailbreak information across both modalities. Its central claim is that a universal adversarial image plus a short 10-token adversarial suffix, optimized jointly through iterative image-text interaction on a small 7B surrogate model, transfers to other open MLLMs (from 7B to 34B parameters) and produces responses that more genuinely fulfill harmful instructions than either text-only or image-only jailbreak attacks. The authors measure this with ASR-G, a metric in which GPT-4 judges whether a response actually carries out the requested harmful behavior, and report higher ASR-G scores than the GCG and Visual-jailbreak baselines. A sympathetic reader would care because current safety mechanisms largely screen text and images separately, and the result suggests that cross-modal attacks can slip through these individual filters.","feed_headline":"One image plus a short suffix jailbreaks multimodal AI from 7B to 34B","feed_subtitle":"Crafted on a small 7B model, the pair transfers to larger open models and beats text-only attacks on response quality.","key_machinery":"The carrying mechanism is the alternating image-text interaction optimization loop (Algorithms 1–3). In each outer iteration, the current adversarial suffix is held fixed while the adversarial image is optimized to maximize the average per-token log-likelihood of a target harmful response; then the image is held fixed while the suffix token embeddings are optimized by greedy coordinate gradient search, which computes token-substitution candidates from the gradient and selects the best replacement by loss. Both steps use variance-tuned gradients: the update direction is the raw gradient minus the average gradient over K (image) or M (suffix) random neighboring samples, combined with a momentum term. This variance tuning is what the paper credits for making the adversarial image and suffix transfer across different MLLM architectures and sizes. The adversarial loss itself is the normalized negative log-likelihood of the target harmful response conditioned on the concatenation of the harmful query and suffix, with the adversarial image as an additional conditioning input.","core_discovery":"The central discovery is that 'align is not enough': aligning an MLLM's safety behavior on text and images separately leaves a gap at the image-text interaction, and this gap can be weaponized. The paper proposes an attack that alternates between optimizing an adversarial image with projected gradient descent and optimizing a 10-token adversarial suffix with greedy coordinate gradient search, using the current state of the other modality as the conditioning input at each step. To keep the adversarial pair transferable across different MLLMs, both optimization steps replace the raw gradient with a variance-tuned gradient obtained by sampling neighboring inputs, which steers updates toward flatter loss landscapes. Trained on 25 harmful behaviors from the AdvBench subset, the universal image-suffix pair is then evaluated on unseen harmful prompts and transferred to models like LLaVA, MiniGPT-v2, MiniGPT4, InstructBLIP, Yi-VL, mPLUG-Owl2, MiniCPM, CogVLM, and Qwen2-VL. The paper reports that this method achieves higher ASR-G than GCG and Visual-jailbreak in both white-box and transfer settings, and that the shorter suffix lowers prompt perplexity, making the attack harder to detect.","pith_inferences":["If the GPT-4-based ASR-G judge is validated against human raters, the paper's emphasis on response quality rather than mere refusal absence would give the field a more meaningful evaluation standard for jailbreak attacks.","A simple testable extension is to run the same alternating cross-modal optimization against a closed model such as GPT-4V through its API; the paper's in-context experiments with GPT-4O hint at partial transfer but do not claim full jailbreak.","Because variance-tuned gradients are borrowed from transferability research, a plausible defense is adversarial training on multimodal inputs that explicitly flattens the loss landscape; the paper proposes this idea but does not test it.","The authors note that resource constraints limited surrogates to 7B models; using a larger surrogate or an ensemble of surrogates might restore transfer to very large models, but this is left untested."],"forward_implications":["A universal adversarial image plus a 10-token suffix crafted on a 7B surrogate transfers to larger open MLLMs, including 34B-parameter models, with higher ASR-G than text-only GCG or image-only Visual-jailbreak.","Because the resulting suffix is half the length of GCG's 20-token suffix, the concatenated prompt has lower perplexity, which should make the attack less noticeable to perplexity-based defenses.","The attack generalizes to multimodal in-context learning: adding adversarial image-text demonstrations raises ASR and ASR-G on models such as MiniCPM-V2.6, Qwen2-VL, and LLaVA-NeXT.","Single-modality safety filters are insufficient; the authors argue defenses should be designed against cross-modal adversarial interactions, for example by cross-modal adversarial fine-tuning or multimodal input sanitization.","Attack success diminishes as MLLMs grow larger, so model scale provides some safety benefit, but the vulnerability persists across sizes."],"supporting_citations":[{"why":"Supplies the GCG adversarial-suffix baseline and the AdvBench corpus used for training and evaluation.","marker":"[8]"},{"why":"Supplies the Visual-jailbreak baseline and the universal adversarial image approach the paper extends.","marker":"[11]"},{"why":"Supplies the variance-tuned gradient technique used to improve transferability of the adversarial image and suffix.","marker":"[38]"},{"why":"Supplies the ASR-G metric, in which GPT-4 judges whether a response fulfills the harmful instruction.","marker":"[21]"},{"why":"Prior multimodal jailbreak attack work that the paper compares against and extends to universal cross-modal attacks.","marker":"[22]"}],"fun_headline_variants":["Image-text gap enables universal jailbreak across multimodal LLMs","Jailbreak MLLMs with one adversarial image and a short suffix","Universal image-suffix attack beats text-only jailbreaks on MLLMs","Transferable multimodal jailbreak exploits image-text interaction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that GPT-4's judgment of whether a response actually fulfills a harmful instruction (the ASR-G metric) is accurate and unbiased; the paper provides no human agreement study, calibration, or error analysis for this judge, so if the judge is too lenient or too strict, the claimed superiority of the multimodal attack over GCG on response quality could reverse.","fun_headline_variants_meta":{"raw":{"variants":["Image-text gap enables universal jailbreak across multimodal LLMs","Jailbreak MLLMs with one adversarial image and a short suffix","Universal image-suffix attack beats text-only jailbreaks on MLLMs","Transferable multimodal jailbreak exploits image-text interaction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000607,"raw_usage":{"total_tokens":2880,"prompt_tokens":1046,"completion_tokens":1834,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":1763}},"tokens_in":662,"tokens_out":1834,"duration_ms":14263,"temperature":1.0,"reasoning_tokens":1763,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:45:20.911140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of model outputs from the transfer experiments, have independent human annotators rate whether each response genuinely fulfills the corresponding harmful instruction, and compare their labels with GPT-4's ASR-G labels. If human raters disagree with GPT-4 on a substantial fraction of cases, or if the ordering between the proposed method and GCG reverses under human labels, the paper's central quality claim collapses.","supporting_citations":[{"cited_title":"Visual adversarial examples jailbreak aligned large language models,","cited_arxiv_id":null,"evidence_quote":"Supplies the Visual-jailbreak baseline and the universal adversarial image approach the paper extends."},{"cited_title":"Enhancing the transferability of adversarial attacks through variance tuning,","cited_arxiv_id":null,"evidence_quote":"Supplies the variance-tuned gradient technique used to improve transferability of the adversarial image and suffix."}],"review_version":1}